Automatic testing method and device, electronic equipment and storage medium

Abstract

The invention provides an automatic testing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining metadata attributes and health degree evaluation indexes of use cases in a use case library; screening a target use case set from the use case library based on the metadata attribute, the health degree evaluation index and the influence range of the current code change; executing the target use case set to obtain a test result; and analyzing the test result to identify at least one type of fault features and generate corresponding optimization suggestions. Therefore, the problem of insufficient efficiency and accuracy caused by unreasonable test resource allocation in the prior art is solved, and synchronous optimization of test accuracy and execution efficiency is realized.

Description

Technical Field

[0001] This application relates to the field of testing technology, and in particular to an automated testing method, apparatus, electronic device, and storage medium. Background Technology

[0002] As software systems become increasingly complex, automated testing has become a crucial technical means to ensure software quality. In the Continuous Integration and Continuous Delivery (CI / CD) process, testing systems need to quickly and accurately verify the correctness of code changes, which places higher demands on testing efficiency and accuracy.

[0003] In existing technologies, automated testing systems typically employ either a full-scale testing strategy or a code-change-based testing strategy. A full-scale testing strategy executes all test cases in the test case library, while a code-change-based testing strategy selects test cases to execute by analyzing code differences.

[0004] However, the existing technology lacks a multi-dimensional evaluation mechanism for the selection and execution of test cases, resulting in unreasonable allocation of test resources and affecting test efficiency and accuracy. Summary of the Invention

[0005] The purpose of this application is to provide an automated testing method, apparatus, electronic device, and storage medium to address the problem in the prior art where the screening and execution process of test cases lacks a multi-dimensional evaluation mechanism, leading to unreasonable allocation of test resources and affecting testing efficiency and accuracy. The specific technical solution is as follows:

[0006] Firstly, this application provides an automated testing method, including:

[0007] Obtain the metadata attributes and health assessment metrics of the use cases in the use case library;

[0008] Based on the metadata attributes, the health assessment indicators, and the impact scope of the current code changes, a target set of use cases is selected from the use case library;

[0009] Execute the target test case set to obtain test results;

[0010] The test results are analyzed to identify at least one type of fault characteristic and generate corresponding optimization suggestions.

[0011] In one possible implementation, obtaining the health assessment metrics for each use case in the use case library includes:

[0012] For each use case, a stability evaluation value is calculated based on the number of consecutive failures of the use case, wherein the stability evaluation value decreases as the number of consecutive failures increases;

[0013] Based on the business weights and interface levels covered by the use cases, a business value assessment value is calculated, wherein the business weights are used to characterize the importance of the corresponding business functions in the system, and the interface levels are used to characterize the criticality level of the interfaces covered by the use cases.

[0014] Based on the stability assessment value and the business value assessment value, a health assessment index is calculated.

[0015] In one possible implementation, the step of filtering the target use case set from the use case library based on the metadata attributes, the health assessment metrics, and the impact scope of the current code change includes:

[0016] Based on the interface association characteristics of the test cases analyzed by the metadata attributes, a first candidate set related to the current test requirements is selected;

[0017] Based on the health assessment indicators, the use cases in the first candidate set are evaluated for stability, and a second candidate set that meets the health standards is selected.

[0018] Based on the scope of impact, a change correlation analysis is performed on the second candidate set to filter out the target use case set that matches the code change.

[0019] In one possible implementation, executing the target test case set to obtain test results includes:

[0020] The target use case set is asynchronously distributed to the execution cluster via a message middleware;

[0021] Obtain the node processing capacity information of the currently available execution nodes in the execution cluster;

[0022] Analyze the interface call relationships among the various use cases in the target use case set;

[0023] The number of shards is calculated based on the node processing capacity information and the interface call correlation.

[0024] Tasks are allocated based on the number of shards to execute the target test case set and obtain test results.

[0025] In one possible implementation, the method further includes:

[0026] The response time of each node is monitored in real time using service probes;

[0027] When the response time of any node exceeds a preset threshold: stop assigning new tasks to the node, and / or migrate the tasks already assigned to the node to a healthy node, and / or update the status of the node and generate a node anomaly alarm notification.

[0028] In one possible implementation, the analysis of the test results to identify at least one type of fault characteristic includes:

[0029] Extract abnormal features from the test results, including abnormal response codes and abnormal response times;

[0030] Cluster analysis is performed on the abnormal features to identify clusters of fault modes with similar characteristics;

[0031] The fault mode clusters are classified and labeled based on the historical fault knowledge base to obtain at least one type of fault feature.

[0032] In one possible implementation, generating the corresponding optimization suggestions includes:

[0033] For service deployment anomalies, generate service restart and version rollback suggestions;

[0034] For database connection leakage-related faults, generate connection pool tuning and resource monitoring recommendations;

[0035] For interface contract change-related faults, generate update suggestions for expected test case results.

[0036] Secondly, this application provides an automated testing apparatus, comprising:

[0037] The acquisition module is used to acquire metadata attributes and health assessment metrics of test cases in the test case library;

[0038] The filtering module is used to filter target use case sets from the use case library based on the metadata attributes, the health assessment indicators, and the impact scope of the current code changes;

[0039] The execution module is used to execute the target test case set and obtain test results;

[0040] The analysis module is used to analyze the test results to identify at least one type of fault characteristic and generate corresponding optimization suggestions.

[0041] In one possible implementation, the acquisition module is specifically used for:

[0042] For each use case, a stability evaluation value is calculated based on the number of consecutive failures of the use case, wherein the stability evaluation value decreases as the number of consecutive failures increases;

[0043] Based on the business weights and interface levels covered by the use cases, a business value assessment value is calculated, wherein the business weights are used to characterize the importance of the corresponding business functions in the system, and the interface levels are used to characterize the criticality level of the interfaces covered by the use cases.

[0044] Based on the stability assessment value and the business value assessment value, a health assessment index is calculated.

[0045] In one possible implementation, the filtering module is specifically used for:

[0046] Based on the interface association characteristics of the test cases analyzed by the metadata attributes, a first candidate set related to the current test requirements is selected;

[0047] Based on the health assessment indicators, the use cases in the first candidate set are evaluated for stability, and a second candidate set that meets the health standards is selected.

[0048] Based on the scope of impact, a change correlation analysis is performed on the second candidate set to filter out the target use case set that matches the code change.

[0049] In one possible implementation, the execution module is specifically used for:

[0050] The target use case set is asynchronously distributed to the execution cluster via a message middleware;

[0051] Obtain the node processing capacity information of the currently available execution nodes in the execution cluster;

[0052] Analyze the interface call relationships among the various use cases in the target use case set;

[0053] The number of shards is calculated based on the node processing capacity information and the interface call correlation.

[0054] Tasks are allocated based on the number of shards to execute the target test case set and obtain test results.

[0055] In one possible implementation, the device further includes a monitoring module for:

[0056] The response time of each node is monitored in real time using service probes;

[0057] When the response time of any node exceeds a preset threshold: stop assigning new tasks to the node, and / or migrate the tasks already assigned to the node to a healthy node, and / or update the status of the node and generate a node anomaly alarm notification.

[0058] In one possible implementation, the analysis module is specifically used for:

[0059] Extract abnormal features from the test results, including abnormal response codes and abnormal response times;

[0060] Cluster analysis is performed on the abnormal features to identify clusters of fault modes with similar characteristics;

[0061] The fault mode clusters are classified and labeled based on the historical fault knowledge base to obtain at least one type of fault feature.

[0062] In one possible implementation, the analysis module is further configured to:

[0063] For service deployment anomalies, generate service restart and version rollback suggestions;

[0064] For database connection leakage-related faults, generate connection pool tuning and resource monitoring recommendations;

[0065] For interface contract change-related faults, generate update suggestions for expected test case results.

[0066] Thirdly, an electronic device is provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0067] Memory, used to store computer programs;

[0068] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0069] Fourthly, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0070] Fifthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform any of the automated testing methods described above.

[0071] Beneficial effects of the embodiments in this application:

[0072] This application provides an automated testing method, apparatus, electronic device, and storage medium. First, by acquiring the metadata attributes and health assessment indicators of test cases, a multi-dimensional evaluation system covering the functional attributes and quality status of test cases is established, solving the problem of single-dimensional test case selection in existing technologies. Second, based on a triple-condition approach of metadata attributes, health assessment indicators, and the impact scope of code changes, a target test case set is dynamically selected, achieving precise test case selection and avoiding resource waste caused by inefficient or redundant testing. Third, by executing the selected target test case set, test execution efficiency is significantly improved while ensuring test coverage. Finally, by intelligently analyzing test results to identify fault characteristics and generate optimization suggestions, the accuracy of problem localization and repair is improved. Thus, the problem of insufficient efficiency and accuracy due to unreasonable allocation of test resources in existing technologies is solved, achieving simultaneous optimization of test accuracy and execution efficiency.

[0073] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0074] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0075] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0076] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0077] Figure 1 A flowchart illustrating an automated testing method provided in this application embodiment;

[0078] Figure 2 A flowchart illustrating another automated testing method provided in this application embodiment;

[0079] Figure 3 A flowchart illustrating yet another automated testing method provided in this application embodiment;

[0080] Figure 4 A system processing flow diagram for automated testing provided in this application embodiment;

[0081] Figure 5 This is a schematic diagram of the structure of an automated testing device provided in an embodiment of this application;

[0082] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0084] The following disclosure provides numerous different embodiments or examples for implementing various structures of the invention. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of the invention. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0085] Figure 1 This is a flowchart illustrating an automated testing method provided in an embodiment of this application. This method can be applied to one or more electronic devices such as smartphones, laptops, desktop computers, portable computers, and servers. Furthermore, the execution entity of this method can be hardware or software. When the execution entity is hardware, it can be one or more of the aforementioned electronic devices. For example, a single electronic device can execute this method, or multiple electronic devices can cooperate with each other to execute this method. When the execution entity is software, this method can be implemented as multiple software programs or software modules, or as a single software program or software module. No specific limitations are imposed here.

[0086] like Figure 1 As shown, the method specifically includes:

[0087] S101. Obtain the metadata attributes and health assessment metrics of the use cases in the use case library.

[0088] Metadata attributes refer to structured data that describes the characteristics of test cases, including test case version number, request protocol type, and associated sub-service call information.

[0089] Health assessment metrics are comprehensive evaluation parameters that reflect the reliability and business value of test cases, and consist of stability scores and value scores.

[0090] In this embodiment, a complete use case feature database can be established by collecting metadata attributes of use cases (such as version change records and interface call relationships) and calculating health assessment indicators (including stability assessment based on the number of consecutive failures and value assessment based on business weights), providing data support for intelligent screening. In specific implementation, the system will periodically scan the use case database, extract the latest metadata, and update the health score.

[0091] Specifically, the health assessment metrics for each use case in the use case library can be obtained through the following steps: For each use case, calculate a stability assessment value based on the number of consecutive failures, wherein the stability assessment value decreases as the number of consecutive failures increases; calculate a business value assessment value based on the business weight and interface level covered by the use case, wherein the business weight is used to characterize the importance of the corresponding business function in the system, and the interface level is used to characterize the criticality level of the interface covered by the use case; calculate the health assessment metric based on the stability assessment value and the business value assessment value.

[0092] Among them, the stability assessment value is a quantitative indicator reflecting the reliability of test cases. It is calculated based on the number of consecutive failures using the algorithm W_f = 0.6^(n-1) (where n is the number of consecutive failures). This algorithm ensures that the assessment value decreases exponentially with the increase of the number of failures. The business value assessment value is a quantitative indicator reflecting the importance of test cases. It is calculated by the formula V = Σ(business weight × coverage interface level) based on the business weight (set according to the criticality of business functions) and the coverage interface level (divided according to the criticality of the interface in the system).

[0093] In practice, the system periodically scans test case execution records, automatically updates the consecutive failure count (n), and calculates various parameters based on a preset business weight table (e.g., core transaction business weight is 1.0, auxiliary functions are 0.5) and interface level table (e.g., payment interface is level one, query interface is level two). Finally, it generates a health assessment index by weighted summation of the stability assessment value and the business value assessment value. This scheme strengthens the negative impact of consecutive failures on stability through the W_f = 0.6^(n-1) algorithm, making the assessment more consistent with actual reliability characteristics. Simultaneously, the Σ(business weight × coverage interface level) formula ensures that high-value businesses and key interfaces receive higher weights. Finally, through the synergistic effect of the dual assessment dimensions (stability + business value), the health index accurately reflects the actual quality status of test cases, providing a reliable basis for intelligent screening.

[0094] S102. Based on the metadata attributes, the health assessment indicators, and the impact scope of the current code change, select a target use case set from the use case library.

[0095] The scope of impact of code changes refers to the set of functional modules and their associated interfaces that have been modified in this code commit.

[0096] The target test case set is a set of test cases that have been selected and determined to be executed.

[0097] In one embodiment, S102 may include the following steps: analyzing the interface association characteristics of the test cases based on the metadata attributes, and filtering out a first candidate set related to the current testing requirements; performing a stability assessment on the test cases in the first candidate set based on the health assessment indicators, and filtering out a second candidate set that meets the health standards; performing a change association analysis on the second candidate set based on the scope of influence, and filtering out a target test case set that matches the code change.

[0098] Interface association features refer to the call relationship data between use cases and various interfaces of the system, which is obtained by parsing the request protocol and sub-call information in the metadata; health status meets the standard, which means that the health evaluation index of the use case exceeds the preset threshold (e.g., ≥60 points); change correlation analysis refers to comparing the intersection of the interface covered by the use case and the interface modified in the code.

[0099] In practice: First, based on the interface call chain in the metadata, test cases involving the interfaces of the current testing requirements are selected to form a first candidate set. Then, low-quality test cases are filtered out according to health indicators to generate a second candidate set. Finally, by analyzing the diff results of the three most recent code changes, a complete mapping relationship between code modifications and interface calls is established. Test cases directly related to the changed interfaces are accurately selected from the second candidate set: for core interface changes, all upstream and downstream related test cases are retained; for important interface changes, the main test scenarios are matched; and for ordinary interface changes, key verification test cases are selectively retained, forming a target test case set that is both complete and efficient. This solution ensures coverage of the impact of changes through interface association analysis, reduces inefficient testing through health filtering, and accurately locates the interfaces affected by changes through code diff analysis. It ensures a high degree of matching between the target test case set and code modifications, avoiding resource waste caused by irrelevant testing.

[0100] S103. Execute the target test case set to obtain the test results.

[0101] The test results include detailed information such as the execution status of the test cases (success / failure), response time, and error codes.

[0102] In this embodiment, the target test case set is executed in a distributed testing environment, and the execution process is monitored in real time, recording complete test data. During execution, the system adopts an intelligent hierarchical scheduling strategy: 50% of resources are fixedly allocated to the core payment link, and the remaining resources are dynamically allocated according to the health of the test cases; the Jitter algorithm is enabled to retry abnormal requests (base latency ± random perturbation) to avoid retry storms; at the same time, the node status is monitored in real time, and when the response time exceeds the threshold, the system automatically circuit breaks and migrates the task to a healthy node. All test results are stored in a structured manner, including raw response data, execution logs, and environmental monitoring indicators, providing a data foundation for subsequent analysis.

[0103] S104. Analyze the test results to identify at least one type of fault characteristic and generate corresponding optimization suggestions.

[0104] Fault characteristics refer to common abnormal patterns extracted from failed use cases, such as specific error code combinations or response time distribution characteristics.

[0105] Optimization suggestions are specific improvement plans proposed for the identified faults.

[0106] In this embodiment, the test results are first preprocessed (e.g., data cleaning and feature extraction); then, a clustering algorithm (e.g., DBSCAN) is used to identify significant failure modes; next, the most likely failure causes are matched according to a preset root cause knowledge base; and finally, corresponding optimization suggestions are generated.

[0107] Specifically, generating corresponding optimization suggestions can include the following steps: for service deployment anomaly-type failures, generate suggestions for service restart and version rollback; for database connection leakage-type failures, generate suggestions for connection pool tuning and resource monitoring; for interface contract change-type failures, generate suggestions for updating expected test case results.

[0108] Service deployment anomaly failures refer to system anomalies caused by service version or configuration issues; database connection leakage failures refer to performance problems caused by database connections not being properly released; and interface contract change failures refer to validation failures caused by changes in interface definitions.

[0109] In practice, the system first identifies fault types through cluster analysis, then determines the most likely cause of the fault based on a predefined root cause localization matrix, and finally generates targeted suggestions based on the fault category: for service deployment anomalies, it automatically generates service restart instructions containing specific rollback version numbers; for connection leaks, it outputs connection pool parameter optimization schemes and monitoring indicator configuration suggestions; for contract changes, it provides interface document comparison results and use case expected result update guidance. This solution significantly improves the efficiency of problem localization and repair by classifying typical fault scenarios and generating directly actionable repair suggestions based on intelligent analysis, while greatly reducing the workload of manual diagnosis, making system maintenance more efficient and reliable.

[0110] In another embodiment, generating optimization recommendations further includes: setting priorities based on the historical frequency of occurrence of fault categories; generating automated repair scripts for high-frequency fault categories; and storing the recommendations in association with code change records.

[0111] Historical frequency refers to the statistical number of times various faults occur within a preset time period; automated repair scripts refer to the set of code instructions that can automatically perform repair operations; code change records refer to the history of code version updates recorded by the system.

[0112] In practice, the system maintains a historical statistical database of fault categories and dynamically adjusts processing priorities based on fault occurrence frequency. For high-frequency faults, the system automatically generates executable scripts containing complete repair procedures. Simultaneously, the generated suggestions are associated with and stored along with relevant code commit records, forming a complete fault-repair knowledge base. This solution ensures that critical faults are handled first through dynamic priority settings, improves the repair efficiency of high-frequency issues through automated scripts, and achieves a complete closed loop for fault tracing by combining associated storage of change records. This builds an intelligent operation and maintenance system with self-learning capabilities, improving system stability and maintainability.

[0113] Furthermore, the analysis phase can generate multi-dimensional quality reports by integrating test results, environmental monitoring, and code change data: Test case effectiveness: assessing test value based on defect detection rate and false alarm rate; Environmental stability: comprehensively scoring based on node anomaly rate and response time fluctuation; Top 10 risky interfaces: ranked by failure rate and business impact. Report data is fed back to the health assessment model in real time, forming a closed-loop optimization and improving fault diagnosis accuracy.

[0114] In this embodiment, firstly, by acquiring the metadata attributes and health assessment indicators of test cases, a multi-dimensional evaluation system covering the functional attributes and quality status of test cases is established, solving the problem of single-dimensional test case selection in existing technologies. Secondly, based on the triple conditions of metadata attributes, health assessment indicators, and the impact scope of code changes, the target test case set is dynamically filtered, achieving precise selection of test cases and avoiding resource waste caused by inefficient or redundant testing. Thirdly, by executing the filtered target test case set, test execution efficiency is significantly improved while ensuring test coverage. Finally, by intelligently analyzing the test results to identify fault characteristics and generate optimization suggestions, the accuracy of problem localization and repair is improved. Thus, the problem of insufficient efficiency and accuracy caused by unreasonable allocation of test resources in existing technologies is solved, achieving simultaneous optimization of test accuracy and execution efficiency.

[0115] In another embodiment, before S101, that is, before performing the test, the system first performs an environment pre-check phase to ensure the reliability of the test environment. This pre-check phase is deeply integrated with the overall architecture of the present invention, forming a complete quality assurance closed loop, specifically including:

[0116] I. Environmental Health Inspection

[0117] The following core metrics are automatically verified via the Kubernetes API: Node resource availability: including CPU load, memory usage, and disk I / O health; Service deployment integrity: verifying the running status of all dependent Pods and the reachability of service endpoints; Network topology connectivity: checking cross-node communication latency and service mesh configuration.

[0118] II. Deployment of Smart Probes

[0119] The probe type is dynamically selected based on the technical characteristics of the service being tested: for Java / SpringBoot services, bytecode is injected to enhance the proxy; for Go services, kernel-level monitoring based on eBPF is deployed; for containerized services, a sidecar container is attached to intercept traffic.

[0120] III. Establishment of Environmental Baselines

[0121] Generate a pre-inspection report containing the following dimensions: Resource baseline: normal distribution interval of CPU / memory utilization of each node; Performance baseline: P99 response time threshold of core interfaces; Topology baseline: dependency graph of inter-service call relationships.

[0122] This pre-inspection phase forms a triple collaborative mechanism with subsequent testing processes: First, node health data is synchronized to the intelligent scheduling module in real time to dynamically adjust the task sharding strategy; second, benchmark indicators collected by traffic probes are used to calibrate the anomaly judgment criteria during the execution phase; and finally, service topology data enhances the accuracy of root cause localization during the analysis phase.

[0123] See Figure 2 This is a flowchart illustrating an embodiment of another automated testing method provided in this application. Figure 2 The process shown above Figure 1 Based on the illustrated process, describe how to execute the target test case set and obtain the test results. For example... Figure 2 As shown, the process may include the following steps:

[0124] S201. The target use case set is asynchronously distributed to the execution cluster through a message middleware.

[0125] Message middleware refers to software infrastructure that enables asynchronous communication between distributed systems, such as Kafka and RabbitMQ; execution cluster refers to a pool of computing resources composed of multiple test execution nodes.

[0126] In this embodiment of the application, the task instructions of the target use case set are asynchronously sent to the execution cluster through the publish / subscribe mode of the message queue, thereby achieving decoupling and load balancing and avoiding blocking problems caused by synchronous calls.

[0127] S202. Obtain the node processing capability information of the currently available execution nodes in the execution cluster.

[0128] Node processing capacity information refers to the amount of test tasks that an execution node can process per unit of time, which is usually determined by hardware indicators such as the number of CPU cores and memory capacity.

[0129] In this embodiment of the application, by calling the cluster management interface, the resource utilization rate and current load of each node are obtained in real time, providing a data foundation for dynamic sharding.

[0130] S203. Analyze the interface call correlation of each use case in the target use case set.

[0131] Interface call dependency refers to the dependency relationship formed between test cases through system interfaces.

[0132] In this embodiment of the application, by analyzing the call chain information in the use case metadata, a dependency graph between use cases is constructed to identify strongly related use case groups that must be executed on the same node.

[0133] S204. Calculate the number of shards based on the node processing capability information and the interface call correlation.

[0134] The number of slices refers to the number of parallel execution units into which the test task is divided.

[0135] In this embodiment, the basic number of shards is calculated based on the node processing capacity information, and the sharding strategy is adjusted according to the interface call correlation to ensure that a single shard does not exceed the node load limit and that strongly related use cases are assigned to the same shard.

[0136] Specifically, S204 may include the following steps: determining the maximum use case capacity of a single node based on the node processing capacity information; analyzing the dependency relationship between use cases based on the interface call correlation; and determining a sharding configuration that meets the following conditions based on the maximum use case capacity and the dependency relationship: the total number of shards is not less than a preset minimum parallelism threshold; the number of use cases in a single shard does not exceed the maximum use case capacity; and use cases with strong dependencies are assigned to the same shard.

[0137] The maximum test case capacity of a single node refers to the maximum number of test cases that a single execution node can process simultaneously while ensuring test quality. It is calculated by combining the node's hardware configuration (such as the number of CPU cores and memory size) and benchmark test data. Dependency refers to the execution association between test cases due to shared data or call order constraints. It is determined by analyzing the interface call chain. The minimum parallelism threshold is the minimum number of shards required to ensure test efficiency.

[0138] In practical implementation, the theoretical carrying capacity is first calculated based on node resource configuration. Then, the interface call dependency graph between test cases is analyzed, ultimately generating a sharding scheme that meets the following constraints: 1) Ensuring sufficient concurrency by meeting or exceeding the minimum parallelism threshold; 2) Limiting the number of test cases per shard to the node's carrying capacity to prevent overload; 3) Assigning test cases with strong dependencies (such as those sharing test data) to the same shard to ensure correct execution. This scheme achieves intelligent generation of sharding schemes by quantifying node processing capacity information and accurately modeling test case dependencies. It fully utilizes the parallel computing advantages of distributed clusters while avoiding test failures caused by resource overload or improper dependency handling. In practical applications, it can significantly improve test throughput while maintaining the accuracy and consistency of test results. In application, the system dynamically adjusts sharding parameters to adapt to changes in cluster load, ensuring efficient and stable execution of test tasks.

[0139] S205. Assign tasks according to the number of shards to execute the target test case set and obtain test results.

[0140] Task allocation refers to scheduling the sharded test tasks to specific nodes for execution.

[0141] In this embodiment, the shards are evenly distributed to available nodes through a cluster scheduler, the test execution is started, and the test results such as response data and logs are collected in real time.

[0142] Figure 2The process illustrated utilizes a message middleware to distribute tasks with high concurrency, combines a dynamic sharding algorithm to optimize resource utilization, and ensures test integrity based on interface relationships. Ultimately, it significantly improves execution efficiency while maintaining test accuracy, effectively supporting large-scale automated testing needs. In practice, the system continuously monitors node status and automatically reassigns tasks when a faulty node is detected, ensuring high availability throughout the testing process.

[0143] See Figure 3 This is a flowchart illustrating another embodiment of the automated testing method provided in this application. Figure 3 The process shown above Figure 1 Based on the illustrated process, describe how to analyze the test results to identify at least one type of fault characteristic. For example... Figure 3 As shown, the process may include the following steps:

[0144] S301. Extract abnormal features from the test results, wherein the abnormal features include abnormal response codes and abnormal response times.

[0145] Anomaly features refer to abnormal indicator data extracted from test results, including abnormal response codes (such as HTTP 500 error codes) and abnormal response times (such as delays exceeding three times the normal threshold).

[0146] In this embodiment, feature extraction and standardization of the original test data transforms the scattered test results into structured feature vectors, establishing a data foundation for subsequent analysis. Specifically, the system parses test logs, extracts key performance indicators, and marks and normalizes outliers.

[0147] S302. Perform cluster analysis on the abnormal features to identify fault mode clusters with similar features.

[0148] Cluster analysis refers to the process of classifying similar abnormal features using unsupervised learning algorithms; a fault mode cluster refers to a set of anomalies that share common features.

[0149] In this embodiment, a density-based clustering algorithm (such as DBSCAN) is used to automatically identify abnormal groups with similar characteristics by calculating the distribution density of feature vectors in a multidimensional space. During implementation, the system dynamically adjusts the clustering parameters to adapt to test data of different scales.

[0150] Specifically, S302 may include the following steps: calculating the density distribution of abnormal features in a multidimensional space; identifying environmental jitter outliers with a density below a threshold; and reconstructing the fault mode cluster after removing the outliers.

[0151] Multidimensional spatial density distribution refers to the statistical distribution formed after mapping characteristic parameters such as abnormal response codes and response times to a mathematical space; environmental jitter outliers refer to isolated abnormal data points caused by temporary environmental fluctuations (such as network interruptions), whose density is significantly lower than the clustered areas formed by real faults; fault mode cluster reconstruction refers to the set of common anomalies that are re-divided after filtering out interfering data.

[0152] This scheme first projects standardized anomaly features onto a multi-dimensional feature space and calculates the point density of each region using methods such as kernel density estimation. Then, it identifies and removes discrete points with densities below a preset threshold (caused by environmental fluctuations). Finally, it re-clusters the remaining data points to generate a purer cluster of fault modes. This scheme effectively distinguishes between environmental noise and real faults through density analysis, significantly improving the accuracy of fault mode identification. The outlier removal mechanism avoids interference from occasional anomalies in the analysis results. The reconstructed fault mode clusters have higher homogeneity and representativeness, providing a more reliable data foundation for subsequent root cause analysis and remediation suggestion generation. In practical applications, this can reduce false alarm rates and improve fault location efficiency. In specific implementation, the system dynamically adjusts the density threshold to adapt to the characteristics of different testing environments, ensuring the adaptability of the analysis.

[0153] S303. Classify and label the fault mode clusters based on the historical fault knowledge base to obtain at least one type of fault feature.

[0154] A historical fault knowledge base is a structured database that contains known fault characteristics and solutions.

[0155] In this embodiment, the clustering results are matched with fault features in the knowledge base, and each fault mode cluster is labeled with a type tag (such as "service timeout", "database deadlock", etc.). New fault modes are also recorded to expand the knowledge base. The system generates a confidence score for each labeled result for subsequent decision-making reference. In practice, the system establishes a complete fault handling closed loop, feeding back the analysis results to the test execution and system operation and maintenance stages in real time.

[0156] Figure 3 The process described, through automated feature extraction and cluster analysis, can accurately identify various system failure modes and effectively distinguish between occasional anomalies such as environmental fluctuations and real system defects. Combined with the annotation mechanism of the historical knowledge base, it not only improves the accuracy of fault diagnosis but also enables continuous self-optimization of the knowledge base. This significantly reduces the workload of manual analysis, substantially improves the accuracy of fault identification, and provides a reliable basis for subsequent automatic repair.

[0157] In another embodiment of this application, the method may further include the following steps: monitoring the response time of each node in real time through a service probe; when the response time of any node exceeds a preset threshold: stopping the allocation of new tasks to the node, and / or migrating the tasks already allocated to the node to a healthy node, and / or updating the status of the node and generating a node anomaly alarm notification.

[0158] Service probes are lightweight monitoring agents deployed on various nodes to collect performance metrics in real time; healthy nodes are execution nodes with normal response times and resource utilization within a safe range; node anomaly alarm notifications are early warning messages that include fault details and handling suggestions.

[0159] In this embodiment, node response times are periodically collected by probes. When a response time exceeds a preset threshold (dynamically calculated based on historical baselines), a three-level protection mechanism is automatically triggered: 1) new tasks are stopped from being distributed to abnormal nodes to prevent the problem from worsening; 2) tasks to be executed are rescheduled to healthy nodes with normal load to ensure test continuity; 3) the node status is updated to unavailable and an alarm is generated to notify maintenance personnel to intervene. This solution effectively prevents false alarms caused by unstable test environments through real-time monitoring and automatic circuit breaking mechanisms. The task migration function ensures that test tasks are not interrupted due to single-point failures. The status update and alarm mechanisms provide real-time basis for maintenance decisions. The entire solution significantly improves the stability of the test environment while reducing the need for manual intervention. In specific implementation, the system records complete failover logs for subsequent analysis and strategy optimization.

[0160] In applications, probes can periodically collect performance metrics such as CPU utilization and memory utilization. CPU utilization reflects the computational load of a node's processor; high CPU utilization can lead to task processing delays. Memory utilization indicates the node's memory resource usage; insufficient memory can trigger frequent garbage collection or even process crashes. By continuously monitoring these metrics, the system can more accurately determine the root cause of node anomalies (such as distinguishing between network latency and insufficient computing resources) and provide a basis for dynamic task scheduling decisions. Specifically, when response time is abnormal, combining CPU / memory metrics can determine whether it is due to node overload or external dependency issues, thereby triggering targeted handling strategies (such as load balancing or service restart). This multi-dimensional monitoring significantly improves the accuracy of fault diagnosis and the rationality of resource scheduling.

[0161] Optionally, embodiments of this application also provide a system processing flow diagram for automated testing, such as... Figure 4 As shown, this system is a three-stage intelligent optimization system, which includes:

[0162] I. Dynamic Test Case Management Module

[0163] Metadata-driven version control unit

[0164] The configuration is as follows: Uses a YAML+JSON hybrid description format to manage the context information of test cases, including request protocols, version identifiers, and sub-call relationships; supports version comparison based on commit history; and records the complete change trajectory of test cases.

[0165] Health assessment unit

[0166] The configuration is as follows: based on the number of consecutive failures of the test case, the stability weight is calculated using the algorithm W_f = 0.6^(n-1) (where n is the number of consecutive failures); based on the business weight and interface coverage level, the value score is calculated using V = Σ(business weight × interface coverage level); a comprehensive test case health assessment index is generated; and the necessity of test case retention is determined based on the health index.

[0167] Dynamic Filtering Unit

[0168] The configuration includes: analyzing the impact of code changes; associating and filtering test cases to be executed; implementing non-full execution to improve efficiency; and supporting precise matching of the impact scope of changes.

[0169] II. Intelligent Scheduling and Execution Engine

[0170] Distributed Execution Unit

[0171] The configuration is as follows: asynchronous task distribution is based on Kafka; dynamic sharding is achieved using the Max(ceil(total number of test cases / (number of nodes × 2)), 5) algorithm; execution parameters are adjusted in real time; and task parallel efficiency is optimized.

[0172] Environmental monitoring unit

[0173] The configuration is as follows: deploy service probes for real-time monitoring; trigger a circuit breaker mechanism when the response time is >30s; ensure environmental availability; and automatically handle abnormal nodes.

[0174] III. Intelligent Analysis and Decision-Making System

[0175] Error Analysis Unit

[0176] The configuration is as follows: use the DBSCAN algorithm to cluster and analyze error codes, response times, and logs; identify false positive failures caused by environmental jitter; and separate key error pattern clusters.

[0177] Root cause localization unit

[0178] The configuration includes: establishing a mapping matrix between anomaly characteristics and possible causes; providing confidence level assessments (e.g., a sudden increase in 500 error rate corresponds to service deployment anomalies, with a confidence level of 92%); and supporting multi-dimensional fault diagnosis. For example: Anomaly characteristic: a sudden increase in 500 error rate, possible cause: service deployment anomalies, confidence level: 92%; Anomaly characteristic: high response time dispersion, possible cause: database connection leakage, confidence level: 87%; Anomaly characteristic: concentrated validation failures, possible cause: interface contract changes, confidence level: 95%.

[0179] Repair suggestion unit

[0180] The configuration includes: automatically generating defect correlation maps; providing hot repair suggestions (such as automatic parameter correction); and supporting the generation of adaptive repair schemes.

[0181] Through the collaborative work of the above modules, this system achieves intelligent management of the entire lifecycle of test cases, significantly improving testing efficiency and accuracy.

[0182] Based on the same technical concept, embodiments of this application also provide an automated testing device, such as... Figure 5 As shown, the device includes:

[0183] Module 51 is used to obtain the metadata attributes and health assessment metrics of the use cases in the use case library;

[0184] The filtering module 52 is used to filter target use case sets from the use case library based on the metadata attributes, the health assessment indicators, and the impact scope of the current code changes.

[0185] Execution module 53 is used to execute the target test case set and obtain test results;

[0186] The analysis module 54 is used to analyze the test results to identify at least one type of fault characteristic and generate corresponding optimization suggestions.

[0187] In one possible implementation, the acquisition module is specifically used for:

[0188] For each use case, a stability evaluation value is calculated based on the number of consecutive failures of the use case, wherein the stability evaluation value decreases as the number of consecutive failures increases;

[0189] Based on the business weights and interface levels covered by the use cases, a business value assessment value is calculated, wherein the business weights are used to characterize the importance of the corresponding business functions in the system, and the interface levels are used to characterize the criticality level of the interfaces covered by the use cases.

[0190] Based on the stability assessment value and the business value assessment value, a health assessment index is calculated.

[0191] In one possible implementation, the filtering module is specifically used for:

[0192] Based on the interface association characteristics of the test cases analyzed by the metadata attributes, a first candidate set related to the current test requirements is selected;

[0193] Based on the health assessment indicators, the use cases in the first candidate set are evaluated for stability, and a second candidate set that meets the health standards is selected.

[0194] Based on the scope of impact, a change correlation analysis is performed on the second candidate set to filter out the target use case set that matches the code change.

[0195] In one possible implementation, the execution module is specifically used for:

[0196] The target use case set is asynchronously distributed to the execution cluster via a message middleware;

[0197] Obtain the node processing capacity information of the currently available execution nodes in the execution cluster;

[0198] Analyze the interface call relationships among the various use cases in the target use case set;

[0199] The number of shards is calculated based on the node processing capacity information and the interface call correlation.

[0200] Tasks are allocated based on the number of shards to execute the target test case set and obtain test results.

[0201] In one possible implementation, the device further includes a monitoring module for:

[0202] The response time of each node is monitored in real time using service probes;

[0203] When the response time of any node exceeds a preset threshold: stop assigning new tasks to the node, and / or migrate the tasks already assigned to the node to a healthy node, and / or update the status of the node and generate a node anomaly alarm notification.

[0204] In one possible implementation, the analysis module is specifically used for:

[0205] Extract abnormal features from the test results, including abnormal response codes and abnormal response times;

[0206] Cluster analysis is performed on the abnormal features to identify clusters of fault modes with similar characteristics;

[0207] The fault mode clusters are classified and labeled based on the historical fault knowledge base to obtain at least one type of fault feature.

[0208] In one possible implementation, the analysis module is further configured to:

[0209] For service deployment anomalies, generate service restart and version rollback suggestions;

[0210] For database connection leakage-related faults, generate connection pool tuning and resource monitoring recommendations;

[0211] For interface contract change-related faults, generate update suggestions for expected test case results.

[0212] Based on the same technical concept, embodiments of this application also provide an electronic device, such as... Figure 6 As shown, it includes a processor 111, a communication interface 112, a memory 113, and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0213] Memory 113 is used to store computer programs;

[0214] When processor 111 executes a program stored in memory 113, it performs the following steps:

[0215] Obtain the metadata attributes and health assessment metrics of the use cases in the use case library;

[0216] Based on the metadata attributes, the health assessment indicators, and the impact scope of the current code changes, a target set of use cases is selected from the use case library;

[0217] Execute the target test case set to obtain test results;

[0218] The test results are analyzed to identify at least one type of fault characteristic and generate corresponding optimization suggestions.

[0219] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0220] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0221] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0222] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0223] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described automated testing methods.

[0224] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the automated testing methods described above.

[0225] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0226] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0227] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0228] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An automated testing method, characterized in that, The method includes: Obtain the metadata attributes and health assessment metrics of the use cases in the use case library; Based on the metadata attributes, the health assessment indicators, and the impact scope of the current code changes, a target set of use cases is selected from the use case library; Execute the target test case set to obtain test results; The test results are analyzed to identify at least one type of fault characteristic and generate corresponding optimization suggestions.

2. The method according to claim 1, characterized in that, The health assessment metrics for each use case in the use case library include: For each use case, a stability evaluation value is calculated based on the number of consecutive failures of the use case, wherein the stability evaluation value decreases as the number of consecutive failures increases; Based on the business weights and interface levels covered by the use cases, a business value assessment value is calculated, wherein the business weights are used to characterize the importance of the corresponding business functions in the system, and the interface levels are used to characterize the criticality level of the interfaces covered by the use cases. Based on the stability assessment value and the business value assessment value, a health assessment index is calculated.

3. The method according to claim 1, characterized in that, The step of filtering target use case sets from the use case library based on the metadata attributes, the health assessment indicators, and the impact scope of the current code changes includes: Based on the interface association characteristics of the test cases analyzed by the metadata attributes, a first candidate set related to the current test requirements is selected; Based on the health assessment indicators, the use cases in the first candidate set are evaluated for stability, and a second candidate set that meets the health standards is selected. Based on the scope of impact, a change correlation analysis is performed on the second candidate set to filter out the target use case set that matches the code change.

4. The method according to claim 1, characterized in that, The execution of the target test case set to obtain test results includes: The target use case set is asynchronously distributed to the execution cluster via a message middleware; Obtain the node processing capacity information of the currently available execution nodes in the execution cluster; Analyze the interface call relationships among the various use cases in the target use case set; The number of shards is calculated based on the node processing capacity information and the interface call correlation. Tasks are allocated based on the number of shards to execute the target test case set and obtain test results.

5. The method according to claim 1, characterized in that, The method further includes: The response time of each node is monitored in real time using service probes; When the response time of any node exceeds a preset threshold: stop assigning new tasks to the node, and / or migrate the tasks already assigned to the node to a healthy node, and / or update the status of the node and generate a node anomaly alarm notification.

6. The method according to claim 1, characterized in that, The analysis of the test results to identify at least one type of fault characteristic includes: Extract abnormal features from the test results, including abnormal response codes and abnormal response times; Cluster analysis is performed on the abnormal features to identify clusters of fault modes with similar characteristics; The fault mode clusters are classified and labeled based on the historical fault knowledge base to obtain at least one type of fault feature.

7. The method according to claim 1, characterized in that, The generation of corresponding optimization suggestions includes: For service deployment anomalies, generate service restart and version rollback suggestions; For database connection leakage-related faults, generate connection pool tuning and resource monitoring recommendations; For interface contract change-related faults, generate update suggestions for expected test case results.

8. An automated testing device, characterized in that, The device includes: The acquisition module is used to acquire metadata attributes and health assessment metrics of test cases in the test case library; The filtering module is used to filter target use case sets from the use case library based on the metadata attributes, the health assessment indicators, and the impact scope of the current code changes; The execution module is used to execute the target test case set and obtain test results; The analysis module is used to analyze the test results to identify at least one type of fault characteristic and generate corresponding optimization suggestions.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the automated testing method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the automated testing method according to any one of claims 1-7.

Images