System testing method and apparatus

Abstract

The application relates to the technical field of data testing, and provides a system testing method and device. The method comprises the following steps: according to an obtained to-be-tested system, querying a historical testing system in a historical system database to obtain a similar system set of the to-be-tested system; and according to an initial fault testing scheme executed by each system in the similar system set and a historical fault testing record of the to-be-tested system itself, generating a candidate fault testing scheme set; taking each candidate fault testing scheme in the candidate fault testing scheme set as a target fault testing scheme in sequence, determining a recommendation score of the target fault testing scheme according to an utility value of the target fault testing scheme and the similarity between each system in the similar system set and the to-be-tested system, and determining a final fault testing scheme from the candidate fault testing scheme set based on the recommendation score to test the to-be-tested system. The application can efficiently select a fault testing scheme suitable for the to-be-tested system, and improves the efficiency of system testing.

Description

Technical Field

[0001] This application belongs to the field of data testing technology, and in particular relates to a system testing method and apparatus. Background Technology

[0002] With the deepening of industrial informatization, system stability faces severe challenges. Reliability testing, which actively injects faults to verify the fault tolerance of a system, is an important means of ensuring system stability. However, traditional reliability testing suffers from inefficient fault selection, and effective fault experience cannot be reused across different systems. Faults that have been verified as invalid or of low value are repeatedly executed, resulting in wasted resources and making it difficult to guarantee the stable operation of the system after it goes live. Summary of the Invention

[0003] In view of the shortcomings of the prior art, this application provides a system testing method and apparatus, which at least solves the problems of low efficiency in fault scheme selection, inability to reuse effective fault experience between different systems, repeated execution of verified invalid or low-value faults, resulting in resource waste and difficulty in ensuring the stable operation of the system after it goes online.

[0004] Firstly, this application provides a system testing method, including: Based on the obtained system to be tested, query the historical test systems in the historical system database to obtain a set of similar systems that match the system to be tested; Obtain the initial fault test plans executed by each system in the set of similar systems, and generate a set of candidate fault test plans based on the initial fault test plans and the historical fault test records of the system under test. Each candidate fault test scheme in the candidate fault test scheme set is taken as the target fault test scheme in turn. The recommended score of the target fault test scheme is determined based on the utility value of the target fault test scheme and the similarity between each system in the similar system set and the system to be tested. The utility value represents the degree of contribution of the target fault test scheme to the discovery of system faults. Based on the recommended scores, the final fault test scheme is determined from the set of candidate fault test schemes, and the final fault test scheme is used to test the system under test.

[0005] In one embodiment of this application, a set of candidate fault test plans is generated based on an initial fault test plan and the historical fault test records of the system under test, including: If the system under test does not have historical fault test records, a set of candidate fault test schemes is determined based on the initial fault test scheme. If the system under test has historical fault test records, valid fault test schemes from the historical fault test records are obtained, where valid fault test schemes are schemes with utility values ​​greater than or equal to a preset utility threshold. A set of candidate fault test schemes is determined based on the initial fault test scheme and the valid fault test schemes.

[0006] In one embodiment of this application, a recommended score for the target fault testing scheme is determined based on the utility value of the target fault testing scheme and the similarity between each system in the similar system set and the system to be tested, including: Obtain the first utility value of the target fault test plan for each system in the similar system set; determine the recommended score based on the similarity between each system in the similar system set and the system to be tested and the first utility value.

[0007] In one embodiment of this application, a recommended score for the target fault testing scheme is determined based on the utility value of the target fault testing scheme and the similarity between each system in the similar system set and the system to be tested, including: Obtain the second utility value of the system under test executing the target fault test plan in historical fault tests, and obtain the third utility value of each system in the set of similar systems executing the target fault test plan; determine the initial recommended score based on the similarity between each system in the set of similar systems and the system under test and the third utility value; determine the recommended score based on the second utility value and the initial recommended score.

[0008] In one embodiment of this application, determining the recommendation score based on the second utility value and the initial recommendation score includes: Based on historical fault test records, determine the number of historical fault tests conducted on the system under test; based on the number of historical fault tests conducted, determine the first weighting coefficient and the second weighting coefficient; using the first weighting coefficient and the second weighting coefficient, perform a weighted summation of the second utility value and the initial recommended score to obtain the recommended score.

[0009] In one embodiment of this application, based on the acquired system to be tested, historical test systems are queried from the historical system database to obtain a set of similar systems that match the system to be tested, including: Based on the acquired system to be tested, the target feature vector of the system to be tested is extracted; the similarity between the target feature vector and the historical feature vector is queried to obtain the similarity calculation result, wherein the historical feature vector corresponds one-to-one with each system in the historical system database; based on the similarity calculation result, the systems with similarity greater than or equal to the preset similarity threshold are taken as the set of similar systems.

[0010] In one embodiment of this application, extracting the target feature vector of the system under test includes: Extract the first feature vector and the second feature vector of the system under test, where the first feature vector is the type feature vector of the system under test and the second feature vector is the numerical feature vector of the system under test; convert the first feature vector into a one-hot encoded vector and normalize the second feature vector to obtain the second normalized vector; concatenate the one-hot encoded vector and the normalized second feature vector to obtain the target feature vector.

[0011] In one embodiment of this application, after testing the system under test using the final fault test scheme, the method further includes: Obtain the test results and determine the original utility value of each fault test scheme in the final fault test scheme based on the test results; in response to the received tagging information, correct the original utility value of each fault test scheme to obtain the utility value of each fault test scheme in the final fault test scheme.

[0012] In one embodiment of this application, the method further includes: When any fault test scheme in the final fault test scheme is executed, the associated faults related to any fault test scheme are determined based on the pre-built fault similarity matrix, and the associated faults are displayed through the target terminal.

[0013] Secondly, this application also provides a system testing apparatus, comprising: The query module is configured to query the historical system database for historical test systems based on the obtained system to be tested, and obtain a set of similar systems that match the system to be tested; The generation module is configured to obtain the initial fault test plans executed by each system in the set of similar systems, and generate a set of candidate fault test plans based on the initial fault test plans and the historical fault test records of the system under test. The recommendation module is configured to sequentially take each candidate fault test scheme in the candidate fault test scheme set as the target fault test scheme, and determine the recommendation score of the target fault test scheme based on the utility value of the target fault test scheme and the similarity between each system in the similar system set and the system to be tested. The utility value represents the degree of contribution of the target fault test scheme to the discovery of system faults. The testing module is configured to determine the final fault test plan from the set of candidate fault test plans based on the recommended score, and then use the final fault test plan to test the system under test.

[0014] Thirdly, this application also provides an electronic device, which includes: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the steps of the above-described system testing method.

[0015] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a computer's processor, causes the computer to implement the steps of the above-described system testing method.

[0016] The beneficial effects of this technical solution are as follows: After obtaining the system to be tested, a set of similar systems matching the system to be tested is obtained by querying historical test systems in the historical system database; the initial fault test plans executed by each system in the similar system set are obtained, and a set of candidate fault test plans is generated based on the initial fault test plans and the historical fault test records of the system to be tested; each candidate fault test plan in the set of candidate fault test plans is used as the target fault test plan in turn, and the recommended score of the target fault test plan is determined based on the utility value of the target fault test plan and the similarity between each system in the similar system set and the system to be tested, where the utility value represents the degree of contribution of the target fault test plan to the discovery of system faults; based on the recommended score, the final fault test plan is determined from the set of candidate fault test plans, and the system to be tested is tested using the final fault test plan. In this way, without relying on expert experience, suitable fault testing schemes for the system under test can be selected efficiently. This avoids, to some extent, the problem of invalid fault testing schemes occupying system operating resources and low-value fault testing schemes being repeatedly executed, thereby improving the utilization efficiency of test resources and the efficiency of system testing. At the same time, it can cover high-risk faults within a limited test window, fully verify the fault tolerance capability of the system, and ensure the stable operation of the system after it goes live.

[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings: Figure 1 This is a schematic flowchart illustrating a system testing method in an exemplary embodiment of this application; Figure 2 This is a schematic diagram illustrating the structure of a system testing apparatus according to an exemplary embodiment of this application; Figure 3 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0019] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0020] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. The drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the shape, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0021] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the present application. However, it will be apparent to those skilled in the art that embodiments of the present application may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the present application.

[0022] Please see Figure 1 , Figure 1 This is a flowchart illustrating a system testing method as shown in an exemplary embodiment of this application. Figure 1 As shown, in an exemplary embodiment, the system testing method includes steps S110 to S140, and each step is described in detail below.

[0023] S110, Based on the obtained system to be tested, query the historical test systems in the historical system database to obtain a set of similar systems that match the system to be tested; The historical system database is a pre-built database that stores historical experimental data. This allows for quick retrieval of matching historical experimental data when testing new systems to be tested, thereby improving the efficiency of test plan generation.

[0024] The historical system database includes three core tables: the system table, the fault table, and the experiment table.

[0025] The `system` table stores system characteristics, the `fault` table stores the definitions of fault scenarios, and the `experiment` table stores detailed records and utility values ​​for each experiment. After each experiment is completed, the experimental data (including system identifiers, feature vectors, fault test schemes, experimental results, etc.) are stored in the historical experiment database, and the corresponding utility values ​​are calculated and recorded, thus forming a unified data standard and providing a data foundation for queries.

[0026] The utility value is a quantitative indicator that characterizes the contribution of the fault testing scheme to the discovery of system faults. The utility value can be calculated based on the test results after each historical test experiment is performed. The detailed calculation process will be explained in subsequent embodiments and will not be elaborated here.

[0027] In some embodiments, based on the acquired system to be tested, historical test systems are queried from the historical system database to obtain a set of similar systems that match the system to be tested, including: Based on the acquired system to be tested, the target feature vector of the system to be tested is extracted; the similarity between the target feature vector and the historical feature vector is queried to obtain the similarity calculation result, wherein the historical feature vector corresponds one-to-one with each system in the historical system database; based on the similarity calculation result, the systems with similarity greater than or equal to the preset similarity threshold are taken as the set of similar systems.

[0028] The target feature vector refers to the vector representation obtained after feature extraction from the original data of the system under test, which is used to characterize the essential attributes or content features of the system under test.

[0029] Specifically, after acquiring the system to be tested, the target feature vector can be extracted from the system. Then, the similarity between the target feature vector and the historical feature vector is queried in the historical system database. The historical feature vectors of each system in the historical experimental database are pre-stored in the system table of data storage unit 1 in a one-to-one correspondence. The construction method of the historical feature vector is the same as that of the target feature vector, ensuring that the historical feature vector and the target feature vector are consistent in vector dimension and scale.

[0030] In some examples, cosine similarity can be used to measure the similarity between systems by calculating the cosine of the angle between the target feature vector and the historical feature vectors. The specific calculation method is as follows:

[0031] in, Let T be the similarity between the system to be tested and system s in the set of similar systems. For the target feature vector, Let be the historical eigenvectors of system s.

[0032] After obtaining the similarity calculation results, systems with a similarity greater than or equal to a preset similarity threshold (e.g., 0.7) are selected as a set of similar systems.

[0033] Additionally, if the set of similar systems is empty after the query, meaning there are no systems in the historical experimental database that are sufficiently similar to the system to be tested, then the top 3 systems with the highest similarity ranking can be selected as the set of similar systems, and a warning log can be recorded to indicate that the reference systems for subsequent recommendation results have limited similarity, so that testers can make a comprehensive judgment when adopting recommendations.

[0034] In this way, we can ensure the similarity between each system in the set of similar systems and the system under test in normal scenarios, and also ensure that the recommendation process can continue to be executed in scenarios without highly similar systems, thus balancing recommendation quality and process robustness.

[0035] In some embodiments, extracting the target feature vector of the system under test includes: Extract the first and second feature vectors of the system under test; convert the first feature vector into a one-hot encoded vector and normalize the second feature vector to obtain a second normalized vector; concatenate the one-hot encoded vector and the normalized second feature vector to obtain the target feature vector.

[0036] After obtaining the system to be tested, extract information such as the programming language, architecture pattern, communication protocol, business type, whether there is caching and caching type, whether there is a database and database type, whether there is a message queue, the average number of requests per second (QPS) over the past 7 days, the 99th percentile latency (P99) baseline, the average error rate, the average central processing unit utilization (CPU) utilization, and the average memory utilization.

[0037] If any information is missing, its value is set to 0 to ensure that the dimension of the feature vector remains complete even if some feature information is not fully collected, thus avoiding the problem that subsequent vector operations cannot be performed due to the lack of individual features.

[0038] The extracted information includes a first feature vector and a second feature vector. The first feature vector is the type feature vector of the system under test, including programming language, architectural pattern, communication protocol, business type, and type features of middleware such as cache, database, and message queue; the second feature vector is the numerical feature vector of the system under test, including operational metrics such as average QPS, P99 latency baseline, average error rate, average CPU utilization, and average memory utilization.

[0039] For the first feature vector, also known as the type feature vector, it is converted into a one-hot encoded vector. For example, if there are 6 possible programming languages ​​(Java, Go, Python, C++, Node.js, and others), a 6-dimensional vector is generated, with the language position being 1 and the rest being 0. The same method is used for architecture patterns, communication protocols, business types, cache types, database types, and message queue types, with each dimension encoded independently. All one-hot vectors are concatenated in a fixed order.

[0040] The second eigenvector, i.e., the numerical eigenvector, is normalized to obtain the second normalized vector. For example, for QPS and P99 latency, since the numerical range may span several orders of magnitude, logarithmic normalization is used: first calculate log... 10 (x+1), then scaled to the [0,1] range using min-max. The global minimum and maximum values ​​are obtained from the historical database; if the historical database is empty, empirical values ​​are used (QPS logarithm range 0-6, P99 latency logarithm range 0-4). Error rate, CPU utilization, and memory utilization are already in the [0,1] range and require no additional processing.

[0041] The one-hot encoded vector and the normalized second feature vector are concatenated to obtain the target feature vector. For example, all the one-hot vectors and the normalized numerical vectors are concatenated in a fixed order to form a 46-dimensional (dynamically adjusted) vector, where each dimension takes values ​​between [0,1]. This vector serves as the target feature vector of the system T under test.

[0042] In this way, the discrete semantics of type feature vectors are transformed into equidistant relations in vector space, the cross-order-of-magnitude differences of numerical feature vectors can be effectively compressed, the dimensional differences are uniformly eliminated, and the two are ensured to be comparable in a unified dimensional space.

[0043] S120: Obtain the initial fault test plans executed by each system in the set of similar systems, and generate a set of candidate fault test plans based on the initial fault test plans and the historical fault test records of the system under test. Fault testing schemes refer to a fault scenario and its parameter configuration that are actively injected into the system during reliability testing, such as network latency injection, process termination, disk fullness, etc.

[0044] In some embodiments, a set of candidate fault test schemes is generated based on the initial fault test scheme and the historical fault test records of the system under test, including: If the system under test does not have historical fault test records, a set of candidate fault test schemes is determined based on the initial fault test scheme. If the system under test has historical fault test records, valid fault test schemes from the historical fault test records are obtained, where valid fault test schemes are schemes with utility values ​​greater than or equal to a preset utility threshold. A set of candidate fault test schemes is determined based on the initial fault test scheme and the valid fault test schemes.

[0045] First, traverse each system in the set of similar systems, extract all the initial fault test plans executed by each system, and at the same time determine the historical fault test records of the system under test.

[0046] If the system under test has no historical fault test records, it indicates that the system is in a cold start scenario and has no experience to refer to. In this case, a set of candidate fault test schemes can be determined based on the initial fault test scheme. It is recommended to rely entirely on the fault test experience of historical systems in the set of similar systems, so that the new system can obtain an evidence-based recommendation even without any experimental data of its own.

[0047] In some examples, the initial fault test schemes for each system can be sorted from high to low according to their utility value, and the top 5 fault test schemes in each system can be extracted. All the top 5 fault test schemes can be merged, and duplicate fault test schemes can be deduplicated to obtain a set of candidate fault test schemes.

[0048] If the system under test has historical fault test records, valid fault test schemes can be obtained from these records. Valid fault test schemes are those with a utility value greater than or equal to a preset utility threshold (e.g., 20). Simultaneously, the initial fault test schemes for each system in the similar system set are sorted from highest to lowest utility value, and the top 5 fault test schemes in each system are extracted. All top 5 fault test schemes are merged, and duplicate fault test schemes are deduplicated. The valid fault test schemes and the deduplicated fault test schemes are then merged into a candidate fault test scheme set.

[0049] In this way, by distinguishing between two scenarios—whether the system under test has historical records or not—targeted candidate sets are generated, ensuring that all solutions in the candidate fault test solution set have high potential value, thereby improving the quality of recommendation results and the utilization efficiency of test resources.

[0050] S130, each candidate fault test scheme in the candidate fault test scheme set is taken as the target fault test scheme in turn. Based on the utility value of the target fault test scheme and the similarity between each system in the similar system set and the system to be tested, the recommendation score of the target fault test scheme is determined. The utility value represents the degree of contribution of the target fault test scheme to the discovery of system faults. In some embodiments, a recommended score for the target fault test scheme is determined based on the utility value of the target fault test scheme and the similarity between each system in the set of similar systems and the system under test, including: Obtain the first utility value of the target fault test plan for each system in the similar system set; determine the recommended score based on the similarity between each system in the similar system set and the system to be tested and the first utility value.

[0051] If the system under test does not have any historical fault test records, the first utility value of each system in the similar system set for executing the target fault test plan is obtained. Using the first calculation formula, the recommended score is determined based on the similarity between each system in the similar system set and the system under test, and the first utility value. The first calculation formula is as follows:

[0052] in, Recommended score, Let s be a set of similar systems, and let s be any system in the set of similar systems. Let T be the similarity between the system to be tested and the system s. The first utility value for executing fault f in system s (or 0 if not executed).

[0053] In some embodiments, a recommended score for the target fault test scheme is determined based on the utility value of the target fault test scheme and the similarity between each system in the set of similar systems and the system under test, including: Obtain the second utility value of the system under test executing the target fault test plan in historical fault tests, and obtain the third utility value of each system in the set of similar systems executing the target fault test plan; determine the initial recommended score based on the similarity between each system in the set of similar systems and the system under test and the third utility value; determine the recommended score based on the second utility value and the initial recommended score.

[0054] If the system under test has historical fault test records, then obtain the second utility value of the system under test executing the target fault test plan in the historical fault test, and obtain the third utility value of each system in the set of similar systems executing the target fault test plan; Using the first calculation formula, an initial recommendation score is determined based on the similarity between each system in the similar system set and the system to be tested, and the third utility value. Then, using the second calculation formula, a recommendation score is determined based on the second utility value and the initial recommendation score. The second calculation formula is shown below:

[0055] in, The second utility value, The initial recommendation score, As the first weighting coefficient, This is the second weighting coefficient.

[0056] In some embodiments, determining the recommendation score based on the second utility value and the initial recommendation score includes: Based on historical fault test records, determine the number of historical fault tests conducted on the system under test; based on the number of historical fault tests conducted, determine the first weighting coefficient and the second weighting coefficient; using the first weighting coefficient and the second weighting coefficient, perform a weighted summation of the second utility value and the initial recommended score to obtain the recommended score.

[0057] Based on the historical fault test records of the system under test, the number of historical fault tests actually performed on the system is counted from the historical experimental database. Then, according to the interval in which the number of historical fault tests falls, a first weighting coefficient α and a second weighting coefficient β are determined according to a preset grading rule. Finally, using the first and second weighting coefficients, the second utility value and the initial recommended score are weighted and summed to obtain the recommended score.

[0058] For example, the grading rules can be as follows: when the number of historical fault tests is less than 5, α is 0.3 and β is 0.7; when the number of historical fault tests is 5 to 10, α is 0.5 and β is 0.5; when the number of historical fault tests is more than 10, α is 0.7 and β is 0.3. The sum of α and β is always 1. When the number of historical fault tests is less than 5, the data sample of the system under test is insufficient, and the statistical reliability of its own experience is low. Therefore, the recommendation result focuses on the set of similar systems, i.e., β is set to a larger value. When the number of historical fault tests is 5 to 10, the system's own data has accumulated to a certain scale, and a balanced strategy is adopted, with α and β each set to 0.5. When the number of historical fault tests is more than 10, the system's own experience has been fully accumulated, and its adaptability to the characteristics of the system under test is higher than the indirect experience of the set of similar systems. Therefore, the recommendation result focuses on the system's own historical experience, i.e., α is set to a larger value.

[0059] In this way, based on the number of historical fault tests, the first weight coefficient and the second weight coefficient are determined. As the number of historical fault tests of the system under test accumulates, the credibility of its own experience gradually increases, and the weight gradually shifts towards the first weight coefficient. This improves the accuracy of the recommendation results in adapting to the characteristics of the system under test and addresses the problem that the fixed weight strategy cannot take into account the recommendation quality at different data accumulation stages.

[0060] S140, based on the recommended score, determine the final fault test plan from the set of candidate fault test plans, and use the final fault test plan to test the system under test.

[0061] Based on the recommendation scores, each candidate fault test plan in the candidate fault test plan set is ranked, and the final fault test plan is determined according to the ranking results. For example, the top 5 candidate fault test plans with the highest recommendation scores can be used as the final fault test plans for testing the system under test.

[0062] At the same time, when determining the final fault test plan, recommendation information can be output simultaneously. The recommendation information can include the name, parameters, recommendation score and reasons for the final fault test plan, so as to enhance the interpretability of the recommendation results and make it easier for testers to understand and adopt the recommendation results.

[0063] In some embodiments, after testing the system under test using the final failure test scheme, the method further includes: Obtain the test results and determine the original utility value of each fault test scheme in the final fault test scheme based on the test results; in response to the received tagging information, correct the original utility value of each fault test scheme to obtain the utility value of each fault test scheme in the final fault test scheme.

[0064] After testing the system under test using the final failure test plan, test results can be obtained. These results include four elements: whether a problem was found, its severity level, whether a new problem was found, and the time taken. Then, using the utility value calculation formula, the original utility value of each failure test plan in the final failure test plan is determined based on the test results. The utility value calculation formula is as follows:

[0065] in, The above formula represents the original utility value. If a problem is found, the value is 30; otherwise, it is 0. If the problem is new, the value is 0; the severity level ranges from 1 to 5, with 5 being the most severe; if the problem is new, the value is 25, and if the problem is not new, the value is 0; the time unit is minutes.

[0066] The labeling information can characterize the testers' evaluation of this fault test plan.

[0067] After receiving the tagging information input by the tester, the original utility value of each fault test plan is corrected using this tagging information. For example, if the tagging information is valid, the utility value of the fault test plan is... If the flag is invalid, the utility value of the fault test plan is... If the user does not input any marking information, the utility value of this fault test plan is... .

[0068] The utility values ​​of each candidate fault test scheme are written into the historical experimental database, replacing or supplementing the corresponding utility value records for use in the next round of recommendations. In this way, the experimental results generated after executing the final fault test scheme are transformed into utility value updates, ensuring that the utility values ​​continuously reflect the actual value of the fault test schemes based on experimental feedback, providing more reliable input data for calculating recommendation scores.

[0069] In some embodiments, the method further includes: When any fault test scheme in the final fault test scheme is executed, the associated faults related to any fault test scheme are determined based on the pre-built fault similarity matrix, and the associated faults are displayed through the target terminal.

[0070] Specifically, the Pearson correlation coefficients of the utility values ​​between each pair of fault test schemes in the historical experimental database can be calculated in advance, and a fault similarity matrix can be constructed using the Pearson correlation coefficients as matrix elements.

[0071] When any fault test scheme in the final fault test scheme is executed, based on the pre-built fault similarity matrix, fault test schemes with a Pearson correlation coefficient greater than or equal to a preset correlation coefficient threshold (e.g., 0.7) with the fault test scheme are searched from the matrix. The fault test schemes that meet the condition are sorted from high to low according to the correlation coefficient, and the top 3 fault test schemes with the highest correlation are selected as associated faults.

[0072] By displaying associated faults to users through the target terminal, users can easily view and select associated fault test plans when executing any fault test plan in the final fault test plan, improving the operational convenience of combined fault testing. For example, when a user executes a fault test plan and discovers a problem, executing highly related associated faults is expected to further uncover similar problems and trigger deeper defects that can only be exposed through the combined effect of multiple faults.

[0073] In this way, by exploring the combined effects between faults, we can expand the breadth of recommendations, improve the problem that a single fault test plan is insufficient to fully expose system defects, and enhance the defect detection capability in combined fault scenarios.

[0074] According to the technical solution provided in the embodiments of this application, after obtaining the system to be tested, a set of similar systems matching the system to be tested is obtained by querying historical test systems in the historical system database; the initial fault test schemes executed by each system in the similar system set are obtained, and a set of candidate fault test schemes is generated based on the initial fault test schemes and the historical fault test records of the system to be tested itself; each candidate fault test scheme in the set of candidate fault test schemes is used as a target fault test scheme in turn, and a recommended score for the target fault test scheme is determined based on the utility value of the target fault test scheme and the similarity between each system in the similar system set and the system to be tested, wherein the utility value represents the degree of contribution of the target fault test scheme to the discovery of system faults; based on the recommended score, a final fault test scheme is determined from the set of candidate fault test schemes, and the system to be tested is tested using the final fault test scheme. In this way, without relying on expert experience, suitable fault testing schemes for the system under test can be selected efficiently. This avoids, to some extent, the problem of invalid fault testing schemes occupying system operating resources and low-value fault testing schemes being repeatedly executed, thereby improving the utilization efficiency of test resources and the efficiency of system testing. At the same time, it can cover high-risk faults within a limited test window, fully verify the fault tolerance capability of the system, and ensure the stable operation of the system after it goes live.

[0075] It should be understood that the order of the steps in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the process of the embodiments of this application.

[0076] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0077] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0078] Figure 2 This is a schematic diagram of the structure of a system testing apparatus shown in an exemplary embodiment of this application. Figure 2 As shown, the exemplary system testing apparatus includes: The query module 210 is configured to query the historical test systems in the historical system database based on the obtained system to be tested, and obtain a set of similar systems that match the system to be tested; The generation module 220 is configured to obtain the initial fault test plans executed by each system in the set of similar systems, and generate a set of candidate fault test plans based on the initial fault test plans and the historical fault test records of the system under test. The recommendation module 230 is configured to sequentially take each candidate fault test scheme in the candidate fault test scheme set as the target fault test scheme, and determine the recommendation score of the target fault test scheme based on the utility value of the target fault test scheme and the similarity between each system in the similar system set and the system to be tested. The utility value represents the degree of contribution of the target fault test scheme to the discovery of system faults. Test module 240 is configured to determine the final fault test scheme from the set of candidate fault test schemes based on the recommended score, and use the final fault test scheme to test the system under test.

[0079] In some embodiments, the generation module 220 is further configured to: if the system under test does not have historical fault test records, determine a set of candidate fault test schemes based on the initial fault test scheme; if the system under test has historical fault test records, obtain valid fault test schemes from the historical fault test records, wherein a valid fault test scheme is a scheme with a utility value greater than or equal to a preset utility threshold, and determine a set of candidate fault test schemes based on the initial fault test scheme and the valid fault test schemes.

[0080] In some embodiments, the recommendation module 230 is further configured to obtain the first utility value of each system in the set of similar systems performing the target fault test scheme; and to determine the recommendation score based on the similarity between each system in the set of similar systems and the system to be tested and the first utility value.

[0081] In some embodiments, the recommendation module 230 is further configured to obtain a second utility value of the system under test executing the target fault test scheme in historical fault tests, and to obtain a third utility value of each system in the set of similar systems executing the target fault test scheme; determine an initial recommendation score based on the similarity between each system in the set of similar systems and the system under test and the third utility value; and determine a recommendation score based on the second utility value and the initial recommendation score.

[0082] In some embodiments, the recommendation module 230 is further configured to determine the number of historical fault tests of the system under test based on historical fault test records; determine a first weighting coefficient and a second weighting coefficient based on the number of historical fault tests; and use the first weighting coefficient and the second weighting coefficient to perform a weighted summation of the second utility value and the initial recommendation score to obtain a recommendation score.

[0083] In some embodiments, the query module 210 is further configured to extract the target feature vector of the system to be tested based on the acquired system to be tested; query the similarity between the target feature vector and the historical feature vector to obtain the similarity calculation result, wherein the historical feature vector corresponds one-to-one with each system in the historical system database; and, based on the similarity calculation result, take the systems with a similarity greater than or equal to a preset similarity threshold as the set of similar systems.

[0084] In some embodiments, the query module 210 is further configured to extract a first feature vector and a second feature vector of the system under test, wherein the first feature vector is a type feature vector of the system under test and the second feature vector is a numerical feature vector of the system under test; convert the first feature vector into a one-hot encoded vector and normalize the second feature vector to obtain a second normalized vector; and concatenate the one-hot encoded vector and the normalized second feature vector to obtain a target feature vector.

[0085] In some embodiments, the test module 240 is further configured to acquire test results, determine the original utility value of each fault test scheme in the final fault test scheme based on the test results, and correct the original utility value of each fault test scheme in response to the received tagging information to obtain the utility value of each fault test scheme in the final fault test scheme.

[0086] In some embodiments, the test module 240 is further configured to, when executing any fault test scheme in the final fault test scheme, determine the associated faults associated with any fault test scheme based on a pre-built fault similarity matrix, and display the associated faults through the target terminal.

[0087] Embodiments of this application also provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to implement the methods provided in the above embodiments.

[0088] Figure 3 A schematic diagram of a computer system suitable for implementing the embodiments of this application is shown. It should be noted that... Figure 3 The computer system 300 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0089] like Figure 3As shown, the computer system 300 includes a Central Processing Unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 302 or programs loaded from storage portion 308 into Random Access Memory (RAM) 303, such as executing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An Input / Output (I / O) interface 305 is also connected to the bus 304.

[0090] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0091] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs various functions defined in the system of this application.

[0092] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0094] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0095] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0096] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods described in the various embodiments above.

[0097] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the steps of this application.

Claims

1. A system testing method, characterized in that, include: Based on the obtained system to be tested, query the historical test systems in the historical system database to obtain a set of similar systems that match the system to be tested; Obtain the initial fault test plans executed by each system in the set of similar systems, and generate a set of candidate fault test plans based on the initial fault test plans and the historical fault test records of the system under test. Each candidate fault test scheme in the candidate fault test scheme set is taken as the target fault test scheme in turn. The recommendation score of the target fault test scheme is determined based on the utility value of the target fault test scheme and the similarity between each system in the similar system set and the system to be tested. The utility value represents the degree of contribution of the target fault test scheme to the discovery of system faults. Based on the recommended score, a final fault test scheme is determined from the set of candidate fault test schemes, and the final fault test scheme is used to test the system under test.

2. The method according to claim 1, characterized in that, The step of generating a candidate fault test scheme set based on the initial fault test scheme and the historical fault test records of the system under test includes: If the system under test does not have any historical fault test records, then a set of candidate fault test schemes is determined based on the initial fault test scheme. If the system under test has historical fault test records, then the effective fault test schemes in the historical fault test records are obtained, wherein the effective fault test schemes are schemes with a utility value greater than or equal to a preset utility threshold. Based on the initial fault test schemes and the effective fault test schemes, a set of candidate fault test schemes is determined.

3. The method according to claim 2, characterized in that, The step of determining the recommended score of the target fault test scheme based on the utility value of the target fault test scheme and the similarity between each system in the similar system set and the system to be tested includes: Obtain the first utility value of the target fault test scheme for each system in the set of similar systems; The recommendation score is determined based on the similarity between each system in the set of similar systems and the system to be tested, and the first utility value.

4. The method according to claim 2, characterized in that, The step of determining the recommended score of the target fault test scheme based on the utility value of the target fault test scheme and the similarity between each system in the similar system set and the system to be tested includes: Obtain the second utility value of the system under test executing the target fault test scheme in historical fault tests, and obtain the third utility value of each system in the set of similar systems executing the target fault test scheme; The initial recommendation score is determined based on the similarity between each system in the set of similar systems and the system to be tested, and the third utility value. The recommendation score is determined based on the second utility value and the initial recommendation score.

5. The method according to claim 4, characterized in that, The step of determining the recommendation score based on the second utility value and the initial recommendation score includes: Based on the historical fault test records, determine the number of historical fault tests performed on the system under test; Based on the number of historical fault tests, determine the first weighting coefficient and the second weighting coefficient; The second utility value and the initial recommendation score are weighted and summed using the first weight coefficient and the second weight coefficient to obtain the recommendation score.

6. The method according to claim 1, characterized in that, The step involves querying historical test systems in the historical system database based on the acquired system to be tested, to obtain a set of similar systems that match the system to be tested, including: Based on the acquired system to be tested, extract the target feature vector of the system to be tested; The similarity between the target feature vector and the historical feature vector is queried to obtain the similarity calculation result, wherein the historical feature vector corresponds one-to-one with each system in the historical system database; Based on the similarity calculation results, systems with a similarity greater than or equal to a preset similarity threshold are classified as a set of similar systems.

7. The method according to claim 6, characterized in that, The extraction of the target feature vector of the system under test includes: Extract a first feature vector and a second feature vector from the system under test, wherein the first feature vector is a type feature vector of the system under test, and the second feature vector is a numerical feature vector of the system under test; The first feature vector is converted into a one-hot encoded vector, and the second feature vector is normalized to obtain a second normalized vector. The target feature vector is obtained by concatenating the one-hot encoded vector and the normalized second feature vector.

8. The method according to claim 1, characterized in that, After testing the system under test using the final fault test scheme, the method further includes: Obtain the test results, and determine the original utility value of each fault test scheme in the final fault test scheme based on the test results; In response to the received tagging information, the original utility value of each fault test scheme is corrected to obtain the utility value of each fault test scheme in the final fault test scheme.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: When any of the fault test schemes in the final fault test scheme is executed, the associated faults related to any of the fault test schemes are determined based on the pre-constructed fault similarity matrix, and the associated faults are displayed through the target terminal.

10. A system testing apparatus, characterized in that, include: The query module is configured to query the historical test systems in the historical system database based on the obtained system to be tested, and obtain a set of similar systems that match the system to be tested; The generation module is configured to obtain the initial fault test plans executed by each system in the set of similar systems, and generate a set of candidate fault test plans based on the initial fault test plans and the historical fault test records of the system under test. The recommendation module is configured to sequentially take each candidate fault test scheme in the candidate fault test scheme set as the target fault test scheme, and determine the recommendation score of the target fault test scheme based on the utility value of the target fault test scheme and the similarity between each system in the similar system set and the system to be tested, wherein the utility value represents the degree of contribution of the target fault test scheme to the discovery of system faults; The testing module is configured to determine a final fault test scheme from the set of candidate fault test schemes based on the recommended score, and to test the system under test using the final fault test scheme.

Images