Method and device for automatically testing service reliability

By building a test scenario library and automated testing methods, the existing service reliability testing problems are solved, and efficient and accurate service reliability evaluation is achieved, which is suitable for complex service systems.

CN119003372APending Publication Date: 2024-11-22JINAN JOVISION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411382051.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-30
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing service reliability testing relies on manual operations, consumes a lot of human resources, is inefficient, and it is difficult to fully cover abnormal situations, resulting in poor test stability and unreliable results.

Method used

Build a test scenario library, simulate various abnormal situations through automated testing methods, formulate a test plan based on service reliability testing instructions, call the tested service information and related dependency service information of the test scenario for abnormal simulation, and evaluate the abnormal recovery time to evaluate Service reliability.

Benefits of technology

It reduces human resources waste, improves testing efficiency and accuracy, ensures comprehensiveness and consistency of testing, can run 24 hours a day, provides quantitative reliability evaluation standards, and is suitable for various service types.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003372A_ABST
    Figure CN119003372A_ABST
Patent Text Reader

Abstract

The invention discloses an automatic test method and device for service reliability, and belongs to the field of electric digital data processing, and the method comprises the steps: constructing a test scene library which comprises tested service information and related dependent service information of a plurality of test scenes; a service reliability test instruction is obtained, a test plan is made according to the service reliability test instruction, and the test plan comprises at least one test scene; according to the test plan, calling tested service information and related dependent service information corresponding to the test scene, and carrying out test scene abnormity simulation; and evaluating the service reliability according to the abnormal recovery time of the test scene. According to the abnormal recovery time, service reliability is evaluated, a reliability evaluation rule is set, and an evaluation result is more objective and fair through a quantitative standard. The test scene library is constructed, multiple test scenes are preset, the complexity of the service reliability test is reduced, it is ensured that the test covers various possible fault points, and the comprehensiveness and depth of the test are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of electrical digital data processing, and in particular relates to a service reliability automatic testing method and device. Background Art

[0002] Service reliability testing is a crucial step in the service development process. It aims to evaluate the performance and reliability of services under various conditions by subjecting them to a series of tests. Traditionally, service reliability testing verifies the error handling capabilities of services by providing abnormal inputs to cause application-level dependencies to fail or hang. At the same time, it also simulates usage under extreme conditions to accelerate the aging process of the product, thereby evaluating the life and reliability of the product in a shorter period of time. However, as service systems become more and more complex, existing reliability testing faces many challenges.

[0003] Existing service reliability testing mainly relies on manual operations, such as manually writing and executing Linux commands or shell scripts to simulate middleware failures, dependent service unavailability, server resource limitations, etc. This method not only consumes a lot of human resources, but is also inefficient and difficult to fully cover all possible abnormal situations.

[0004] Current automated testing technology has obvious deficiencies in service reliability testing. Existing automated testing solutions often fail to fully consider various abnormal situations that may be encountered during service operation, such as service downtime, middleware anomalies, service network anomalies, etc. When there are too many factors to consider in the test, the situation becomes more complicated. It is difficult for traditional methods to effectively determine the stability of the service based on multiple factors. In addition, due to the level of testers, different testers obtain different reliability test results for the same service, and the test stability is low. Summary of the invention

[0005] The present application provides a service reliability automated testing method and device to solve the problems of reliability testing consuming a large amount of human resources and being inefficient, as well as the problems of poor test stability and unreliable test results caused by the complex service system structure and the need to consider many factors in reliability testing.

[0006] The technical solution adopted in this application is:

[0007] A service reliability automated testing method, comprising:

[0008] Constructing a test scenario library, wherein the test scenario library includes the tested service information and related dependent service information of multiple test scenarios;

[0009] Obtaining a service reliability test instruction, and formulating a test plan according to the service reliability test instruction, wherein the test plan includes at least one test scenario;

[0010] According to the test plan, calling the tested service information and the related dependent service information corresponding to the test scenario to perform test scenario abnormal simulation;

[0011] Evaluate service reliability based on abnormal recovery time in test scenarios.

[0012] The service reliability automated testing method of the present application also has the following additional technical features:

[0013] According to the test plan, the tested service information and the related dependent service information corresponding to the test scenario are called to perform test scenario abnormal simulation, specifically:

[0014] According to the test plan, calling a plurality of the test scenarios in the test scenario library for service reliability testing;

[0015] Determine the calling order of the multiple test scenarios, and call the test scenarios according to the calling order to perform test scenario abnormality simulation.

[0016] Calling the test scenario in the calling sequence to perform test scenario abnormal simulation also includes:

[0017] Setting the calling time interval of the test scenario;

[0018] If the abnormal simulation time of the test scenario exceeds the calling time interval, the abnormal simulation of the test scenario is stopped, the service is restored to a normal state, and the next test scenario is called in the calling sequence to perform the test scenario abnormal simulation.

[0019] The calling time interval is counted from the start time of the abnormal simulation of the test scenario.

[0020] Determine the calling order of the multiple test scenarios, specifically:

[0021] The test scenarios are set with corresponding importance levels and risk levels;

[0022] Determining a calling order of the test scenarios according to the importance levels;

[0023] If the importance levels of the multiple test scenarios are consistent, the calling order of the test scenarios is further determined according to the risk levels.

[0024] According to the test plan, the tested service information and the related dependent service information corresponding to the test scenario are called to perform test scenario abnormal simulation, specifically:

[0025] According to the test plan, one of the test scenarios in the test scenario library is called for service reliability testing.

[0026] Obtaining a service reliability test instruction, and formulating a test plan according to the service reliability test instruction, wherein the test plan includes at least one test scenario and further includes:

[0027] Obtaining flag information corresponding to the test scenario;

[0028] According to the flag information, comparing the test scenarios in the test plan with the test scenarios in the test scenario library;

[0029] According to the comparison result, the test scenario information in the test scenario library is updated and maintained.

[0030] Evaluate service reliability based on the abnormal recovery time of the test scenario, specifically:

[0031] Setting a maximum tolerable recovery time, and comparing the abnormal recovery time of the test scenario with the maximum tolerable recovery time;

[0032] If the abnormal recovery time of the test scenario is greater than the maximum tolerable recovery time, it is determined that the simulation test of the test scenario fails;

[0033] If the abnormal recovery time of the test scenario is less than or equal to the maximum tolerable recovery time, the test scenario is scored according to a preset scoring rule;

[0034] The test results of multiple test scenarios are combined to give a comprehensive score for the reliability of the service.

[0035] If the abnormal recovery time of the test scenario is less than or equal to the maximum tolerable recovery time, the test scenario is scored according to a preset scoring rule, specifically:

[0036] The time interval between the abnormal recovery time of the test scenario and the maximum tolerable recovery time is calculated, and the smaller the time interval is, the lower the score of the test scenario is.

[0037] This application also adopts a service reliability automated testing device, including:

[0038] A test task scheduling module is used to obtain a service reliability test instruction and formulate a test plan according to the service reliability test instruction;

[0039] The exception simulation module is used to call the tested service information and related dependent service information corresponding to the test scenario according to the test plan to simulate the test scenario exception;

[0040] The result analysis and report generation module is used to evaluate service reliability based on the abnormal recovery time of the test scenario;

[0041] The monitoring and logging module is used to monitor the service system status, service response, exception information, etc. during the test process and generate a test log.

[0042] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0043] 1. In this application, by building a test scenario library, multiple test scenarios are pre-set, and by calling the tested service information and related dependent service information corresponding to the test scenario, test scenario exception simulation is performed. This reduces the complexity of service reliability testing. Testers only need to focus on the tested service itself and issue targeted reliability test instructions. This method can automatically call the test scenario to simulate the test scenario exception, and then generate service reliability evaluation results. Testers do not need to pay attention to the specific details of the test scenario exception simulation, reducing the waste of human resources caused by manual simulation of various test scenarios during testing.

[0044] In addition, according to the obtained service reliability test instructions, the test plan is automatically formulated, the tested service information and related dependent service information corresponding to the test scenario are called, and the test scenario abnormality simulation is performed to evaluate the service reliability. In other words, the user only needs to provide the service reliability test instructions, and this method can automatically execute the test task, reducing manual intervention. In this method, the reliability test can run 24 hours a day, which greatly improves the test efficiency and reduces the time required for the test.

[0045] 2. In this application, the service reliability is evaluated based on the abnormal recovery time of the test scenario. The service reliability is evaluated by recording the abnormal recovery time of the test scenario during the abnormal simulation of the test scenario. Among them, the abnormal recovery time is an important indicator for measuring service reliability, which directly reflects the recovery ability of the service when encountering abnormal situations. When the service can recover from the abnormality quickly, it shows that it has good robustness and robustness. And for users, the shorter the service interruption time, the better the user experience. Therefore, the abnormal recovery time is a key factor in evaluating service reliability. Moreover, no matter what type of service, the abnormal recovery time is a generally applicable evaluation indicator, which is applicable to various test scenarios. Therefore, the abnormal recovery time of the test scenario as an element of service reliability evaluation not only improves the accuracy of this test method, but also improves the versatility of this test method.

[0046] Through the abnormal recovery time, reliability evaluation rules can be set to convert the service reliability evaluation into a quantitative standard, making the evaluation results more objective and fair. In addition, the service reliability under different test scenarios can be directly compared, which is conducive to horizontally evaluating the reliability differences between different services or different versions of services.

[0047] 3. In this application, a test scenario library is constructed, and the test scenario library includes the tested service information and related dependent service information of multiple test scenarios. The test scenario library includes multiple test scenarios, which can ensure that the test covers various possible failure points, improve the comprehensiveness and depth of the test, help discover more potential problems, and improve the overall reliability of the test method. In addition, by building a test scenario library, the test scenarios can be standardized to ensure that each test follows the same standards and processes, thereby improving the repeatability and consistency of the test results. With the continuous updating and maintenance of the test scenario library, a large number of test scenarios have been accumulated, which can be used for the test design of new services to avoid duplication of work.

[0048] In addition, the test scenario library can support parameterized settings, allowing users to flexibly configure test scenarios according to different needs and reduce the complexity of scenario simulation. The test scenario library contains detailed information about the tested services and related dependent services, which can help simulate various abnormal situations more accurately, ensure the consistency of the test environment and the production environment, and improve the credibility of the test method. Moreover, the defined test scenarios can be quickly reused, which can greatly improve the test efficiency, especially when performing regression testing or regular health checks. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0050] Figure 1 A schematic diagram of a process flow of the service reliability automation testing method according to an implementation mode of the present invention;

[0051] Figure 2 It is a schematic diagram of the structure of the service reliability automatic testing device in one implementation mode of the present invention.

[0052] in:

[0053] 11 test task scheduling module; 12 exception simulation module; 13 result analysis and report generation module; 14 monitoring and logging module. DETAILED DESCRIPTION

[0054] In order to more clearly illustrate the overall concept of the present invention, a detailed description is given below in an exemplary manner in conjunction with the accompanying drawings.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0056] like Figure 1 As shown, a service reliability automated testing method includes:

[0057] S101: Build a test scenario library, which includes tested service information and related dependent service information of multiple test scenarios.

[0058] This step aims to build a test scenario library containing multiple test scenarios, so that when performing service reliability testing, the test scenarios in the test scenario library can be called to simulate exceptions, thereby improving the efficiency of service reliability testing. In addition, the test scenario library records the information of the tested service and its related dependent services in detail to ensure the comprehensiveness and accuracy of the test scenarios, thereby improving the quality of service reliability testing.

[0059] Specifically, building a test scenario library includes:

[0060] Define test scenarios that are designed to simulate the performance of the service under different abnormal conditions. The design of the test scenarios should cover the main functional modules of the service and the services it depends on to ensure that the reliability of the service can be fully evaluated.

[0061] For each test scenario, collect basic information about the tested service, such as service name, version, deployment environment, functional module, etc. This information helps to clarify the test object and its characteristics.

[0062] In addition to the service under test, it is also necessary to collect information about related dependent services, including but not limited to middleware, databases, external APIs, etc. The status of these dependent services may directly affect the behavior and performance of the service under test, so they also need to be included in the test scenario library.

[0063] The collected test scenario information is stored in a structured manner in the database to form a test scenario library. Each test scenario should include detailed information such as abnormal operations and expected goals.

[0064] It should be noted that the test scenario library needs to be regularly maintained and updated to reflect changes in services and their dependencies. For example, when a service is upgraded or a new dependent service is introduced, the test scenario library should also be updated accordingly.

[0065] In this step, the test scenario library includes multiple test scenarios, which can ensure that the test covers various possible failure points, improve the comprehensiveness and depth of the test, help discover more potential problems, and improve the overall reliability of the test method. In addition, by building a test scenario library, the test scenarios can be standardized to ensure that each test follows the same standards and processes, thereby improving the repeatability and consistency of the test results. With the continuous updating and maintenance of the test scenario library, a large number of test scenarios have been accumulated, which can be used in the test design of new services to avoid duplication of work.

[0066] In addition, the test scenario library can support parameterized settings, allowing users to flexibly configure test scenarios according to different needs and reduce the complexity of scenario simulation. The test scenario library contains detailed information about the tested services and related dependent services, which can help simulate various abnormal situations more accurately, ensure the consistency of the test environment and the production environment, and improve the credibility of the test method. Moreover, the defined test scenarios can be quickly reused, which can greatly improve the test efficiency, especially when performing regression testing or regular health checks.

[0067] S102: Obtain a service reliability test instruction, and formulate a test plan according to the service reliability test instruction, wherein the test plan includes at least one test scenario.

[0068] This step aims to obtain service reliability test instructions and develop a test plan containing at least one test scenario based on the test instructions to ensure the systematic and targeted nature of the test, thereby comprehensively evaluating the reliability of the service.

[0069] It should be noted that the service reliability test instruction is input by the user. The service reliability test instruction includes the tested service information to specify the tested service. The service reliability test instruction may also include test scenario flag information to facilitate calling the test scenario in the test scenario library for abnormal simulation testing.

[0070] Specifically, this step includes obtaining a service reliability test instruction input by a user.

[0071] Parse the service information under test in the service reliability test instruction to clarify the name of the service under test, so as to distinguish different services, specify the service version to be tested, and ensure that the test is targeted at the correct version.

[0072] Parse the test scenario identification information in the service reliability test instruction to facilitate the reference and management of the test scenarios in this reliability test.

[0073] Include test scenarios in the test plan so that abnormal simulation tests can be carried out in the order of the test plan to avoid omissions or confusion of test scenarios.

[0074] It should be noted that the service reliability test instruction submitted by the user may not include the test scenario identification information. In this case, the method calls the test scenario according to the general test template to perform reliability test on the service. The general test template includes multiple general test scenarios. This further reduces the number of service reliability test instructions that the user needs to enter.

[0075] In addition, users can also define the test scenarios in the test template by themselves and set personalized test templates to improve the pertinence of service reliability testing. When the structures of the services to be tested are the same or similar, the same test template can be used for testing.

[0076] This step formulates a test plan based on the service reliability test instructions to ensure that the test instructions fully meet the actual needs of users, improve the pertinence of the test, and thus improve user satisfaction. At the same time, it can also avoid unnecessary tests for users, reduce the number of invalid tests, and make the test process more efficient.

[0077] By developing a detailed test plan, you can ensure that the test covers all the test scenarios specified by the user, thereby improving the comprehensiveness and completeness of the test. You can also avoid confusion in test scenario calls and ensure that the test scenarios in the test plan are executed in sequence.

[0078] S103: According to the test plan, the tested service information and related dependent service information corresponding to the test scenario are called to perform abnormal simulation of the test scenario.

[0079] The purpose of this step is to call the test scenario from the test scenario library according to the test plan, perform test scenario exception simulation according to the corresponding service information under test and related dependent service information, and ensure accurate execution of the test scenario for exception simulation.

[0080] Specifically, this step includes, reading the test plan and parsing each test scenario in the test plan;

[0081] According to the test scenario identification information in the test plan, locate the corresponding test scenario information from the test scenario library;

[0082] According to the tested service information and related dependent service information corresponding to the test scenario, perform test scenario exception simulation, such as disconnecting the network connection, shutting down the middleware service, etc.

[0083] It should be noted that the test plan includes test scenario identification information, call time interval, call sequence, and recovery mechanism, etc. The test scenario identification information is used to locate the test scenario from the test scenario library; the call time interval and call sequence are used to execute the test scenario in an orderly manner, maintain the orderliness and independence of each test scenario, and avoid test confusion caused by parallel test scenarios; the recovery mechanism is used to perform recovery operations after each test scenario is completed to ensure that the test environment returns to the initial state and does not affect subsequent tests.

[0084] This step simulates the abnormality of the test scenario by calling the tested service information and related dependent service information corresponding to the test scenario. This reduces the complexity of the service reliability test. The tester only needs to focus on the tested service itself and issue targeted reliability test instructions. This method can automatically call the test scenario to simulate the abnormality of the test scenario, and then generate the service reliability evaluation results. The tester does not need to pay attention to the specific details of the abnormal simulation of the test scenario, reducing the waste of human resources caused by manual simulation of various test scenarios during the test.

[0085] In addition, according to the obtained service reliability test instructions, the test plan is automatically formulated, the test scenario is called to call the tested service information and related dependent service information corresponding to the test scenario, and the test scenario abnormality simulation is performed to evaluate the service reliability. In other words, the user only needs to provide the service reliability test instructions, and this method can automatically execute the test task, reducing manual intervention. In this method, the reliability test can run 24 hours a day, which greatly improves the test efficiency and reduces the time required for the test.

[0086] S104: Evaluate service reliability based on the abnormal recovery time of the test scenario.

[0087] This step aims to evaluate the reliability of the service based on the abnormal recovery time in the test scenario, and to ensure that the service can meet the expected reliability requirements by analyzing the recovery performance of the service under abnormal conditions.

[0088] Specifically, this step includes recording the specific time when the service recovers from the abnormal state to the normal state after executing the abnormal simulation in each test scenario;

[0089] Compare the actual abnormal recovery time with the preset maximum tolerance recovery time to determine whether the service recovery time meets the preset threshold or standard, and score each test scenario;

[0090] The reliability of the service is comprehensively scored based on the test results of multiple test scenarios.

[0091] In this step, the abnormal recovery time is an important indicator for measuring service reliability, which directly reflects the service's ability to recover when encountering abnormal situations. When a service can recover quickly from an abnormality, it indicates that it has good robustness and robustness. Moreover, for users, the shorter the service interruption time, the better the user experience. Therefore, the abnormal recovery time is a key factor in evaluating service reliability. Moreover, regardless of the type of service, the abnormal recovery time is a universally applicable evaluation indicator that is suitable for various application scenarios. Therefore, the abnormal recovery time of the test scenario as an element of service reliability evaluation not only improves the accuracy of this test method, but also improves the versatility of this test method.

[0092] Through the abnormal recovery time, reliability evaluation rules can be set to convert the service reliability evaluation into a quantitative standard, making the evaluation results more objective and fair. In addition, the service reliability under different test scenarios can be directly compared, which is conducive to horizontally evaluating the reliability differences between different services or different versions of services.

[0093] As a preferred implementation of the present application, according to the test plan, the tested service information and related dependent service information corresponding to the test scenario are called to perform abnormal simulation of the test scenario, specifically:

[0094] According to the test plan, multiple test scenarios in the test scenario library are called for service reliability testing;

[0095] Determine the calling order of multiple test scenarios, and call the test scenarios in the calling order to simulate test scenario exceptions.

[0096] This step is used to locate the required test scenarios from the test scenario library according to the test plan, and call the test scenarios in the calling sequence to simulate the test scenario exceptions.

[0097] Specifically, parse each test scenario in the test plan, including test scenario flag information, operation time interval, call sequence, and recovery mechanism;

[0098] Locating the test scene from the test scene library according to the test scene mark information;

[0099] Formulate the execution order of test scenarios to ensure that the test scenarios do not interfere with each other and can fully cover all test requirements;

[0100] Execute each test scenario in turn according to the execution order to simulate exceptions;

[0101] After each test scenario is completed, perform recovery operations to ensure that the test environment returns to its initial state and does not affect subsequent tests;

[0102] The abnormal recovery time of each test scenario is recorded for service reliability evaluation.

[0103] In this step, by calling the test scenarios in the test scenario library, we can ensure that the abnormal simulation is carried out strictly according to the test plan, avoiding human errors. And each test is performed according to the same test plan, ensuring the consistency of the test results, which is convenient for horizontal comparison of the reliability performance of services in different versions or at different time points.

[0104] In addition, by performing recovery operations, it is ensured that the test environment returns to the initial state after each test, without affecting subsequent tests. Moreover, each test scenario is executed in sequence according to the execution order to perform abnormal simulation, avoiding the mutual influence of multiple test scenarios and ensuring the accuracy and reliability of the test results. In addition, by automating the abnormal simulation of test scenarios in the test plan, the test efficiency can be improved and the time and cost of manual intervention can be reduced.

[0105] As an example of this implementation, calling the test scenarios in the calling sequence to perform test scenario abnormal simulation also includes:

[0106] Set the calling time interval of the test scenario;

[0107] If the exception simulation time of the test scenario exceeds the calling time interval, stop the exception simulation of the test scenario, restore the service to normal state, and call the next test scenario in the calling order to perform the test scenario exception simulation.

[0108] In this step, a call interval is set for each test scenario according to the test requirements. The call interval is determined based on the service's recovery capability and test requirements to ensure that the service has enough time to recover from abnormal conditions while avoiding a long call interval that affects test efficiency.

[0109] Specifically, the calling time interval is counted from the start time of the abnormal simulation of the test scenario. When each test scenario starts to simulate the abnormality, the time record is triggered and the timing starts to accurately record the abnormal simulation time of the test scenario.

[0110] If the exception simulation time exceeds the preset call interval, the timeout mechanism is triggered to immediately stop the exception simulation of the current test scenario. The service is restored to normal state to ensure that it will not affect the execution of subsequent test scenarios, so as to improve the controllability and reliability of the test.

[0111] This step can prevent a test scenario from taking up too much time and affecting the test efficiency of this method by calling the timeout mechanism set by the time interval. When the abnormal simulation time of a test scenario times out, the service can be restored in time to avoid being in an abnormal state for a long time and affecting the accuracy of subsequent tests.

[0112] By setting a reasonable calling time interval, we can avoid interference between multiple test scenarios and ensure that the effect of each test scenario can be observed and evaluated separately, thereby improving the accuracy of reliability testing.

[0113] As another example of this implementation, the calling order of multiple test scenarios is determined as follows:

[0114] The test scenarios are set with corresponding importance levels and risk levels;

[0115] Determine the order in which test scenarios are called according to their importance level;

[0116] If the importance levels of multiple test scenarios are the same, the order in which the test scenarios are called is further determined based on the risk levels.

[0117] It should be noted that the importance level in this step is determined comprehensively based on the impact of the test scenario on the business and the user's attention level. First, the importance level is determined based on the user's attention level, and the test scenarios that the user cares about are given a higher importance level. For test scenarios that users are equally concerned about, the importance level is further determined based on the impact of the test scenario on the business, and the scenario with greater impact is given a higher importance level.

[0118] The risk level is determined based on the failure probability and impact range of the test scenario. Test scenarios with a higher probability of abnormal occurrence are assigned a higher risk level; test scenarios with a wider impact range are assigned a higher risk level.

[0119] This step manages the test scenarios by setting the importance level, ensuring that the scenarios with the greatest impact on the business are tested first, ensuring that the key test scenarios are fully tested, and improving the pertinence of the reliability test of this method. In addition, the test scenarios that users are most concerned about are tested first to ensure the reliability of the service in these test scenarios and improve user satisfaction.

[0120] In addition, by determining the calling order according to the importance level and risk level, test resources can be reasonably allocated to ensure that limited test resources are used for the most critical parts and avoid wasting too many resources on low-importance and low-risk test scenarios.

[0121] Thirdly, prioritizing testing scenarios with high importance and high risk can reduce redundant testing, improve testing efficiency, and quickly identify potential problems and make timely service repairs.

[0122] The present application does not limit the number of test scenarios that are called. As a preferred implementation of the present application, according to the test plan, a test scenario in the test scenario library is called for service reliability testing.

[0123] When using this method to conduct targeted testing on a service, only one test scenario in the test scenario library is called for service reliability testing. This can improve the accuracy of calling the test scenario and avoid redundant testing caused by calling multiple test scenarios. In particular, if one of the test scenarios fails the abnormal simulation after a reliability test, it is suitable to conduct another reliability test on the single test scenario after optimizing and upgrading the service.

[0124] This method also ensures that the abnormal simulation is strictly carried out according to the test scenario by calling the test scenario in the test scenario library, thereby reducing human errors.

[0125] As a preferred implementation of the present application, a service reliability test instruction is obtained, and a test plan is formulated according to the service reliability test instruction. The test plan includes at least one test scenario and also includes:

[0126] Get the sign information corresponding to the test scenario;

[0127] Compare the test scenarios in the test plan with the test scenarios in the test scenario library according to the flag information;

[0128] According to the comparison results, the test scenario information in the test scenario library is updated and maintained.

[0129] In this step, the flag information corresponding to the test scenario is obtained. The flag information is a unique identifier for each test scenario. According to the flag information in the test plan, the corresponding test scenario is searched in the test scenario library. The test scenario information in the test plan is compared with the information in the test scenario library to ensure the consistency between the two.

[0130] If the test scenarios in the test plan are consistent with the information in the test scenario library, no changes are required. If the test scenarios in the test plan are inconsistent with the information in the test scenario library, update the information in the test scenario library to make it consistent with the test plan.

[0131] If the test plan contains new test scenarios that are not in the test scenario library, the new scenarios are added to the test scenario library.

[0132] This step can ensure that the information in the test plan is consistent with the information in the test scenario library by comparing the test scenarios in the test plan with the scenarios in the test scenario library, thereby improving the accuracy of the test plan. In addition, based on the comparison results, the test scenario information in the test scenario library is updated and maintained to ensure that the information in the test scenario library is always up to date and reflects the latest test requirements and test scenario changes. Moreover, each test is based on the latest test scenario library, which improves the accuracy of the test results.

[0133] In addition, according to the changes in the service reliability test instructions, the test scenario information in the test scenario library is dynamically adjusted to ensure that the test can flexibly respond to different test requirements. In addition, the test scenario library supports adding new test scenarios at any time to enhance the scalability of the test scenario library.

[0134] As a preferred implementation of the present application, the service reliability is evaluated according to the abnormal recovery time of the test scenario, specifically:

[0135] Set the maximum tolerable recovery time and compare the abnormal recovery time of the test scenario with the maximum tolerable recovery time;

[0136] If the abnormal recovery time of the test scenario is greater than the maximum tolerance recovery time, the test scenario simulation test is considered to have failed;

[0137] If the abnormal recovery time of the test scenario is less than or equal to the maximum tolerance recovery time, the test scenario is scored according to the preset scoring rules;

[0138] The reliability of the service system is comprehensively scored based on the test results of multiple test scenarios.

[0139] In this step, the maximum tolerable recovery time for each test scenario is defined according to the service level agreement (SLA) or other business requirements. Of course, the same maximum tolerable recovery time can also be defined for multiple test scenarios, and this application does not limit this.

[0140] The abnormal recovery time of the test scenario starts from the time when the abnormal simulation of the test scenario begins and ends when the service returns to normal.

[0141] If the abnormal recovery time is greater than the maximum tolerable recovery time in a certain test scenario, it is determined that the service has not recovered to normal under the abnormal simulation of the test scenario, or the service takes too long to recover to normal, and the service is unreliable. If the service is unreliable, it may cause confusion when the service encounters an abnormal situation, or the interruption time is too long, which greatly affects the customer experience. At this time, the tested service needs to be optimized and improved accordingly to improve the reliability and stability of the tested service. After optimization, the test can be resubmitted and the reliability test can be performed again.

[0142] If the abnormal recovery time of the test scenario is less than or equal to the maximum tolerable recovery time. According to the service level agreement (SLA) and other business requirements, preset scoring rules are set and the service is scored according to the preset scoring rules. Generally, the score range can be from 60 to 100 points, and the higher the score, the better the reliability of the service.

[0143] The scores of all test scenarios are weighted averaged to obtain a comprehensive score. Based on the comprehensive score, the reliability of the entire service is comprehensively evaluated. In this step, the average score of all test scenario scores can be taken as the comprehensive score. Alternatively, each test scenario can be weighted according to the importance level and risk level, and the scores of all test scenarios are weighted averaged to obtain a comprehensive score.

[0144] This step can quantitatively evaluate the reliability of the service by setting the maximum tolerance recovery time and comparing it with the abnormal recovery time, avoiding errors caused by subjective judgment. And the maximum tolerance recovery time is used as an objective standard, making the evaluation process more objective and fair, avoiding different test results due to different comprehensive qualities of testers.

[0145] By comparing the abnormal recovery time and the maximum tolerable recovery time for each test scenario, we can quickly locate that the service has lower stability in a certain test scenario, and then make improvement suggestions based on the evaluation results to facilitate targeted maintenance and improvement of the service.

[0146] As a preferred embodiment of this implementation, if the abnormal recovery time of the test scenario is less than or equal to the maximum tolerance recovery time, the test scenario is scored according to the preset scoring rules, specifically:

[0147] Calculate the time interval between the abnormal recovery time and the maximum tolerable recovery time of the test scenario. The smaller the time interval, the lower the score of the test scenario.

[0148] In this step, time interval = maximum tolerable recovery time - abnormal recovery time. Score = f(time interval), where f(x) is a decreasing function, that is, the larger the time interval, the higher the score; the smaller the time interval, the lower the score. Because the smaller the time interval, the closer the time it takes for the service to return to normal in this test scenario is to the user's threshold and the user's tolerance limit. In this state, the user experience is worse and the service reliability is lower.

[0149] In one embodiment, when the time interval is equal to 0, the score is 60 points; when the time interval is equal to the maximum tolerable recovery time, the score is 100 points; f(x) is a linear decreasing function.

[0150] This step uses time intervals to score, which can more carefully evaluate the performance of the service in terms of recovery time. Even if the recovery time is abnormal within the maximum tolerance recovery time, the performance differences of the service in different test scenarios can be distinguished through scoring, rather than simply being classified as "reliable" or "unreliable."

[0151] The relationship between time intervals and scores can more scientifically reflect the actual situation of service reliability. Through specific scoring results, we can find out in which test scenarios the service recovers more slowly, providing a basis for subsequent optimization.

[0152] like Figure 2 As shown, the present application also provides a service reliability automated testing device, comprising:

[0153] The test task scheduling module 11 is used to obtain the service reliability test instructions and formulate a test plan according to the service reliability test instructions;

[0154] The abnormal simulation module 11 is used to call the tested service information and related dependent service information corresponding to the test scenario according to the test plan to perform abnormal simulation of the test scenario;

[0155] The result analysis and report generation module 13 is used to evaluate the service reliability according to the abnormal recovery time of the test scenario;

[0156] The monitoring and logging module 14 is used to monitor the service system status, service response, abnormal information, etc. during the test process and generate a test log.

[0157] This device can achieve any technical effect in the above method, which will not be described in detail here.

[0158] Anything not described in the present invention can be achieved by adopting or drawing on existing technologies.

[0159] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0160] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A service reliability automated testing method, characterized in that: include: Constructing a test scenario library, wherein the test scenario library includes the tested service information and related dependent service information of multiple test scenarios; Obtaining a service reliability test instruction, and formulating a test plan according to the service reliability test instruction, wherein the test plan includes at least one test scenario; According to the test plan, calling the tested service information and the related dependent service information corresponding to the test scenario to perform test scenario abnormal simulation; Evaluate service reliability based on abnormal recovery time in test scenarios.

2. The service reliability automated testing method according to claim 1, characterized in that: According to the test plan, the tested service information and the related dependent service information corresponding to the test scenario are called to perform test scenario abnormal simulation, specifically: According to the test plan, calling a plurality of the test scenarios in the test scenario library for service reliability testing; Determine the calling order of the multiple test scenarios, and call the test scenarios according to the calling order to perform test scenario abnormality simulation.

3. The service reliability automated testing method according to claim 2, characterized in that: Calling the test scenario in the calling sequence to perform test scenario abnormal simulation also includes: Setting the calling time interval of the test scenario; If the abnormal simulation time of the test scenario exceeds the calling time interval, the abnormal simulation of the test scenario is stopped, the service is restored to a normal state, and the next test scenario is called in the calling sequence to perform the test scenario abnormal simulation.

4. The service reliability automated testing method according to claim 3, characterized in that: The calling time interval is counted from the start time of the abnormal simulation of the test scenario.

5. The service reliability automated testing method according to claim 2, characterized in that: Determine the calling order of the multiple test scenarios, specifically: The test scenarios are set with corresponding importance levels and risk levels; Determining a calling order of the test scenarios according to the importance levels; If the importance levels of the multiple test scenarios are consistent, the calling order of the test scenarios is further determined according to the risk levels.

6. The service reliability automated testing method according to claim 1, characterized in that: According to the test plan, the tested service information and the related dependent service information corresponding to the test scenario are called to perform test scenario abnormal simulation, specifically: According to the test plan, one of the test scenarios in the test scenario library is called for service reliability testing.

7. The service reliability automated testing method according to claim 1, characterized in that: Obtaining a service reliability test instruction, and formulating a test plan according to the service reliability test instruction, wherein the test plan includes at least one test scenario and further includes: Obtaining flag information corresponding to the test scenario; According to the flag information, comparing the test scenarios in the test plan with the test scenarios in the test scenario library; According to the comparison result, the test scenario information in the test scenario library is updated and maintained.

8. The service reliability automated testing method according to claim 1, characterized in that: Evaluate service reliability based on the abnormal recovery time of the test scenario, specifically: Setting a maximum tolerable recovery time, and comparing the abnormal recovery time of the test scenario with the maximum tolerable recovery time; If the abnormal recovery time of the test scenario is greater than the maximum tolerable recovery time, it is determined that the simulation test of the test scenario fails; If the abnormal recovery time of the test scenario is less than or equal to the maximum tolerable recovery time, the test scenario is scored according to a preset scoring rule; The test results of multiple test scenarios are combined to give a comprehensive score for the reliability of the service.

9. The service reliability automated testing method according to claim 8, characterized in that: If the abnormal recovery time of the test scenario is less than or equal to the maximum tolerable recovery time, the test scenario is scored according to a preset scoring rule, specifically: The time interval between the abnormal recovery time of the test scenario and the maximum tolerable recovery time is calculated, and the smaller the time interval is, the lower the score of the test scenario is.

10. A service reliability automated testing device, characterized in that: include: A test task scheduling module is used to obtain a service reliability test instruction and formulate a test plan according to the service reliability test instruction; The exception simulation module is used to call the tested service information and related dependent service information corresponding to the test scenario according to the test plan to simulate the test scenario exception; The result analysis and report generation module is used to evaluate service reliability based on the abnormal recovery time of the test scenario; The monitoring and logging module is used to monitor the service system status, service response, exception information, etc. during the test process and generate a test log.