Service robustness test method and device, equipment, storage medium and program product
By obtaining the directed dependency graph of BMC services for fault injection, the problem of low efficiency in BMC service testing is solved, achieving efficient and accurate service robustness testing and fault isolation verification, and providing a quantitative standard for service robustness testing.
Patent Information
- Application Number
- CN202511407481.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-28
- Publication Date
- 2026-01-20
AI Technical Summary
In existing technologies, the testing and verification of BMC services relies on manual selection of test targets, which is inefficient. There is a lack of systematic and automated service isolation testing methods, which cannot verify complex dependencies between services. There is a lack of quantitative evaluation standards for self-recovery capabilities, and there is a lack of scalable and configurable testing frameworks.
By obtaining a directed dependency graph between multiple services, performing fault injection based on the directed dependency graph, acquiring status monitoring data, determining service robustness test results, using fault injection weights for fault injection, and dynamically adjusting test priorities, a complete service robustness testing methodology is provided.
It enables efficient and accurate service robustness testing, improves testing efficiency, provides quantitative standards for service robustness testing, and supports automated testing and fault isolation verification of BMC services.
Smart Images
Figure CN121364976A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computers, and in particular to a service robustness testing method and device, equipment, a storage medium and a program product. BACKGROUND
[0002] With the increasing complexity of data centers and server architectures, the BMC (Baseboard Management Controller) is a dedicated controller in the server for monitoring and managing hardware. As a key component for implementing out-of-band management of servers, its reliability and robustness have become a core consideration factor for system design. Multiple management services are usually integrated in the BMC, such as IPMI (Intelligent Platform Management Interface), Redfish and KVM (Kernel-based Virtual Machine) services, etc.
[0003] However, with the increasing integration of BMC functions, there may be coupling relationships such as resource sharing, dependent calls and process communication between services, which may affect other services when a service fails, and thus affect the stability and reliability of the BMC system as a whole. Therefore, it is necessary to test and verify the robustness and isolation of BMC services, but currently the testing and verification of BMC services usually relies on test personnel to select test targets, which is not efficient. SUMMARY
[0004] Therefore, it is necessary to provide a service robustness testing method, device, equipment, storage medium and program product to solve the above technical problems.
[0005] In a first aspect, the present application provides a service robustness testing method, comprising:
[0006] obtaining a directed dependency graph between a plurality of services, the directed dependency graph being used to represent the dependency relationship between the plurality of services and the fault injection weight of each service;
[0007] According to the directed dependency graph, fault injection is performed on the plurality of services;
[0008] obtaining state monitoring data of the plurality of services;
[0009] According to the state monitoring data, determining the service robustness testing result of each service in the plurality of services.
[0010] In the above embodiment, firstly, a directed dependency graph between a plurality of services is obtained to represent the dependency relationship between the plurality of services and the bug injection weight of each service, then, the plurality of services are fault injected according to the directed dependency graph, at the same time, the state monitoring data of the plurality of services are obtained, and finally, the service robustness test result of each service in the plurality of services is determined according to the state monitoring data. In this way, the plurality of services can be fault injected according to the bug injection weight of each service in the directed dependency graph, and then the state monitoring data of the plurality of services are obtained to determine the service robustness test result of each service, thereby realizing a complete service robustness test method. At the same time, the plurality of services are fault injected by the bug injection weight without manual experience, thereby improving the efficiency and accuracy of service robustness test.
[0011] In one of the embodiments, the directed dependency graph between the plurality of services is obtained, comprising:
[0012] According to the configuration file of each service in the plurality of services, the start order dependency relationship and the self-recovery capability index information of each service are determined;
[0013] According to the running data of each service in the plurality of services, the service call frequency of each service is determined;
[0014] According to the start order dependency relationship of each service in the plurality of services, an initial directed dependency graph is generated;
[0015] According to the start order dependency relationship of each service in the plurality of services, the self-recovery capability index information of each service, and the service call frequency of each service, the bug injection weight of each service is calculated and added to the initial directed dependency graph to obtain the directed dependency graph.
[0016] In the above embodiment, the start order dependency relationship of each service in the plurality of services, the self-recovery capability index information of each service, and the service call frequency of each service are obtained by analyzing the files, logs, etc. of each service, and the bug injection weight of each service is further calculated, and then the bug injection weight of each service is added to the initial directed dependency graph to obtain the directed dependency graph. The directed dependency graph is generated in a lightweight, low-overhead and portable manner without calling tracking or patching each service, and the system stability is not affected.
[0017] In one of the embodiments, the plurality of services are fault injected according to the directed dependency graph, comprising:
[0018] According to the directed dependency graph, a target service is determined;
[0019] According to the fault priority corresponding to each fault mode, a target fault mode is determined;
[0020] Based on the target fault mode, the target service is fault injected.
[0021] In the above embodiment, the target service is tested by determining the target fault mode and the target service, and injecting faults into the target service through the target fault mode.
[0022] In one of the embodiments, the method further comprises:
[0023] Obtaining a current test case corresponding to the target fault mode;
[0024] Obtaining a historical test record of the target service;
[0025] Adjusting a fault priority of the target fault mode according to a similarity degree between the current test case and the historical test record.
[0026] In the above embodiment, the test frequency penalty algorithm can intelligently manage the fault priority of the fault mode through a double-layer weight reduction mechanism, and perform weight reduction operation on the fault mode for similar dependency topology and similar historical test case.
[0027] In one of the embodiments, the method further comprises:
[0028] Obtaining a test result of the target service, the test result comprising a service recovery time and an influence range of the target service;
[0029] Determining a key service degree index of the target service according to the service recovery time and the influence range of the target service;
[0030] Updating an error injection weight of the target service in the directed dependency graph according to the key service degree index of the target service.
[0031] In the above embodiment, the error injection weight of the target service in the directed dependency graph is updated through the historical test record, so as to dynamically adjust the test priority of each service.
[0032] In one of the embodiments, the service robustness test result of each service in the plurality of services is determined according to the state monitoring data, comprising:
[0033] The recovery completeness score, the recovery time score, the recovery insensitivity degree score, the recovery success rate score, the isolation effect score and the alarm completeness score of each service in the plurality of services are determined according to the state monitoring data;
[0034] The recovery completeness score, the recovery time score, the recovery insensitivity degree score, the recovery success rate score, the isolation effect score and the alarm completeness score of each service are weighted and calculated to obtain the robustness test result of each service.
[0035] In the above embodiment, according to the state monitoring data, the plurality of index scores of each service are determined, and the robustness test result of each service is obtained, thereby providing a complete service robustness test evaluation and quantitative standard.
[0036] In a second aspect, the present application further provides a service robustness test device, comprising:
[0037] A first obtaining module is configured to obtain a directed dependency graph between the plurality of services, the directed dependency graph being used to represent the dependency relationship between the plurality of services and the error injection weight of each service;
[0038] An injection module is configured to perform fault injection on the plurality of services according to the directed dependency graph;
[0039] A second obtaining module is configured to obtain state monitoring data of the plurality of services;
[0040] A determining module is configured to determine a service robustness test result of each service in the plurality of services according to the state monitoring data.
[0041] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method of any one of the above first aspect when executing the computer program.
[0042] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the method of any one of the above first aspect when executed by a processor.
[0043] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, and the computer program implements the steps of the method of any one of the above first aspect when executed by a processor.
[0044] The service robustness test method, device, equipment, storage medium and program product, first, the directed dependency graph between the plurality of services is obtained, the directed dependency graph being used to represent the dependency relationship between the plurality of services and the error injection weight of each service, then, fault injection is performed on the plurality of services according to the directed dependency graph, at the same time, state monitoring data of the plurality of services is obtained, finally, a service robustness test result of each service in the plurality of services is determined according to the state monitoring data. In this way, the plurality of services can be fault injected according to the error injection weight of each service of the directed dependency graph, then the state monitoring data of the plurality of services is obtained, the service robustness test result of each service is determined, a complete service robustness test method is implemented, at the same time, the plurality of services are fault injected through the error injection weight without manual experience, thereby improving the efficiency and accuracy of service robustness test. BRIEF DESCRIPTION OF DRAWINGS
[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained based on these drawings without creative labor.
[0046] Figure 1 An application environment diagram of the service robustness test method in one embodiment;
[0047] Figure 2 A flowchart of the service robustness test method in one embodiment;
[0048] Figure 3 A flowchart of the directed dependency graph generation step in one embodiment;
[0049] Figure 4 A flowchart of the fault injection step in one embodiment;
[0050] Figure 5 A flowchart of the fault priority adjustment step in one embodiment;
[0051] Figure 6 A flowchart of the fault injection weight adjustment step of the service in one embodiment;
[0052] Figure 7 A flowchart of the service robustness test result calculation step in one embodiment;
[0053] Figure 8 A flowchart of the service robustness automatic test system in one embodiment;
[0054] Figure 9 A structural diagram of the service dependency modeling module in one embodiment;
[0055] Figure 10 A structural diagram of the adaptive fault injection module in one embodiment;
[0056] Figure 11 A flowchart of the service state monitoring module in one embodiment;
[0057] Figure 12 A flowchart of the system robustness evaluation module in one embodiment;
[0058] Figure 13 A page diagram of the test report in one embodiment;
[0059] Figure 14 A page diagram of the test report in one embodiment;
[0060] Figure 15 a page diagram of a test report in an embodiment;
[0061] Figure 16 a structural block diagram of a service robustness test device in an embodiment;
[0062] Figure 17 an internal structural diagram of a computer device in an embodiment. DETAILED DESCRIPTION
[0063] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.
[0064] It should be noted that the terms "first", "second", and the like used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the options or any combination of multiple options.
[0065] With the improvement of the integration of BMC functions, there can be coupling relationships such as resource sharing, dependent calling, and process communication between services, which can affect other services when a certain service fails, and further affect the stability and reliability of the overall BMC system. Therefore, the robustness and isolation of BMC services need to be tested and verified, but currently the testing and verification of BMC services usually depends on the selection of test targets by test personnel, and the testing efficiency is not high. At the same time, the testing of BMC services in the related art also faces some problems.
[0066] (1) Lack of systematic and automated service isolation testing methods, usually only single-service functional testing or whole-machine stability testing, narrow test range, unable to verify the complex dependency relationship between services.
[0067] (2) Coarse test granularity, lack of quantitative evaluation standards for self-recovery capability.
[0068] (3) Lack of scalable and configurable test framework, lack of modular design.
[0069] Therefore, the application provides a service robustness test method, which can perform fault injection on multiple services according to fault injection weights of each service in a directed dependency graph, then obtain state monitoring data of the multiple services, and determine service robustness test results of each service, thereby realizing a complete service robustness test method, and improving the efficiency and accuracy of service robustness test by performing fault injection on the multiple services through the fault injection weights without manual experience.
[0070] The service robustness test method provided in the embodiments of the application can be applied to an application environment as shown in Figure 1 . The management server 102 communicates with the server 104 through a network. A data storage system can store data required to be processed by the management server 102. The data storage system can be integrated on the management server 102, or placed on a cloud or other network server. The management server 102 and the server 104 can be provided with a BMC. The BMC is a special microcontroller used for monitoring the state of server hardware and providing remote management functions, wherein the BMC integrates and runs multiple services. The management server 102 and the server 104 can be independent physical servers, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0071] In an exemplary embodiment, as shown in Figure 2 , a service robustness test method is provided. The method is applied to the management server in Figure 1 , and includes the following steps 201 to 204. Wherein:
[0072] Step 201, obtaining a directed dependency graph between multiple services.
[0073] The directed dependency graph is used to represent the dependency relationship between the multiple services and the fault injection weight of each service. The multiple services are services running in the BMC. At present, the verification strategy for BMC services is usually single-service function test or whole-machine stability test, which cannot verify the risks in complex dependency relationships. Therefore, in order to verify the coupling relationship between services, the directed dependency graph between the multiple services is obtained in the embodiment.
[0074] Optionally, the dependency relationship between each service is extracted, and a weighted directed graph is used, for example, modeling is performed through the NetworkX library. Each node in the directed dependency graph represents an independent service, and the directed edge between the nodes represents the dependency relationship direction between the services. For example, if there is an A→B edge, it means that the service B depends on the service A. If it is determined that there is a dependency relationship between the services, the function of the other services depending on the service needs to be checked after subsequent fault injection, thereby realizing the test on the coupling relationship between the multiple services.
[0075] Meanwhile, the error injection weight of each service can be calculated according to the importance of each service, so that the service to be injected with a fault can be determined according to the error injection weight when the fault injection is performed, or the service to be checked can be determined when the service state detection is performed, without manual selection according to experience, and the efficiency is higher.
[0076] In step 202, the fault injection is performed on the plurality of services according to the directed dependency graph.
[0077] For example, the fault injection can be performed on the plurality of services by using different fault types to verify the fault tolerance capability of the services, the isolation between the services, and the self-recovery mechanism. Meanwhile, the fault injection can be performed on the services according to the error injection weight of each service in the directed dependency graph, for example, the fault injection is preferentially performed on the service with a high error injection weight. For the plurality of services without a dependency relationship, the fault injection can be performed in parallel. The services to be injected with a fault can be automatically determined by using the directed dependency graph, without manual selection of the target.
[0078] In step 203, the state monitoring data of the plurality of services is obtained.
[0079] During the service robustness test, the running state of each service in the whole process before, during, and after the fault injection is monitored in multiple dimensions, and the state monitoring data of the plurality of services is obtained.
[0080] In step 204, the service robustness test result of each service in the plurality of services is determined according to the state monitoring data.
[0081] Optionally, the robustness of a service is the ability of the service to stably run and normally provide services when facing abnormal conditions such as input errors and hardware faults. The robustness of a service reflects the tolerance and recovery capability of the service to errors. The isolation between services means that the services are independent of each other, and a fault does not affect each other. The service robustness test result can include a self-recovery capability standard and a fault isolation capability standard. The self-recovery capability standard is used to indicate the recovery capability of the service when an abnormality occurs, and can include recovery integrity, recovery time, recovery insensitivity, and recovery success rate. The fault isolation capability standard is used to indicate the isolation effect when a fault occurs in the service and whether the alarm is normal. The state monitoring data of each service is obtained, data cleaning and preprocessing are performed on the state monitoring data, the state monitoring data is converted into formatted data, and then the service robustness test result of each service is obtained according to the state monitoring data. In order to facilitate viewing, the service robustness test result of each service is displayed through a page.
[0082] In the above embodiment, firstly, the directed dependency graph between the plurality of services is obtained to represent the dependency relationship between the plurality of services and the bug injection weight of each service, then the fault injection is performed on the plurality of services according to the directed dependency graph, at the same time, the state monitoring data of the plurality of services is obtained, and finally, the service robustness test result of each service in the plurality of services is determined according to the state monitoring data. In this way, the fault injection can be performed on the plurality of services according to the bug injection weight of each service in the directed dependency graph, and then the state monitoring data of the plurality of services is obtained to determine the service robustness test result of each service, so that a complete service robustness test method is realized, and the fault injection is performed on the plurality of services by the bug injection weight without manual experience, thereby improving the efficiency and accuracy of the service robustness test.
[0083] In the embodiment of the present application, the step of obtaining the directed dependency graph between the plurality of services includes the following steps as shown in Figure 3
[0084] Step 301, determining the startup order dependency relationship and self-recovery capability index information of each service according to the configuration file of each service in the plurality of services.
[0085] The startup order dependency relationship of each service can represent other services that need to be depended on when starting, and the self-recovery capability index information of each service can include automatic restart time, data recovery speed, network connection recovery speed and the like after system failure.
[0086] Optionally, the configuration file of each service in the plurality of services is statically analyzed, for example, each service unit file such as *.service or *.target file is parsed, the [Unit] segment content is parsed, the content of After=, Before=, Requires= and the like dependency instruction fields is extracted according to the keyword, and the startup order dependency relationship of each service is generated. At the same time, the [Service] segment content is parsed, the content of Restart=, RestartSec= and the like fields is extracted according to the keyword, and the self-recovery capability index of each service is obtained.
[0087] Step 302, determining the service call frequency of each service according to the running data of each service in the plurality of services.
[0088] The service call frequency of each service can represent the number of times of being called by other services in a unit of time. By obtaining the call frequency of each service, the potential communication dependency of each service can be identified. Optionally, the communication frequency of each service can be counted by periodically running commands such as netstat, tcpdump, and ps, and the process behavior can be monitored by using log commands such as journalctl, and the content of the field is obtained by using preset key log keywords such as "Start handler", "Waiting for XXX", and the like, and the call logic and control flow dependency between services are analyzed according to the content of the field. At the same time, the service startup failure log is analyzed, and the error content in the service startup failure log due to the non-starting of other services is obtained, so as to further determine the dependency path and call relationship of the services, and the service call frequency of each service is obtained according to the above information.
[0089] In step 303, an initial directed dependency graph is generated according to the startup order dependency relationship of each service in the plurality of services.
[0090] According to the startup order dependency relationship of each service and the potential communication dependency of each service obtained by the above analysis, a weighted directed graph is used for modeling to generate an initial directed dependency graph. Each node represents an independent service, and the directed edge between the nodes represents the dependency relationship direction between the services, for example, if there is an A→B edge, it means that the service B depends on the service A. In the initial directed dependency graph, the node is generated to record the results of the analysis in the above steps, including the service name, the self-recovery capability index information, the call frequency, and the out-degree weight, etc. Among them, the out-degree weight is used to represent the total weight of how many services each service depends on.
[0091] Then, the orderliness of the initial directed dependency graph is verified by performing a topological sorting operation. During the sorting process, if a circular dependency is detected that causes the sorting to be infeasible, for example, NetworkXUnfeasible exception information is captured, the conflict dependency edge forming a closed loop is automatically stripped, the structure of the initial directed graph is re-adjusted to eliminate the circular dependency, and the conflict information is recorded for subsequent traceability analysis.
[0092] In step 304, the bug injection weight of each service is calculated according to the startup order dependency relationship of each service in the plurality of services, the self-recovery capability index information of each service, and the service call frequency of each service, and the bug injection weight of each service is added to the initial directed dependency graph to obtain a directed dependency graph.
[0093] After generating the initial directed dependency graph, the bug injection weight of each service node can be calculated according to the information of each service node, and the bug injection weight calculation formula of each service node is as follows.
[0094]
[0095] wherein Weight(service) is the weight of the service. Other parameters can be as described in Table 1 below.
[0096] Table 1 Formula for calculating error injection weight of service Index
[0097]
[0098] The value of is_critical can be set according to the actual requirements of the service, or can be dynamically adjusted during the test process. 、 and is the coefficient corresponding to each index, which can be selected can take the value 0.7, can be set according to the test mode, for example, the value of the fault isolation test mode or the self-recovery capability test mode is ±0.4, is 0.2. It can be understood that for different test environments, the formula coefficients should be flexibly changed according to the test needs, and the embodiments of the present application do not limit this.
[0099] In the above embodiments, by analyzing the files, logs and the like of each service, the startup order dependency relationship of each service in the plurality of services, the self-recovery capability index information of each service and the service call frequency of each service are obtained, and the error injection weight of each service is further calculated, and then the error injection weight of each service is added to the initial directed dependency graph to obtain the directed dependency graph. Without calling tracking or plugging each service, the directed dependency graph is generated in a lightweight, low-overhead and portable manner, which does not affect the stability of the system.
[0100] In one embodiment, in order to accurately perform fault injection on the plurality of services, the plurality of services are subjected to fault injection according to the directed dependency graph, as shown in Figure 4 The steps include:
[0101] Step 401, determining a target service according to the directed dependency graph.
[0102] Optionally, the target service is determined according to the error injection weight of each service in the directed dependency graph, and the target service is the service currently subjected to fault injection. The fault scenario can be injected to the service with high error injection weight first, and then the error injection weight of each service and the parameter value of each fault mode are updated in real time according to the feedback of the test, so as to avoid redundant test and cover the link of the key service.
[0103] Step 402, determining a target fault mode according to the fault priority corresponding to each fault mode.
[0104] The parameter value of each fault mode includes a fault priority corresponding to each fault mode. When each service is subjected to fault injection, the target fault mode can be selected according to the fault priority corresponding to each fault mode. Optionally, the fault mode with a high fault priority can be preferentially selected for fault injection. The type of each fault mode can be as shown in Table 2.
[0105] Table 2: Fault injection type classification
[0106]
[0107] In step 403, the target service is subjected to fault injection based on the target fault mode.
[0108] By simulating the fault condition of the target fault mode, the performance and recovery capability of the target service under the target fault mode injection are analyzed, so that the service robustness of the target service can be determined. Optionally, if the target fault mode is process termination injection or service stop injection, parallel injection can be supported, that is, by selecting other services that have no overlapping dependency with the target service that is being subjected to fault injection, the fault is injected in parallel without coupling.
[0109] The other services that have no overlapping dependency with the target service can be obtained by a service independence evaluation algorithm. The service independence evaluation algorithm is to find all services that have direct and indirect dependencies with the target service A and store them in a related service set. All services are traversed. If the current service is not in the related service set and all dependencies of the current service are also not in the related service set, the current service is added to an independent service set until all services are traversed. The services in the independent service set are the services that have no overlapping dependency with the target service and can be subjected to fault injection in parallel.
[0110] For example, to avoid damaging the BMC core service or known behaviors that will cause downtime risks and affect the efficiency of parallel testing, a whitelist protection mechanism is also supported in fault injection. The whitelist protection mechanism allows services to be added to the whitelist according to requirements, and skips the services recorded in the whitelist and their dependencies during fault injection. Testers can release the services from the whitelist after parallel error injection for testing, thereby achieving the purpose of saving testing time.
[0111] In the above embodiment, the target fault mode and the target service are determined, and the target service is subjected to fault injection by the target fault mode, thereby realizing the testing of the target service. Further, the fault injection supports multiple fault modes, and supports parallel execution and whitelist mechanism.
[0112] In one embodiment, the fault priority corresponding to each fault mode can be adjusted according to the test situation, such as negative feedback adjustment by a test frequency punishment algorithm, such asFigure 5 As shown, the steps include:
[0113] Step 501, obtaining a current test case corresponding to a target fault mode.
[0114] The current test case is a test case used when the target fault mode is injected with faults.
[0115] Step 502, obtaining a historical test record of a target service.
[0116] The historical test record includes test records when faults of all fault modes performed on the target service are injected, and the historical test record includes a historical test case and a test result corresponding to the historical test case.
[0117] Step 503, adjusting a fault priority corresponding to the target fault mode according to a similarity degree of the current test case and the historical test record.
[0118] Optionally, the similarity degree of the current test case and the historical test case is calculated, if the current test case and the historical test case are the same type of cases, that is, the target fault mode has been injected with faults to the target service for test, the fault priority corresponding to the target fault mode is greatly de-weighted. If the current test case and the historical test case are similar topology cases, the fault priority corresponding to the target fault mode is de-weighted. It can be understood that the fault priority of the target fault mode after being greatly de-weighted is less than the fault priority after being de-weighted.
[0119] In the above embodiment, the test frequency punishment algorithm can intelligently manage the fault priority of the fault mode through a double-layer de-weighting mechanism, and perform de-weighting operation on the fault mode for similar dependency topology and similar historical test case.
[0120] In one embodiment, the fault injection weight corresponding to each service can be feedback adjusted according to the test situation, such as negative feedback adjusting the fault injection weight of each service through a historical data weighting algorithm, such as Figure 6 As shown, the steps include:
[0121] Step 601, obtaining a test result of a target service.
[0122] The test result includes a service recovery time and an influence range of the target service. According to the test result of the target service, the key service degree index of the target service is dynamically adjusted, thereby being used to guide the priority in subsequent fault injection.
[0123] Step 602, determining a key service degree index of a target service according to a service recovery time and an influence range of the target service.
[0124] It can be understood that the adjustment range of the key service degree indicator of the target service is determined by the service recovery time and the influence range, and the target service with a long service recovery time and a large influence range corresponds to a more promoted key service degree indicator. The service recovery time of the target service is compared with the restart time RestartSec in the configuration, and individual tolerance is realized for different services, that is, different services judge whether the service recovery time is too long through different threshold values. The historical data weighting algorithm ensures that the key service degree indicator fluctuates within a reasonable range, for example, by setting upper and lower range threshold values, to avoid unlimited expansion of the key service degree indicator and to retain controllability in the gradual updating process. The specific algorithm implementation and parameters of each adjustment can be set according to actual test requirements, and the embodiments of the present application do not limit this.
[0125] In step 603, the bug injection weight of the target service in the directed dependency graph is updated according to the key service degree indicator of the target service.
[0126] The bug injection weight of the target service is recalculated according to the key service degree indicator of the target service, and the bug injection weight of the target service in the directed dependency graph is updated according to the recalculated bug injection weight, so as to update the priority of the target service in the subsequent fault injection process.
[0127] In the above embodiments, the bug injection weight of the target service in the directed dependency graph is updated through historical test records, and the test priority of each service is dynamically adjusted.
[0128] In the embodiments of the present application, the service robustness test results of each service in the plurality of services are determined according to the state monitoring data, as shown in the following table 1. Figure 7
[0129] In step 701, the recovery completeness score, the recovery time score, the recovery imperceptibility score, the recovery success rate score, the isolation effect score and the alarm completeness score of each service in the plurality of services are determined according to the state monitoring data.
[0130] The running states of each service before, during and after fault injection are monitored in multiple dimensions, and the state monitoring data of the plurality of services are obtained. The data collection sources of the state monitoring data can include process-level indicators, functional indicators, system resources, interface response information and log analysis. By classifying and analyzing the collected state monitoring data, the recovery completeness score, the recovery time score, the recovery imperceptibility score, the recovery success rate score, the isolation effect score and the alarm completeness score of each service are determined. The meanings and score evaluation standards of each indicator are shown in the following table 3.
[0131] Table 3 Service evaluation indicator explanation
[0132]
[0133] Step 702, the recovery completeness score, the recovery time score, the recovery imperceptibility score, the recovery success rate score, the isolation effect score and the alarm completeness score of each service are weighted and calculated to obtain the robustness test result of each service.
[0134] The recovery completeness score, the recovery time score, the recovery imperceptibility score and the recovery success rate score of each service can reflect the self-recovery capability standard corresponding to each service, and the isolation effect score and the alarm completeness score of each service can reflect the fault isolation capability standard of each service. The above scores are weighted and calculated to obtain the robustness test result of each service.
[0135] In the above embodiment, according to the state monitoring data, the multiple index scores of each service are determined, and the robustness test result of each service is obtained, which provides a complete service robustness test evaluation and quantitative standard.
[0136] In the embodiments of the present application, a service robustness test method is provided, which can be applied to a service robustness automatic test system. The system can include a service dependency modeling module, an adaptive fault injection module, a service state monitoring module, a system robustness evaluation module and a report generation module. Through the cooperative work of the above modules, the service robustness automatic test system can dynamically select the target service for fault injection, automatically analyze the influence on other services, evaluate the self-recovery capability of the target service to be tested, and generate quantitative robustness test results. Optionally, the service robustness automatic test system can be applied to BMC firmware development, regression test, integration verification and other stages. For example, it can be applied to BMC or general Linux platform after release, service fault isolation line and self-recovery capability verification, instead of manual test review, saving labor cost and increasing result reliability. It can also be applied to CI / CD pipeline integration, supporting error interception in pre-release link to reduce error flow into production department. The system flow chart can be as shown in Figure 8 .
[0137] The service dependency modeling module can include three sub-function modules: static analysis module, dynamic analysis module and graph generation module, which are used to automatically analyze and model the dependency relationship between services in the BMC multi-service environment, and finally output a directed dependency graph for the subsequent adaptive fault injection module. The directed dependency graph can reflect the start order, dependency relationship and fault injection weight between services. The structure diagram of the service dependency modeling module can be as shown in Figure 9As shown, the start order dependency relationship and self-recovery capability index information of each service are determined by the static analysis module, the service call frequency of each service is analyzed by the dynamic analysis module, and the directed dependency graph is generated by the graph generation module according to the outputs of the static analysis module and the dynamic analysis module.
[0138] The adaptive fault injection module can include three sub-function modules, i.e., a target selection module, a fault injection module, and a strategy optimization module, for performing a strategic and controlled fault injection experiment in the BMC multi-service system to verify the fault tolerance capability of the system, the isolation between services, and the self-recovery mechanism. The target selection module determines the target service according to the directed dependency graph input by the service dependency modeling module, the fault injection module performs fault injection on the target service, and the strategy optimization module dynamically adjusts the fault injection weight of the target service and the fault priority of the fault mode through a historical data weighting algorithm and a test frequency penalty algorithm. The structure diagram of the adaptive fault injection module can be as shown in Figure 10 .
[0139] The service state monitoring module is used for multi-dimensional monitoring of the running state and behavior of each service in the whole process before, during, and after fault injection. The flowchart of the service state monitoring module can be as shown in Figure 11 . In the initialization stage, the detection configuration is loaded, the service state storage is initialized, and preparation is made for subsequent data collection. Monitoring is performed through a double-path parallel monitoring process, one side listens to fault injection events and generates special monitoring tasks for specific services, and the other side sets a timer to periodically perform regular detection on all target services at a set frequency. In the data collection execution stage, data is collected through different data sources, and fault injection event information and service dependency information are obtained. The state monitoring data analysis and structured processing process the collected original state monitoring data, judge whether there is an abnormality, and convert it into a structured format such as JSON format for subsequent analysis. The result output, the final monitoring result will be used as the input of the system robustness evaluation and other analysis modules to support the judgment of the service recovery capability and fault isolation capability.
[0140] The system robustness evaluation module determines the score of each index according to the obtained state monitoring data, realizes the quantitative evaluation of the recovery capability and robustness of each service under multiple fault scenarios, and focuses on the self-recovery capability and business recovery process of the service after suffering fault injection. The flowchart of the system robustness evaluation module can be as shown in Figure 12 .
[0141] The report generation module outputs the test results of the system robustness module in the form of a report to show the test personnel in an intuitive way. Among them, the test report can be as shown in Figure 13 , Figure 14 and Figure 15As shown in the middle.
[0142] In the above embodiments, by constructing a directed dependency graph, the target service needing to be injected with faults is automatically selected by combining the fault injection weights of each service. Meanwhile, based on a feedback adjustment algorithm, the fault injection weights of each service and the fault priority of each fault mode are dynamically adjusted.
[0143] The state of each service is monitored in a triggered and timed manner, the state monitoring data is determined, the robustness test result of each service is obtained according to the state monitoring data, and the result is output to a page for display. A test result report with higher readability can be generated based on HTML.
[0144] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order limitation for the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.
[0145] Based on the same inventive concept, the embodiments of the present application also provide a service robustness test device for implementing the service robustness test method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more service robustness test device embodiments provided below can refer to the limitations of the service robustness test method in the above text, which will not be repeated here.
[0146] In one exemplary embodiment, as Figure 16 shown, a service robustness test device 1600 is provided, comprising: a first acquisition module 1601, an injection module 1602, a second acquisition module 1603, and a determination module 1604, wherein:
[0147] The first acquisition module 1601 is configured to acquire a directed dependency graph between a plurality of services, the directed dependency graph being used to represent a dependency relationship between the plurality of services and a fault injection weight of each service;
[0148] The injection module 1602 is configured to perform fault injection on the plurality of services according to the directed dependency graph.
[0149] The second obtaining module 1603 is configured to obtain state monitoring data of a plurality of services.
[0150] The determining module 1604 is configured to determine service robustness test results of each service in the plurality of services according to the state monitoring data.
[0151] In one embodiment, the first obtaining module 1601 is specifically configured to determine the start order dependency relationship and the self-recovery capability index information of each service according to a configuration file of each service in the plurality of services; determine the service call frequency of each service according to running data of each service in the plurality of services; generate an initial directed dependency graph according to the start order dependency relationship of each service in the plurality of services; and calculate the error injection weight of each service according to the start order dependency relationship of each service in the plurality of services, the self-recovery capability index information of each service, and the service call frequency of each service, and add the error injection weight of each service to the initial directed dependency graph to obtain the directed dependency graph.
[0152] In one embodiment, the injecting module 1602 is specifically configured to determine a target service according to the directed dependency graph; determine a target fault mode according to the fault priority corresponding to each fault mode; and perform fault injection on the target service based on the target fault mode.
[0153] In one embodiment, the apparatus further includes an adjusting module configured to obtain a current test case corresponding to the target fault mode; obtain a historical test record of the target service; and adjust the fault priority corresponding to the target fault mode according to the similarity between the current test case and the historical test record.
[0154] In one embodiment, the apparatus further includes an updating module configured to obtain a test result of the target service, the test result including a service recovery time and an influence range of the target service; determine a key service degree index of the target service according to the service recovery time and the influence range of the target service; and update the error injection weight of the target service in the directed dependency graph according to the key service degree index of the target service.
[0155] In one embodiment, the determining module 1604 is specifically configured to determine a recovery completeness score, a recovery time score, a recovery imperceptibility score, a recovery success rate score, an isolation effect score, and an alarm completeness score of each service in the plurality of services according to the state monitoring data; and perform weighted calculation on the recovery completeness score, the recovery time score, the recovery imperceptibility score, the recovery success rate score, the isolation effect score, and the alarm completeness score of each service to obtain the robustness test result of each service.
[0156] Each of the modules in the service robustness testing apparatus can be implemented by software, hardware, and combinations thereof, in whole or in part. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be invoked by the processor to perform the operations corresponding to the modules.
[0157] In an example embodiment, a computer device, which can be a server, has an internal structure diagram as shown in Figure 16 The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store test data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a service robustness testing method.
[0158] Those skilled in the art can understand that Figure 16 The structure shown in the above
[0159] In an example embodiment, a computer device includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the following steps: obtaining a directed dependency graph between a plurality of services, the directed dependency graph being configured to represent dependency relationships between the plurality of services and fault injection weights of the services; performing fault injection on the plurality of services according to the directed dependency graph; obtaining state monitoring data of the plurality of services; and determining service robustness testing results of the services in the plurality of services according to the state monitoring data.
[0160] In an embodiment, the processor, when executing the computer program, further implements the following steps: determining, according to the configuration files of the services in the plurality of services, the start order dependency relationship and the self-recovery capability index information of the services; determining, according to the running data of the services in the plurality of services, the service call frequency of the services; generating an initial directed dependency graph according to the start order dependency relationship of the services in the plurality of services; calculating the bug injection weight of the services according to the start order dependency relationship of the services in the plurality of services, the self-recovery capability index information of the services, and the service call frequency of the services, and adding the bug injection weight of the services to the initial directed dependency graph to obtain the directed dependency graph.
[0161] In an embodiment, the processor, when executing the computer program, further implements the following steps: determining the target service according to the directed dependency graph; determining the target fault mode according to the fault priority corresponding to each fault mode; and performing fault injection on the target service based on the target fault mode.
[0162] In an embodiment, the processor, when executing the computer program, further implements the following steps: obtaining the current test case corresponding to the target fault mode; obtaining the historical test record of the target service; and adjusting the fault priority corresponding to the target fault mode according to the similarity between the current test case and the historical test record.
[0163] In an embodiment, the processor, when executing the computer program, further implements the following steps: obtaining the test result of the target service, the test result including the service recovery time and the influence range of the target service; determining the key service degree index of the target service according to the service recovery time and the influence range of the target service; and updating the bug injection weight of the target service in the directed dependency graph according to the key service degree index of the target service.
[0164] In an embodiment, the processor, when executing the computer program, further implements the following steps: determining, according to the state monitoring data, the recovery completeness score, the recovery time score, the recovery insensitivity degree score, the recovery success rate score, the isolation effect score, and the alarm completeness score of each service in the plurality of services; and performing weighted calculation on the recovery completeness score, the recovery time score, the recovery insensitivity degree score, the recovery success rate score, the isolation effect score, and the alarm completeness score of each service to obtain the robustness test result of each service.
[0165] In an embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any one of the above method embodiments.
[0166] In an embodiment, a computer program product is provided, and the computer program product includes a computer program. The computer program is executed by a processor to implement the steps of any one of the above method embodiments.
[0167] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.
[0168] It can be understood by those skilled in the art that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing related hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments of each method. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.
[0169] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described above, however, any combination of these technical features is deemed to be within the scope of the present application.
[0170] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.
Claims
1. A service robustness testing method, characterized by, The method comprises: obtaining a directed dependency graph between a plurality of services, the directed dependency graph being used to represent dependency relationships between the plurality of services and fault injection weights of the services; performing fault injection on the plurality of services according to the directed dependency graph; obtaining state monitoring data of the plurality of services; determining service robustness test results of the services in the plurality of services according to the state monitoring data.
2. The method of claim 1, wherein, The obtaining of the directed dependency graph between the plurality of services comprises: determining startup sequence dependency relationships and self-recovery capability index information of the services according to configuration files of the services in the plurality of services; determining service call frequencies of the services according to running data of the services in the plurality of services; generating an initial directed dependency graph according to the startup sequence dependency relationships of the services; calculating fault injection weights of the services according to the startup sequence dependency relationships of the services, the self-recovery capability index information of the services, and the service call frequencies of the services, and adding the fault injection weights of the services to the initial directed dependency graph to obtain the directed dependency graph.
3. The method of claim 1, wherein, The performing of fault injection on the plurality of services according to the directed dependency graph comprises: determining a target service according to the directed dependency graph; determining a target fault mode according to fault priorities corresponding to the fault modes; performing fault injection on the target service based on the target fault mode.
4. The method of claim 3, wherein, The method further comprises: obtaining a current test case corresponding to the target fault mode; obtaining a historical test record of the target service; adjusting the fault priority of the target fault mode according to a similarity degree between the current test case and the historical test record.
5. The method of claim 3, wherein, The method further comprises: obtaining a test result of the target service, the test result comprising a service recovery time and an influence range of the target service; determining a key service degree index of the target service according to the service recovery time and the influence range of the target service; updating the fault injection weight of the target service in the directed dependency graph according to the key service degree index of the target service.
6. The method of claim 3, wherein, The determining of the service robustness test results of the services in the plurality of services according to the state monitoring data comprises: determining recovery completeness scores, recovery time scores, recovery insensitivity degree scores, recovery success rate scores, isolation effect scores, and alarm completeness scores of the services in the plurality of services according to the state monitoring data; performing weighted calculation on the recovery completeness scores, the recovery time scores, the recovery insensitivity degree scores, the recovery success rate scores, the isolation effect scores, and the alarm completeness scores of the services to obtain the robustness test results of the services.
7. A service robustness testing apparatus characterized by comprising: The apparatus comprises: a first obtaining module configured to obtain a directed dependency graph between a plurality of services, the directed dependency graph being used to represent dependency relationships between the plurality of services and fault injection weights of the services; an injection module configured to perform fault injection on the plurality of services according to the directed dependency graph; a second obtaining module configured to obtain state monitoring data of the plurality of services; A determining module configured to determine a service robustness test result of each service in the plurality of services according to the state monitoring data.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.