Automatic testing method and system for OpenStack stability

Abstract

The invention discloses an automatic OpenStack stability test method and system, and relates to the technical field of cloud platform stability test.The method comprises the steps that test configuration information including test scene configuration parameters, flow model parameters, expected monitoring indexes and test execution strategies is acquired; initializing a tested system environment and test scene deployment based on the test scene configuration parameters; generating and sending test traffic to the tested system based on the traffic model parameters; in the testing process, actual monitoring index data of the tested system are collected, and when a specified abnormal alarm is triggered, testing is stopped, and a field testing environment state is reserved; comparing the actual monitoring index data with the expected monitoring index, and recording an abnormal log entry according to a comparison result; and generating a test report based on the test process data and the exception log entry. According to the method, the stability evaluation capability and the operation and maintenance response level of the cloud platform in a complex scene are enhanced while the test efficiency and the result credibility are improved.

Description

Technical Field

[0001] This invention relates to the field of cloud platform stability testing technology, and in particular to an automated testing method and system for OpenStack stability. Background Technology

[0002] With the widespread adoption of open-source cloud platforms like OpenStack in enterprise private and hybrid cloud construction, their stability, reliability, and performance have become crucial for ensuring the continuity of upper-layer services. The OpenStack platform has numerous functional modules and complex inter-component collaboration, making it difficult to effectively evaluate its stability under long-term operation, high concurrency, and dynamic loads using traditional testing methods. Currently, stability testing for the OpenStack platform generally faces the following limitations:

[0003] First, the automation level of test scenario construction and deployment is insufficient. Traditional methods mainly rely on manual operation to call OpenStack APIs (Application Programming Interfaces) to complete tasks such as virtual machine creation and network configuration. This method is not only inefficient, but also struggles to automatically complete the construction of complex network topologies and environment initialization before testing, making it difficult to guarantee the consistency of the test environment and affecting the reliability of test results.

[0004] Secondly, the simulated test traffic is disconnected from real-world business scenarios. Existing load testing tools for OpenStack struggle to simulate dynamic data flows based on real-world business logic. Furthermore, test traffic often fails to effectively cover real-world scenarios such as inter-virtual machine communication and cross-subnet forwarding, resulting in test results that do not reflect the platform's stability performance in complex network environments.

[0005] Furthermore, the processes of test execution, monitoring, and result analysis are disconnected from each other. When anomalies occur during testing, manual intervention is often required for log investigation and result judgment, resulting in a delayed response and making it difficult to achieve closed-loop automation of testing, monitoring, and analysis.

[0006] In addition, once a test fails, the test system usually automatically or manually cleans up the test environment, resulting in the loss of key on-site information such as the virtual machine state, network configuration, and traffic snapshots at the time of the anomaly. This greatly increases the difficulty of subsequent problem localization and anomaly analysis, and prolongs the fault recovery cycle. Summary of the Invention

[0007] (a) Technical problems to be solved

[0008] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present invention provides an automated testing method and system for OpenStack stability, which solves the technical problems of low deployment efficiency and poor consistency of test environment, inability of test traffic to simulate real business scenarios, disconnect between test monitoring and result analysis, and difficulty in fault location due to the difficulty in preserving abnormal conditions.

[0009] (II) Technical Solution

[0010] To achieve the above objectives, the main technical solutions adopted by the present invention include:

[0011] In a first aspect, embodiments of the present invention provide an automated testing method for OpenStack stability, comprising:

[0012] Obtain test configuration information, which includes at least test scenario configuration parameters, traffic model parameters, expected monitoring metrics, and test execution strategies;

[0013] Based on the test scenario configuration parameters, the environment initialization and virtual test scenario deployment of the system under test are automatically executed by calling the OpenStack interface.

[0014] Based on the traffic model parameters and the preset test traffic, a resource pool is generated, and test traffic is dynamically generated and sent to the system under test.

[0015] According to the test execution strategy, the stability test of the system under test is initiated. During the test execution, the actual monitoring index data of the system under test is collected at a preset period, and the test is stopped and the state of the on-site test environment is preserved when a specified abnormal alarm is triggered.

[0016] The actual monitoring indicator data is compared with the expected monitoring indicator in real time. The test is judged based on the comparison results, and the test events that fail are recorded as abnormal log entries.

[0017] After the test is completed, a test report is generated based on the test process data and abnormal log entries, and the abnormal log entries that have increased are marked in the test report.

[0018] Optionally, obtaining test configuration information includes:

[0019] Receive and parse the test task instructions input by the user. The test task instructions contain the test task identifier of the system under test.

[0020] Based on the test task identifier, load the test configuration information associated with the test task identifier from the preset test configuration database. The test configuration information includes test scenario configuration parameters, traffic model parameters, expected monitoring indicators, and test execution strategies.

[0021] The test scenario configuration parameters include: virtual machine specifications and network topology.

[0022] Traffic model parameters include: traffic protocol type, packet size, and traffic source and destination endpoint identifiers;

[0023] Expected monitoring metrics include: CPU utilization, memory utilization, disk utilization, API call latency and inter-virtual machine network communication latency, virtual machine instance status, and the number of abnormal log entries;

[0024] The test execution strategy includes: test execution order, test duration for each scenario, number of test loops, failure retry strategy, and environment cleanup and recovery strategy.

[0025] Optionally, based on the test scenario configuration parameters, the environment initialization and virtual test scenario deployment of the system under test can be automatically performed by calling the OpenStack interface, including:

[0026] Initialize the test environment of the system under test according to the environment cleanup and recovery strategy in the test scenario configuration parameters;

[0027] By calling the OpenStack API interface, virtual machine instances of the corresponding specifications are created according to the virtual machine specification parameters and network topology in the test scenario configuration parameters, and network connections between virtual machines and between virtual machines and external networks are configured to build a virtual test scenario.

[0028] Send forwarding rules corresponding to the network topology to the virtual switches in the system under test so that test traffic can be forwarded to the target virtual machine instance according to the preset path;

[0029] After the virtual test scenario is deployed, a health check is performed on all created virtual machine instances to confirm that each virtual machine instance is in a normal operating state.

[0030] If any of the following abnormal situations occur during the deployment process: virtual machine creation failure, network configuration error, or virtual switch rule delivery failure, the current deployment process will be paused, the abnormal event information will be recorded in the abnormal log entry, and the corresponding deployment steps will be re-executed after the abnormality is repaired, according to the failure retry policy in the test execution strategy.

[0031] Optionally, based on traffic model parameters and a preset test traffic resource pool, test traffic is dynamically generated and sent to the system under test, including:

[0032] Configure the test traffic generation template based on the traffic protocol type and packet size in the traffic model parameters;

[0033] Dynamically allocate at least one traffic generator from the test traffic generation resource pool and deploy the traffic generator within the system under test or in an external network;

[0034] The generated template is sent to the traffic generator, and the transmission path of the test traffic is determined based on the traffic source and target endpoint identifiers in the traffic model parameters.

[0035] The control traffic generator dynamically sends test traffic to the target virtual machine instance at a preset traffic rate according to the test execution order and duration in the test execution strategy;

[0036] During the test traffic transmission process, the transmission status of the traffic generator and the network link quality are monitored in real time. If a traffic interruption or packet loss rate exceeds the threshold, the traffic parameters are automatically adjusted or the traffic generator is switched according to the failure retry policy, and abnormal traffic events are recorded.

[0037] Optionally, according to the test execution strategy, stability testing of the system under test is initiated. During the test execution, actual monitoring index data of the system under test are collected at preset intervals, and the test is stopped and the state of the on-site test environment is preserved when a specified abnormal alarm is triggered.

[0038] According to the test execution order defined in the test execution strategy, the stability tests of each virtual test scenario in the system under test are started sequentially.

[0039] During the execution of each test scenario, the timing for that virtual test scenario is started according to the test duration set in the test execution strategy.

[0040] Based on the preset collection period configured in the test execution strategy, actual monitoring metric data are collected by calling the OpenStack API interface;

[0041] Each time the actual monitoring indicator data is collected is associated with the corresponding test scenario information and the collection timestamp and stored to obtain time-series monitoring data records;

[0042] If any virtual machine instance is found to be in an abnormal state or the test traffic is interrupted during the test, the system will automatically decide whether to pause the current test and retain the state of the on-site test environment, perform a retry, or jump to the next virtual test scenario, based on the failure retry policy in the test execution strategy, and record the abnormal event in the abnormal log entry.

[0043] Optionally, the actual monitoring indicator data is compared with the expected monitoring indicator in real time, and the test is judged as passed based on the comparison results. Test events that fail are recorded as exception log entries, including:

[0044] The actual monitoring indicator data is categorized by virtual machine instance and time period, and compared in real time with the preset threshold range of the corresponding virtual machine instance in the expected monitoring indicators.

[0045] If any of the actual monitored metrics exceeds the corresponding preset threshold range, the virtual machine instance is determined to have failed the test in the current test cycle, and an abnormal log entry containing the virtual machine identifier, abnormal metric, exceeding value and occurrence timestamp is generated according to preset rules.

[0046] If all actual monitoring metrics are within the corresponding preset threshold range, the virtual machine instance is deemed to have passed the test within the current testing period.

[0047] The judgment results and corresponding exception log entries in each test cycle are associated with the test scenario identifier and stored to generate test result logs;

[0048] If the same virtual machine instance exhibits the same abnormal metric in multiple consecutive test cycles, it will be marked as a persistent abnormal event in the abnormal log entry, and the virtual test scenario will be automatically retested according to the failure retry policy in the test execution strategy.

[0049] Optionally, after the test is completed, a test report is generated based on the test process data and exception log entries, and the exception log entries that have increased in the test report include:

[0050] Based on the time-series monitoring data records, test result logs, and exception log entries stored during the test process, data is aggregated and statistically analyzed according to test scenarios and time dimensions to generate a structured test report;

[0051] If the test report contains any abnormal log entry that shows an increase, an abnormal event alarm will be triggered to the user's terminal via email or system notification according to the preset alarm rules.

[0052] Upon triggering the alarm, the resource reclamation and environment cleanup processes of the system under test are stopped, and the on-site test environment status at the time of the anomaly is retrieved and archived in the form of an environment snapshot, and a corresponding fault investigation entry link is generated.

[0053] Secondly, embodiments of the present invention provide an automated testing system for OpenStack stability, comprising:

[0054] The user-side interface is used to receive test task instructions input by the user and to display test results and alarm notifications.

[0055] The test process generates a resource pool to provide dynamically allocable traffic generators and send test traffic to the system under test.

[0056] The OpenStack component resource pool provides virtual machine, network, and storage resources to support the deployment and operation of virtual test scenarios.

[0057] The main controller is used to execute the automated testing method for OpenStack stability described above.

[0058] Optionally, the main controller includes:

[0059] The configuration acquisition module is used to acquire test configuration information, which includes at least test scenario configuration parameters, traffic model parameters, expected monitoring metrics, and test execution strategies.

[0060] The test scenario deployment module is used to automatically perform environment initialization and virtual test scenario deployment of the system under test by calling the OpenStack interface based on the test scenario configuration parameters.

[0061] The test traffic module is used to generate a resource pool based on traffic model parameters and preset test traffic, and dynamically generate and send test traffic to the system under test.

[0062] The test monitoring module is used to initiate stability tests on the system under test according to the test execution strategy. During the test execution, it collects the actual monitoring index data of the system under test at a preset period, and stops the test and retains the state of the on-site test environment when a specified abnormal alarm is triggered.

[0063] The anomaly notification module is used to compare the actual monitoring indicator data with the expected monitoring indicator in real time, determine whether the test passes based on the comparison results, and record the failed test events as anomaly log entries.

[0064] The test report module is used to generate a test report based on the test process data and exception log entries after the test is completed, and to mark the exception log entries that have increased in the test report.

[0065] (III) Beneficial Effects

[0066] The beneficial effects of this invention are as follows: The automated testing method for OpenStack stability proposed in this invention, by employing configurable test scenario parameters and a dynamic traffic model, combined with automated deployment and closed-loop monitoring and evaluation processes, achieves a fully automated closed-loop process from environment initialization, traffic injection, and metric collection to result judgment and on-site retention, compared to existing technologies. Furthermore, this invention transforms OpenStack testing from isolated scenario verification into a complete quality assurance system. Through integrated process control, it significantly enhances the stability assessment capabilities and operational response levels of cloud platforms in complex scenarios while improving testing efficiency and result reliability. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating an automated testing method for OpenStack stability according to an embodiment of the present invention.

[0068] Figure 2 This is an example diagram of OpenStack stability test points provided in an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of the composition of an automated testing system for OpenStack stability provided in an embodiment of the present invention. Detailed Implementation

[0070] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] refer to Figures 1 to 3 As shown in the embodiment of the present invention, an automated testing method for OpenStack stability includes: acquiring test configuration information, which includes at least test scenario configuration parameters, traffic model parameters, expected monitoring indicators, and test execution strategies; automatically executing environment initialization and virtual test scenario deployment of the system under test by calling OpenStack interfaces based on the test scenario configuration parameters; dynamically generating and sending test traffic to the system under test based on the traffic model parameters and a preset test traffic generation resource pool; initiating stability testing of the system under test according to the test execution strategy, collecting actual monitoring indicator data of the system under test at preset intervals during test execution, and stopping the test and retaining the state of the on-site test environment when a specified abnormal alarm is triggered; comparing the actual monitoring indicator data with the expected monitoring indicators in real time, determining whether the test passes based on the comparison results, and recording failed test events as abnormal log entries; generating a test report based on the test process data and abnormal log entries after the test, and marking the rising abnormal log entries in the test report.

[0072] This embodiment proposes an automated testing method for OpenStack stability. By employing configurable test scenario parameters and a dynamic traffic model, combined with automated deployment and closed-loop monitoring and evaluation processes, it achieves a fully automated closed-loop process from environment initialization, traffic injection, and metric collection to result judgment and on-site retention, compared to existing technologies. Furthermore, this embodiment transforms OpenStack testing from isolated scenario verification into a complete quality assurance system. Through integrated process control, it significantly enhances the stability assessment capabilities and operational response levels of the cloud platform in complex scenarios, while improving testing efficiency and result reliability.

[0073] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present invention can be understood more clearly and thoroughly, and that the scope of the present invention can be fully conveyed to those skilled in the art.

[0074] Specifically, refer to Figure 1 As shown in the figure, this embodiment proposes an automated testing method for OpenStack stability, which may include the following steps S100 to S600:

[0075] S100. Obtain test configuration information. The test configuration information includes at least the test scenario configuration parameters, traffic model parameters, expected monitoring indicators, and test execution strategies.

[0076] In this embodiment, the test configuration information of the automated testing solution not only covers the construction of test scenarios but also delves into the fine-grained configuration of test cases, including but not limited to the network topology configuration of the tested environment, virtual resource specification settings, and traffic model parameters. This enables the entire testing process to automatically complete the deployment and activation of test cases from the initialization of the test environment, execute predefined test steps to collect various metrics, and perform environment cleanup or state switching according to preset environment cleanup and recovery strategies before testing, thereby ensuring the consistency and reusability of the test environment. Specifically, step S100 may include the following sub-steps S110 to S120:

[0077] S110. Receive and parse the test task instruction input by the user. The test task instruction contains the test task identifier of the system under test.

[0078] S120. Based on the test task identifier, load the test configuration information associated with the test task identifier from the preset test configuration database. The test configuration information includes test scenario configuration parameters, traffic model parameters, expected monitoring indicators, and test execution strategies.

[0079] Furthermore, the test scenario configuration parameters include: virtual machine specifications and network topology. Traffic model parameters include: traffic protocol type, packet size, and traffic source and destination endpoint identifiers. Expected monitoring metrics include: CPU utilization, memory utilization, disk utilization, API call latency and inter-virtual machine network communication latency, virtual machine instance status, and the number of exception log entries. Test execution strategies include: test execution order, test duration for each scenario, number of test loops, failure retry strategy, and environment cleanup and recovery strategy.

[0080] S200 automatically performs environment initialization and virtual test scenario deployment of the system under test by calling the OpenStack interface based on the test scenario configuration parameters.

[0081] In this embodiment, during initial runtime, the test environment of the system under test needs to be restored to its initial state according to a predefined environment cleanup and recovery strategy. This involves cleaning up legacy virtual machines, network configurations, and other test resources to ensure that each test is conducted in an independent and consistent environment, preventing residual data from interfering with the current test results. For virtual test scenario deployment, this embodiment employs automated orchestration technology. By programmatically calling the API interfaces of various OpenStack services, end-to-end automation from resource scheduling and network configuration to health checks is achieved. This transforms the construction of the test environment from manual operation to a fast, accurate, and repeatable automated process, significantly improving testing efficiency and environmental reliability. Specifically, step S200 may include the following sub-steps S210 to S240:

[0082] S210. Based on the environment cleanup and recovery strategy in the test scenario configuration parameters, initialize the test environment of the system under test.

[0083] Furthermore, the environment cleanup and recovery strategy includes: automatically identifying and deleting all virtual machine instances created by historical test tasks; cleaning up virtual networks, subnets, routers, and security group rules related to historical tests; and resetting the flow table rules of the underlying virtual switches to their initial state according to the test configuration.

[0084] S220. Call the OpenStack API interface, create virtual machine instances of the corresponding specifications according to the virtual machine specification parameters and network topology in the test scenario configuration parameters, and configure the network connections between virtual machines and between virtual machines and external networks to build a virtual test scenario.

[0085] It is worth mentioning that, in some virtual test scenarios, if parallel execution is used when automatically configuring the switch, conflicts may occur on the switch. Therefore, in order to avoid failures caused by switch configuration conflicts, each virtual test scenario in this embodiment adopts a serial execution strategy when it is executed.

[0086] For example, building a virtual test scenario includes: Test Scenario 1: Virtual machine 1 (e.g., firewall component) is configured with 4 cores and 8 storage, Virtual machine 2 (e.g., WAF component) is configured with 4 cores and 8 storage, and Virtual machine 1 and Virtual machine 2 are combined. The test time, test traffic volume, test traffic type, and metrics to be monitored and collected are set for this scenario. Test Scenario 2: Virtual machine 3 (firewall component only) is configured with 8 cores and 16 storage. The test time, test traffic volume, test traffic type, and metrics to be monitored and collected are set for this scenario.

[0087] S230. Send forwarding rules corresponding to the network topology to the virtual switch in the system under test so that the test traffic can be forwarded to the target virtual machine instance according to the preset path.

[0088] Furthermore, the forwarding rules include: configuring flow table entries based on the IP address and MAC address of the target virtual machine instance; setting priorities and forwarding ports for test traffic of different protocol types (such as TCP, UDP, ICMP); and configuring necessary network service quality policies to simulate bandwidth limitations or latency in a real network environment.

[0089] S240. After the virtual test scenario is deployed, perform a health status check on all created virtual machine instances to confirm that each virtual machine instance is in a normal operating state.

[0090] Furthermore, the health status check includes: first, querying the status of the virtual machine instance through the API interface; then, performing a network connectivity test on the virtual machine instance; and finally, verifying whether the basic services inside the virtual machine have started normally through the cloud initialization script.

[0091] It's worth noting that if any of the following anomalies occur during deployment: virtual machine creation failure, network configuration error, or virtual switch rule delivery failure, the current deployment process will be paused. The anomaly event information (such as error code, resource identifier, and timestamp) will be recorded in the anomaly log entry. Following the failure retry policy in the test execution strategy, the corresponding deployment steps will be re-executed after the anomaly is resolved. For example, if the initial virtual machine creation fails due to insufficient resources, the system will automatically wait for a predetermined time interval before re-initiating the creation request. If the creation still fails after the maximum of three retries, the deployment will be stopped, and the anomaly event will be reported to the final test report.

[0092] S300 generates a resource pool based on traffic model parameters and preset test traffic, and dynamically generates and sends test traffic to the system under test.

[0093] In this embodiment, the traffic simulation process no longer relies on a single, preset static traffic script, but is dynamically generated through flexibly configurable traffic model parameters, and deployed and scheduled in conjunction with a scalable test traffic generation resource pool. This allows the test traffic to be dynamically adjusted in multiple dimensions such as protocol type, packet characteristics, sending path, rate, and duration according to the needs of the actual test scenario, thereby simulating complex traffic patterns that more closely resemble the characteristics of real business loads, and achieving a refined evaluation of the stability and performance of OpenStack network components and the overall platform under diverse traffic pressures. Specifically, step S300 may include the following sub-steps S310 to S350:

[0094] S310. Configure the test traffic generation template according to the traffic protocol type and data packet size in the traffic model parameters.

[0095] Furthermore, based on the traffic protocol type, the corresponding traffic generation library is invoked to construct the basic message structure. Combined with the data packet size parameter, core fields such as payload length, flag bits, and sequence number for each data packet are set to generate a reusable traffic generation template. This traffic generation template can further support the configuration of traffic characteristics, such as the data packet sending interval, burst mode, and connection concurrency.

[0096] S320. Dynamically allocate at least one traffic generator from the test traffic generation resource pool and deploy the traffic generator within the system under test or in the external network.

[0097] Furthermore, the test traffic generation resource pool consists of a set of dynamically startable and stopable virtual machines or containerized instances, each pre-installed with traffic generation software. Based on the required traffic volume and topology for this test, the system automatically selects an appropriate number of traffic generators from the resource pool and deploys them within the system under test or in the external network via API interfaces, ensuring flexibility in traffic injection points.

[0098] S330. The generated template is sent to the traffic generator, and the transmission path of the test traffic is determined based on the traffic source and target endpoint identifiers in the traffic model parameters.

[0099] Furthermore, the traffic generation template and specific sending tasks generated in step S310 are distributed to each deployed traffic generator. Simultaneously, based on the network topology, the traffic source and destination endpoint identifiers are parsed, and the expected network path of the traffic is automatically calculated to provide a basis for subsequent link quality monitoring and problem localization.

[0100] S340: Control the traffic generator to dynamically send test traffic to the target virtual machine instance at a preset traffic rate according to the test execution order and duration in the test execution strategy.

[0101] S350: During the test traffic transmission process, the transmission status of the traffic generator and the network link quality are monitored in real time. If a traffic interruption or packet loss rate exceeds the threshold, the traffic parameters are automatically adjusted or the traffic generator is switched according to the failure retry policy, and abnormal traffic events are recorded.

[0102] Furthermore, while testing traffic transmission, the system collects real-time data on the transmission status of each traffic generator (e.g., number of packets sent, transmission rate) and network link quality (e.g., round-trip latency, packet loss rate, jitter). If a traffic interruption is detected, or the packet loss rate exceeds a preset alarm threshold (e.g., packet loss rate greater than 1% for three consecutive sampling periods), an anomaly handling process is automatically triggered: First, traffic parameters are adjusted according to the failure retry strategy (e.g., reducing the transmission rate, changing the target port); if the problem persists after adjustment, the traffic task is automatically migrated to a backup traffic generator in the resource pool. Finally, a detailed traffic anomaly event log is generated for the entire process, recording the anomaly time, type, involved resources, and handling measures for subsequent analysis and report generation.

[0103] S400. Based on the test execution strategy, start the stability test of the system under test. During the test execution, collect the actual monitoring index data of the system under test according to the preset cycle, and stop the test and retain the state of the on-site test environment when a specified abnormal alarm is triggered.

[0104] In this embodiment, after the test starts, the test process is automatically executed according to the scenario sequence, duration, and data collection cycle defined in the strategy. Simultaneously, multi-dimensional system monitoring data is collected, ensuring the timing accuracy, process traceability, and timely anomaly response of the test process. This allows stability assessment to be based on real, continuous system state data, rather than discrete test points. Detailed test points can be found in [reference needed]. Figure 2 As shown. This embodiment effectively solves the problems of test and monitoring disconnect, data timestamp misalignment, and delayed anomaly response in traditional methods by integrating test execution and monitoring data acquisition. Simultaneously, if the entire test process is interrupted due to certain anomalies, a pause alarm is triggered and the site environment is preserved for easy manual intervention, analysis, and adjustment. Specifically, step S400 may include the following sub-steps S410 to S440:

[0105] S410. According to the test execution order defined in the test execution strategy, start the stability test of each virtual test scenario in the system under test in sequence.

[0106] For example, the test execution order is as follows: first, perform basic network connectivity tests, then perform high-concurrency API stress tests, and finally perform long-term continuous load tests.

[0107] S420. During the execution of each test scenario, the timing for the virtual test scenario is started according to the test duration set in the test execution strategy.

[0108] S430: Based on the preset collection period configured in the test execution strategy, collect actual monitoring metric data by calling the OpenStack API interface.

[0109] Furthermore, during the execution of the test scenario, data collection tasks are initiated periodically according to the preset collection cycle configured in the test execution strategy. The collection tasks obtain actual monitoring metrics such as CPU utilization, memory utilization, disk utilization, API call latency, inter-virtual machine network communication latency, and virtual machine instance status by calling the monitoring API interfaces of various OpenStack services.

[0110] S440. Associate and store the actual monitoring indicator data collected each time with the corresponding test scenario information and collection timestamp to obtain time-series monitoring data records.

[0111] Furthermore, after each successful data collection, key metadata is automatically appended to the data record, including: the test scenario identifier to which this collection belongs, the collection timestamp, and the identifier of the virtual machine instance or resource that generated the data. Subsequently, this indicator data with rich contextual information is written into the time-series database, thus forming monitoring data records with clear test phases and time-series markers, providing a structured data source for subsequent data backtracking, trend analysis, and performance bottleneck localization by scenario.

[0112] It's worth noting that if any virtual machine instance is detected as abnormal or test traffic is interrupted during testing, the system will automatically decide whether to pause the current test and preserve the current test environment state, retry, or jump to the next virtual test scenario, based on the failure retry policy in the test execution strategy. The abnormal event will also be recorded in the exception log entry. For example, the failure retry policy may include: for transient anomalies (i.e., anomalies not recorded in the initial exception log entry or anomalies not occurring in the previous round of testing), the current test will be automatically paused, and the scenario will be retried after a short period, or the test will be restarted after manual analysis and adjustment; for persistent anomalies (i.e., anomalies that occur in multiple rounds of scenario testing), the anomaly will be recorded, and the system will jump to the next test scenario to continue execution, avoiding blocking the entire test process.

[0113] S500 compares the actual monitoring indicator data with the expected monitoring indicator data in real time, determines whether the test passes based on the comparison results, and records the test events that fail as exception log entries.

[0114] In this embodiment, by comparing the real-time collected actual monitoring indicators with predefined expected monitoring indicators in real time, an automated evaluation loop of testing, analysis, and judgment is achieved, thereby significantly shortening the delay from problem discovery to alarm, making the testing process itself a proactive means of stability detection and anomaly perception. Simultaneously, this embodiment not only records simple pass / fail statuses but also triggers corresponding automated responses based on preset strategies, thus elevating the traditional passive result recording to a proactive test recording process. Specifically, step S500 may include the following sub-steps S510 to S530:

[0115] S510. Classify the actual monitoring indicator data according to virtual machine instances and time periods, and compare them in real time with the preset threshold range of the corresponding virtual machine instances in the expected monitoring indicators.

[0116] S520a. If any indicator in the actual monitoring data exceeds the corresponding preset threshold range, it is determined that the virtual machine instance has failed the test in the current test cycle, and an abnormal log entry containing the virtual machine identifier, abnormal indicator, exceeding value and occurrence timestamp is generated according to the preset rules.

[0117] S520b: If all actual monitoring metrics are within the corresponding preset threshold range, then the virtual machine instance is determined to have passed the test within the current test cycle.

[0118] S530. The judgment results and corresponding exception log entries in each test cycle are associated with the test scenario identifier and stored to generate test result logs.

[0119] Furthermore, after completing the comparison and judgment for each test cycle, the overall judgment result (e.g., in scenario A at time T, 8 out of 10 virtual machines pass and 2 fail) and all generated exception log entries are associated with the currently executing test scenario identifier and stored uniformly as a test result log. This allows subsequent analysis to clearly trace which stage of which test scenario each exception occurred in, facilitating problem aggregation and test evaluation by scenario.

[0120] It's worth noting that if the same virtual machine instance exhibits the same abnormal metric across multiple consecutive test cycles, it will be marked as a persistent anomaly in the anomaly log entry. Furthermore, the virtual test scenario will be automatically retested according to the failure retry policy in the test execution strategy. For example, the policy might be set to automatically clean up and rebuild the current scenario environment when a persistent API latency anomaly occurs, and then re-execute the scenario test to verify whether it is an intermittent environmental issue, thereby achieving a higher level of automated troubleshooting and verification.

[0121] S600. After the test is completed, a test report is generated based on the test process data and abnormal log entries, and the abnormal log entries that have increased are marked in the test report.

[0122] In this embodiment, by automatically summarizing and analyzing the process data generated throughout the entire testing cycle, a structured test report is generated, transforming the original monitoring stream and event logs into conclusive information that can be directly used for quality assessment and decision-making. More importantly, once the report shows an increase of one abnormal log entry compared to the previous round of testing results, the system immediately initiates an automated response chain that includes immediate alarms and site protection. This breaks through the limitation of the traditional testing process where the environment is destroyed immediately after the test ends. By proactively preserving a complete system snapshot at the moment the anomaly occurs, a failed test is transformed into a fault reproduction opportunity for in-depth analysis, greatly improving the operations and maintenance team's ability to anticipate potential online risks and the efficiency of diagnosing existing problems. Specifically, step S600 may include the following sub-steps S610 to S630:

[0123] S610. Based on the time-series monitoring data records, test result logs, and exception log entries stored during the test process, aggregate and statistically analyze the data according to the test scenario and time dimension to generate a structured test report.

[0124] S620. If the test report contains any abnormal log entry that shows an increase, an abnormal event alarm will be triggered to the user terminal via email or system notification according to the preset alarm rules.

[0125] S630. Upon triggering an alarm, stop the resource reclamation and environment cleanup process of the system under test, retrieve the on-site test environment status at the time of the anomaly and archive it in the form of an environment snapshot, and generate a corresponding fault troubleshooting entry link.

[0126] Furthermore, to prevent the fault scene from being destroyed by subsequent automated cleanup processes, this embodiment simultaneously stops any pre-planned resource reclamation and environment cleanup operations targeting the system under test when an alarm is triggered. Simultaneously, it retrieves and freezes the entire on-site test environment state at the moment the first or most severe anomaly is recorded. Then, this state data is packaged and stored as an environment snapshot in a dedicated archive storage. Finally, a unique and accessible fault diagnosis link is generated from the snapshot and associated with the alarm information and test report. This allows maintenance personnel to easily restore the fault scene with a single click via the link after receiving an alarm, enabling interactive debugging, in-depth log extraction, or performance profiling analysis, thereby efficiently locating the cause of the anomaly.

[0127] In addition, refer to Figure 3 As shown, this embodiment also proposes an automated testing system for OpenStack stability, including:

[0128] The client-side interface is used to receive test task instructions input by the user and to display test results and alarm notifications.

[0129] The testing process generates a resource pool to provide dynamically allocated traffic generators and send test traffic to the system under test.

[0130] The OpenStack component resource pool provides virtual machine, network, and storage resources to support the deployment and operation of virtual test scenarios.

[0131] The main controller is used to execute the automated testing method for OpenStack stability described above.

[0132] Furthermore, the main controller includes:

[0133] The configuration acquisition module is used to acquire test configuration information, which includes at least test scenario configuration parameters, traffic model parameters, expected monitoring metrics, and test execution strategies.

[0134] The test scenario deployment module is used to automatically perform environment initialization and virtual test scenario deployment of the system under test by calling the OpenStack interface based on the test scenario configuration parameters.

[0135] The test traffic module is used to dynamically generate and send test traffic to the system under test based on traffic model parameters and preset test traffic to create a resource pool.

[0136] The test monitoring module is used to initiate stability tests on the system under test according to the test execution strategy. During the test execution, it collects the actual monitoring index data of the system under test at a preset period, and stops the test and retains the state of the on-site test environment when a specified abnormal alarm is triggered.

[0137] The anomaly notification module is used to compare the actual monitoring indicator data with the expected monitoring indicator in real time, determine whether the test passes based on the comparison results, and record the failed test events as anomaly log entries.

[0138] The test report module is used to generate a test report based on the test process data and exception log entries after the test is completed, and to mark the exception log entries that have increased in the test report.

[0139] In summary, this invention proposes an automated testing method and system for OpenStack stability. By integrating configurable test scenario parameters, dynamic traffic models, and automated deployment, monitoring, and anomaly detection processes, it constructs a closed-loop cloud platform stability assessment system. This method not only automates the entire process from test environment construction, traffic injection, and metric monitoring to result analysis, but also transforms the traditional decentralized, passive, and result-oriented testing model into a proactive, closed-loop automated testing system centered on quality insight and fault prevention through intelligent anomaly identification, real-time alerts, and on-site snapshot retention mechanisms. This invention systematically solves key problems in existing technologies such as low environment deployment efficiency, inaccurate test traffic, disconnect between monitoring and execution, and difficulty in reproducing anomalies, significantly improving the efficiency, accuracy, and operational response capabilities of OpenStack cloud platform stability assessment in complex and dynamic business scenarios.

[0140] Since the systems / devices described in the above embodiments of the present invention are systems / devices used to implement the methods of the above embodiments of the present invention, those skilled in the art can understand the specific structure and modifications of the systems / devices based on the methods described in the above embodiments of the present invention, and therefore will not be repeated here. All systems / devices used in the methods of the above embodiments of the present invention fall within the scope of protection of the present invention.

[0141] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0142] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions.

[0143] It should be noted that in the description of this invention, the word "a" or "an" preceding a component does not exclude the existence of multiple such components. This invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. The use of terms such as first, second, third, etc., is merely for convenience and does not indicate any order. These terms can be understood as part of the component names.

[0144] Furthermore, it should be noted that in the description of this specification, the terms "one embodiment," "some embodiments," "embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0145] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning of the basic inventive concept, can make other changes and modifications to these embodiments.

[0146] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from the spirit and scope of the invention.

Claims

1. An automated testing method for OpenStack stability, characterized in that, include: Obtain test configuration information, which includes at least test scenario configuration parameters, traffic model parameters, expected monitoring metrics, and test execution strategies; Based on the test scenario configuration parameters, the environment initialization and virtual test scenario deployment of the system under test are automatically executed by calling the OpenStack interface. Based on the traffic model parameters and the preset test traffic, a resource pool is generated, and test traffic is dynamically generated and sent to the system under test. According to the test execution strategy, the stability test of the system under test is initiated. During the test execution, the actual monitoring index data of the system under test is collected at a preset period, and the test is stopped and the state of the on-site test environment is preserved when a specified abnormal alarm is triggered. The actual monitoring indicator data is compared with the expected monitoring indicator in real time. The test is judged based on the comparison results, and the test events that fail are recorded as abnormal log entries. After the test is completed, a test report is generated based on the test process data and abnormal log entries, and the abnormal log entries that have increased are marked in the test report.

2. The automated testing method for OpenStack stability as described in claim 1, characterized in that, Obtaining test configuration information includes: Receive and parse the test task instructions input by the user. The test task instructions contain the test task identifier of the system under test. Based on the test task identifier, load the test configuration information associated with the test task identifier from the preset test configuration database. The test configuration information includes test scenario configuration parameters, traffic model parameters, expected monitoring indicators, and test execution strategies. The test scenario configuration parameters include: virtual machine specifications and network topology. Traffic model parameters include: traffic protocol type, packet size, and traffic source and destination endpoint identifiers; Expected monitoring metrics include: CPU utilization, memory utilization, disk utilization, API call latency and inter-virtual machine network communication latency, virtual machine instance status, and the number of abnormal log entries; The test execution strategy includes: test execution order, test duration for each scenario, number of test loops, failure retry strategy, and environment cleanup and recovery strategy.

3. The automated testing method for OpenStack stability as described in claim 1, characterized in that, Based on the test scenario configuration parameters, the system automatically performs environment initialization and virtual test scenario deployment of the system under test by calling the OpenStack interface, including: Initialize the test environment of the system under test according to the environment cleanup and recovery strategy in the test scenario configuration parameters; By calling the OpenStack API interface, virtual machine instances of the corresponding specifications are created according to the virtual machine specification parameters and network topology in the test scenario configuration parameters, and network connections between virtual machines and between virtual machines and external networks are configured to build a virtual test scenario. Send forwarding rules corresponding to the network topology to the virtual switches in the system under test so that test traffic can be forwarded to the target virtual machine instance according to the preset path; After the virtual test scenario is deployed, a health check is performed on all created virtual machine instances to confirm that each virtual machine instance is in a normal operating state. If any of the following abnormal situations occur during the deployment process: virtual machine creation failure, network configuration error, or virtual switch rule delivery failure, the current deployment process will be paused, the abnormal event information will be recorded in the abnormal log entry, and the corresponding deployment steps will be re-executed after the abnormality is repaired, according to the failure retry policy in the test execution strategy.

4. The automated testing method for OpenStack stability as described in claim 1, characterized in that, Based on traffic model parameters and a pre-defined test traffic pool, test traffic is dynamically generated and sent to the system under test, including: Configure the test traffic generation template based on the traffic protocol type and packet size in the traffic model parameters; Dynamically allocate at least one traffic generator from the test traffic generation resource pool and deploy the traffic generator within the system under test or in an external network; The generated template is sent to the traffic generator, and the transmission path of the test traffic is determined based on the traffic source and target endpoint identifiers in the traffic model parameters. The control traffic generator dynamically sends test traffic to the target virtual machine instance at a preset traffic rate according to the test execution order and duration in the test execution strategy; During the test traffic transmission process, the transmission status of the traffic generator and the network link quality are monitored in real time. If a traffic interruption or packet loss rate exceeds the threshold, the traffic parameters are automatically adjusted or the traffic generator is switched according to the failure retry policy, and abnormal traffic events are recorded.

5. The automated testing method for OpenStack stability as described in claim 1, characterized in that, According to the test execution strategy, stability testing of the system under test is initiated. During the test execution, actual monitoring index data of the system under test are collected at preset intervals, and the test is stopped and the on-site test environment state is preserved when a specified abnormal alarm is triggered. According to the test execution order defined in the test execution strategy, the stability tests of each virtual test scenario in the system under test are started sequentially. During the execution of each test scenario, the timing for that virtual test scenario is started according to the test duration set in the test execution strategy. Based on the preset collection period configured in the test execution strategy, actual monitoring metric data are collected by calling the OpenStack API interface; Each time the actual monitoring indicator data is collected is associated with the corresponding test scenario information and the collection timestamp and stored to obtain time-series monitoring data records; If any virtual machine instance is found to be in an abnormal state or the test traffic is interrupted during the test, the system will automatically decide whether to pause the current test and retain the state of the on-site test environment, perform a retry, or jump to the next virtual test scenario, based on the failure retry policy in the test execution strategy, and record the abnormal event in the abnormal log entry.

6. The automated testing method for OpenStack stability as described in claim 1, characterized in that, The actual monitoring data is compared with the expected monitoring data in real time. Based on the comparison results, the test is judged to pass or fail, and test events that fail are recorded as exception log entries, including: The actual monitoring indicator data is categorized by virtual machine instance and time period, and compared in real time with the preset threshold range of the corresponding virtual machine instance in the expected monitoring indicators. If any of the actual monitored metrics exceeds the corresponding preset threshold range, the virtual machine instance is determined to have failed the test in the current test cycle, and an abnormal log entry containing the virtual machine identifier, abnormal metric, exceeding value and occurrence timestamp is generated according to preset rules. If all actual monitoring metrics are within the corresponding preset threshold range, the virtual machine instance is deemed to have passed the test within the current testing period. The judgment results and corresponding exception log entries in each test cycle are associated with the test scenario identifier and stored to generate test result logs; If the same virtual machine instance exhibits the same abnormal metric in multiple consecutive test cycles, it will be marked as a persistent abnormal event in the abnormal log entry, and the virtual test scenario will be automatically retested according to the failure retry policy in the test execution strategy.

7. The automated testing method for OpenStack stability as described in claim 1, characterized in that, After the test is completed, a test report is generated based on the test process data and exception log entries. The test report will mark the exception log entries that have increased in number, including: Based on the time-series monitoring data records, test result logs, and exception log entries stored during the test process, data is aggregated and statistically analyzed according to test scenarios and time dimensions to generate a structured test report; If the test report contains any abnormal log entry that shows an increase, an abnormal event alarm will be triggered to the user's terminal via email or system notification according to the preset alarm rules. Upon triggering the alarm, the resource reclamation and environment cleanup processes of the system under test are stopped, and the on-site test environment status at the time of the anomaly is retrieved and archived in the form of an environment snapshot, and a corresponding fault investigation entry link is generated.

8. An automated testing system for OpenStack stability, characterized in that, include: The user-side interface is used to receive test task instructions input by the user and to display test results and alarm notifications. The test process generates a resource pool to provide dynamically allocable traffic generators and send test traffic to the system under test. The OpenStack component resource pool provides virtual machine, network, and storage resources to support the deployment and operation of virtual test scenarios. The main controller is used to execute an automated testing method for OpenStack stability as described in any one of claims 1-7.

9. The automated testing system for OpenStack stability as described in claim 8, characterized in that, The main controller includes: The configuration acquisition module is used to acquire test configuration information, which includes at least test scenario configuration parameters, traffic model parameters, expected monitoring metrics, and test execution strategies. The test scenario deployment module is used to automatically perform environment initialization and virtual test scenario deployment of the system under test by calling the OpenStack interface based on the test scenario configuration parameters. The test traffic module is used to generate a resource pool based on traffic model parameters and preset test traffic, and dynamically generate and send test traffic to the system under test. The test monitoring module is used to initiate stability tests on the system under test according to the test execution strategy. During the test execution, it collects the actual monitoring index data of the system under test at a preset period, and stops the test and retains the state of the on-site test environment when a specified abnormal alarm is triggered. The anomaly notification module is used to compare the actual monitoring indicator data with the expected monitoring indicator in real time, determine whether the test passes based on the comparison results, and record the failed test events as anomaly log entries. The test report module is used to generate a test report based on the test process data and exception log entries after the test is completed, and to mark the exception log entries that have increased in the test report.

Images