Automated Testing Method and System for Large-Scale Cluster Performance Based on Cloud Services

Abstract

The present invention discloses a large-scale automated cluster performance testing method and system based on cloud services. The method includes batch-generating computing nodes on a physical server based on an operating system installed with test software; designating the test software for one of the generated computing nodes as a server and the test software for the remaining computing nodes as clients; issuing a test program and test tasks based on the test program to the designated clients via the server, and synchronously obtaining the resource usage status of the client and the physical server / client during the execution of the test tasks by each client. The method can define the performance boundaries that can be achieved by cloud computing solutions on the same and different hardware, and the performance boundaries that can be achieved by different cloud computing solutions on the same hardware, thereby achieving high testing efficiency and low testing error.

Description

Technical Field

[0001] The present invention relates to a large-scale cluster performance testing technology based on cloud services, and in particular to a large-scale cluster performance automated testing method and system based on cloud services. Background Art

[0002] Large-scale cloud-based clusters are currently the fundamental architecture for providing cloud services. For example, virtual machines are provisioned on physical servers as compute nodes, and physical clients log in to the compute nodes to implement various remote operations or computing, including cloud desktops. Currently, large-scale cloud-based clusters face the following challenges: It's difficult to define the upper limit of resources available for running virtual machines in large-scale cloud-based clusters; it's difficult to define the computing resources required for specific operations within virtual machines; it's difficult to accurately define the network resources required for remote operations on virtual machines; it's difficult to effectively understand the maximum performance achievable by cloud server hardware, and it's difficult to effectively detect performance degradation due to hardware aging. It's difficult to compare the maximum performance boundaries achievable by different hardware in large-scale cloud-based clusters; it's difficult to compare the computing resource consumption required for the same operation on software service systems from different vendors using the same hardware; it's difficult to compare the time and performance of operations under the same load; and it's difficult to compare the maximum performance boundaries achievable. Performance testing is difficult to evaluate, as everyone's subjective judgment of performance makes it difficult to establish a unified standard for test data. Every company claims their hardware and software combination performs better. For large-scale cluster testing based on cloud services, comparative testing is mostly conducted by testers. Each tester has his or her own testing methods and testing habits. The response time and test monitoring duration will cause human interference errors in the test results. In addition, different test scripts currently on the market cannot be written and executed on multiple platforms at the same time. For example, a test script written on Windows 7 cannot be tested on Windows 10, Windows 2008, or Linux. Summary of the Invention

[0003] The technical problem to be solved by the present invention is as follows: In response to the above-mentioned problems of the prior art, a large-scale cluster performance automation testing method and system based on cloud services are provided. The present invention can define the respective performance boundaries that cloud computing solutions can achieve under the same hardware and different hardware, and define the respective performance boundaries that different cloud computing solutions can achieve under the same hardware. The testing efficiency is high and the testing error is small.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] A large-scale cluster performance automation testing method based on cloud services, comprising:

[0006] 1) Prepare an operating system with test software installed, and generate computing nodes in batches on a physical server based on the operating system with test software installed;

[0007] 2) Designate the test software of one computing node as the server and the test software of the remaining computing nodes as the clients among the generated computing nodes;

[0008] 3) Send the test program and the test task based on the test program to the designated client through the server, and synchronously obtain the resource usage status of the client and the physical server / physical client during the execution of the test task by each client.

[0009] Optionally, step 1) includes: 1.1) preparing an operating system with test software installed; 1.2) creating a source virtual machine on a physical server, installing the operating system with test software in the created source virtual machine, or importing a virtual machine that has already been installed with the operating system with test software installed; 1.3) utilizing the batch publishing function of the virtualization system on the physical server to copy batch virtual machine copies based on the source virtual machine to obtain a batch of computing nodes with the operating system with test software installed.

[0010] Optionally, step 2) designating the test software of one computing node as the server and the test software of the remaining computing nodes as the client in the generated computing nodes means: remotely logging into the designated computing node on the physical client, and the test software on the designated computing node triggers the test software of the computing node to send a broadcast message to the remaining computing nodes through a specified operation, so that the test software of the computing node serves as the server and the test software of the remaining computing nodes serves as the client.

[0011] Optionally, after step 2) and before step 3), the step of each client periodically sending its own identity information, IP address and resource occupancy status to the server is also included. The resource occupancy status includes at least one of CPU, memory, disk I / O and network I / O status data. The CPU status data refers to the CPU occupancy ratio, the memory status data includes the total memory and space memory size, the disk I / O status data includes the number of read bytes and written bytes, and the network I / O status data includes the number of sent bytes and received bytes.

[0012] Optionally, the test program in step 3) is one or more of an exe executable file, a python executable file, a shell executable file, a cmd executable file, and an execution script program generated by a recorded manual operation sequence; the operations performed by the test program in step 3) include one or more of window operations, drawing program operations, picture browsing operations, video playback operations, browser operations, resource browser operations, office software operations, maximum computing load stress test operations, maximum disk read and write stress test operations, and maximum network bandwidth stress test operations.

[0013] Optionally, the attributes of the test task based on the test program in step 3) include the test task name, the selected test program, the execution mode and execution parameters of the test program, the execution mode includes immediate execution mode and timed execution mode, and the execution parameters include the number of repeated executions, execution interval time and start execution time.

[0014] Optionally, step 3) synchronously obtains the resource occupancy status of each client and the physical server / physical client during the execution of the test task by each client, including: while sending the test task based on the test program to the designated client through the server, simultaneously obtains the resource occupancy status on the physical server / physical client through the SSH protocol, and starts recording the resource occupancy status of the client; after receiving the notification of completion of the test task sent by any client, stops recording the resource occupancy status of the client, thereby obtaining the resource occupancy status of the client during the execution of the test task; after receiving the notification of completion of the test task sent by all clients, stops recording the resource occupancy status on the physical server / physical client, thereby obtaining the resource occupancy status of the physical server / physical client during the execution of the test task by each client; the resource occupancy status includes at least one of CPU, memory, disk I / O and network I / O status data, the CPU status data refers to the CPU occupancy ratio, the memory status data includes the total memory and spatial memory size, the disk I / O status data includes the number of read bytes and written bytes, and the network I / O status data includes the number of sent bytes and received bytes.

[0015] Optionally, after step 2) and before step 3), a step of batch updating clients is also included: the distribution address of the new version of the test software is sent to each client on the server side, and each client obtains the new version of the test software based on the received distribution address and completes the update and upgrade of the local test software.

[0016] In addition, the present invention also provides a large-scale cluster performance automation testing system based on cloud services, including an interconnected microprocessor and a memory, characterized in that the microprocessor is programmed or configured to execute the steps of the large-scale cluster performance automation testing method based on cloud services.

[0017] In addition, the present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program that is programmed or configured to execute the large-scale cluster performance automation testing method based on cloud services.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] 1. The present invention can define the performance boundaries that cloud computing solutions can achieve under the same hardware and different hardware, and define the performance boundaries that different cloud computing solutions can achieve under the same hardware. The testing process is simple and fast, which can reduce the complexity of the operation process.

[0020] 2. In cloud computing, there can be thousands of cloud virtual machines. Manual testing consumes a lot of human resources and requires a lot of testing time. This invention can quickly meet the rigid requirements of cluster large-scale performance testing, quickly deploy the test environment, improve test efficiency, shorten the test cycle, and reduce the human resource consumption of testing.

[0021] 3. The present invention generates computing nodes in batches on a physical server based on an operating system installed with test software, sends test programs and test tasks based on the test programs to designated clients through a server, and synchronously obtains the resource occupancy status of the client and the physical server / physical client during the execution of the test tasks by each client, thereby ensuring the same test environment and test method, eliminating the influence of human factors caused by human participation, the errors caused by human performance recording, and the interference of human factors in server performance evaluation, with the advantages of small test errors and reliable and stable test results. By making each performance test comparable and ensuring that the actions of multiple tests are exactly the same, the performance optimization level of hardware and software can be evaluated more objectively.

[0022] 4. Traditional testing software does not evaluate the performance of transmission protocols, nor can it determine the success of protocol optimization. As R&D departments continuously iterate and release new versions, it is very difficult to determine whether the network bandwidth and system performance consumed by the same desktop operation have been optimized or eliminated. The present invention can synchronously obtain the resource usage status of each client and the physical server / client during the execution of test tasks. This can be used to compare the performance and advantages and disadvantages of network transmission protocols and image compression protocols, thereby optimizing network transmission protocols and image compression protocols. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 Schematic diagram of the basic process of the method of the embodiment of the present invention.

[0024] Figure 2 Schematic diagram of the topological structure of the system in the embodiment of the present invention.

[0025] Figure 3 Schematic diagram of the performance test principle of the system in the embodiment of the present invention.

[0026] Figure 4 This is a schematic diagram of obtaining resource occupancy status on a physical server / physical client in the prior art.

[0027] Figure 5 This is a schematic diagram of a command window for obtaining memory occupancy status in an embodiment of the present invention.

[0028] Figure 6 This is a schematic diagram of a command window for obtaining CPU occupancy status in an embodiment of the present invention.

[0029] Figure 7 This is a schematic diagram of a command window for obtaining network usage status data in an embodiment of the present invention.

[0030] Figure 8 This is a schematic diagram of synchronously acquiring resource occupancy status on a physical server / physical client in an embodiment of the present invention.

[0031] Figure 9 This is a topology diagram of remote desktop traffic monitoring in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] like Figure 1 、 Figure 2 and Figure 3 As shown, the large-scale cluster performance automation testing method based on cloud services in this embodiment includes:

[0033] 1) Prepare an operating system with test software installed, and generate computing nodes in batches on a physical server based on the operating system with test software installed;

[0034] 2) Designate the test software of one computing node as the server and the test software of the remaining computing nodes as the clients among the generated computing nodes;

[0035] 3) Send the test program and the test task based on the test program to the designated client through the server, and synchronously obtain the resource usage status of the client and the physical server / physical client during the execution of the test task by each client.

[0036] In this embodiment, the test software is specifically named KylinTest, which is made based on Qt graphical interface technology, batch nested multi-threading technology, network remote communication, Linux and Windows system kernel API interfaces, and supports macro recording and screen recording technologies to generate test programs.

[0037] In this embodiment, step 1) includes: 1.1) preparing an operating system with test software installed; 1.2) creating a source virtual machine on a physical server, installing an operating system with test software in the created source virtual machine or importing a virtual machine with an operating system already installed with test software; 1.3) utilizing the batch publishing function of the virtualization system on the physical server, copying batch virtual machine copies based on the source virtual machine, and obtaining batch computing nodes with operating systems installed with test software. Among them, the virtualization system can adopt KVM, or openstack, or KSVD Kylin Cloud Desktop Server, or Sangfor Cloud Solution as needed. After creating the source virtual machine, install the KylinTest operating system in the created source virtual machine or import a virtual machine with an operating system already installed with KylinTest; then utilize the batch publishing function of the virtualization system on the physical server to quickly copy a large number of operating system copies from the source virtual machine and start it. After startup, KylinTest will start automatically.

[0038] In this embodiment, step 2) designating the test software of one of the generated computing nodes as a server and the test software of the remaining computing nodes as clients refers to remotely logging into the designated computing node on the physical client, and triggering the test software on the designated computing node to send a broadcast message to the remaining computing nodes through a specified operation, thereby making the test software of the designated computing node serve as a server and the test software of the remaining computing nodes serve as clients. In this embodiment, all computing nodes are computing nodes in the cloud service cluster, and one node is used as a server to control all other computing nodes, thereby uniformly mobilizing the computing nodes in the cloud server cluster.

[0039] In this embodiment, after step 2) and before step 3), the step of each client periodically sending its own identity information, IP address, and resource occupancy status to the server is also included. In this embodiment, the resource occupancy status includes CPU, memory, disk I / O, and network I / O status data (at least one of which can be used as needed). CPU status data refers to the CPU occupancy ratio, memory status data includes total memory and free memory size, disk I / O status data includes the number of read bytes and written bytes, and network I / O status data includes the number of sent bytes and received bytes. As an optional implementation method, in this embodiment, the server also includes generating an Excel file based on the resource occupancy status reported by the client.

[0040] In this embodiment, step 3) sends the test program to the designated client through the server, and can quickly send the latest test script to each client, thereby reducing the workload of manually deploying automated test scripts and quickly implementing the deployment of new test projects.

[0041] In this embodiment, the test program in step 3) is one or more of an exe executable file, a python executable file, a shell executable file, a cmd executable file, and an execution script program generated by a recorded manual operation sequence, so that subsequent testers can reduce the test workload while ensuring that each test is the same operation time and operation speed, eliminating the differences in the tests and providing a basis for performance comparison; the operations performed by the test program in step 3) include window operations, drawing program operations, picture browsing operations, video playback operations, browser operations, resource browser operations, office software operations, maximum computing load stress test operations, maximum disk read and write stress test operations, and maximum network bandwidth stress test operations. In this embodiment, the operations performed by the test program are different from ordinary stress tests and random tests, but strive to truly restore normal office processes. The test program can not only import existing executable files (exe executable files, python executable files, shell executable files, cmd executable files, and execution script programs generated by recorded manual operation sequences), but also can be generated by on-site recorded manual operation sequences. It imitates the operations commonly used by people using the desktop and most realistically restores human operations. For example, a cloud desktop user can simulate the stress of their office desktop. Examples include playing videos, turning pages in PowerPoint presentations, Word documents, and Excel spreadsheets, browsing the Taobao homepage and turning pages in a browser, zooming in and out on images, dragging windows, and other common office operations. A variety of executable files are available, enabling comprehensive testing of Windows and Linux operating systems. The test program performs operations such as maximum computing load stress testing, maximum disk read / write stress testing, and maximum network bandwidth stress testing, testing the upper limits of the virtual machine system under maximum office demands. By testing the upper limits of the virtual machine system through common office operations, a more accurate understanding of the upper limits can be achieved. By simulating real human operations, the computing and network concurrency stresses associated with normal computer use can be replicated. Traditional functional testing cannot guarantee that each test is performed identically by the same person, nor can the testing process be identical across different testers. The time intervals and steps between each tester can vary. For example, if tester A performs a Word document and records the hardware performance consumed as A1, then the hardware performance report obtained by tester A performing the Word document again may differ. If tester B tests Word and records the hardware performance consumed by Word as B1, this B1 may be different from A1 recorded by tester A. If there is no guarantee that A1, A2, and B2 are different, then after a hardware upgrade or software performance optimization, the test report cannot verify that performance has been improved.The test program in this embodiment can use an execution script generated by recording the manual operation sequence, recording the tester's operations in a 1:1 ratio as a template for hardware and software performance testing under multiple iterations. As an optional implementation, the test program in this embodiment is shown in Table 1.

[0042] Table 1: Example of test procedure.

[0043]

[0044]

[0045]

[0046]

[0047] When a test task requires a new test program for simulating real user operations, the test program can be sent to the specified client through the server, so that the client can obtain the new test program. For example, there are 300 test nodes in the cluster. When a new test requirement arises and a new script is needed, an execution script program generated by a recorded manual operation sequence can be added according to the custom script. Click the start recording function on the server to record all keyboard and mouse operations, and record them into a script, which is directly added to the script management page and provided for use in custom tasks. The execution script program generated by the recorded manual operation sequence on the server is then sent to each client to quickly update the test script. It should be noted that in this embodiment, the test program used to simulate real user operations is a preferred implementation method, but based on the concept of the method of this embodiment, the test program in this embodiment does not rely on simulating real user operations, and can also be an ordinary stress test operation or other test operation. Traditional stress tests, large CPU computing requests, high network occupancy pressure, high disk I / O read and write stress tests, etc., are different from normal users' use of the desktop. Normal users' use of the desktop is fluctuating, with peaks and valleys. Therefore, the stress tests on the market do not simulate real desktop usage habits, and tests without human operation similarity are not referenceable. On the other hand, the performance of the CPU, memory, disk, etc. are written on the hardware product and clearly stated in the hardware product manual. There is no need to verify the software. Even if verification is done, it is not very meaningful because the performance must serve actual use. The purpose of the test in this embodiment is to simulate real human use to test the upper limit of the system's support in actual business. Therefore, the maximum concurrent performance can be tested. For example, how many people can play PPT at the same time, and how many people can write Word documents, use browsers, Excel, etc. at the same time.

[0048] In the present embodiment, the attributes of the test task based on the test program in step 3) include the test task name, the selected test program, the execution mode and execution parameters of the test program, the execution mode includes the immediate execution mode and the timed execution mode, and the execution parameters include the number of repeated executions, the execution interval time and the start execution time. For example, since some tests need to be performed in an environment with less access, the task execution time can be set to start at 0:00 p.m. according to the time setting in the newly created test task. When the task is released, the client will determine whether it has reached 0:00 p.m. If the time reaches 0:00 p.m., it will start to execute the task released by the server. The server will record the resource occupancy status reported by the client at 0:00 p.m., and will also record the resource occupancy status on the physical server / physical client for easy viewing the next day.

[0049] In this embodiment, step 3) sends a test task based on the test program to the designated client through the server, and uses the custom task as the test unit to realize multi-threading of the custom script, custom task execution time, custom number of repetitions, and custom time interval between each test. In combination with the selection of the test program, and synchronous acquisition of the resource occupancy status of each client and the physical server / physical client during the execution of the test task by each client, it is ensured that the recorded performance changes are caused by the execution of the task. Through cluster concurrent testing, each client is managed and the client is controlled to execute the custom task to conduct large-scale performance stress testing. By having a large number of virtual machines acting as clients execute a certain operation at the same time, the maximum support boundary that the cluster can achieve for a certain operation is calculated; and the performance of different systems under the same amount of custom automatic test operations.

[0050] In this embodiment, step 3) synchronously obtaining the resource occupancy status of each client and the physical server / physical client during the execution of the test task by each client includes: while sending the test task based on the test program to the designated client through the server, simultaneously obtaining the resource occupancy status on the physical server / physical client through the SSH protocol, and starting to record the resource occupancy status of the client; after receiving the notification of completion of the test task sent by any client, ending the recording of the resource occupancy status of the client, thereby obtaining the resource occupancy status of the client during the execution of the test task; after receiving the notification of completion of the test task sent by all clients, ending the recording of the resource occupancy status on the physical server / physical client, thereby obtaining the resource occupancy status of the physical server / physical client during the execution of the test task by each client; the resource occupancy status includes at least one of CPU, memory, disk I / O and network I / O status data, the CPU status data refers to the CPU occupancy ratio, the memory status data includes the total memory and spatial memory size, the disk I / O status data includes the number of read bytes and written bytes, and the network I / O status data includes the number of sent bytes and received bytes. When obtaining the resource occupancy status on the physical server / physical client via the SSH protocol, the server will automatically input the username and login password of the Linux server to be monitored, as well as the monitoring terminal commands for the various data to be monitored, ensuring that the resource occupancy status of each client and the physical server / physical client is synchronously obtained during the execution of the test task by each client. In this embodiment, SSH remote control is used as the monitoring basis for the physical server / physical client. When the task is executed, the physical server / physical client is monitored. According to the specific requirements of each test, the resource occupancy status of the system is freely customized to be monitored and the monitored data is recorded. After the server is connected to the physical server / physical client at the same time via the SSH protocol, whether the task sent to the client is executed immediately or at a scheduled time, as soon as the task starts, various information of the physical server / physical client will be recorded based on the monitoring information and an Excel file will be generated. A line graph will also be generated to facilitate subsequent performance analysis of large-scale tests. The continuous analysis structure can derive the pros and cons of different versions, different systems, and different hardware.

[0051] In this embodiment, in step 3), the resource occupancy status of each client and the physical server / physical client during the execution of the test task by each client can eliminate the influence of human factors caused by human participation and the errors caused by human performance recording. For example, when playing a PPT in a virtual machine, it is necessary to monitor the performance changes of the virtualized server, and the resource occupancy status of the server consumed by the virtual machine playing the PPT in the server, such as Figure 4As shown, the normal process of manual monitoring includes: Step 1: Enable server memory monitoring: Use the SSH tool and use script commands in the SSH tool to monitor performance, such as Figure 5 As shown, the command used is "free-s 3|grep Mem"; Step 2: Enable server CPU monitoring: Use the SSH tool and use script commands in the SSH tool to monitor performance, such as Figure 6 As shown, the command used is "top|grep qemu"; Step 3: Open server disk monitoring: Use SSH tool, and use script commands in SSH tool to monitor performance; Step 4: Open server network monitoring: Use SSH tool, and use script commands in SSH tool to monitor performance, such as Figure 7 As shown, the command used is "iftop–n–t|grep cumulative"; Step 5: Play PPT inside the virtual machine; Step 6: Close PPT inside the virtual machine; Step 7: Close server network monitoring: Enter the SSH tool, and use ctrl+c to terminate monitoring; Step 8: Close server disk monitoring: Enter the SSH tool, and use ctrl+c to terminate monitoring; Step 9: Close server CPU monitoring: Enter the SSH tool, and use ctrl+c to terminate monitoring; Step 10: Close server memory monitoring: Enter the SSH tool, and use ctrl+c to terminate monitoring. This step has time errors for multiple nodes. The monitoring has started but the PPT has not started playing, and the PPT has been closed but the monitoring has not stopped. The error of memory monitoring is the largest. Truly valid data must start and end at the same time, such as Figure 8 As shown, in this embodiment, the steps for synchronously monitoring the resource usage status of each client and the physical server / client during the execution of the test task by each client include: Step 1: At the same time as the test starts, memory monitoring, CPU monitoring, disk monitoring, and network monitoring are turned on, and the PPT playback in the virtual machine is turned off; Step 2: At the same time as the test ends, memory monitoring, CPU monitoring, disk monitoring, and network monitoring are turned off, and the PPT playback in the virtual machine is turned off. These steps can eliminate the influence of human factors caused by human participation and errors caused by manual performance recording.

[0052] In this embodiment, after step 2) and before step 3), a batch client update step is also included: the server sends the distribution address of the new version of the test software to each client. Each client obtains the new version of the test software based on the received distribution address and completes the local test software update. This allows for rapid updates to the test software itself, reducing the workload of iterative software updates. If the test software itself is updated during the test process, testers do not need to manually replace and upgrade each cloud node one by one. They only need to place the test software at a unified address and send an update command from the server, and all clients will be automatically updated. For example, if there are 300 computing nodes in the cluster, and each computing node has installed the KylinTest version 0.5 test system. When the system needs to be updated to version 0.6, the KylinTest version 0.6 file is placed on an Http file server, and then according to the above automatic update function, the network address of KylinTest is entered into the update address of the server, and the update button is clicked to quickly update the version of the test system, and the distribution address of KylinTest version 0.6 is sent to each client. Each client obtains the KylinTest version 0.6 based on the received distribution address and completes the update and upgrade of the local KylinTest.

[0053] In summary, there is currently no system tool in China that can monitor both cloud server backend performance and cloud computing node performance, and perform cloud computing performance boundary stress testing by uniformly issuing batch stress testing tasks. The large-scale cluster performance automation testing method based on cloud services in this embodiment can monitor both cloud server backend performance and cloud computing node performance, and perform cloud computing performance boundary stress testing by uniformly issuing batch stress testing tasks. The custom scripts used in this embodiment simulate human desktop usage, such as Word, Excel, and PowerPoint presentations, image viewing, web browsing, and window dragging. Rather than simply performing I / O read / write stress testing or computational testing, they can simulate the usage of an average cloud desktop user and accurately deduce the cloud server's true capacity limit. Since each software uses a different image compression algorithm, the amount of traffic transmitted over the network by the compressed data varies. This embodiment's method can simulate the average traffic consumed by different software applications and calculate the network bandwidth consumed and differences when performing similar operations. For example, Word2007 / 2010 / 2013, Excel2007 / 2010 / 2013, PPT2007 / 2010 / 2013, wps11, etc., all use different compression algorithms and interface refresh algorithms to execute the same file. Therefore, the network bandwidth consumed when browsing the same file and performing the same operation is different. This can indicate the software that needs to be optimized for the optimization of the relevant protocol algorithms of various remote cloud desktops. The method of this embodiment can monitor the instantaneous network traffic when the same software is used. By detecting the correspondence between the lost network traffic and the time nodes of the software operation, the network traffic required for a specific operation can be obtained, such as: Excel maximization and minimization, Word inserting pictures, PPT page turning, PPT dragging pictures, etc. The network traffic consumed when such operations occur will reach an accurately measurable peak value. In protocol optimization, the image algorithm of such operations can be targetedly optimized, thereby indicating the key points that can be optimized in the protocol. The method of this embodiment can evaluate the performance of cloud services, cloud virtual machines, and the performance of certain software operations in the virtual operating system in an unattended environment. It can be used at night when no one is using the system to execute scheduled test tasks, simulate large-scale office usage scenarios, and automatically detect various performance issues. It can verify the high availability of the system before the actual cloud service system is officially launched.

[0054] Furthermore, the method of this embodiment can provide virtualization performance consumption testing capabilities. KVM, VirtManager, Sangfor Virtual Platform, Kylin KSVD Virtualization Platform, and OpenStack are all virtualization implementation frameworks that virtualize hardware devices into multiple software devices for use by virtualized systems and applications. This virtualization process consumes some performance, and the quality of performance consumption is particularly evident in large-scale virtualized cloud platforms. It's difficult to measure the performance consumed by virtualization on standard virtualization platforms. For example, on a KVM-QEMU platform, virtualization only involves a single QEMU process. The system CPU, memory, and disk IO used by this QEMU process are the sum of the internal consumption of virtual machines within virtualization systems like Windows and Linux, as well as the performance consumed by the virtualization platform's virtualization processes. This makes measuring the performance loss consumed by virtualization difficult, making it a difficult area for testing in the industry. Furthermore, in large-scale clusters, KVM, OpenStack, Sangfor, and Kylin KSVD platforms all utilize features like Numa CPU core splitting, dynamically allocating computing resources from idle virtual machines to more computationally intensive ones. This makes measuring the peak performance of virtualization cluster platforms even more difficult. To address these two points, the method of this embodiment allows all virtual machines in a cluster to simultaneously run the same application operations, perform the same computations, and occupy the same computing resources. This prevents resource misappropriation in cloud servers, as there are no idle virtual machines in the cluster. The upper limit of concurrency that the cluster can support for a certain application operation can be calculated; the performance value consumed by cluster hardware virtualization into software can be calculated based on the physical performance consumption of the physical server cluster minus the total operation performance consumption reported by the virtual machine (this value record has become a key reference for the competition among virtualization manufacturers. The lower the performance value, the better the virtualization algorithm). Figure 9 As shown: After the personal computer collects the performance of the physical server and the performance of the virtual machine, it can obtain (the total performance of the physical server A-the total performance of the virtual machine=the hardware virtualization performance consumption).

[0055] In addition, this embodiment also provides a large-scale cluster performance automation testing system based on cloud services, including an interconnected microprocessor and a memory, wherein the microprocessor is programmed or configured to execute the steps of the aforementioned large-scale cluster performance automation testing method based on cloud services.

[0056] In addition, this embodiment also provides a computer-readable storage medium, which stores a computer program programmed or configured to execute the aforementioned large-scale cluster performance automation testing method based on cloud services.

[0057] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0058] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the concept of the present invention are within the scope of protection of the present invention. It should be noted that for those skilled in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A large-scale cluster performance automation testing method based on cloud services, characterized in that: include: 1) Preparing an operating system with test software installed, and generating batches of compute nodes on a physical server based on the operating system with test software installed, including: 1.1) Preparing an operating system with test software installed; 1.2) Creating a source virtual machine on the physical server, installing the operating system with test software in the created source virtual machine, or importing a virtual machine that already has the operating system installed with test software installed; 1.3) Using the batch publishing function of the virtualization system on the physical server, creating batches of virtual machine copies based on the source virtual machine, thereby obtaining batches of compute nodes with the operating system with test software installed; 2) designating the test software of one computing node among the generated computing nodes as a server and the test software of the remaining computing nodes as clients, including: remotely logging into the designated computing node on a physical client, and triggering the test software on the designated computing node to send a broadcast message to the remaining computing nodes through a designated operation, so that the test software of the designated computing node serves as a server and the test software of the remaining computing nodes serves as clients; 3) Send the test program and the test task based on the test program to the designated client through the server, and synchronously obtain the resource occupancy status of the client and the physical server during the execution of the test task on each client; let all virtual machines in the cluster run the same application operation, the same calculation, and occupy the same computing resources at the same time, so there is no misappropriation of computing resources on the cloud server and there are no idle virtual machines in the cluster, so as to calculate the concurrency limit that the cluster can support for a certain application operation; the operations performed by the test program in step 3) include one or more of window operations, drawing program operations, image browsing operations, video playback operations, browser operations, resource browser operations, office software operations, maximum computing load stress test operations, maximum disk read and write stress test operations and maximum network bandwidth stress test operations; the computing node is a virtual machine, and also includes calculating the performance value consumed by cluster hardware virtualization into software based on the physical performance consumption of the physical server cluster minus the sum of the operation performance consumption reported internally by the virtual machine.

2. The large-scale cluster performance automation testing method based on cloud services according to claim 1 is characterized in that: After step 2) and before step 3), the process also includes a step in which each client periodically sends its own identity information, IP address, and resource occupancy status to the server. The resource occupancy status includes at least one of CPU, memory, disk I / O, and network I / O status data. The CPU status data refers to the CPU occupancy ratio, the memory status data includes the total memory and free memory size, the disk I / O status data includes the number of read bytes and written bytes, and the network I / O status data includes the number of sent bytes and received bytes.

3. The large-scale cluster performance automation testing method based on cloud services according to claim 1 is characterized in that: The test program in step 3) is one or more of an exe executable file, a python executable file, a shell executable file, a cmd executable file, and an execution script program generated by a recorded manual operation sequence.

4. The large-scale cluster performance automation testing method based on cloud services according to claim 1 is characterized in that: The attributes of the test task based on the test program in step 3) include the test task name, the selected test program, the execution mode and execution parameters of the test program. The execution mode includes immediate execution mode and timed execution mode. The execution parameters include the number of repeated executions, execution interval time and start execution time.

5. The large-scale cluster performance automation testing method based on cloud services according to claim 1 is characterized in that: In step 3), synchronously obtaining the resource occupancy status of each client and the physical server during the execution of the test task by each client includes: while sending the test task based on the test program to the designated client through the server, simultaneously obtaining the resource occupancy status on the physical server through the SSH protocol, and starting to record the resource occupancy status of the client; after receiving the notification of completion of the test task sent by any client, ending the recording of the resource occupancy status of the client, thereby obtaining the resource occupancy status of the client during the execution of the test task; after receiving the notification of completion of the test task sent by all clients, ending the recording of the resource occupancy status on the physical server, thereby obtaining the resource occupancy status of the physical server during the execution of the test task by each client; the resource occupancy status includes at least one of CPU, memory, disk I / O and network I / O status data, the CPU status data refers to the CPU occupancy ratio, the memory status data includes the total memory and space memory size, the disk I / O status data includes the number of read bytes and written bytes, and the network I / O status data includes the number of sent bytes and received bytes.

6. The large-scale cluster performance automation testing method based on cloud services according to claim 1 is characterized in that: After step 2) and before step 3), the step of batch updating clients is also included: the distribution address of the new version of the test software is sent to each client on the server side, and each client obtains the new version of the test software based on the received distribution address and completes the update and upgrade of the local test software.

7. A large-scale cluster performance automation test system based on cloud services, comprising interconnected microprocessors and memories, characterized in that: The microprocessor is programmed or configured to execute the steps of the large-scale cluster performance automation testing method based on cloud services as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program that is programmed or configured to execute the large-scale cluster performance automation testing method based on cloud services as described in any one of claims 1 to 6.

Images