Python-Redis-based concurrency performance test method
Through the Python-Redis concurrent performance testing method, the problem of inaccurate conversion of the existing interface testing framework relying on Fiddler and HAR formats is solved, and distributed stress testing and detailed performance analysis is realized, which simplifies the test process and improves accuracy and reliability. .
Patent Information
- Application Number
- CN202411440799.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-15
- Publication Date
- 2025-01-10
AI Technical Summary
The existing interface testing framework relies on the Fiddler package grabbing tool, and the conversion from HAR format to YAML format may be inaccurate, resulting in the complex process of testing script generation and data import, requiring debugging and configuration between multiple tools and formats.
Using Python-Redis concurrency performance testing method, by installing Python, Unittest and Locust, using Redis as message middleware, implementing distributed testing and coordination, writing test scripts and Locustfiles, starting the master and slave nodes, and setting concurrent parameters using the Locust Web interface. And monitor and analyze performance data.
Distributed stress testing is implemented, which avoids excessive burden on the system being tested, ensures the accuracy and reliability of the test, provides detailed performance test reports and real-time monitoring, and simplifies the test process and exception handling.
Smart Images
Figure CN119292934A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of performance testing, and specifically is a concurrent performance testing method based on Python-Redis. Background Art
[0002] The existing interface testing framework generally uses the fiddler program to capture packets and export the captured packets in HAR format, converts the HAR format data packets into readable YAML format, imports them into HttpRunner to generate test scripts, and then introduces HTMLTestRunner and mkdocs programs to define the generation and presentation of test reports, and then executes the test scripts, runs the test cases through HttpRunner, and obtains the corresponding test logs and test reports.
[0003] However, this solution first relies on the Fiddler packet capture tool. If the test environment does not install or use other packet capture tools, it is impossible to correctly capture the interface data and perform automated script conversion. In addition, the conversion process from HAR format to YAML format may have certain inaccuracies, including the conversion of request parameters, data format, etc., which may introduce some potential errors. In addition, the entire process involves conversion between multiple tools and formats, which requires certain debugging and configuration to ensure correct data import and script generation. Summary of the invention
[0004] The purpose of the present invention is to provide a Python-Redis concurrency performance testing method to solve the problems raised in the above background technology.
[0005] In order to achieve the above object, the present invention provides the following technical solution: a Python-Redis concurrency performance testing method, comprising the following steps:
[0006] Step 1: Install the environment
[0007] Install the Python environment, and install Unittest, Locust, and Redis systems in the Python environment, and make sure they are all configured and running correctly;
[0008] Step 2: Write a test script
[0009] Use Python to write interface automation test scripts and encapsulate log modules and email modules, and execute test scripts. In the scripts, you can use Python libraries to send HTTP requests, parse responses, and process data. You can use the functions of the Locust framework to define simulated user behaviors and set load models, such as the number of requests initiated per second and the number of concurrent users. The test script should include information such as the interface to be tested, parameter settings, and expected results.
[0010]
[0011]
[0012]
[0013] Step 3: Integrate Redis
[0014] Locust supports distributed testing using Redis to achieve coordination of concurrent users and task scheduling. Connect to Redis and configure Locust in the test script to use Redis as the message middleware. In this way, multiple Locust nodes can connect to the same Redis instance and communicate and coordinate through Redis.
[0015] Step 4: Write Locustfile
[0016] In the same directory as the test script, create a file named "Locustfile.py". This file contains the class used to define the Locust test. In the class, use the decorators and methods provided by Locust to define user behaviors, tasks, and performance parameters;
[0017] Step 5: Start the Locust master node
[0018] Start the Locust master node using the following command in the command line:
[0019] locust-f Locustfile.py--master
[0020] This will start the Locust master node and connect to the Redis instance. The master node is responsible for task distribution and centralized monitoring. It will send tasks to other Locust slave nodes;
[0021] Step 6: Start the Locust slave node
[0022] Start the Locust slave node on other machines using the following command, specifying the IP address and port number of the master node:
[0023] Locust -f Locustfile.py --slave --master-host=<master node IP address> --master-port=<master node port number>
[0024] This will start the Locust slave node and connect to the master node. The slave node receives commands for task distribution and execution. Multiple Locust slave nodes can be started on multiple machines to implement distributed stress testing.
[0025] Step 7: Access the Locust Web Interface
[0026] Enter the IP address and port number of the master node in the web browser to access the Locust web interface: http: / / 127.0.0.1:8089. On the web interface, you can set test parameters such as the number of concurrent users and the number of new users per second, and start the test.
[0027] Step 8: Monitor and collect results
[0028] In the Locust web interface, you can monitor performance data in real time, such as request response time, error rate, throughput, etc. After the test, you can export a detailed performance test result report, including the response time distribution of each request, error log, etc.
[0029] Step 9: Analyze and optimize
[0030] Analyze and optimize performance issues based on the collected performance test data and reports. Determine which interfaces have performance bottlenecks or errors based on performance indicators and expected results. Use related functions and libraries of Redis, Locust, and Python to further troubleshoot and optimize performance issues.
[0031] Preferably, the Python described in step 1 has a rich standard library, and the Python can be used to write programs suitable for different systems.
[0032] Preferably, the Unittest described in step 1 can perform automated testing, and the Unittest can be used to capture and fix errors in the code.
[0033] Preferably, the test script described in step 2 can be run in different system environments, and the test script can be modified and adjusted at any time.
[0034] Preferably, the Locust described in step 1 can perform stress testing on different types of systems, and the test intensity of the Locust can be adjusted at any time.
[0035] Preferably, the Redis system described in step one can store or quickly read the test data, and the Redis system can integrate the test data as needed and generate a test report.
[0036] Preferably, the test type of the distributed test described in step three can be adjusted, and the test methods of the distributed test are divided into static test, dynamic test and mixed test.
[0037] Preferably, the message middleware described in step three can perform message transmission, and the message middleware can communicate between different platforms.
[0038] Preferably, the performance test data and reports described in step nine are provided with a periodic deletion function, and the period of periodic deletion of the performance test data and reports is six months.
[0039] The beneficial effects of the present invention are as follows:
[0040] Use Python libraries to send HTTP requests, parse responses, and process data to write test scripts. Locust provides convenient APIs and tools for simulating a large number of users accessing the system concurrently, generating and distributing test requests, and counting and reporting performance indicators. Redis is a high-performance open source in-memory database that can be used as Locust's message middleware for distributed testing and coordination. Through the configuration and use of Redis and the method of integrating with Locust, distributed stress testing is achieved by running Locust slave nodes on multiple nodes, and reasonable management of system resource consumption is achieved by configuring and managing the communication and coordination between Locust master and slave nodes, avoiding excessive burden on the tested system or test environment, and ensuring that the test does not have an adverse impact on other systems or networks. In addition, sufficient exception handling and error logging are performed throughout the test process, which can ensure that abnormal situations are captured and handled in a timely manner during the test, and record relevant information for subsequent analysis and troubleshooting. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 It is a distributed test execution flow chart of the present invention;
[0042] Figure 2 This is a schematic diagram of the distribution of Locust nodes of the present invention. DETAILED DESCRIPTION
[0043] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0044] like Figure 1 to Figure 2 As shown, an embodiment of the present invention provides a Python-Redis concurrency performance testing method, comprising the following steps:
[0045] Step 1: Install the environment
[0046] Install the Python environment, and install Unittest, Locust, and Redis systems in the Python environment, and make sure they are all configured and running correctly;
[0047] Step 2: Write a test script
[0048] Use Python to write interface automation test scripts and encapsulate log modules and email modules, and execute test scripts. In the scripts, you can use Python libraries to send HTTP requests, parse responses, and process data. You can use the functions of the Locust framework to define simulated user behaviors and set load models, such as the number of requests initiated per second and the number of concurrent users. The test script should include information such as the interface to be tested, parameter settings, and expected results.
[0049]
[0050]
[0051]
[0052] Step 3: Integrate Redis
[0053] Locust supports distributed testing using Redis to achieve coordination of concurrent users and task scheduling. Connect to Redis and configure Locust in the test script to use Redis as the message middleware. In this way, multiple Locust nodes can connect to the same Redis instance and communicate and coordinate through Redis.
[0054] Step 4: Write Locustfile
[0055] In the same directory as the test script, create a file named "Locustfile.py". This file contains the class used to define the Locust test. In the class, use the decorators and methods provided by Locust to define user behaviors, tasks, and performance parameters;
[0056] Step 5: Start the Locust master node
[0057] Start the Locust master node using the following command in the command line:
[0058] locust-f Locustfile.py--master
[0059] This will start the Locust master node and connect to the Redis instance. The master node is responsible for task distribution and centralized monitoring. It will send tasks to other Locust slave nodes;
[0060] Step 6: Start the Locust slave node
[0061] Start the Locust slave node on other machines using the following command, specifying the IP address and port number of the master node:
[0062] Locust -f Locustfile.py --slave --master-host=<master node IP address> --master-port=<master node port number>
[0063] This will start the Locust slave node and connect to the master node. The slave node receives commands for task distribution and execution. Multiple Locust slave nodes can be started on multiple machines to implement distributed stress testing.
[0064] Step 7: Access the Locust Web Interface
[0065] Enter the IP address and port number of the master node in the web browser to access the Locust web interface: http: / / 127.0.0.1:8089. On the web interface, you can set test parameters such as the number of concurrent users and the number of new users per second, and start the test.
[0066] Step 8: Monitor and collect results
[0067] In the Locust web interface, you can monitor performance data in real time, such as request response time, error rate, throughput, etc. After the test, you can export a detailed performance test result report, including the response time distribution of each request, error log, etc.
[0068] Step 9: Analyze and optimize
[0069] Analyze and optimize performance issues based on the collected performance test data and reports. Determine which interfaces have performance bottlenecks or errors based on performance indicators and expected results. Use related functions and libraries of Redis, Locust, and Python to further troubleshoot and optimize performance issues.
[0070] Using Redis as the message middleware, distributed performance testing can be implemented. Multiple Locust nodes can be connected to Redis, and Redis can be used to schedule tasks and summarize results. This makes it possible to simulate large-scale concurrent requests, perform load testing in a distributed environment, and collect and analyze results. Locust is a load testing tool that uses Python for test script writing. It can define user behavior based on specific business needs and scenarios, and can customize and edit user behavior. This makes it possible to simulate real user behavior and traffic patterns, perform performance testing and load stress testing more accurately, and support rapid development and execution of performance testing. Combining Redis and Locust, an efficient performance testing solution can be implemented, and performance analysis and optimization can be easily performed. Because it provides an easy-to-understand and visual Web interface, users can view and analyze performance indicators such as request response time, error rate, and throughput in real time. In addition, Locust can also generate detailed test reports, including the response time distribution of each request, error logs, etc., which enables users to discover performance problems in a timely manner and perform performance optimization and adjustment.
[0071] Among them, Python in step one has a rich standard library, and Python can write programs suitable for different systems.
[0072] The Python standard library is very large. With the help of various standard libraries, users can easily handle various tasks, including unit testing, threads, databases, and other system-related operations.
[0073] Among them, Unittest in step one can perform automated testing. Unittest can be used to capture and fix errors in the code.
[0074] By writing unit tests through Unittest, you can verify whether each part of the program can run correctly under different circumstances.
[0075] Among them, the test script of step 2 can be run in different system environments, and the test script can be modified and adjusted at any time.
[0076] The modifiable design of the test script allows users to adjust the test content at any time.
[0077] Among them, Locust in step one can perform stress testing on different types of systems, and the test intensity of Locust can be adjusted at any time.
[0078] Locust can simulate large-scale concurrent users and distribute stress load to the system or cluster to test the load capacity of websites and systems.
[0079] Among them, the Redis system in step one can store or quickly read the test data, and the Redis system can integrate the test data as needed and produce a test report.
[0080] Redis supports multiple data structures, including strings, hash tables, lists, sets, and sorted sets. Each data structure has corresponding commands and operation methods. Redis's data structure is not limited to simple key-value pair storage. It can also achieve complex data processing and query by operating and combining data structures.
[0081] Among them, the test type of the distributed test in step three can be adjusted, and the scheduling methods of the distributed test are divided into static scheduling, dynamic scheduling and hybrid scheduling.
[0082] Distributed testing can simulate various actual work scenarios through methods such as stress testing, load testing, fault tolerance testing, and recovery testing to discover performance bottlenecks, errors, and security risks of the system under various conditions.
[0083] Among them, the message middleware in step three can perform message transmission, and the message middleware can communicate between different platforms.
[0084] Message middleware can shield the characteristics between various platforms and protocols, realize rapid collaboration between applications, and can transmit or store and forward messages at any time.
[0085] Among them, the performance test data and reports in step nine are set with a periodic deletion function, and the period of periodic deletion of the performance test data and reports is six months.
[0086] The design of periodic deletion ensures that old performance test data and reports do not occupy excessive server storage space, thereby improving the utilization rate of server resources.
[0087] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or apparatus.
[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A concurrent performance testing method based on Python-Redis, characterized in that: The following steps are involved: Step 1: Install the environment Install the Python environment, and install Unittest, Locust, and Redis systems in the Python environment, and make sure they are all configured and running correctly; Step 2: Write a test script Use Python to write interface automation test scripts and encapsulate log modules and email modules, and execute test scripts. In the scripts, you can use Python libraries to send HTTP requests, parse responses, and process data. You can use the functions of the Locust framework to define simulated user behaviors and set load models, such as the number of requests initiated per second and the number of concurrent users. The test script should include information such as the interface to be tested, parameter settings, and expected results. #Import required libraries from locus t import HttpUser,between,task import random import json import redis #Create a Redis connection redis_client=redis.Redis(host='127.0.0.1',port=6379) #Custom user class inherits from HttpUser class UserBehavior(HttpUser): wait_time=between(1,3) @task def perform_login(self): #Get the login credentials required for interface automated testing from Redis token=redis_client.get('login_token').decode('utf-8')#Send login request headers={'Authorization':f'Bearer{token}'} response=self.client.post(' / login', headers=headers) #Parse the response if response.status_code==200: json_response=json.loads(response.text) user_id=json_response['user id'] #Store the user ID in Redis for later use redis_client.set('user_id',user_id) else: print('Login failed') @task def perform_action(self): #Get the user ID from Redis user_id=redis_client.get('user_id').decode('utf-8')#Construct request data data = { 'user id':user_id 'action':'perform_action', #Other parameters.... } #Send request response=self.client.post(' / action',json=data) #Processing the response if response.status_code==200: print('Action performed successfully') else: print('Action failed') #Run performance test class MyUser(HttpUser): tasks = [UserBehavior] wait_time=between(1,2) Step 3: Integrate Redis Locust supports distributed testing using Redis to achieve coordination of concurrent users and task scheduling. Connect to Redis and configure Locust in the test script to use Redis as the message middleware. In this way, multiple Locust nodes can connect to the same Redis instance and communicate and coordinate through Redis. Step 4: Write Locustfile In the same directory as the test script, create a file named "Locustfile.py". This file contains the class used to define the Locust test. In the class, use the decorators and methods provided by Locust to define user behaviors, tasks, and performance parameters; Step 5: Start the Locust master node Start the Locust master node using the following command in the command line: locust-f Locustfile.py--master This will start the Locust master node and connect to the Redis instance. The master node is responsible for task distribution and centralized monitoring. It will send tasks to other Locust slave nodes; Step 6: Start the Locust slave node Start the Locust slave node on other machines using the following command, specifying the IP address and port number of the master node: Locust -f Locustfile.py --slave --master-host=<master node IP address> --master-port=<master node port number> This will start the Locust slave node and connect to the master node. The slave node receives commands for task distribution and execution. Multiple Locust slave nodes can be started on multiple machines to implement distributed stress testing. Step 7: Access the Locust Web Interface Enter the IP address and port number of the master node in the web browser to access the Locust web interface: http: / / 127.0.0.1:8089. On the web interface, you can set test parameters such as the number of concurrent users and the number of new users per second, and start the test. Step 8: Monitor and collect results In the Locust web interface, you can monitor performance data in real time, such as request response time, error rate, throughput, etc. After the test, you can export a detailed performance test result report, including the response time distribution of each request, error log, etc. Step 9: Analyze and optimize Analyze and optimize performance issues based on the collected performance test data and reports. Determine which interfaces have performance bottlenecks or errors based on performance indicators and expected results. Use related functions and libraries of Redis, Locust, and Python to further troubleshoot and optimize performance issues.
2. A Python-Redis concurrency performance testing method according to claim 1, characterized in that: The Python described in step 1 has a rich standard library, and the Python can be used to write programs suitable for different systems.
3. A Python-Redis concurrency performance testing method according to claim 1, characterized in that: The Unittest described in step 1 can perform automated testing, and the Unittest can be used to capture and fix errors in the code.
4. A Python-Redis concurrency performance testing method according to claim 1, characterized in that: The test script described in step 2 can be run in different system environments, and the test script can be modified and adjusted at any time.
5. A Python-Redis based concurrent performance testing method according to claim 1, characterized in that: The Locust described in step 1 can perform stress testing on different types of systems, and the test intensity of the Locust can be adjusted at any time.
6. A Python-Redis based concurrent performance testing method according to claim 1, characterized in that: The Redis system described in step 1 can store or quickly read the test data, and the Redis system can integrate the test data as needed and generate a test report.
7. A Python-Redis based concurrent performance testing method according to claim 1, characterized in that: The test type of the distributed test described in step three can be adjusted, and the scheduling methods of the distributed test are divided into static scheduling, dynamic scheduling and hybrid scheduling.
8. A Python-Redis based concurrent performance testing method according to claim 1, characterized in that: The message middleware described in step three can perform message transmission, and the message middleware can communicate between different platforms.
9. A Python-Redis based concurrent performance testing method according to claim 1, characterized in that: The performance test data and reports described in step nine are provided with a periodic deletion function, and the periodic deletion time of the performance test data and reports is six months.