Database pressure testing method and device, electronic equipment and storage medium
By automatically generating database instances and test configuration files, the inefficiency of stress testing caused by manual deployment in the prior art is solved, and efficient and accurate database stress testing is achieved.
Patent Information
- Application Number
- CN202510591119.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2025-08-08
AI Technical Summary
The existing database stress testing method requires users to manually deploy database instances, which leads to cumbersome and time-consuming deployment process and low stress testing efficiency.
By obtaining the database name and type of the test database, extracting connection and tool parameters from the test configuration file, automatically generate simulated business data and query statements, automatically connect to the database and perform stress testing, avoiding manual deployment.
Improves the efficiency and accuracy of database stress testing, reduces deployment complexity, and achieves compatibility and flexibility for multiple database types.
Smart Images
Figure CN120448233A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology and is applied to the fields of financial technology and digital medicine, and in particular to a database stress testing method, device, electronic device and storage medium. Background Art
[0002] Each user may initiate multiple database requests simultaneously. To ensure that the database can respond to all user requests in a timely manner, the database needs to withstand high concurrent access pressure. For example, in fintech scenarios, the database needs to handle various types of concurrent transactions in real time, such as stock trading, bank transfers, and online payments. Another example is digital healthcare, where the database needs to process large amounts of medical data in real time, such as electronic medical records and health test results. To ensure that the database can operate stably under high load conditions, it is necessary to stress test the database. Existing stress testing methods require users to manually deploy database instances, a cumbersome and time-consuming deployment process that results in low stress testing efficiency. Summary of the Invention
[0003] The main purpose of the embodiments of the present application is to provide a database stress testing method, device, electronic device and storage medium, aiming to improve the efficiency of database stress testing.
[0004] To achieve the above objectives, a first aspect of an embodiment of the present application provides a database stress testing method, the method comprising:
[0005] Get the database name and database type of the test database;
[0006] Extracting database connection parameters from a pre-acquired test configuration file according to the database name, and extracting test tool parameters from the test configuration file according to the database type;
[0007] Calling the stress testing tool according to the test tool parameters to generate original simulated business data and test query statements;
[0008] Connecting to the test database according to the database connection parameters;
[0009] storing the original simulated business data in the test partition of the connected test database to obtain reference simulated business data;
[0010] A stress test is performed on the connected test database according to the test query statement and the reference simulated business data of the test partition.
[0011] In some embodiments, performing a stress test on the connected test database according to the test query statement and the reference simulated business data of the test partition includes:
[0012] Extracting thread concurrency and test duration from the test configuration file;
[0013] Start the target thread according to the thread concurrency;
[0014] Establishing a connection between the target thread and the test database according to the database connection parameters;
[0015] A stress test is performed on the connected test database according to the test query statement, the reference simulated business data of the test partition and the test duration.
[0016] In some embodiments, performing a stress test on the connected test database according to the test query statement, the reference simulated business data of the test partition, and the test duration includes:
[0017] Performing a database query operation on the reference simulated business data according to the test query statement and the test duration, and extracting a tool status parameter from the test configuration file;
[0018] calling a performance analysis tool according to the tool status parameter;
[0019] The performance analysis tool is used to perform performance testing on the test database.
[0020] In some embodiments, performing a database query operation on the reference simulated business data according to the test query statement and the test duration includes:
[0021] Classifying the reference simulated business data to obtain test hot data and test cold data;
[0022] Calculating a hot data ratio between the test hot data and the test cold data;
[0023] Classifying the test query statements according to the hot and cold data ratio to obtain hot query statements and cold query statements;
[0024] The database query operation is performed on the test hot data using the hot query statement and the test duration, and the database query operation is performed on the test cold data using the cold query statement and the test duration.
[0025] In some embodiments, classifying the reference simulated business data to obtain test hot data and test cold data includes:
[0026] Calculating the data access frequency of the reference simulated business data according to a preset data distribution model;
[0027] Calculating the mean of the data access frequencies to obtain a target access frequency;
[0028] The reference simulated business data is classified according to the target access frequency to obtain the test hot data and the test cold data; wherein the target access frequency of the test hot data is greater than the target access frequency of the test cold data.
[0029] In some embodiments, storing the original simulated business data in a test partition of the connected test database to obtain reference simulated business data includes:
[0030] Storing the original simulated service data in a temporary buffer to obtain cached simulated service data;
[0031] Starting an asynchronous thread, and reading a preset batch of cached simulated business data from the temporary buffer through the asynchronous thread to obtain selected simulated business data;
[0032] The selected simulated business data is stored in the test partition through the asynchronous thread to obtain the reference simulated business data.
[0033] In some embodiments, calling the stress testing tool according to the testing tool parameters to generate original simulated business data and test query statements includes:
[0034] Obtaining a database table structure and query log from a production database; wherein the production database stores real business data, and the database table structure is the table structure of the real business data;
[0035] Calling a stress testing tool according to the testing tool parameters;
[0036] Filling the database table structure with the stress testing tool to obtain the original simulated business data;
[0037] A production query statement is extracted from the query log, and statement processing is performed on the production query statement through the stress testing tool to obtain the test query statement.
[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a database stress testing device, the database stress testing device comprising:
[0039] The acquisition module is used to obtain the database name and database type of the test database;
[0040] A parameter extraction module is used to extract database connection parameters from a pre-acquired test configuration file according to the database name, and to extract test tool parameters from the test configuration file according to the database type;
[0041] A generation module is used to call the stress test tool according to the test tool parameters to generate original simulated business data and test query statements;
[0042] A connection module, configured to connect to the test database according to the database connection parameters;
[0043] A storage module, configured to store the original simulated service data in a test partition of the connected test database to obtain reference simulated service data;
[0044] A testing module is used to perform a stress test on the connected test database according to the test query statement and the reference simulated business data of the test partition.
[0045] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.
[0046] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.
[0047] The database stress testing method, database stress testing device, electronic device and computer-readable storage medium proposed in the embodiments of the present application obtain the database name and database type of the test database to locate the test database according to the database name and database type. In order to realize automated database stress testing, a test configuration file is introduced to manage the parameters required for stress testing through the test configuration file and dynamically adjust the parameters, thereby reducing the complexity of database instance deployment. According to the database name, the database connection parameters are extracted from the pre-acquired test configuration file to establish a connection with the test database according to the database connection parameters, and the connected test database is stress tested. According to the database type, the test tool parameters are extracted from the test configuration file, and the stress testing tool is called according to the test tool parameters to generate original simulated business data and test query statements to automatically generate a database instance instead of manually deploying the database instance. According to the database connection parameters, the test database is connected, and the original simulated business data is stored in the test partition of the connected test database to obtain reference simulated business data, so as to stress test the test database based on the reference simulated business data instead of the real business data, thereby ensuring the flexibility, security and repeatability of the stress test. The connected test database is stress tested based on the automatically generated test query statements and the reference simulated business data of the test partition, thereby avoiding the problem of low test efficiency caused by manual deployment of the database embodiment and improving the efficiency of the database stress test. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 This is a flowchart of the database stress testing method provided by an embodiment of the present application;
[0049] Figure 2 yes Figure 1 Flowchart of step S130 in FIG.
[0050] Figure 3 yes Figure 1 Flowchart of step S150 in FIG.
[0051] Figure 4 yes Figure 1 Flowchart of step S160 in FIG.
[0052] Figure 5 yes Figure 4 Flowchart of step S440 in FIG.
[0053] Figure 6 yes Figure 5 Flowchart of step S510 in FIG.
[0054] Figure 7 yes Figure 6 Flowchart of step S610 in FIG.
[0055] Figure 8 Schematic diagram of the structure of the database stress testing device provided in the embodiment of the present application;
[0056] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0057] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.
[0058] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.
[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.
[0060] Each user may initiate multiple database requests simultaneously. To ensure that the database can respond to all user requests in a timely manner, the database needs to withstand high concurrent access pressure. For example, in fintech scenarios, the database needs to handle various types of concurrent transactions in real time, such as stock trading, bank transfers, and online payments. Another example is digital healthcare, where the database needs to process large amounts of medical data in real time, such as electronic medical records and health test results. To ensure that the database can operate stably under high load conditions, it is necessary to stress test the database. Existing stress testing methods require users to manually deploy database instances, a cumbersome and time-consuming deployment process that results in low stress testing efficiency.
[0061] Based on this, embodiments of the present application provide a database stress testing method, a database stress testing device, an electronic device, and a computer-readable storage medium, aiming to improve the efficiency of database stress testing.
[0062] The database stress testing method, database stress testing device, electronic device, and computer-readable storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the database stress testing method in the embodiments of the present application is described.
[0063] The database stress testing method provided in the embodiment of the present application relates to the field of computer technology. The database stress testing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the database stress testing method, etc., but is not limited to the above forms.
[0064] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.
[0065] Figure 1 This is an optional flowchart of the database stress testing method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S160.
[0066] Step S110, obtaining the database name and database type of the test database;
[0067] Step S120, extracting database connection parameters from the pre-acquired test configuration file according to the database name, and extracting test tool parameters from the test configuration file according to the database type;
[0068] Step S130: calling the stress test tool according to the test tool parameters to generate original simulated business data and test query statements;
[0069] Step S140, connecting to the test database according to the database connection parameters;
[0070] Step S150, storing the original simulated business data into the test partition of the connected test database to obtain reference simulated business data;
[0071] Step S160 , performing a stress test on the connected test database according to the test query statement and the reference simulated business data of the test partition.
[0072] In steps S110 to S160, as shown in the embodiment of the present application, parameters related to database instance generation are managed through a test configuration file, so that a database instance is automatically generated based on the parameters, thereby reducing the complexity and time required to deploy the database instance. When using the automatically generated database instance for database stress testing, the efficiency of the database stress test is greatly improved.
[0073] In step S110 of some embodiments, stress testing involves testing the stability and reliability of a database under high stress. To ensure the proper execution of the stress test, the database name and database type of the test database are obtained. Different database stress tests can be performed based on different database names and types, thus accommodating a wide range of database tests, rather than just testing a single database. The test database is the database to be stress-tested. A database is a system used to store, manage, and retrieve business data. Taking the FinTech scenario as an example, the test database stores business data such as financial transaction data, user data, and financial product data, enabling real-time processing of user transaction requests such as deposits, withdrawals, and transfers based on this business data. Financial transaction data includes real-time transaction information for financial products such as stocks and bonds. User data includes basic user information such as name and contact information, account information such as bank accounts and fund accounts, and credit records. Financial product data includes basic product information such as product name and type, as well as product return data such as historical returns, expected returns, and risk level. Taking the digital healthcare scenario as an example, the test database stores basic patient information, diagnosis and treatment data, medical imaging data, and electronic medical records to manage patient data and provide healthcare services. The database name is a unique name used to identify the test database. Database types include relational databases and non-relational databases. Relational databases include MySQL and PostgreSQL, while non-relational databases include MongoDB and Redis.
[0074] In step S120 of some embodiments, a test configuration file is introduced to support the stress testing requirements of multiple databases. The test configuration file records the test configuration parameters required for stress testing of different databases, and the test tool or test framework can be guided to perform stress testing according to the test configuration file. Specifically, the database connection parameters of the test database are extracted from the test configuration file according to the database name. The database connection parameters are used to establish a connection between the test tool or test framework and the test database. The database connection parameters include the network address of the database, port number, database name, database type, etc. The test tool parameters are extracted from the test configuration file according to the database type. The test tool parameters are used to indicate the type of stress testing tool, such as stress testing tool A and stress testing tool B.
[0075] A testing tool or testing framework integrates various stress testing tools. These tools can be invoked based on test tool parameters. The stress testing tools generate database instances, which include raw simulated business data and test queries. The raw simulated business data is generated to simulate a real-world business environment during database stress testing scenarios. Test queries are generated using the standard programming language of the test database. By executing these test queries against the raw simulated business data, operations such as create, query, update, and delete can be performed on the data. If the test database is a MySQL database, the standard programming language is Structured Query Language (SQL), and the test queries are SQL statements. For example, in a financial business scenario, when inserting new records into a database, such as creating a new customer account or applying for a loan, a query containing a create operation can be used. To query existing records from the database, such as checking account balances and transaction history, a query containing a query operation can be used. To modify existing records in the database, such as updating account balances, contact information, or loan application status, a query containing an update operation can be used. To delete existing records from the database, such as canceling a loan application or closing an account, use a query statement containing the delete operation.
[0076] See also Figure 2 In some embodiments, step S130 may include but is not limited to steps S210 to S240:
[0077] Step S210: Obtain a database table structure and query log from a production database; wherein the production database stores real business data, and the database table structure is the table structure of the real business data;
[0078] Step S220, calling the stress testing tool according to the testing tool parameters;
[0079] Step S230: Fill the database table structure using a stress testing tool to obtain original simulated business data;
[0080] Step S240 , extracting production query statements from the query log, and performing statement processing on the production query statements through a stress testing tool to obtain test query statements.
[0081] In step S210 of some embodiments, the production database is a database used in an actual business operation environment, and the production database is used to store and manage real business data, such as transaction records, user information, etc. In order to improve the accuracy of database stress testing, it is necessary to generate simulated business data that is highly similar to the real business data, so that the test environment is close to the production environment. Specifically, the database table structure is obtained from the production database through a database management tool or database instructions. The database table structure is the table structure of the real business data. The database table structure defines the organization of the tables in the production database, including table names, column names, data types, constraints, etc. The constraints include primary keys, foreign keys, uniqueness, non-empty, etc. The production database has a query log function, and the query log is obtained by calling the interface of the query log function. The query log records all query statements executed by the production database.
[0082] In step S220 of some embodiments, a stress test tool is invoked based on the test tool parameters to perform a stress test. For example, if the test tool parameter obtained from a MySQL database is A, then stress test tool A is invoked to perform a stress test; if the test tool parameter obtained from a PostgreSQL database is B, then stress test tool B is invoked to perform a stress test. Using the test configuration file, stress tests can be performed on test databases of various database types, fully realizing the scalability and compatibility of stress testing, thereby meeting the needs of incremental stress testing.
[0083] In step S230 of some embodiments, the database table structure provides a template for generating business data, parameterizes each element in the database table structure, converts each element into an operational parameter, generates a random field through a stress testing tool, and fills the operational parameters in the database table structure according to the random field to obtain the original simulated business data.
[0084] In step S240 of some embodiments, a query statement is extracted from the query log using a text processing tool such as grep or awk to obtain a production query statement. The variable portion of the production query statement is parameterized, and the parameterized production query statement is saved as a template. The template is then filled with placeholders using a stress testing tool to obtain a test query statement.
[0085] Through the above steps S210 to S240, a database instance that is highly similar to real data can be automatically generated, avoiding the problems of high deployment complexity and long deployment time caused by manual deployment of database instances, and performing stress testing based on database instances that are highly similar to real data, thereby improving the accuracy of database stress testing.
[0086] In step S140 of some embodiments, a connection between a test tool or a test framework and a test database is established through database connection parameters, so that a stress test is performed on the connected test database through the test tool or the test framework.
[0087] To prevent the stress testing process from impacting the actual operational business of the test database, a test partition is created in the test database. The original simulated business data is stored in the test partition of the connected test database to generate reference simulated business data. This ensures that the stress testing process only queries the original simulated business data in the test partition, ensuring the security of the actual business data. The test partition is additional storage space independent of the actual business storage space.
[0088] See also Figure 3 In some embodiments, step S150 may include but is not limited to steps S310 to S330:
[0089] Step S310, storing the original simulated service data in a temporary buffer to obtain cached simulated service data;
[0090] Step S320: start an asynchronous thread, and read a preset batch of cached simulated business data from a temporary buffer through the asynchronous thread to obtain selected simulated business data;
[0091] Step S330: Store the selected simulated business data into the test partition through the asynchronous thread to obtain reference simulated business data.
[0092] In some embodiments, if each database instance is immediately written to the test partition in step S310, a large number of database write operations will be generated, increasing the burden on the test database. To improve the efficiency of stress testing, the original simulated business data is stored in a temporary buffer. Multiple database instances are accumulated in the temporary buffer to obtain cached simulated business data, thereby reducing the number of database writes. At the same time, when an error occurs during the database write process, the database instance can be temporarily stored in the temporary buffer and retried after the database is restored, avoiding data loss and ensuring that the stress test can be executed normally.
[0093] In step S320 of some embodiments, the database instance generation process and the database writing process involve different threads. If the database instance generation thread directly performs the database writing operation, the thread will be blocked, affecting the efficiency of database writing. The data writing operation can be asynchronous through a temporary buffer. The database instance generation thread only needs to put the database instance into the buffer without waiting for the database write operation to be completed, thereby improving the efficiency of writing the original simulated business data into the test partition. Specifically, an asynchronous thread is started to enable the database instance generation thread to execute the simulated business data generation task, and the asynchronous threads execute the database writing tasks respectively. The overall efficiency of the stress test is improved through the parallel processing of the database instance generation thread and the asynchronous thread. The cached simulated business data can be read in batches from the temporary buffer through the asynchronous thread, and the preset batch of cached simulated business data can be used as the selected simulated business data.
[0094] In step S330 of some embodiments, the selected simulated business data is written into the test partition through an asynchronous thread to obtain reference simulated business data.
[0095] Through the above steps S310 to S330, the speed of writing simulated business data into the test partition can be increased, thereby improving the efficiency of the stress test.
[0096] Execute test query statements against the reference simulated business data in the test partition of the connected test database to perform stress testing on the test database. The specific process of stress testing is described in detail below.
[0097] See also Figure 4 In some embodiments, step S160 may include but is not limited to steps S410 to S440:
[0098] Step S410, extracting thread concurrency and test duration from the test configuration file;
[0099] Step S420, starting the target thread according to the thread concurrency;
[0100] Step S430, establishing a connection between the target thread and the test database according to the database connection parameters;
[0101] Step S440 , performing a stress test on the connected test database according to the test query statement, the reference simulated business data of the test partition, and the test duration.
[0102] In step S410 of some embodiments, in order to detect the system performance of the database in a high-concurrency scenario, a high-concurrency thread is used to simulate a situation where a large number of users access the database at the same time, such as simulating a stock trading scenario where a large number of users perform stock buying and selling operations at the same time. Specifically, the thread concurrency and test duration are extracted from the test configuration file, the thread concurrency is used to indicate the number of concurrencies of the threads, and the test duration is used to indicate the test duration for performing the stress test under the thread concurrency. Multiple thread concurrencies can be set in the test configuration file, and the stress test tasks under each thread concurrency are executed in turn to test the system performance of the database under different concurrencies. For example, by setting the thread concurrency to 1 and 4, the performance impact of 1 thread and 4 threads on the test database can be tested respectively.
[0103] In step S420 of some embodiments, a target thread is started based on the thread concurrency. The target thread includes at least one sub-thread, and the thread concurrency is the number of sub-threads. If the thread concurrency is 4, four sub-threads are started. If the thread concurrency is 1 or 4, the stress test task for the four sub-threads is executed only after the stress test task for the one sub-thread is completed.
[0104] In step S430 of some embodiments, a connection is established between each sub-thread and the test database according to database connection parameters, and a stress test is performed on the test database through multiple sub-threads to obtain the system performance of the test database when facing multiple sub-threads.
[0105] In step S440 of some embodiments, each sub-thread executes a test query statement against the reference simulated business data of the test partition, and performs a stress test on the connected test database for a preset duration, where the preset duration is the test duration, thereby completing the stress test task at the thread concurrency. If there are multiple thread concurrencies, the stress test task at the next thread concurrency is executed with reference to steps S410 to S440 until the execution of each thread concurrency is completed, thereby obtaining the system performance of the test database at each thread concurrency.
[0106] The above steps S410 to S440 simulate a high-concurrency scenario through thread concurrency to test the system performance of the database in a high-concurrency scenario, implement a fully automated testing process, and greatly improve the efficiency of stress testing.
[0107] See also Figure 5 In some embodiments, step S440 may include but is not limited to steps S510 to S530:
[0108] Step S510 , performing a database query operation on the reference simulated business data according to the test query statement and the test duration, and extracting tool status parameters from the test configuration file;
[0109] Step S520, calling the performance analysis tool according to the tool status parameter;
[0110] Step S530: Perform performance testing on the test database using a performance analysis tool.
[0111] In step S510 of some embodiments, a database query operation is performed on the reference simulated business data according to the test query statement. The database query operation can be at least one of creation, deletion, update and query operations, until the execution time of the database query operation is greater than or equal to the test time, and the number of queries per second (Query Per Seconds, QPS) and the number of transactions per second (Transaction Per Seconds, TPS) of the test database are obtained. In addition to obtaining the request processing status of the test database for high-concurrency requests, it is also necessary to care about the hardware server performance indicators of the test database. Therefore, while executing the database query operation, the step of extracting the tool status parameter from the test configuration file is concurrently executed. The tool status parameter is used to indicate whether the performance analysis tool is in the on state or the off state. While executing the database query operation, the system performance information is synchronously collected, which will have an impact on the database query process. Therefore, the tool status parameter is used to determine whether to turn on or off the collection of system performance information to meet the collection requirements of multiple performance indicators.
[0112] In step S520 of some embodiments, the performance analysis tool includes a first performance analysis tool and a second performance analysis tool, and the tool state parameter includes a first state parameter of the first performance analysis tool and a second state parameter of the second performance analysis tool, the first state parameter is used to indicate whether the first performance analysis tool is in an on state or an off state, and the second state parameter is used to indicate whether the second performance analysis tool is in an on state or an off state. The first performance analysis tool is used to test the impact of the target thread on the hardware server performance of the test database, and can be selected according to actual conditions. The second performance analysis tool is used to indicate the hardware service resources of the test database, and can be selected according to actual conditions. It should be noted that the second performance analysis tool can generate a flame graph that is easy to intuitively analyze, so as to perform performance analysis on the test database based on the flame graph and obtain the pressure bottleneck that the test database can cope with.
[0113] If the first status parameter indicates that the first performance analysis tool is in the enabled state, the first performance analysis tool is called to monitor the system performance indicator; otherwise, the first performance analysis tool is not called to collect system performance information. If the second status parameter indicates that the second performance analysis tool is in the enabled state, the second performance analysis tool is called to monitor the system performance indicator; otherwise, the second performance analysis tool is not called to collect system performance information.
[0114] In step S530 of some embodiments, the thread ID of the target thread is input into a first performance analysis tool for performance testing to obtain the target thread's usage of hardware server resources of the test database, such as the target thread's processor usage, memory usage, I / O usage, and network usage of the test database. A second performance analysis tool is used to perform performance testing on the test database to obtain the test database's hardware server resource usage, such as the test database's processor usage, memory usage, I / O usage, and network usage.
[0115] It's important to note that after the stress test is complete, the test results from the entire process need to be analyzed and summarized, extracting key performance indicators from each test process to create a test report. The detailed test results should also be compressed and saved. Intermediate data and instances generated during the test should be cleared to ensure that no residual data will affect the initialization of the test instance during the next test.
[0116] Through the above steps S510 to S530, various system performance indicators of the test database can be obtained during the stress test, so as to perform a comprehensive test on the test database and improve the accuracy of the stress test.
[0117] See also Figure 6 In some embodiments, step S510 may include but is not limited to steps S610 to S640:
[0118] Step S610: classify the reference simulated business data to obtain test hot data and test cold data;
[0119] Step S620, calculating the hot and cold data ratio between the test hot data and the test cold data;
[0120] Step S630, classifying the test query statements according to the ratio of hot and cold data to obtain hot query statements and cold query statements;
[0121] Step S640 , executing a database query operation on the test hot data using the hot query statement and the test duration, and executing a database query operation on the test cold data using the cold query statement and the test duration.
[0122] In step S610 of some embodiments, in order to more comprehensively and accurately evaluate the system performance of the test database under various load conditions, access to hot and cold data is simulated during stress testing to make the test environment closer to the production environment and obtain the test database's processing performance for different types of data. Specifically, the reference simulated business data is classified according to data heat to obtain test hot data and test cold data. Test hot data refers to active data that is frequently accessed and requires fast reading and writing, such as online transaction data and current account balances. Test cold data refers to archived data that is less frequently accessed, such as historical transaction data and historical account balances.
[0123] In some embodiments, in step S620, the amount of hot test data is counted to obtain a first value. The amount of cold test data is counted to obtain a second value. The ratio between the first value and the second value is calculated to obtain a hot-cold data ratio. If the amount of hot test data is w1 and the amount of cold test data is w2, the hot-cold data ratio is expressed as w1 / w2.
[0124] In step S630 of some embodiments, the test query statements are classified according to the hot / cold data ratio to obtain hot query statements and cold query statements. The hot query statements are used to access the hot test data, while the cold query statements are used to access the cold test data. For example, if the hot / cold data ratio is 80%, 80% of the test query statements are classified as hot query statements, and 20% of the test query statements are classified as cold query statements.
[0125] In step S640 of some embodiments, in order to simulate a large number of users accessing the test hot data in the test database at the same time, a database query operation is performed on the test hot data through a hot query statement according to the test duration; in order to simulate a small number of users accessing the test cold data in the test database, a database query operation is performed on the test cold data through a cold query statement according to the test duration.
[0126] Through the above steps S610 to S640, the system performance of the test database when processing simulated business data of different popularity types can be obtained, thereby achieving a more comprehensive stress test on the database.
[0127] See also Figure 7 In some embodiments, step S610 may include but is not limited to steps S710 to S730:
[0128] Step S710, calculating the data access frequency of the reference simulation business data according to a preset data distribution model;
[0129] Step S720, performing mean calculation on the data access frequency to obtain a target access frequency;
[0130] Step S730 , classifying the reference simulated business data according to the target access frequency to obtain test hot data and test cold data; wherein the target access frequency of the test hot data is greater than the target access frequency of the test cold data.
[0131] In step S710 of some embodiments, certain data is frequently accessed within a certain period of time. As time passes, the access frequency gradually decreases, and this data becomes cold data. Certain data is then re-accessed within a certain period of time, becoming hot data. The alternation of hot and cold data exhibits a degree of randomness and dynamic variation. To more accurately simulate hot and cold data, multiple data distribution models are obtained. Each data distribution model is used to assign values to reference simulated business data, resulting in multiple data access frequencies for the reference simulated business data. Data distribution models can include uniform distribution, normal distribution, Poisson distribution, and the like.
[0132] In step S720 of some embodiments, the data access frequencies output by each data distribution model are averaged to obtain a target access frequency for the reference simulated business data. If there are three data distribution models, and the data access frequencies output by these models are z1, z2, and z3, respectively, the target access frequency is expressed as (z1+z2+z3) / 3.
[0133] In step S730 of some embodiments, an access frequency threshold is set. If the target access frequency is greater than or equal to the access frequency threshold, the reference simulated business data is designated as test hot data. If the target access frequency is less than the access frequency threshold, the reference simulated business data is designated as test cold data. The target access frequency of the test hot data is greater than the target access frequency of the test cold data.
[0134] Through the above steps S710 to S730, the randomness of hot and cold data can be accurately simulated, making the test environment closer to the real environment, thereby improving the accuracy and reliability of the stress test.
[0135] The related technology can only perform stress testing on a single database. For example, a certain tool is usually used for stress testing of MySQL databases. If it is necessary to perform stress testing on other databases such as PostgreSQL databases, other plug-ins need to be introduced, which makes the testing process more complicated. The database stress testing method of the embodiment of the present application introduces a configuration file for the entire test process, and provides the configuration of accessing various stress testing tools and the configuration of various databases in the configuration file to be compatible with various stress testing tools, and implements stress testing support for various databases through the adapter mode. The entire stress testing process is controlled by configuring indicators such as the number of test threads, so that different test results can be obtained for different numbers of threads and pressures in a single test. At the same time, stress testing is performed through customized query statements and customized simulated business data, thereby achieving the scalability and compatibility of stress testing.
[0136] See also Figure 8 The present application also provides a database stress testing device that can implement the above database stress testing method. The database stress testing device includes:
[0137] An acquisition module 810 is used to acquire the database name and database type of the test database;
[0138] Parameter extraction module 820, for extracting database connection parameters from a pre-acquired test configuration file based on the database name, and extracting test tool parameters from the test configuration file based on the database type;
[0139] A generation module 830 is used to call a stress testing tool according to test tool parameters to generate original simulated business data and test query statements;
[0140] A connection module 840 is used to connect to a test database according to database connection parameters;
[0141] The storage module 850 is used to store the original simulated business data into the test partition of the connected test database to obtain reference simulated business data;
[0142] The testing module 860 is used to perform stress testing on the connected test database according to the test query statement and the reference simulated business data of the test partition.
[0143] The specific implementation of the database stress testing device is basically the same as the specific embodiment of the above-mentioned database stress testing method, and will not be repeated here.
[0144] The present application also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-described database stress testing method when executing the computer program. The electronic device can be any smart terminal, such as a tablet computer or an in-vehicle computer.
[0145] See also Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:
[0146] The processor 910 may be implemented as a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.
[0147] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 920 and is called by the processor 910 to execute the database stress testing method of the embodiments of this application.
[0148] Input / output interface 930, used to implement information input and output;
[0149] Communication interface 940, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);
[0150] bus 950 , which transmits information between various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );
[0151] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .
[0152] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned database stress testing method is implemented.
[0153] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0154] The database stress testing method, database stress testing device, electronic device, and computer storage medium provided in the embodiments of the present application manage parameters related to database instance generation through a test configuration file, automatically generating database instances based on the parameters. This reduces the complexity and time required to deploy the database instances. When using the automatically generated database instances for database stress testing, the efficiency of database stress testing is greatly improved.
[0155] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0156] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0157] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.
[0158] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.
[0159] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0160] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0161] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0162] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0163] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0164] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0165] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.
Claims
1. A database stress testing method, characterized in that: The method comprises: Get the database name and database type of the test database; Extracting database connection parameters from a pre-acquired test configuration file according to the database name, and extracting test tool parameters from the test configuration file according to the database type; Calling the stress testing tool according to the test tool parameters to generate original simulated business data and test query statements; Connecting to the test database according to the database connection parameters; storing the original simulated business data in the test partition of the connected test database to obtain reference simulated business data; A stress test is performed on the connected test database according to the test query statement and the reference simulated business data of the test partition.
2. The method according to claim 1, characterized in that The stress testing of the connected test database according to the test query statement and the reference simulated business data of the test partition includes: Extracting thread concurrency and test duration from the test configuration file; Start the target thread according to the thread concurrency; Establishing a connection between the target thread and the test database according to the database connection parameters; A stress test is performed on the connected test database according to the test query statement, the reference simulated business data of the test partition and the test duration.
3. The method according to claim 2, characterized in that The stress testing of the connected test database according to the test query statement, the reference simulated business data of the test partition, and the test duration includes: Performing a database query operation on the reference simulated business data according to the test query statement and the test duration, and extracting a tool status parameter from the test configuration file; calling a performance analysis tool according to the tool status parameter; The performance analysis tool is used to perform performance testing on the test database.
4. The method according to claim 3, characterized in that The performing a database query operation on the reference simulated business data according to the test query statement and the test duration includes: Classifying the reference simulated business data to obtain test hot data and test cold data; Calculating a hot data ratio between the test hot data and the test cold data; Classifying the test query statements according to the hot and cold data ratio to obtain hot query statements and cold query statements; The database query operation is performed on the test hot data using the hot query statement and the test duration, and the database query operation is performed on the test cold data using the cold query statement and the test duration.
5. The method according to claim 4, characterized in that The data classification of the reference simulated business data to obtain test hot data and test cold data includes: Calculating the data access frequency of the reference simulated business data according to a preset data distribution model; Calculating the mean of the data access frequencies to obtain a target access frequency; The reference simulated business data is classified according to the target access frequency to obtain the test hot data and the test cold data; wherein the target access frequency of the test hot data is greater than the target access frequency of the test cold data.
6. The method according to any one of claims 1 to 5, characterized in that The storing the original simulated business data into the test partition of the connected test database to obtain reference simulated business data includes: Storing the original simulated service data in a temporary buffer to obtain cached simulated service data; Starting an asynchronous thread, and reading a preset batch of cached simulated business data from the temporary buffer through the asynchronous thread to obtain selected simulated business data; The selected simulated business data is stored in the test partition through the asynchronous thread to obtain the reference simulated business data.
7. The method according to any one of claims 1 to 5, characterized in that The step of calling the stress testing tool according to the test tool parameters to generate original simulated business data and test query statements includes: Obtaining a database table structure and query log from a production database; wherein the production database stores real business data, and the database table structure is the table structure of the real business data; Calling a stress testing tool according to the testing tool parameters; Filling the database table structure with the stress testing tool to obtain the original simulated business data; A production query statement is extracted from the query log, and statement processing is performed on the production query statement through the stress testing tool to obtain the test query statement.
8. A database stress testing device, characterized in that: The device comprises: The acquisition module is used to obtain the database name and database type of the test database; A parameter extraction module is used to extract database connection parameters from a pre-acquired test configuration file according to the database name, and to extract test tool parameters from the test configuration file according to the database type; A generation module is used to call the stress test tool according to the test tool parameters to generate original simulated business data and test query statements; A connection module, configured to connect to the test database according to the database connection parameters; A storage module, configured to store the original simulated service data in a test partition of the connected test database to obtain reference simulated service data; A testing module is used to perform a stress test on the connected test database according to the test query statement and the reference simulated business data of the test partition.
9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.