Test data processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to the field of big data processing, and particularly discloses a test data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: determining a plurality of database fragments indicated by a test data extraction instruction, and connecting each database fragment; determining a database fragment group needing to be subjected to test data joint query according to association levels among the data stored in the plurality of database fragments; taking each database fragment group as a single query object, and taking each database fragment which is not used for constructing the database fragment group as a single query object; a database query statement is obtained, after the database query statement is called, the database query statement is controlled to be executed circularly, data query is carried out on each query object in sequence, and database fragments to which test data obtained through query belong are identified respectively; and writing the identified test data into a test data storage document. By adopting the scheme, the test efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of big data processing, and in particular to a test data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the advent of the Internet and the era of big data, the amount of data has shown an explosive growth trend. For most database users, a single database system has bottlenecks in data storage, query speed, and data processing capabilities, making it difficult to cope with such a huge amount of data. Based on this, large databases can be split into small database shards, so that each database shard is only responsible for the storage and query of a part of the data, thereby effectively solving the problem of too much data stored in a single database. When using multiple database shards, if the database user wants to use the data stored in multiple database shards for testing, in order to simulate a more realistic operating situation, it is usually necessary to extract a certain amount of data from each database shard as test data.

[0003] In traditional technology, if you want to extract test data from multiple database shards separately, you usually need to repeatedly call the database query statement multiple times, and then manually merge and organize the test data extracted from each database shard. This repeated calling and manual organization process will lead to low test data extraction efficiency, thereby reducing test efficiency. Summary of the invention

[0004] Based on this, it is necessary to provide a test data processing method, apparatus, computer equipment, computer-readable storage medium and computer program product that can improve test efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a test data processing method, comprising:

[0006] Receive a test data extraction instruction, determine multiple database shards indicated by the test data extraction instruction, and connect to each database shard respectively;

[0007] According to the correlation level between the data stored in the multiple database shards, determine the database shard group that needs to perform the test data joint query; each database shard group includes: at least two database shards, and the correlation level between the data stored in at least two database shards reaches a preset level;

[0008] Treat each database shard group as a single query object, and treat each database shard not used to build a database shard group as a single query object;

[0009] Obtain a database query statement. After calling the database query statement, control the database query statement to execute in a loop, query data for each query object in turn, and identify the database shards to which the test data obtained by the query belongs.

[0010] Write the marked test data into the test data storage document.

[0011] In one embodiment, the database query statement is constructed in the following manner:

[0012] Obtain test data screening conditions and database parameters that are compatible with multiple database shards;

[0013] Generate database query statements based on test data screening conditions and database parameters.

[0014] In one embodiment, after calling the database query statement, the database query statement is controlled to be executed cyclically, and data is queried for each query object in turn, including:

[0015] After calling the database query statement, get the total number of shards of multiple database shards;

[0016] In the process of controlling the execution of database query statements, each time a data query on the query object is completed, the total number of queried database shards is counted until the total number of queried database shards is equal to the total number of shards, and the control database query statement stops executing.

[0017] In one embodiment, writing the marked test data into a test data storage file includes:

[0018] For the test data extracted from each database shard, a preset parameterized format is obtained, and according to the data extraction fields set in the preset parameterized format, target field extraction is performed on the marked test data to obtain target test data of the marked database shard;

[0019] Obtain a specified path of a test data storage document and a data writing method matching the test data storage document;

[0020] According to the data writing mode that matches the test data storage document, the target test data is written into the test data storage document according to the specified path.

[0021] In one embodiment, after writing the marked test data into the test data storage file, the method further includes:

[0022] Receive test data retrieval instructions;

[0023] According to the database shard identifier indicated by the test data retrieval instruction, retrieve the test data matching the database shard identifier from the test data storage document;

[0024] The retrieved test data is input into the test environment indicated by the test data retrieval instruction, so as to perform the test using the retrieved test data.

[0025] In one embodiment, after writing the marked test data into the test data storage file, the method further includes:

[0026] Receive test data screening condition modification instructions;

[0027] Modify the database query statement according to the test data screening condition modification instruction to obtain a modified database query statement;

[0028] When the test data extraction instruction is received again, after the modified database query statement is called, the modified database query statement is controlled to be executed cyclically to extract test data from multiple database shards.

[0029] In a second aspect, the present application also provides a test data processing device, comprising:

[0030] A data extraction instruction processing module, used to receive a test data extraction instruction, determine a plurality of database shards indicated by the test data extraction instruction, and connect to each database shard respectively;

[0031] A database group determination module is used to determine a database shard group for which test data joint query is required according to the correlation level between data stored in multiple database shards; each database shard group includes: at least two database shards, and the correlation level between the data stored in at least two database shards reaches a preset level;

[0032] A query object determination module, used to treat each database shard group as a single query object, and treat each database shard not used to construct the database shard group as a single query object;

[0033] The query statement execution module is used to obtain the database query statement. After calling the database query statement, the module controls the database query statement to execute in a loop, performs data query on each query object in turn, and identifies the database shards to which the test data obtained by the query belongs.

[0034] The test data writing module is used to write the marked test data into the test data storage document.

[0035] In one embodiment, the test data processing device further includes:

[0036] The statement parameter acquisition module is used to obtain test data screening conditions and database parameters adapted to multiple database shards;

[0037] The query statement generation module is used to generate database query statements based on test data screening conditions and database parameters.

[0038] In one embodiment, the query statement execution module includes:

[0039] A shard total number acquisition unit, used to acquire the total number of shards of multiple database shards after calling a database query statement;

[0040] The statement execution control unit is used to count the total number of database shards that have been queried each time a data query on the query object is completed during the process of controlling the execution of the database query statement, until the total number of database shards that have been queried is equal to the total number of shards, and control the database query statement to stop executing.

[0041] In one embodiment, the test data writing module includes:

[0042] The target test data extraction unit is used to obtain a preset parameterized format for the test data extracted from each database shard, and perform target field extraction on the identified test data according to the data extraction fields set in the preset parameterized format to obtain the target test data of the identified database shard;

[0043] A write parameter acquisition unit, used to acquire a specified path of a test data storage document and a data writing method matching the test data storage document;

[0044] The test data writing unit is used to write the target test data into the test data storage document according to the data writing mode matching the test data storage document and along the specified path.

[0045] In one embodiment, the test data processing device further includes:

[0046] A call instruction receiving unit, used for receiving a test data call instruction;

[0047] A test data retrieval unit, configured to retrieve test data matching the database shard identifier from a test data storage document according to the database shard identifier indicated by the test data retrieval instruction;

[0048] The test data input unit is used to input the retrieved test data into the test environment indicated by the test data retrieval instruction, so as to perform testing through the retrieved test data.

[0049] In one embodiment, the test data processing device further includes:

[0050] A modification instruction receiving unit, used for receiving a test data screening condition modification instruction;

[0051] A query statement modification unit, used to modify the instruction according to the test data screening condition, modify the database query statement, and obtain a modified database query statement;

[0052] The modified statement execution control unit is used to control the modified database query statement to be executed cyclically after calling the modified database query statement when the test data extraction instruction is received again, so as to extract test data from multiple database shards.

[0053] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned test data processing method when executing the computer program.

[0054] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-mentioned test data processing method when executed by a processor.

[0055] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the above-mentioned test data processing method when executed by a processor.

[0056] The above-mentioned test data processing method, device, computer equipment, computer-readable storage medium and computer program product first determine the multiple database shards indicated by the test data extraction instruction, connect each database shard respectively, and then determine the database shard group that needs to perform test data joint query according to the correlation level between the data stored in the multiple database shards, each database shard group includes: at least two database shards, and the correlation level between the data stored in at least two database shards reaches a preset level. Further, each database shard group can be used as a single query object. On the one hand, it is convenient to perform joint query on database shards with associated stored data, so that test data with association but different sources can be quickly obtained during testing without manual integration, thereby improving test efficiency. On the other hand, by using the database shard group as a single query object for data query, data query can be performed on multiple database shards in parallel, thereby improving data query efficiency and further improving test efficiency. Furthermore, each database shard that is not used to build a database shard group is treated as a single query object, and a database query statement is obtained. After calling the database query statement, the database query statement is controlled to be executed in a loop, and data queries are performed on each query object in turn, and the database shards to which the test data obtained by the query belong are respectively identified, so that during the test, the test data queried from any database shard can be quickly retrieved according to the identification to improve the test efficiency. Finally, the marked test data is efficiently written into the test data storage document without manual document sorting. Therefore, by using the above method, the processes of data query, data format processing, and data writing can be connected in series with one click, and the required test data can be efficiently queried and written into the document, which provides convenience for subsequent testing and improves the test efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] FIG1 is an application environment diagram of a test data processing method in one embodiment;

[0059] FIG2 is a schematic diagram of a flow chart of a test data processing method in one embodiment;

[0060] FIG3 is a schematic diagram of setting configuration parameters in one embodiment;

[0061] FIG4 is a schematic diagram of a flow chart of controlling the cyclic execution of database query statements in one embodiment;

[0062] FIG5 is a schematic diagram of a process of formatting test data and then writing it into a document in one embodiment;

[0063] FIG6 is a schematic diagram of a flow chart of processing a test data retrieval instruction in one embodiment;

[0064] FIG7 is a schematic diagram of a flow chart of processing a test data screening condition modification instruction in one embodiment;

[0065] FIG8 is a schematic flow chart of a test data processing method in another embodiment;

[0066] FIG9 is a flow chart of implementing test data processing by a test tool in one embodiment;

[0067] FIG10 is a block diagram of a test data processing device in one embodiment;

[0068] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0070] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with relevant provisions of national laws and regulations.

[0071] The test data processing method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 can receive the test data extraction instruction triggered by the terminal 102, determine the multiple database shards indicated by the test data extraction instruction, and connect to each database shard respectively, and determine the database shard group that needs to perform test data joint query based on the correlation level between the data stored in the multiple database shards, wherein each database shard group includes: at least two database shards, and the correlation level between the data stored in at least two database shards reaches a preset level. Furthermore, the server 104 can treat each database shard group as a single query object, and treat each database shard that is not used to construct the database shard group as a single query object, then obtain the database query statement, and after calling the database query statement, control the database query statement to execute in a loop, perform data query on each query object in turn, and identify the database shard to which the test data obtained by the query belongs, and finally write the identified test data into the test data storage document. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices, etc. The server 104 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0072] In an exemplary embodiment, as shown in FIG2 , a test data processing method is provided, which is described by taking the method applied to the server 104 in FIG1 as an example, and includes the following steps 202 to 210. Among them:

[0073] Step 202, receiving a test data extraction instruction, determining a plurality of database shards indicated by the test data extraction instruction, and connecting to each database shard respectively.

[0074] The test data extraction instruction carries: various configuration parameters configured by the user in the test tool through the terminal, including but not limited to: identity authentication information required to access the database shard, the name of the database shard, the address of the database shard, thread pool parameters for connecting to the database shard, etc.

[0075] Optionally, the server may receive a test data extraction instruction, parse the test data extraction instruction, determine various configuration parameters configured in the test tool, and verify the identity authentication information carried in the configuration parameters. After the verification is passed, the server may determine the multiple database shards indicated by the test data extraction instruction according to the names of the multiple database shards carried in the configuration parameters, and connect to each database shard respectively through the address of each database shard carried in the configuration parameters. In addition, during the connection process, the connection is performed in accordance with the thread pool parameters carried in the configuration parameters.

[0076] Exemplarily, as shown in FIG3 , a schematic diagram for setting configuration parameters is provided. A user can access the test tool Jmeter (a stress testing tool developed based on Java and can be used to perform stress testing on software) deployed on the server through a terminal, add a configuration element (JDBC Connection Configuration) in Jmeter, and set the following configuration parameters in the configuration element: the connection pool name (Variable Name) of the database shard, so that different database shard connections can be selected by selecting different connection pool names; the maximum number of thread connections of the connection pool of the database shard (MaxNumber of Connections=0); the maximum waiting time of threads of the connection pool of the database shard (Max Wait=10000 milliseconds); the idle time of threads (Time Between Eviction Runs=60000 milliseconds); automatically submit query statements (Auto Commit=True); transaction isolation level (Transaction Isolation=default value DEFAULT); disconnect when the connection is idle (Test While Idle=True); the shortest time in the idle state in the connection pool (Soft MinEvictable Idle Time=5000 milliseconds); the address of the database shard (Database URL); and the identity information to be verified (Username and Password). Based on this, the server can receive the test data extraction instruction, parse the test data extraction instruction, determine the various configuration parameters configured in the test tool, and then determine the multiple database shards indicated by the test data extraction instruction, and connect to each database shard respectively.

[0077] Step 204, determining a database shard group for which test data joint query is required based on the correlation level between data stored in the multiple database shards; each database shard group includes: at least two database shards, and the correlation level between the data stored in at least two database shards reaches a preset level.

[0078] Among them, the correlation level between the data stored in any two database shards represents: the degree of correlation between the data stored in the two database shards. The higher the correlation level, the higher the correlation level, and the more likely it is that the test data stored in the two database shards will be jointly used for testing in the future. Therefore, in this implementation, when querying (extracting) test data from multiple shard databases, the correlation level between the data stored in the database shards is taken into consideration, thereby determining a joint query plan to facilitate the joint use of subsequent test data.

[0079] Optionally, the correlation level between the data stored in any two database shards can be manually pre-configured and stored in the server. Based on this, the server can first determine the database shard group that needs to perform test data joint query according to the correlation level between the data stored in multiple database shards.

[0080] Step 206: Take each database shard group as a single query object, and take each database shard that is not used to construct the database shard group as a single query object.

[0081] Among them, not used to build a database shard group means that the association level between the database shard and other database shards does not reach the preset level, and there is no need to perform joint query with other database shards.

[0082] Optionally, the server can treat each database shard group as a single query object. In this case, the single query object includes at least two shard databases, which facilitates subsequent parallel and efficient data query of at least two shard databases in the query object, and can query and obtain data that is related but comes from different database shards, and improve the efficiency of querying and obtaining test data. The server can also treat each database shard that is not used to build a database shard group as a single query object. In this case, the single query object only includes one shard database.

[0083] Step 208, obtain a database query statement, and after calling the database query statement, control the database query statement to execute in a loop, perform data query on each query object in turn, and identify the database shards to which the test data obtained by the query belongs.

[0084] The database query statement may specifically be a SQL (Structured Query Language) statement that is pre-edited and stored in the server. SQL is a database query and programming language used to access data and query, update and manage database systems. It is a language for operating databases.

[0085] Optionally, the server can first obtain a database query statement, and after calling the database query statement once, control the database query statement to be executed cyclically without calling it multiple times. Further, by controlling the database query statement to be executed cyclically, data is queried for each query object in turn, and the database shards to which the test data obtained by the query belong are respectively identified.

[0086] Exemplarily, for a single query object consisting of a database shard group, at this time, the single query object includes at least two shard databases, and the server can control the database query statement to perform data query on at least two shard databases in the query object in parallel to obtain a data query result, which includes the test data of at least two shard databases, and the test data corresponding to each shard database is respectively identified to which the shard database belongs, and then the test data is bound together and written into the test data storage document. For a single query object consisting of a database shard, at this time, the single query object includes at least one shard database, and the server can control the database query statement to directly perform data query on a shard database in the query object to obtain test data, identify the shard database to which it belongs, and then write the test data storage document.

[0087] Step 210: Write the marked test data into a test data storage file.

[0088] Optionally, each time the server obtains test data from a query object, it will identify the database shard to which the test data belongs, and write the identified test data into a test data storage document. For a query object including at least two database shards, after obtaining test data from the query object and identifying the database shards to which the test data belongs, the server can bind the identified test data from different database shards together and write them into the test data storage document, so as to facilitate combining them for testing.

[0089] Exemplarily, the test data storage document may be a document created based on the java.io package. The java.io package may provide a set of classes and interfaces for reading and writing data streams. In the present embodiment, the test data storage document may be used for writing test data.

[0090] For example, if database shard 1 stores data generated when running business process A in the first half of this year, database shard 2 stores data generated when running business process A in the second half of this year, and other database shards store data generated when running other business processes this year, if it is necessary to extract data generated when running different business processes from multiple database shards as test data, the performance of the system when running all business processes can be tested. In this case, because there is a strong correlation between the data stored in database shard 1 and database shard 2, they should be combined as test data. The correlation level between database shard 1 and database shard 2 can be set to reach a preset level, so that when performing data query, database shard 1 and database shard 2 can be jointly queried, so that the test data in the joint query result is complete and generated when running business process A throughout the year, and after identifying which part of the test data belongs to which database shard, they are bound together and written into the test data storage document so that they can be put into testing together later. Among them, after identifying which part of the test data belongs to which database shard, it is convenient to retrieve the test data corresponding to any database shard separately in special circumstances for testing.

[0091] For example, if database shard 1 stores data generated by executing the buying process, database shard 2 stores data generated by executing the selling process, and other database shards store data generated in non-buying and non-selling processes, if it is necessary to extract data generated by different processes as test data, the performance of the system running all processes can be tested. In this case, because there is a strong correlation between the data stored in database shard 1 and database shard 2, they should be combined together as test data, and the correlation level between database shard 1 and database shard 2 can be set to reach a preset level, so that after performing a joint query on database shard 1 and database shard 2, the test data in the joint query result is complete test data generated by the buying and selling process, and after identifying which part of the test data belongs to which database shard, they are bound together and written into the test data storage document so that they can be put into testing together later.

[0092] The above test data processing method first determines the multiple database shards indicated by the test data extraction instruction, connects each database shard respectively, and then determines the database shard group that needs to perform test data joint query according to the correlation level between the data stored in the multiple database shards, each database shard group includes: at least two database shards, and the correlation level between the data stored in at least two database shards reaches a preset level. Further, each database shard group can be used as a single query object. On the one hand, it is convenient to perform joint query on database shards with associated stored data, so that test data with association but different sources can be quickly obtained during testing without manual integration, thereby improving test efficiency. On the other hand, by using the database shard group as a single query object for data query, data query can be performed on multiple database shards in parallel, thereby improving data query efficiency and further improving test efficiency. Furthermore, each database shard that is not used to build a database shard group is treated as a single query object, and a database query statement is obtained. After calling the database query statement, the database query statement is controlled to be executed in a loop, and data queries are performed on each query object in turn, and the database shards to which the test data obtained by the query belong are respectively identified, so that during the test, the test data queried from any database shard can be quickly retrieved according to the identification to improve the test efficiency. Finally, the marked test data is efficiently written into the test data storage document without manual document sorting. Therefore, by using the above method, the processes of data query, data format processing, and data writing can be connected in series with one click, and the required test data can be efficiently queried and written into the document, which provides convenience for subsequent testing and improves the test efficiency.

[0093] In one embodiment, the database query statement is constructed in the following manner:

[0094] Obtain test data screening conditions and database parameters that are compatible with multiple database shards;

[0095] Generate database query statements based on test data screening conditions and database parameters.

[0096] The test data screening conditions include, but are not limited to: the data under which fields need to be screened as test data, the data volume or number of test data screened from each database shard. Database parameters include: the address, port number, name of the database shard, the user name and password required to access the database shard, etc.

[0097] Optionally, the server may first obtain pre-configured test data screening conditions and database parameters adapted to multiple database shards, and obtain a query statement code template, and insert the test data screening conditions and database parameters involved in the test data screening conditions as parameters into the string in the query statement code template, thereby automatically generating the required database query statement. The query statement code template may be pre-configured in the server to improve the efficiency of generating database query statements.

[0098] Exemplarily, after the server automatically generates the database query statement, a manual verification and correction step may be added to ensure the accuracy of the database query statement finally generated.

[0099] In this embodiment, taking into account the requirements for screening test data and the compatibility of the database, a database query statement that can accurately query test data from the data stored in the database shards can be generated based on the test data screening conditions and database parameters adapted to the database shards, ensuring that the generated database query statement can accurately and efficiently return the required test data, ensuring the smooth progress of the data query process, thereby improving test efficiency.

[0100] In one embodiment, as shown in FIG. 4 , a flow chart of controlling the cyclic execution of a database query statement is provided, that is, after calling the database query statement, the database query statement is controlled to be cyclically executed to perform data query on each query object in turn, including:

[0101] Step 402, after calling the database query statement, obtain the total number of shards of the multiple database shards.

[0102] Optionally, after calling the database query statement, the server may obtain the total number of shards of the plurality of database shards from the stored data, so as to determine the execution stop condition of the database query statement according to the total number of shards.

[0103] Step 404, controlling the execution of the database query statement, and performing data query on the query objects in sequence.

[0104] Optionally, after calling the database query statement once, the server can control the database query statement to start looping execution to query the query objects in sequence without repeated calls. The query order of the query objects can be determined randomly or according to a pre-configured order.

[0105] In the process of controlling the execution of the database query statement, step 406 is executed to count the total number of database shards that have been queried each time a data query on the query object is completed.

[0106] Optionally, each time a data query for a query object is completed, the server may count the total number of database shards that have been queried. When a query is completed for a query object consisting of a database shard group, the increase in the number of database shards that have been queried = the number of database shards in the database shard group.

[0107] If the total number of queried database shards is less than the total number of shards, return to step 404.

[0108] If the total number of queried database shards is equal to the total number of shards, step 408 is executed to stop executing the database query statement.

[0109] Optionally, when the total number of queried database shards is equal to the total number of shards, the server may determine that the query has been completed for all database shards, thereby controlling the database query statement to stop executing.

[0110] In this embodiment, the loop execution process of the database query statement can be accurately controlled according to the total number of shards of the database shards, thereby ensuring the smooth progress of the data query process and improving the subsequent test efficiency.

[0111] In some embodiments, based on FIG. 2 , as shown in FIG. 5 , step 210 may further include the following steps:

[0112] Step 502: Obtain a preset parameterized format for the test data extracted from each database shard.

[0113] Among them, the preset parameterized format is used to format the test data to extract data that can be directly used for testing, so that when the test data in the test data storage document is used in the subsequent test process, there is no need to convert the format of the test data in the test data storage document.

[0114] Optionally, for the test data extracted from each database shard, the server may obtain a pre-configured preset parameterized format.

[0115] Step 504 , according to the data extraction fields set in the preset parameterized format, target field extraction is performed on the marked test data to obtain the target test data of the marked database shard.

[0116] Optionally, for each test data whose database shard has been identified, the server can first perform field parsing on the test data to determine which fields are included in the test data, and then perform target field extraction on the identified test data according to the data extraction fields set in the preset parameterized format to obtain the target test data of the identified database shard.

[0117] It should be noted that if a data extraction field has different field names in different test data, the server can unify the field name instructions triggered by the user and uniformly process the fields with the same actual meaning but inconsistent field names in each test data, so that their field names are consistent with the names of the corresponding data extraction fields.

[0118] Step 506: Acquire a designated path of the test data storage document and a data writing method matching the test data storage document.

[0119] The designated path of the test data storage document may be pre-stored in the server.

[0120] Optionally, the server may obtain a specified path of the test data storage document from pre-stored data, and identify the format of the test data storage document to determine a data writing method that matches the test data storage document, so as to ensure that the data written into the document is compatible with the format of the document.

[0121] Optionally, the data writing methods that can be adopted in this embodiment include but are not limited to the following: (1) Using byte streams to write data, byte data can be directly operated, and it is suitable for processing non-text files or data that requires precise control of bytes. (2) Using character streams to write data, character streams are more suitable for processing text files, and can handle character encoding and conversion, making read and write operations more convenient and intuitive. (3) When writing data to a document, the data can be written to a buffer first, and when the buffer is full, the data can be written to the document. (4) Use the IO class library in Java to perform document writing operations. (5) Use built-in libraries, such as the open function, to perform document writing operations.

[0122] Step 508 , write the target test data into the test data storage document according to the data writing mode matching the test data storage document and along the specified path.

[0123] Optionally, the server can write the test data of the identified database shard into a test data storage document under a specified path according to a data writing method that matches the test data storage document. When subsequent tests are performed, the input file of the test process can be obtained by calling the test data storage document under the specified path, so that the test process can proceed smoothly.

[0124] In this embodiment, after querying and obtaining test data from multiple database shards, the test data will be further subjected to field extraction according to a preset parameterized format, so that in the subsequent test process, the user can directly use the target test data in the test data storage document without additional field extraction, which is conducive to improving the test efficiency. In addition, in this embodiment, the target test data is written into the test data storage document according to the matching data writing method, which can ensure that the target test data written into the test data storage document is not damaged, further ensuring the smooth progress of the subsequent test process, thereby improving the test efficiency.

[0125] In a possible implementation, based on FIG. 2 , as shown in FIG. 6 , after step 210, the test data processing method may further include the following steps:

[0126] Step 602: Receive a test data retrieval instruction, and retrieve test data matching the database shard identifier from a test data storage document according to the database shard identifier indicated by the test data retrieval instruction.

[0127] The test data storage document stores: test data with identifiers from different database shards.

[0128] Optionally, during subsequent testing, if the user needs to retrieve test data from a certain database shard separately from the test data storage document, the user can use the terminal to trigger a test data retrieval instruction, wherein the test data retrieval instruction carries the database shard identifier corresponding to the test data to be retrieved, for example, the name of the database shard to which it belongs, or the number of the database shard to which it belongs. Based on this, the server can receive the test data retrieval instruction, and search (or match the database shard identifier) ​​in the test data storage document according to the database shard identifier indicated by the test data retrieval instruction, and retrieve the test data that matches the database shard identifier after searching (or matching successfully).

[0129] Step 604, input the retrieved test data into the test environment indicated by the test data retrieval instruction, so as to perform testing using the retrieved test data.

[0130] Optionally, the server may input the retrieved test data into a test environment indicated by the test data retrieval instruction to simulate an actual operating environment, thereby performing testing using the retrieved test data.

[0131] In this embodiment, when it is necessary to retrieve test data extracted from a certain database shard separately, the database shard identifier can be used to accurately and automatically extract the test data corresponding to the database shard identifier from the test data storage document, eliminating the need for users to manually screen the test data corresponding to the required database shard, which is beneficial to improving testing efficiency.

[0132] In one embodiment, based on FIG. 2 , as shown in FIG. 7 , after step 210, the test data processing method may further include the following steps:

[0133] Step 702, receiving a test data screening condition modification instruction, and modifying the database query statement according to the test data screening condition modification instruction to obtain a modified database query statement.

[0134] Optionally, if the user needs to modify the test data screening condition in the database query statement to query and obtain another batch of test data, thereby engaging in another test task, the user can use the terminal to trigger the test data screening condition modification instruction. Based on this, the server can receive the test data screening condition modification instruction, and modify the test data screening condition in the database query statement according to the test data screening condition modification instruction, for example, modify the field used to filter the test data in the test data screening condition to obtain a modified database query statement.

[0135] Step 704, when the test data extraction instruction is received again, after calling the modified database query statement, the modified database query statement is controlled to be executed cyclically to extract test data from multiple database shards.

[0136] Optionally, if the user wants to query the test data based on the modified database query statement, the test data extraction instruction can be triggered again. Based on this, the server can receive the test data extraction instruction, determine the multiple database shards indicated by the test data extraction instruction, and connect to each database shard respectively, and determine the database shard group that needs to perform a joint query of the test data according to the correlation level between the data stored in the multiple database shards, and then use each database shard group as a single query object, and use each database shard that is not used to build the database shard group as a single query object. Further, after calling the modified database query statement, the server can control the modified database query statement to execute in a loop to query the query objects in sequence, and then extract test data from multiple database shards.

[0137] In this embodiment, the user may be allowed to modify the data query statement to adapt to the user's test data extraction requirements. Based on this, the test data extraction process before different test processes may be flexibly implemented.

[0138] In a specific embodiment, as shown in FIG8 , a flow chart of another test data processing method is provided, which mainly includes the following steps:

[0139] Step 802, receiving a test data extraction instruction, determining a plurality of database shards indicated by the test data extraction instruction, and connecting to each database shard respectively;

[0140] Step 804, determining a database shard group for which test data joint query is required according to the association level between the data stored in the multiple database shards;

[0141] Step 806, taking each database shard group as a single query object, and taking each database shard not used to construct the database shard group as a single query object;

[0142] Step 808, obtaining test data screening conditions and database parameters adapted to multiple database shards;

[0143] Step 810, generating a database query statement according to the test data screening conditions and database parameters;

[0144] Step 812, after calling the database query statement, obtain the total number of shards of the multiple database shards;

[0145] Step 814, controlling the execution of a database query statement to query the data of the query object;

[0146] Step 816, each time a data query for a query object is completed, the total number of queried database shards is counted;

[0147] If the total number of queried database shards is equal to the total number of shards, execute step 818 to stop the execution of the database query statement; otherwise, return to step 814;

[0148] Step 820, identifying the database shards to which the test data obtained by the query belongs respectively;

[0149] Step 822, extracting target fields from the marked test data according to the data extraction fields set in the preset parameterized format, and obtaining target test data of the marked database shard;

[0150] Step 824 , according to the data writing mode matching the test data storage document and along the specified path, write the target test data into the test data storage document.

[0151] In a specific application environment, as shown in Figure 9, a flow chart of implementing test data processing through a test tool (Jmeter) is provided, which mainly includes the following processes:

[0152] 1. Open the test tool;

[0153] 2. Add a configuration element and set the configuration parameters in the configuration element to connect to the database shard;

[0154] 3. Add thread groups and write program scripts;

[0155] 4. Set the total number of shards;

[0156] 5. Extract test data from multiple database shards through the loop execution of database query statements;

[0157] 6. Write the test data into the test data storage document in the preset parameterized format;

[0158] 7. Put it to the test.

[0159] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0160] Based on the same inventive concept, the embodiment of the present application also provides a test data processing device for implementing the test data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more test data processing device embodiments provided below can refer to the limitations of the test data processing method above, and will not be repeated here.

[0161] In an exemplary embodiment, as shown in FIG. 10 , a test data processing device is provided, comprising: a data extraction instruction processing module 1002, a database group determination module 1004, a query object determination module 1006, a query statement execution module 1008 and a test data writing module 1010, wherein:

[0162] A data extraction instruction processing module, used to receive a test data extraction instruction, determine a plurality of database shards indicated by the test data extraction instruction, and connect to each database shard respectively;

[0163] A database group determination module is used to determine a database shard group for which test data joint query is required according to the correlation level between data stored in multiple database shards; each database shard group includes: at least two database shards, and the correlation level between the data stored in at least two database shards reaches a preset level;

[0164] A query object determination module, used to treat each database shard group as a single query object, and treat each database shard not used to construct the database shard group as a single query object;

[0165] The query statement execution module is used to obtain the database query statement. After calling the database query statement, the module controls the database query statement to execute in a loop, performs data query on each query object in turn, and identifies the database shards to which the test data obtained by the query belongs.

[0166] The test data writing module is used to write the marked test data into the test data storage document.

[0167] The above-mentioned test data processing device first determines the multiple database shards indicated by the test data extraction instruction, connects each database shard respectively, and then determines the database shard group that needs to perform test data joint query according to the correlation level between the data stored in the multiple database shards, each database shard group includes: at least two database shards, and the correlation level between the data stored in at least two database shards reaches a preset level. Further, each database shard group can be used as a single query object. On the one hand, it is convenient to perform joint query on database shards with associated stored data, so that test data with association but different sources can be quickly obtained during testing without manual integration, thereby improving test efficiency. On the other hand, by using the database shard group as a single query object for data query, data query can be performed on multiple database shards in parallel, thereby improving data query efficiency and further improving test efficiency. Furthermore, each database shard that is not used to build a database shard group is treated as a single query object, and a database query statement is obtained. After calling the database query statement, the database query statement is controlled to be executed in a loop, and data queries are performed on each query object in turn, and the database shards to which the test data obtained by the query belong are respectively identified, so that during the test, the test data queried from any database shard can be quickly retrieved according to the identification to improve the test efficiency. Finally, the marked test data is efficiently written into the test data storage document without manual document sorting. Therefore, by using the above method, the processes of data query, data format processing, and data writing can be connected in series with one click, and the required test data can be efficiently queried and written into the document, which provides convenience for subsequent testing and improves the test efficiency.

[0168] In one of the embodiments, the test data processing device also includes: a statement parameter acquisition module, used to obtain test data screening conditions and database parameters adapted to multiple database shards; a query statement generation module, used to generate a database query statement based on the test data screening conditions and database parameters.

[0169] In one of the embodiments, the query statement execution module includes: a shard total number acquisition unit, used to obtain the total number of shards of multiple database shards after calling a database query statement; a statement execution control unit, used to count the total number of queried database shards each time a data query on the query object is completed during the process of controlling the execution of the database query statement, until the total number of queried database shards is equal to the total number of shards, and control the database query statement to stop execution.

[0170] In one embodiment, the test data writing module includes: a target test data extraction unit, which is used to obtain a preset parameterized format for the test data extracted from each database shard, and perform target field extraction on the identified test data according to the data extraction fields set in the preset parameterized format to obtain the target test data of the identified database shard; a write parameter acquisition unit, which is used to obtain a specified path of a test data storage document and a data writing method that matches the test data storage document; a test data writing unit, which is used to write the target test data into the test data storage document according to the specified path according to the data writing method that matches the test data storage document.

[0171] In one of the embodiments, the test data processing device also includes: a retrieval instruction receiving unit, used to receive a test data retrieval instruction; a test data retrieval unit, used to retrieve test data matching the database shard identifier from a test data storage document according to the database shard identifier indicated by the test data retrieval instruction; and a test data input unit, used to input the retrieved test data into the test environment indicated by the test data retrieval instruction so as to perform testing using the retrieved test data.

[0172] In one of the embodiments, the test data processing device also includes: a modification instruction receiving unit, used to receive a test data screening condition modification instruction; a query statement modification unit, used to modify the database query statement according to the test data screening condition modification instruction to obtain a modified database query statement; a modification statement execution control unit, used to control the modified database query statement to be executed cyclically after calling the modified database query statement when the test data extraction instruction is received again, so as to extract test data from multiple database shards.

[0173] Each module in the above test data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0174] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store test data processing data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a test data processing method is implemented.

[0175] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0176] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0177] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0178] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0179] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0180] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0181] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A test data processing method, characterized in that: The method comprises: Receive a test data extraction instruction, determine a plurality of database shards indicated by the test data extraction instruction, and connect to each of the database shards respectively; According to the correlation level between the data stored in the multiple database shards, determine the database shard group that needs to perform test data joint query; each of the database shard groups includes: at least two database shards, and the correlation level between the data stored in the at least two database shards reaches a preset level; Taking each of the database shard groups as a single query object, and taking each database shard not used to construct the database shard group as a single query object; Obtain a database query statement, and after calling the database query statement, perform data query on each query object in turn by controlling the database query statement to be executed cyclically, and identify the database shards to which the test data obtained by the query belongs respectively; Write the marked test data into the test data storage document.

2. The method according to claim 1, characterized in that The database query statement is constructed in the following way: Acquire test data screening conditions and database parameters adapted to the multiple database shards; The database query statement is generated according to the test data screening condition and the database parameter.

3. The method according to claim 1, characterized in that After calling the database query statement, controlling the database query statement to execute cyclically to query data for each query object in turn includes: After calling the database query statement, obtaining the total number of shards of the multiple database shards; In the process of controlling the execution of the database query statement, each time a data query on the query object is completed, the total number of queried database shards is counted until the total number of queried database shards is equal to the total number of shards, and the database query statement is controlled to stop execution.

4. The method according to claim 1, characterized in that: The step of writing the marked test data into the test data storage document comprises: For the test data extracted from each of the database shards, a preset parameterized format is obtained, and according to the data extraction fields set in the preset parameterized format, target field extraction is performed on the identified test data to obtain target test data of the identified database shard; Acquire a specified path of the test data storage document and a data writing method matching the test data storage document; According to the data writing mode matching the test data storage document, the target test data is written into the test data storage document along the specified path.

5. The method according to claim 1, characterized in that After writing the marked test data into the test data storage file, the method further includes: Receive test data retrieval instructions; According to the database shard identifier indicated by the test data retrieval instruction, retrieve test data matching the database shard identifier from the test data storage document; The retrieved test data is input into the test environment indicated by the test data retrieval instruction, so as to perform testing using the retrieved test data.

6. The method according to claim 1, characterized in that After writing the marked test data into the test data storage file, the method further includes: Receive test data screening condition modification instructions; Modify the database query statement according to the test data screening condition modification instruction to obtain a modified database query statement; When the test data extraction instruction is received again, after calling the modified database query statement, the modified database query statement is controlled to be executed cyclically to extract test data from the multiple database shards.

7. A test data processing device, characterized in that: The device comprises: A data extraction instruction processing module, used for receiving a test data extraction instruction, determining a plurality of database shards indicated by the test data extraction instruction, and connecting to each of the database shards respectively; A database group determination module is used to determine a database shard group for which test data joint query is required according to the correlation level between the data stored in the plurality of database shards; each of the database shard groups includes: at least two database shards, and the correlation level between the data stored in the at least two database shards reaches a preset level; A query object determination module, configured to take each of the database shard groups as a single query object, and take each database shard not used to construct the database shard group as a single query object; A query statement execution module is used to obtain a database query statement, and after calling the database query statement, control the database query statement to execute in a loop, perform data query on each query object in turn, and identify the database shards to which the test data obtained by the query belongs respectively; The test data writing module is used to write the marked test data into the test data storage document.

8. The device according to claim 7, characterized in that The test data processing device also includes: A statement parameter acquisition module, used to acquire test data screening conditions and database parameters adapted to the multiple database shards; The query statement generating module is used to generate the database query statement according to the test data screening condition and the database parameter.

9. The device according to claim 7, characterized in that The query statement execution module includes: A shard total number acquisition unit, configured to acquire the total number of shards of the plurality of database shards after calling the database query statement; A statement execution control unit is used to count the total number of database shards that have been queried each time a data query on the query object is completed during the process of controlling the execution of the database query statement, until the total number of database shards that have been queried is equal to the total number of shards, and control the database query statement to stop execution.

10. The device according to claim 7, characterized in that The test data writing module comprises: A target test data extraction unit is used to obtain a preset parameterized format for the test data extracted from each of the database shards, and perform target field extraction on the identified test data according to the data extraction fields set in the preset parameterized format to obtain the target test data of the identified database shard; A writing parameter acquisition unit, used to acquire a specified path of the test data storage document and a data writing mode matching the test data storage document; The test data writing unit is used to write the target test data into the test data storage document according to the data writing mode matching the test data storage document and along the specified path.

11. The device according to claim 7, characterized in that The test data processing device also includes: A call instruction receiving unit, used for receiving a test data call instruction; A test data retrieval unit, configured to retrieve test data matching the database shard identifier from the test data storage document according to the database shard identifier indicated by the test data retrieval instruction; The test data input unit is used to input the retrieved test data into the test environment indicated by the test data retrieval instruction, so as to perform testing through the retrieved test data.

12. The device according to claim 7, characterized in that The test data processing device also includes: A modification instruction receiving unit, used for receiving a test data screening condition modification instruction; A query statement modification unit, used to modify the database query statement according to the test data screening condition modification instruction to obtain a modified database query statement; The modified statement execution control unit is used to control the modified database query statement to be executed cyclically after calling the modified database query statement when the test data extraction instruction is received again, so as to extract test data from the multiple database shards.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images