A data construction method and device, electronic equipment and storage medium

Abstract

The application discloses a data construction method and device, electronic equipment and a storage medium. The method is applied to a task execution module, and the method comprises the following steps: receiving a construction task of test data distributed by a task management module; the construction task of the test data comprises database instance information and field generation strategy configuration information; and the construction task of the test data is executed based on the database instance information and the field generation strategy configuration information, so as to construct test data for a database. The technical scheme of the embodiment of the application can reduce the configuration work of database information by introducing a database instance, and can quickly and efficiently generate test data for the database by introducing field generation strategy configuration information, thereby providing a stable data basis for subsequent database performance test and the like.

Description

Technical Field

[0001] This application relates to the field of database technology, and in particular to a data construction method, apparatus, electronic device, and storage medium. Background Technology

[0002] Database testing is the testing of a software system's database structure, tables, and the data access relationships between them, based on database design specifications. To effectively test a database, accurate and convenient test data needs to be constructed. Currently, this usually requires manual construction of database test data, resulting in low accuracy and efficiency. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a data construction method, apparatus, electronic device, and storage medium.

[0004] This application also provides a data construction method, which is applied to a task execution module, and the method includes:

[0005] Receive test data construction tasks assigned by the task management module; the test data construction tasks include database instance information and field generation strategy configuration information;

[0006] Based on the database instance information and the field generation strategy configuration information, the test data construction task is executed to construct test data for the database.

[0007] In one optional embodiment of this application, the method includes:

[0008] Monitor the operating status of the database; the database is the database corresponding to the database instance information.

[0009] The speed at which the test data is written to the database instance is adjusted based on the database's operating status.

[0010] In one optional embodiment of this application, the method further includes:

[0011] During the process of constructing the test data, the running status of the task execution module and the task execution record are sent to the task management module.

[0012] In an optional embodiment of this application, the database instance information includes database connection parameter information; before executing the test data construction task based on the database instance information and the field generation strategy configuration information, the method further includes:

[0013] Test the database connection information based on the aforementioned parameter connection information;

[0014] Based on the test results of the connection information, it is determined whether the connection information can connect to the database instance of the database.

[0015] In one optional embodiment of this application, the field generation strategy configuration information includes database information, target table information in the database, target table structure information, and test data generation quantity information.

[0016] This application also provides a data construction method, characterized in that the method is applied to a task management module, and the method includes:

[0017] Obtain a test data construction request; the test data construction request includes database instance information and field generation strategy configuration information.

[0018] Based on the test data, a request is made to allocate a test data construction task to the task execution module, so that the task execution module can execute the test data construction task based on the database instance information and field generation strategy configuration information to construct test data for the database.

[0019] In one optional embodiment of this application, the step of obtaining the test data construction request includes:

[0020] Based on the database instance information and field generation strategy configuration information input by the target object in the information input module, a test data construction request is obtained.

[0021] In an optional embodiment of this application, before allocating a test data construction task to the task execution module based on the test data construction request, the method further includes:

[0022] Obtain the address information of the task execution module from the registration module;

[0023] Accordingly, the construction task of allocating test data to the task execution module based on the test data construction request includes:

[0024] Based on the test data, a construction request is made, and the address information is used to allocate a test data construction task to the task execution module.

[0025] In one optional embodiment of this application, the registration module may deploy at least one task execution module; the task execution module is the task execution module whose running state is determined to be available from the at least one task execution module by the registration module.

[0026] In an optional embodiment of this application, after allocating a test data construction task to the task execution module based on the test data construction request, the method further includes:

[0027] The system receives the running status and task execution record sent by the task execution module; wherein the running status and task execution record are sent by the task execution module during the execution of the test data construction task;

[0028] The execution status of the task for building the test data is determined based on the running status and task execution records.

[0029] Based on the execution status, determine whether to continue executing the test data construction task.

[0030] This application embodiment also provides a data construction apparatus, which is applied to a task execution module, and the apparatus includes:

[0031] The receiving unit is used to receive the test data construction task assigned by the task management module; the test data construction task includes database instance information and field generation strategy configuration information.

[0032] The execution unit is used to execute the test data construction task based on the database instance information and the field generation strategy configuration information, and to construct test data for the database.

[0033] In one optional embodiment of this application, the apparatus further includes:

[0034] A monitoring unit is used to monitor the operating status of the database; the database is the database corresponding to the database instance information; and the speed at which the test data is written to the database instance is adjusted based on the operating status of the database.

[0035] In one optional embodiment of this application, the apparatus further includes:

[0036] The sending unit is used to send the running status of the task execution module and the task execution record to the task management module during the process of executing the task of building the test data.

[0037] In an optional embodiment of this application, the database instance information includes database connection parameter information; before the execution unit executes the test data construction task based on the database instance information and the field generation strategy configuration information, the apparatus further includes:

[0038] The testing unit is used to test the connection information of the database based on the parameter connection information; and to determine whether the connection information can connect to the database instance of the database based on the test results of the connection information.

[0039] In one optional embodiment of this application, the field generation strategy configuration information includes database information, target table information in the database, target table structure information, and test data generation quantity information.

[0040] This application embodiment also provides a data construction apparatus, which is applied to a task management module, and the apparatus includes:

[0041] The acquisition unit is used to acquire a test data construction request; the test data construction request includes database instance information and field generation strategy configuration information.

[0042] The allocation unit is used to allocate a test data construction task to the task execution module based on the test data construction request; wherein, the task execution module is used to execute the test data construction task based on the database instance information and field generation strategy configuration information to construct test data for the database.

[0043] In one optional embodiment of this application, the obtaining unit is specifically used to obtain a test data construction request based on the database instance information and field generation strategy configuration information input by the target object in the information input module.

[0044] In an optional embodiment of this application, before the allocation unit allocates a test data construction task to the task execution module based on the test data construction request, the apparatus further includes:

[0045] The acquisition unit is used to obtain the address information of the task execution module from the registration module;

[0046] Accordingly, the allocation unit is specifically used to: construct a test data construction task allocated to the task execution module based on the test data construction request and the address information.

[0047] In one optional embodiment of this application, the registration module may deploy at least one task execution module; the task execution module is the task execution module whose running state is determined to be available from the at least one task execution module by the registration module.

[0048] In an optional embodiment of this application, after the allocation unit allocates a test data construction task to the task execution module based on the test data construction request, the apparatus further includes:

[0049] A receiving unit is configured to receive the running status and task execution record sent by the task execution module; wherein the running status and the task execution record are sent by the task execution module during the execution of the test data construction task; determine the execution status of the test data construction task based on the running status and the task execution record; and determine whether to continue executing the test data construction task based on the execution status.

[0050] This application also provides an electronic device, which includes a memory and a processor. The memory stores computer-executable instructions, and the processor can implement the data construction method described in the above embodiments when it executes the computer-executable instructions in the memory.

[0051] This application also provides a computer storage medium storing executable instructions, which, when executed by a processor, implement the data construction method described in the above embodiments.

[0052] The technical solution of this application embodiment receives a test data construction task assigned by a task management module through a task execution module. The test data construction task includes database instance information and field generation strategy configuration information. Based on the database instance information and the field generation strategy configuration information, the test data construction task is executed to construct test data for the database. In this way, the configuration work for database information can be reduced by introducing a database instance, and test data for the database can be generated quickly and efficiently by introducing field generation strategy configuration information, providing a stable data foundation for subsequent database performance testing and other related work, thus having good promotional value. Attached Figure Description

[0053] Figure 1 A schematic diagram illustrating the interaction process of each module in the execution data construction method provided in this application embodiment;

[0054] Figure 2 A flowchart illustrating a data construction method provided in this application embodiment. Figure 1 ;

[0055] Figure 3 This is a schematic diagram of the database instance information input interface provided in an embodiment of this application;

[0056] Figure 4 A schematic diagram of the field generation strategy configuration information setting interface provided in this application embodiment;

[0057] Figure 5 This is a schematic diagram illustrating the deployment of the registration center and task executor provided in an embodiment of this application.

[0058] Figure 6This is a schematic diagram illustrating the test data construction process provided in the embodiments of this application;

[0059] Figure 7 This is a schematic diagram illustrating the successful task execution display information provided in an embodiment of this application.

[0060] Figure 8 This is a schematic diagram illustrating a task executor failure as provided in an embodiment of this application.

[0061] Figure 9 A flowchart illustrating a data construction method provided in this application embodiment. Figure 2 ;

[0062] Figure 10 A flowchart illustrating a data construction method provided in this application embodiment. Figure 3 ;

[0063] Figure 11 A schematic diagram of the structural composition of a data construction apparatus provided in this application embodiment. Figure 1 ;

[0064] Figure 12 A schematic diagram of the structural composition of a data construction apparatus provided in this application embodiment. Figure 2 ;

[0065] Figure 13 This is a schematic diagram of the structural composition of an electronic device provided in an embodiment of this application. Detailed Implementation

[0066] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0067] One data extraction method involves four steps: synchronized data configuration, connection parameter verification, execution of data synchronization tasks by the synchronization scheduling engine, and persistent synchronization logs. Synchronized data configuration primarily involves configuring the source database connection information and the database tables and fields to be synchronized to Elasticsearch (ES, a search server based on Lucene). After verification, the database connection information is verified, and the synchronization scheduling engine executes the data extraction and synchronization operations from MySQL to ES. During the data extraction process, the log module is used to persist the execution information of the tasks locally. This method simplifies the complexity of database operations, hides many user operation steps, and significantly improves the user experience. However, the above solution has the following drawbacks:

[0068] 1. In this solution, the process of extracting and synchronizing database data to Elasticsearch is deployed using a singleton. This works normally when the business volume is small. However, as the application scenario becomes more complex, the amount of data to be processed will continue to surge. The performance of the server under the singleton deployment will reach a bottleneck, causing slow data processing and even server paralysis, making it unsuitable for complex business.

[0069] 2. The data synchronization configuration process requires filling in database connection information every time a task is executed, which is cumbersome and has a low user-friendliness. Furthermore, the account password and connection information in the database connection information are not effectively encrypted, making sensitive database information visible to the naked eye at the interface layer, which poses a significant risk to data security.

[0070] 3. There is no checking mechanism for abnormal tasks or tasks that occupy resources for a long time during data processing. Long-term execution may cause abnormal tasks to occupy unnecessary equipment operating resources, reducing the effective utilization rate of equipment operating resources.

[0071] To address the shortcomings of the aforementioned data extraction methods, this application provides a data construction method, specifically a database test data construction method based on simple configuration. Figure 1 This diagram illustrates the interaction process of the modules in the execution data construction method provided in this application embodiment. The method completes the construction of test data for the target table in the database instance through steps such as database instance information input, test target table configuration, test data generation strategy configuration, asynchronous execution of data construction tasks, and persistence of task execution information. The technical solution of this application embodiment can quickly generate and write test data according to user-defined field data generation strategies. This simple configuration enables personalized generation strategies for the field data included in the test data, allowing for rapid and efficient generation of test data for the database, providing a stable data foundation for subsequent database performance testing and other related work.

[0072] The data construction method provided in this application has the following characteristics:

[0073] 1. A multi-instance deployment is adopted to solve the performance bottleneck of single-instance deployment using a distributed approach. Multiple task executors are deployed uniformly, and their Internet Protocol addresses (IP addresses) and forwarding interfaces are registered with the registry center to achieve horizontal scaling of the task executors. When the task manager receives a request to build test data, it will obtain the forwarding information of the currently available task executors from the registry center and perform the task, thereby improving the concurrency of the entire task processing. Multi-instance task executors can effectively solve the performance bottleneck problem in single-machine mode, providing availability and scalability for the entire test data building task processing process.

[0074] 2. Sensitive information transmission between the browser, task manager, and task executor is encrypted using certain encryption methods. Sensitive information in the database connection information is not exposed in the data interface, which can effectively improve the security of the database connection information.

[0075] 3. Employing machine learning methods, the constraints of each field in the database are obtained, and the primary key, foreign key, and other information of the fields are encapsulated as input. Based on the input conditions, the output is restricted to generate a result set of test data fields.

[0076] 4. When writing test data to the database instance, periodically capture the database's running status and adjust the writing speed in real time based on the database's running status to improve data writing efficiency and ensure high efficiency in data writing.

[0077] Figure 1 This is a schematic diagram illustrating the interaction process of each module in the execution data construction method provided in the embodiments of this application, as shown below. Figure 1 As shown, the data construction method provided in this application embodiment is mainly executed through several parts, including a browser, task manager, registry center, task executor, business library, and database instance. Below, we will discuss... Figure 1 The various parts are described below:

[0078] 1. Browser: User operation pages for database instance entry, testing, and building.

[0079] 2. Task Manager: Responsible for recording database instance information, monitoring the execution status of test data build tasks, and distributing test data build tasks. Multiple Task Managers can be deployed in a master-slave mode to achieve high availability and improve the robustness of the entire system.

[0080] 3. The registry center is responsible for heartbeat detection and persistence of task executor IPs and ports. The registry center records the IPs and forwarding port numbers of the task executors. When the Task Manager requests task executor information from the registry center, the registry center can return the IP and port information of an available task executor. The forwarding strategy used by the registry center can be a round-robin strategy, a least-connections strategy, etc.

[0081] 4. The task executor is responsible for the actual execution of database connection tests and test data construction, including two functions: generating test data and writing test data. Horizontal scaling of task execution is achieved by registering new task executors with the registry center, thereby improving task processing capabilities.

[0082] (1) The data generation process uses a machine learning model and combines the database’s own configuration information to generate a result set of test data result fields for the user.

[0083] (2) Monitor the database operation status during the data writing process, dynamically adjust the writing speed to achieve the best database operation status and improve writing efficiency.

[0084] 5. Business database, which records database connection information and execution information of test data building tasks.

[0085] 6. Database instance: The target database instance to be operated on.

[0086] Below, in conjunction with Figure 1 The interaction process of each module in the execution data construction method is shown to illustrate the data construction method of this application embodiment.

[0087] Figure 2 A flowchart illustrating a data construction method provided in this application embodiment. Figure 1 ,like Figure 2 As shown, the data construction method is applied to the task execution module and specifically includes the following steps:

[0088] Step 201: Receive the test data construction task assigned by the task management module; the test data construction task includes database instance information and field generation strategy configuration information.

[0089] In this embodiment, the user can use an information input module to input database instance information. Specifically, this information input module can be... Figure 1 In the browser, entering database instance information in the information input module is mainly for persistent operations on connection parameters such as database type, database version, database login account and password. Users can update the database connection information through the edit button of the database instance information entry, reducing the database information configuration work when building subsequent test data. Figure 3 This is a schematic diagram of the database instance information input interface provided in this application embodiment.

[0090] After entering database instance information using the information input module, users can further open the test data construction page in the information input module and set field generation strategy configuration information on the test data construction page.

[0091] In one optional embodiment of this application, the field generation strategy configuration information includes database information, target table information in the database, target table structure information, and test data generation quantity information.

[0092] Specifically, Figure 4 This is a schematic diagram of a test data construction page provided in an embodiment of this application. On this test data construction page, the user sets field generation strategy configuration information, such as... Figure 4As shown, after entering the test data construction page, the user selects to change the target database and target table, and displays the table structure information of the current target table, including column names, types, constraints and default values. The constraints specifically include information such as whether the field is empty, whether it is auto-incrementing, whether it is a primary key, and whether it is a foreign key.

[0093] In this embodiment, after the user completes the input of database instance information and field generation strategy configuration information in the information input module, the information input module sends the database instance information and field generation strategy configuration information to the task management module, such as... Figure 1 As shown, the task management module can be specifically defined as follows: Figure 1 One or more of Task Manager A and Task Manager B in the system; after receiving the database instance information and field generation strategy configuration information sent by the information input module, the task management module will generate the corresponding test data construction task based on the received information and allocate the test data construction task to the task execution module. Figure 1 In this context, the task execution module can specifically be... Figure 1 One or more of the task executors A, B, and C in the process.

[0094] In this embodiment, to ensure the security of data transmission, sensitive information needs to be encrypted during data transmission between the information input module, task management module, and task scheduling module. In one optional implementation, sensitive information such as database login accounts and passwords can be encrypted using a specific encryption algorithm. For example, the Advanced Encryption Standard (AES) symmetric encryption algorithm can be used to encrypt database login accounts and passwords. This symmetric encryption algorithm provides multiple encryption modes and offers good security, efficiency, and flexibility.

[0095] In this embodiment, after the user completes the input of database instance information and field generation strategy configuration information in the information input module, the information input module can send the database instance information and field generation strategy configuration information to the task management module. In one implementation, the information input module can generate test data construction request information based on the database instance information input by the user. The database login account and password in the test data construction request are encrypted. The encrypted test data construction request information is as follows:

[0096]

[0097] In this embodiment, the number of task management modules is one or more, and the number of task execution modules is also one or more. Each of the multiple task execution modules is uniformly deployed to a registration module, which corresponds to... Figure 1 The registration center in the application registers the IP address and forwarding interface of the task execution module to the registration module. This application utilizes the registration module to achieve horizontal expansion of multiple task execution modules.

[0098] In this embodiment of the application, after the task management module obtains a request to build test data from the information input module, it obtains the forwarding information of the currently available task execution modules from the registration module and allocates the test data construction task to the available task execution modules.

[0099] In this embodiment, for multiple task execution modules uniformly deployed to the registration module, the registration module can perform heartbeat detection for each task execution module and persist the ports of the task execution modules. By recording the IP and forwarding port number of each task execution module, when the task management module requests information about the task execution modules from the registration module, the registration module can return the IP and port information of one or more available task execution modules to the task management module. The forwarding strategy used by the registration center can be a round-robin strategy, a least-connection strategy, etc.

[0100] Specifically, such as Figure 1 As shown, the browser sends database instance information, such as the unique identifier TABLEID, along with field generation strategy configuration information, to Task Manager A via an interface. Task Manager A obtains the connection information for the target table of the current task through TABLEID, including the connection IP, port, connection username and password (encrypted), etc., creates a test data building task, and records information such as the task number TASKID, execution status STATUS (in progress), task creation time CREATETIME, execution log MESSAGE, and field generation strategy. Task Manager A then requests executor information from the registry center to execute the current task.

[0101] Figure 1 In the diagram, the registry center has registered information for three task executors: A, B, and C. The registry center periodically checks the running status of each task executor through heartbeat detection. If all three task executors are in an available state, the registry center will return available task executor information to task manager A according to the forwarding policy.

[0102] Here, the common task forwarding strategies of the task management module are as follows:

[0103] 1. Round-robin method: The requests are distributed in turn to the task executor instances in the registry instance list.

[0104] 2. Random method: The request is randomly assigned to any one of the task executor instances in the registry instance list.

[0105] 3. The least connections method is a dynamic forwarding strategy. When a task executor instance receives a task, the connection count is incremented by 1. When the task executor instance fails, the current instance weight is set to 0, and the registry center will no longer assign tasks to the current task executor instance.

[0106] It should be noted that, in this embodiment of the application, the number of task execution modules deployed in the registration module can be expanded, such as... Figure 5 As shown, when there are many test data construction tasks that need to be executed, and it is necessary to increase the number of task executors, in addition to the already enabled task executors A, B, and C, task executor D can be started and its information registered with the registry center. When task manager A or task manager B requests task executor information from the registry center, it will return the information of one of the four executors, namely task executor A, task executor B, task executor C, and task executor D, according to the forwarding policy.

[0107] In an optional embodiment of this application, the database instance information includes database connection parameter information; before executing step 201 above, the task execution module also specifically executes the following steps:

[0108] The connection information of the database is tested based on the parameter connection information; the database instance of the database is determined based on the test results of the connection information.

[0109] Specifically, before executing the task of building test data, the task execution module needs to determine whether it can connect to the database instance based on the database instance information entered by the user in the information input module, that is, to verify whether the connection information in the database instance information entered by the user in the information input module is valid.

[0110] After receiving the database instance information sent by the task management module, the task execution module concatenates information such as database access IP, access port, useSSL, characterEncoding, serverTimezone, and rewriteBatchedStatements into a connectionURL. Then, it uses the connectionURL and the decrypted username and password to obtain the connection information using DriverManager.

[0111] If no exception is thrown during the task execution module's acquisition of Connection information, it is determined that the connection information provided by the user can connect to the database instance normally. After receiving the message that the task execution module has successfully acquired Connection information, the task management module writes the database instance information into the business database. If an exception is thrown during the task execution module's acquisition of Connection information, it is determined that the database connection information entered by the user cannot connect to the database instance correctly. The task execution module captures the exception information, re-encapsulates it, and returns it to the task management module.

[0112] Step 202: Based on the database instance information and the field generation strategy configuration information, execute the test data construction task to construct test data for the database.

[0113] Figure 6 A schematic diagram illustrating the test data construction process provided in this application embodiment; see also Figure 6 When the task execution module executes the test data construction task, it provides a default field generation strategy. This field generation strategy uses machine learning methods, taking field type, length, constraints (whether it is null, whether it is a primary key, whether it is auto-increment, whether it is a foreign key, etc.), and number of rows to be generated as input parameters, and combining them with the database's own configuration information to generate the result set information for each field.

[0114] If the current target table is the student table, it contains the fields id(bigint, primary key, auto-increment, not null), name(varchar(10), not null), and grade(int, not null, foreign key).

[0115] The grade field is a foreign key, and its value comes from the grade table. The generation of the grade field information must be obtained from the related content of the grade table to ensure the quality of the generated data.

[0116] The name field is varchar(10), not null. For this type of non-foreign key field, the default generation method is to generate a random length of 1 to 10 characters, which is a mix of Chinese and English characters. On this basis, users are provided with multiple industry modes for secondary constraints. For example, if the user selects the e-commerce mode in the default generation strategy, the name field will generate e-commerce product names, such as "TV" or "Sofa". If the user selects the person mode in the default generation strategy, the name field will generate person names, such as "Edison" or "Faraday".

[0117] Based on the default generation method, a CSV file upload is provided for each field, and the range of numerical content of the field is specified by parsing the CSV file of the specified format.

[0118] In an optional embodiment of this application, after constructing the test data for the vehicle against the database, the task execution module further performs the following steps:

[0119] Monitor the operating status of the database; the database is the database corresponding to the database instance information; adjust the speed at which the test data is written to the database instance based on the operating status of the database.

[0120] Specifically, to optimize data writing efficiency, as a concrete implementation method, when the task execution module asynchronously executes the test data construction task, Prometheus (an open-source system and service monitoring system) can be introduced to monitor the database running status information. The buffer pool is a key parameter during database writing. When the task execution module starts writing test data to the database instance, it writes data at an initial speed and monitors the database running status. If the database status is not at full capacity, the task execution module will increase the writing speed of test data to the database instance; conversely, it will decrease the writing speed. This embodiment of the application, by using Prometheus to monitor the database running status in real time, enables each database instance to achieve the ideal writing speed.

[0121] In one optional embodiment of this application, during the process of executing the test data construction task, the task execution module sends the running status of the task execution module and the task execution record to the task management module.

[0122] In this embodiment of the application, after the task execution module successfully executes the test data construction task, it will return task execution information to the task management module, and the task management module will update the execution status and execution information record of the test data construction task. Figure 7 This is a schematic diagram illustrating the successful task execution information provided in an embodiment of this application, wherein the successful task execution information is ultimately displayed on the information input module.

[0123] In this embodiment, the task management module periodically inspects the test data building tasks that are in progress. When a test data building task is in progress for a long time, the module checks the log information of the current test data building task. If the progress of the current test data building task has not been updated for a long time, the module determines that the current test data building task has failed. The task management module then updates the test data building task information and stops the corresponding test data building task.

[0124] In this embodiment of the application, the execution status of the test data construction task mainly includes the following four states:

[0125] 1. In progress: The current test data building task is in progress, and the writing progress is displayed in stages in the task execution information.

[0126] 2. Execution successful. The current test data construction task has been successfully executed. The task execution information displays information such as the total amount of data written and the execution time.

[0127] 3. Execution failed. The current test data building task has been executed, but an exception occurred during the execution process. The task execution information displays the execution progress and exception information.

[0128] 4. Execution timeout: The current test data building task has been executed, but the execution time is too long. This may be due to a fault in the task execution module or other reasons.

[0129] In this embodiment, when a task execution management module fails and cannot provide normal service, the registration module updates the instance information of the failed task management module in the registration module through heartbeat detection. Subsequently, when the task management module requests the registration center to obtain the task execution module information, the information of the failed task management module will no longer be returned.

[0130] like Figure 8 As shown, when task executor A fails and is unable to execute the test data building task, the registry center updates the instance information of task executor A through heartbeat detection and marks the status of task executor A as FAILD. Subsequently, when task manager A or task manager B assigns test data building tasks, the registry center will no longer return relevant information about task executor A to task manager A.

[0131] The technical solution of this application embodiment proposes a distributed, multi-instance, scalable database test data construction method based on simple configuration. It solves the performance bottleneck problem of single-instance service in the original technical solution. By introducing a registration center, the horizontal expansion function of the task execution module is realized. In the field generation configuration, machine learning is used to intelligently generate field values ​​for the database test data based on field information. Furthermore, database running status monitoring is introduced during task execution to adjust the write speed in real time. The task management module performs regular inspections. When the construction task is running for a long time, the current data construction task is abandoned to release running resources.

[0132] Figure 9 A flowchart illustrating a data construction method provided in this application embodiment. Figure 2 ,like Figure 9 As shown, the data construction method is applied to the task management module and specifically includes the following steps:

[0133] Step 901: Obtain test data construction request; the test data construction request includes database instance information and field generation strategy configuration information.

[0134] In an optional embodiment of this application, step 901 specifically includes the following steps: obtaining a test data construction request based on the database instance information and field generation strategy configuration information input by the target object in the information input module.

[0135] In this embodiment, the user can use an information input module to input database instance information. Specifically, this information input module can be... Figure 1 In the browser, entering database instance information in the information input module is mainly for persistent operations on connection parameters such as database type, database version, database login account and password. Users can update the database connection information through the edit button of the database instance information entry, reducing the database information configuration work when building subsequent test data. Figure 3 This is a schematic diagram of the database instance information input interface provided in this application embodiment.

[0136] After entering database instance information using the information input module, users can further open the test data construction page in the information input module and set field generation strategy configuration information on the test data construction page.

[0137] In one optional embodiment of this application, the field generation strategy configuration information includes database information, target table information in the database, target table structure information, and test data generation quantity information.

[0138] Specifically, Figure 4 This is a schematic diagram of a test data construction page provided in an embodiment of this application. On this test data construction page, the user sets field generation strategy configuration information, such as... Figure 4 As shown, after entering the test data construction page, the user selects to change the target database and target table, and displays the table structure information of the current target table, including column names, types, constraints and default values. The constraints specifically include information such as whether the field is empty, whether it is auto-incrementing, whether it is a primary key, and whether it is a foreign key.

[0139] In this embodiment, after the user completes the input of database instance information and field generation strategy configuration information in the information input module, the information input module sends the database instance information and field generation strategy configuration information to the task management module, such as... Figure 1 As shown, the task management module can be specifically defined as follows: Figure 1One or more of Task Manager A and Task Manager B in the system; after receiving the database instance information and field generation strategy configuration information sent by the information input module, the task management module will generate the corresponding test data construction task based on the received information and allocate the test data construction task to the task execution module. Figure 1 In this context, the task execution module can specifically be... Figure 1 One or more of the task executors A, B, and C in the process.

[0140] In this embodiment, to ensure the security of data transmission, sensitive information needs to be encrypted during data transmission between the information input module, task management module, and task scheduling module. In one optional implementation, sensitive information such as database login accounts and passwords can be encrypted using a specific encryption algorithm. For example, the Advanced Encryption Standard (AES) symmetric encryption algorithm can be used to encrypt database login accounts and passwords. This symmetric encryption algorithm provides multiple encryption modes and offers good security, efficiency, and flexibility.

[0141] In this embodiment, after the user completes the input of database instance information and field generation strategy configuration information in the information input module, the information input module can send the database instance information and field generation strategy configuration information to the task management module. In one implementation, the information input module can generate test data construction request information based on the database instance information input by the user. The database login account and password in the test data construction request are encrypted. The encrypted test data construction request information is as follows:

[0142]

[0143] Step 902: Based on the test data, a request is made to allocate a test data construction task to the task execution module, so that the task execution module can execute the test data construction task based on the database instance information and field generation strategy configuration information to construct test data for the database.

[0144] In an optional embodiment of this application, the task management module needs to perform the following steps before executing step 902 above:

[0145] Obtain the address information of the task execution module from the registration module;

[0146] Accordingly, step 902 above specifically refers to: constructing a test data construction task allocated to the task execution module based on the test data construction request and the address information.

[0147] In one optional embodiment of this application, the registration module may deploy at least one task execution module; the task execution module is the task execution module whose running state is determined to be available from the at least one task execution module by the registration module.

[0148] In this embodiment, the number of task management modules is one or more, and the number of task execution modules is also one or more. Each of the multiple task execution modules is uniformly deployed to a registration module, which corresponds to... Figure 1 The registration center in the application registers the IP address and forwarding interface of the task execution module to the registration module. This application utilizes the registration module to achieve horizontal expansion of multiple task execution modules.

[0149] In this embodiment of the application, after the task management module obtains a request to build test data from the information input module, it obtains the forwarding information of the currently available task execution modules from the registration module and allocates the test data construction task to the available task execution modules.

[0150] In this embodiment, for multiple task execution modules uniformly deployed to the registration module, the registration module can perform heartbeat detection for each task execution module and persist the ports of the task execution modules. By recording the IP and forwarding port number of each task execution module, when the task management module requests information about the task execution modules from the registration module, the registration module can return the IP and port information of one or more available task execution modules to the task management module. The forwarding strategy used by the registration center can be a round-robin strategy, a least-connection strategy, etc.

[0151] Specifically, such as Figure 1 As shown, the browser sends database instance information, such as the unique identifier TABLEID, along with field generation strategy configuration information, to Task Manager A via an interface. Task Manager A obtains the connection information for the target table of the current task through TABLEID, including the connection IP, port, connection username and password (encrypted), etc., creates a test data building task, and records information such as the task number TASKID, execution status STATUS (in progress), task creation time CREATETIME, execution log MESSAGE, and field generation strategy. Task Manager A then requests executor information from the registry center to execute the current task.

[0152] Figure 1 In the diagram, the registry center has registered information for three task executors: A, B, and C. The registry center periodically checks the running status of each task executor through heartbeat detection. All three task executors are in an available state. The registry center will return available task executor information to task manager A according to the forwarding policy.

[0153] Here, the common task forwarding strategies of the task management module are as follows:

[0154] 1. Round-robin method: The requests are distributed in turn to the task executor instances in the registry instance list.

[0155] 2. Random method: The request is randomly assigned to any one of the task executor instances in the registry instance list.

[0156] 3. The least connections method is a dynamic forwarding strategy. When a task executor instance receives a task, the connection count is incremented by 1. When the task executor instance fails, the current instance weight is set to 0, and the registry center will no longer assign tasks to the current task executor instance.

[0157] It should be noted that, in this embodiment of the application, the number of task execution modules deployed in the registration module can be expanded, such as... Figure 5 As shown, when there are many test data construction tasks that need to be executed, and it is necessary to increase the number of task executors, in addition to the already enabled task executors A, B, and C, task executor D can be started and its information registered with the registry center. When task manager A or task manager B requests task executor information from the registry center, it will return the information of one of the four executors, namely task executor A, task executor B, task executor C, and task executor D, according to the forwarding policy.

[0158] In an optional embodiment of this application, after performing step 902 above, the task management module further performs the following steps:

[0159] The system receives the running status and task execution record sent by the task execution module; wherein the running status and task execution record are sent by the task execution module during the execution of the test data construction task;

[0160] The execution status of the task for building the test data is determined based on the running status and task execution records.

[0161] Based on the execution status, determine whether to continue executing the test data construction task.

[0162] In this embodiment of the application, after the task execution module successfully executes the test data construction task, it will return task execution information to the task management module, and the task management module will update the execution status and execution information record of the test data construction task. Figure 7 This is a schematic diagram illustrating the successful task execution information provided in an embodiment of this application, wherein the successful task execution information is ultimately displayed on the information input module.

[0163] In this embodiment, the task management module periodically inspects the test data building tasks that are in progress. When a test data building task is in progress for a long time, the module checks the log information of the current test data building task. If the progress of the current test data building task has not been updated for a long time, the module determines that the current test data building task has failed. The task management module then updates the test data building task information and stops the corresponding test data building task.

[0164] In this embodiment of the application, the execution status of the test data construction task mainly includes the following four states:

[0165] 1. In progress: The current test data building task is in progress, and the writing progress is displayed in stages in the task execution information.

[0166] 2. Execution successful. The current test data construction task has been successfully executed. The task execution information displays information such as the total amount of data written and the execution time.

[0167] 3. Execution failed. The current test data building task has been executed, but an exception occurred during the execution process. The task execution information displays the execution progress and exception information.

[0168] 4. Execution timeout: The current test data building task has been executed, but the execution time is too long. This may be due to a fault in the task execution module or other reasons.

[0169] In this embodiment, when a task execution management module fails and cannot provide normal service, the registration module updates the instance information of the failed task management module in the registration module through heartbeat detection. Subsequently, when the task management module requests the registration center to obtain the task execution module information, the information of the failed task management module will no longer be returned.

[0170] like Figure 8 As shown, when task executor A fails and is unable to execute the test data building task, the registry center updates the instance information of task executor A through heartbeat detection and marks the status of task executor A as FAILD. Subsequently, when task manager A or task manager B assigns test data building tasks, the registry center will no longer return relevant information about task executor A to task manager A.

[0171] The technical solution of this application embodiment proposes a distributed, multi-instance, scalable database test data construction method based on simple configuration. It solves the performance bottleneck problem of single-instance service in the original technical solution. By introducing a registration center, the horizontal expansion function of the task execution module is realized. In the field generation configuration, machine learning is used to intelligently generate field values ​​for the database test data based on field information. Furthermore, database running status monitoring is introduced during task execution to adjust the write speed in real time. The task management module performs regular inspections. When the construction task is running for a long time, the current data construction task is abandoned to release running resources.

[0172] Figure 10 A flowchart illustrating a data construction method provided in this application embodiment. Figure 3 The data construction method specifically includes the following steps:

[0173] Step 1001: Enter database instance information.

[0174] User through Figure 1 The browser in the browser is used to enter database instance information.

[0175] Step 1002: Determine if the connection information is valid.

[0176] The database instance information includes the connection information of the database to be tested. After obtaining the database instance information, the Task Manager further encrypts the connection information and sends it to the Task Executor, which verifies whether the connection information is valid.

[0177] Step 1003: Update connection information.

[0178] If the task executor verifies that there is an abnormality in the connection information, the task executor will return the connection information abnormality to the task manager, and the user can then modify the entered database instance information in the browser.

[0179] Step 1004: Configure the target database table information for test data construction.

[0180] Step 1005: Configure the strategy for building and generating test data for database table fields.

[0181] The target database table is the target database or target table to be tested. Steps 1004 and 1005 are both configured by the user by setting the relevant fields in the test data construction page of the browser to generate strategy configuration information.

[0182] Step 1006: Asynchronously execute the test data construction task.

[0183] The Task Manager generates test data building tasks based on the database instance information and field generation strategy configuration information entered by the user in the browser, and assigns different test data building tasks to each task executor.

[0184] Step 1007: Record the execution process and synchronize task execution information.

[0185] After successfully executing the test data construction task, the task execution module will return task execution information to the task management module, which will then update the execution status and execution information records of the test data construction task.

[0186] In the technical solution of this application embodiment, by entering database instance information, configuring target table information, configuring field test data generation strategy, asynchronously executing data construction tasks, and persisting task execution information, test data generation and writing can be completed quickly according to the user-defined field data generation strategy, providing stable data assurance for subsequent database performance testing and other work.

[0187] This application embodiment also provides a data construction apparatus, which is applied to a task execution module, and the apparatus includes:

[0188] The receiving unit 1101 is used to receive the test data construction task assigned by the task management module; the test data construction task includes database instance information and field generation strategy configuration information.

[0189] The execution unit 1102 is used to execute the test data construction task based on the database instance information and the field generation strategy configuration information, and to construct test data for the database.

[0190] In one optional embodiment of this application, the apparatus further includes:

[0191] The monitoring unit 1103 is used to monitor the operating status of the database; the database is the database corresponding to the database instance information; and the speed at which the test data is written to the database instance is adjusted based on the operating status of the database.

[0192] In one optional embodiment of this application, the apparatus further includes:

[0193] The sending unit 1104 is used to send the running status of the task execution module and the task execution record to the task management module during the process of executing the task of building the test data.

[0194] In an optional embodiment of this application, the database instance information includes database connection parameter information; before the execution unit executes the test data construction task based on the database instance information and the field generation strategy configuration information, the apparatus further includes:

[0195] Test unit 1105 is used to test the connection information of the database based on the parameter connection information;

[0196] Based on the test results of the connection information, it is determined whether the connection information can connect to the database instance of the database.

[0197] In one optional embodiment of this application, the field generation strategy configuration information includes database information, target table information in the database, target table structure information, and test data generation quantity information.

[0198] Those skilled in the art should understand that Figure 11 The functions of each unit in the data construction device shown can be understood by referring to the relevant description of the aforementioned data construction method. Figure 11 The functions of each unit in the data construction device shown can be implemented by a program running on a processor or by specific logic circuits.

[0199] This application embodiment also provides a data construction apparatus, which is applied to a task management module, and the apparatus includes:

[0200] The obtaining unit 1201 is used to obtain a test data construction request; the test data construction request includes database instance information and field generation strategy configuration information.

[0201] The allocation unit 1202 is used to allocate a test data construction task to the task execution module based on the test data construction request; wherein, the task execution module is used to execute the test data construction task based on the database instance information and field generation strategy configuration information to construct test data for the database.

[0202] In one optional embodiment of this application, the obtaining unit 1201 is specifically used to obtain a test data construction request based on the database instance information and field generation strategy configuration information input by the target object in the information input module.

[0203] In an optional embodiment of this application, before the allocation unit allocates a test data construction task to the task execution module based on the test data construction request, the apparatus further includes:

[0204] The acquisition unit 1203 is used to obtain the address information of the task execution module from the registration module;

[0205] Accordingly, the allocation unit 1202 is specifically used to: construct a test data construction task allocated to the task execution module based on the test data construction request and the address information.

[0206] In one optional embodiment of this application, the registration module may deploy at least one task execution module; the task execution module is the task execution module whose running state is determined to be available from the at least one task execution module by the registration module.

[0207] In an optional embodiment of this application, after the allocation unit 1202 allocates a test data construction task to the task execution module based on the test data construction request, the apparatus further includes:

[0208] The receiving unit 1204 is configured to receive the running status and task execution record sent by the task execution module; wherein the running status and the task execution record are sent by the task execution module during the execution of the test data construction task; the execution status of the test data construction task is determined based on the running status and the task execution record; and the decision on whether to continue executing the test data construction task is made based on the execution status.

[0209] Those skilled in the art should understand that Figure 12 The functions of each unit in the data construction device shown can be understood by referring to the relevant description of the aforementioned data construction method. Figure 12 The functions of each unit in the data construction device shown can be implemented by a program running on a processor or by specific logic circuits.

[0210] This application also provides an electronic device. Figure 13 This is a schematic diagram of the hardware structure of the electronic device according to an embodiment of this application, such as... Figure 13 As shown, the electronic device includes: a communication component 1303 for data transmission, at least one processor 1301, and a memory 1302 for storing computer programs capable of running on the processor 1301. The various components in the terminal are coupled together via a bus system 1304. It is understood that the bus system 1304 is used to implement communication between these components. In addition to a data bus, the bus system 1304 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 13 The general designated all buses as Bus System 1304.

[0211] Wherein, when the processor 1301 executes the computer program, it executes at least Figure 1 The steps of the method shown.

[0212] It is understood that memory 1302 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 1302 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0213] The methods disclosed in the embodiments of this application can be applied to processor 1301, or implemented by processor 1301. Processor 1301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 1301 or by instructions in the form of software. The processor 1301 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 1301 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 1302. Processor 1301 reads the information in memory 1302 and completes the steps of the aforementioned method in combination with its hardware.

[0214] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned call recording method.

[0215] This application also provides a computer-readable storage medium storing a computer program thereon, characterized in that the program, when executed by a processor, is at least used to perform... Figure 1 The steps of the method are shown. The computer-readable storage medium may specifically be a memory. The memory may be, for example... Figure 13 The memory 1302 shown.

[0216] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0217] In the several embodiments provided in this application, it should be understood that the disclosed methods and smart devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0218] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0219] In addition, each functional unit in the various embodiments of this application can be integrated into a second processing unit, or each unit can be a separate unit, or two or more units can be integrated into a unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0220] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data construction method, characterized in that, The method is applied to a task execution module, and the method includes: Receive test data construction tasks assigned by the task management module; the test data construction tasks include database instance information and field generation strategy configuration information; Based on the database instance information and the field generation strategy configuration information, the test data construction task is executed to construct test data for the database; Monitor the operating status of the database; the database is the database corresponding to the database instance information. Adjust the speed at which the test data is written to the database instance of the database based on the database's operating status; The database instance information includes database connection parameter information; before executing the test data construction task based on the database instance information and the field generation strategy configuration information, the method further includes: Test the database connection information based on the aforementioned parameter connection information; Based on the test results of the connection information, it is determined whether the connection information can connect to the database instance of the database.

2. The method according to claim 1, characterized in that, The method further includes: During the process of constructing the test data, the running status of the task execution module and the task execution record are sent to the task management module.

3. The method according to any one of claims 1 to 2, characterized in that, The field generation strategy configuration information includes database information, target table information in the database, target table structure information, and the amount of test data generated.

4. A data construction method, characterized in that, The method is applied to the task management module, and the method includes: Obtain a test data construction request; the test data construction request includes database instance information and field generation strategy configuration information; the database instance information includes database connection parameter information; the database connection parameter information is used to test the database connection information, and based on the test results of the connection information, determine whether the connection information can connect to the database instance; Based on the test data, a request is made to allocate a test data construction task to the task execution module, so that the task execution module can execute the test data construction task based on the database instance information and the field generation strategy configuration information to construct test data for the database. After allocating a test data construction task to the task execution module based on the test data construction request, the method further includes: The system receives the running status and task execution record sent by the task execution module; wherein the running status and task execution record are sent by the task execution module during the execution of the test data construction task; The execution status of the task for building the test data is determined based on the running status and task execution records. Based on the execution status, determine whether to continue executing the test data construction task.

5. The method according to claim 4, characterized in that, The request to obtain test data includes: Based on the database instance information and field generation strategy configuration information input by the target object in the information input module, a test data construction request is obtained.

6. The method according to claim 4, characterized in that, Before allocating a test data construction task to the task execution module based on the test data construction request, the method further includes: Obtain the address information of the task execution module from the registration module; Accordingly, the construction task of allocating test data to the task execution module based on the test data construction request includes: Based on the test data, a construction request is made, and the address information is used to allocate a test data construction task to the task execution module.

7. The method according to claim 6, characterized in that, The registration module may deploy at least one task execution module; the task execution module is the task execution module whose running state is determined to be available from the at least one task execution module by the registration module.

8. A data construction apparatus, characterized in that, The device is used in a task execution module, and the device includes: The receiving unit is used to receive the test data construction task assigned by the task management module; the test data construction task includes database instance information and field generation strategy configuration information. An execution unit is used to execute the test data construction task based on the database instance information and the field generation strategy configuration information, and to construct test data for the database; The device is used to monitor the operating status of the database; the database is the database corresponding to the database instance information. Adjust the speed at which the test data is written to the database instance of the database based on the database's operating status; The database instance information includes database connection parameter information; before executing the test data construction task based on the database instance information and the field generation strategy configuration information, the device is further configured to: Test the database connection information based on the aforementioned parameter connection information; Based on the test results of the connection information, it is determined whether the connection information can connect to the database instance of the database.

9. A data construction apparatus, characterized in that, The device is used in a task management module, and the device includes: The acquisition unit is used to acquire a test data construction request; the test data construction request includes database instance information and field generation strategy configuration information; the database instance information includes database connection parameter information; the database connection parameter information is used to test the database connection information, and based on the test results of the connection information, it is determined whether the connection information can connect to the database instance of the database; An allocation unit is used to allocate a test data construction task to the task execution module based on the test data construction request; wherein, the task execution module is used to execute the test data construction task based on the database instance information and the field generation strategy configuration information to construct test data for the database; After the device allocates a test data construction task to the task execution module based on the test data construction request, the device is further configured to: The system receives the running status and task execution record sent by the task execution module; wherein the running status and task execution record are sent by the task execution module during the execution of the test data construction task; The execution status of the task for building the test data is determined based on the running status and task execution records. Based on the execution status, determine whether to continue executing the test data construction task.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory stores computer-executable instructions, and the processor, when executing the computer-executable instructions in the memory, can implement the method as claimed in claims 1 to 3, or as claimed in any one of claims 4 to 7.

11. A computer storage medium, characterized in that, The storage medium stores executable instructions that, when executed by a processor, implement the method as claimed in claims 1 to 3, or as claimed in any one of claims 4 to 7.

Images