Test data generation method and device, test data generation equipment and storage medium

Abstract

The invention discloses a test data generation method and device, test data generation equipment and a storage medium, and the method comprises the steps: creating an iterator, a generator, a field processor and a writer through configuration information, and describing the scale, the type, the format, the distribution rule, the constraint between data and the like of test data expected to be generated through the configuration information. Through the configuration information, the corresponding iterator, the generator, the field processor and the writer can be configured to generate the test data meeting the expectation, so that after the user configures the information of the generated test data at the client, the test data covering the scene expected by the user can be generated at the server, the generation efficiency of the test data is improved, and the user experience is improved. In addition, the test data of different service requirements and use scenes can be generated by modifying the configuration information, and the adaptability is high.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a test data generating method, device, test data generating equipment and storage medium. Background Art

[0002] After software-related products are developed, the software needs to be tested to test its performance and discover vulnerabilities in the software. When testing the software, test data is usually required to simulate the software's execution environment.

[0003] Taking the development of vehicle big data related products for urban transportation as an example, a large amount of captured data of motor vehicles, non-motor vehicles, and driver and passenger faces is required as test data during testing. These data have certain correlations and strong time and location sequence requirements. Under certain technical and tactical requirements, the test data needs to be distributed in a certain pattern, such as: hidden vehicles, hiding during the day and going out at night, frequent entry and exit, first entry into the city, foothold, accompanying vehicles, etc. The above data requirements and constraints make it difficult to generate test data. Conventional random data generation tools cannot generate data that can cover the above scenarios, and manual data generation is too laborious. Summary of the invention

[0004] The present invention provides a test data generation method, an apparatus, a test data generation device and a storage medium to solve the problem that randomly generated test data cannot cover product scenarios and that manually generated test data is inefficient and costly.

[0005] In a first aspect, the present invention provides a test data generating method, comprising:

[0006] When receiving an execution instruction of a data generation task, obtaining configuration information of the data generation task, the configuration information including configuration information of an iterator, a generator, a field processor, and a writer;

[0007] Creating an iterator according to the configuration information of the iterator, and running the iterator to obtain iteration result data;

[0008] Creating a generator and a row data generation pipeline according to the configuration information of the generator, inputting the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorting the execution order of the field processors according to the configuration information of the field processors;

[0009] A field processor is created according to the configuration information of the field processor, and the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline;

[0010] The row data generation pipeline assembles the field result set into test data and returns it to the generator, and the generator stores the test data in a storage queue;

[0011] A writer is created according to the configuration information of the writer, and the test data in the storage queue is written into a target data source through the writer.

[0012] In a second aspect, the present invention provides a test data generating device, comprising:

[0013] A configuration information acquisition module, used to acquire configuration information of the data generation task when receiving an execution instruction of the data generation task, wherein the configuration information includes configuration information of an iterator, a generator, a field processor, and a writer;

[0014] A data iteration module, used to create an iterator according to the configuration information of the iterator, and run the iterator to obtain iteration result data;

[0015] A row data generation pipeline initialization module, used to create a generator and a row data generation pipeline according to the configuration information of the generator, input the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors;

[0016] A field processing module, used to create a field processor according to the configuration information of the field processor, wherein the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline;

[0017] A data generation module, used for the row data generation pipeline to assemble the field result set into test data and return it to the generator, and the generator stores the test data in a storage queue;

[0018] The data writing module is used to create a writer according to the configuration information of the writer, and write the test data in the storage queue into the target data source through the writer.

[0019] In a third aspect, the present invention provides a test data generating device, the test data generating device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the test data generating method described in the first aspect of the present invention.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the test data generating method described in the first aspect of the present invention when executed by a processor.

[0024] In a fifth aspect, the present invention provides a computer program product, characterized in that it comprises a computer program, and when the computer program is executed by a processor, it implements the test data generating method according to the first aspect.

[0025] The embodiment of the present invention creates an iterator, a generator, a field processor and a writer through configuration information, can generate iteration result data after data iteration through the iterator, and input the iteration result data into the generator, create a row data generation pipeline and determine the execution order of the field processor according to the configuration information of the field processor. After the generator starts the row data generation pipeline, the field processing logic of the field processor processes the iteration result data to obtain a field result set and returns it to the row data generation pipeline. The row data generation pipeline assembles the field result set into test data and returns it to the generator. The generator stores the test data in a storage queue and writes the test data into a target data source through the writer, so that the scale, type, format, distribution law and constraints between data of the expected test data to be generated can be described through the configuration information. The corresponding iterator, generator, field processor and writer can be configured through the configuration information to generate test data that meets the expectations, so that after the user configures the information of the generated test data on the client, the test data covering the user's expected scenarios can be generated on the server, and the generation efficiency of the test data is improved. In addition, the test data of different business needs and usage scenarios can be generated by modifying the configuration information, and the adaptability is strong.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 is a flow chart of a test data generating method provided by an embodiment of the present invention;

[0029] Figure 2 It is a flowchart of iterative data obtained by iterative iteration;

[0030] Figure 3 It is a flowchart of the field processor performing field processing to generate a field result set;

[0031] Figure 4 It is a flowchart of writing data by the writer;

[0032] Figure 5 is a structural schematic diagram of a test data generating device provided by an embodiment of the present invention;

[0033] Figure 6 It is a structural schematic diagram of a test data generating device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] The test data generation method provided in this embodiment can be executed by a test data generation device, which can be implemented by software and / or hardware, and can be composed of two or more physical entities, or one physical entity. For example, the test data generation device can be a server, or of course a server cluster.

[0036] The test data generating device is installed with at least one type of operating system, wherein the operating system includes but is not limited to Android system, Linux system and Windows system. The test data generating device can install at least one application based on the operating system. The application can be an application that comes with the operating system, or an application downloaded from a third-party device or server. In an embodiment of the present invention, the test data generating device has at least an application that can execute the test data generating method. For ease of understanding, the embodiment of the present invention is described by taking the server as the executor of the test data generating method as an example.

[0037] Figure 1This is a flowchart of a test data generation method provided by an embodiment of the present invention. This embodiment is applicable to the case of generating test data. The method can be executed by a test data generation device. The test data generation device can be implemented in the form of hardware and / or software and can be configured in a test data generation device, such as a server. Figure 1 As shown, the test data generation method includes:

[0038] S101. When receiving an execution instruction of a data generation task, obtain configuration information of the data generation task, where the configuration information includes configuration information of an iterator, a generator, a field processor, and a writer.

[0039] In this embodiment, the test data generation task is a task that the server runs and needs to batch generate data that meets the requirements of expected scale, format and data constraints. Taking the generation of vehicle capture data to test the application as an example, the test data generation task can be to generate capture data of 100,000 vehicles. The capture data is distributed in chronological order in the past month, with at least 1 and at most 100 capture data for each vehicle per day, and the capture positions of all vehicle capture data are generated in the order of a given checkpoint list. Of course, it can also be to generate test data for other business scenarios. This embodiment does not limit the business scenarios of the generated test data.

[0040] Among them, the iterator can be a module that generates data through iterative operations, and the configuration information of the iterator can be information that describes the iterative behavior of the iterator. For example, the configuration information of the iterator can include information such as the iterator type, the starting position of the iteration, the ending position, the iteration step, the number of iterations, the iteration method, and the termination condition. It can also include information such as the data processing stage and filtering conditions.

[0041] A generator may be a module that generates test data using the iterative data output by an iterator as input. The configuration information of the generator may be information that describes the data generation method of the iterator. For example, the configuration information of the generator may include information such as the type of the generator, the data storage location, the method of generating data, and the method of storing data.

[0042] A field processor may be a module that processes fields in iterated data and obtains a field result set and returns it to a generator. The configuration information of a field processor may be information that describes the logic of field processing performed by the field processor. For example, the configuration information of a field processor may include information such as a field list, a field dependency list, and a stage processing list.

[0043] The writer may be a module that writes the test data in the storage queue to the target data source. The configuration information of the writer may be information that describes the strategy of the writer for writing data. For example, the configuration information of the writer may include information such as the write strategy, stage processing list, and target data source.

[0044] In one embodiment, the user can fill in the configuration information of the test data generation task such as iterator (one or more nested), generator, field processor (one or more), writer (one or more) in the interactive interface of the client, and the client generates a configuration file in yaml / json / xml format or other allowed parseable format according to the configuration information filled in by the user, and sends the configuration file to the server in the form of a data generation request. The server can receive the data generation request from the client and create a data generation task, determine the parser and use the parser to parse the configuration file to obtain the configuration information of the data generation task, and store the configuration information in the database. This embodiment greatly simplifies the configuration operation of the test data generation task by providing the client with the configuration information for the user to fill in the test data generation task, which is conducive to improving the user experience.

[0045] Specifically, before configuring the test data generation task, a repository for the test data generation task configuration information can be created first. The server responds to the test data generation task creation instruction, creates a test data generation task and assigns a number to the test data generation task. After receiving the configuration file, the server searches for a match in the existing parser list based on the data source of the configuration file (such as local files, network files, configuration strings, etc.) and the configuration content format, automatically scans the parser interface and attempts to parse the configuration file in turn. If the parsing is successful, the configuration information and the assigned number obtained by parsing the configuration file are stored. Otherwise, the server returns a message to the client indicating that there is no matching parser and the parsing configuration has failed.

[0046] When an execution instruction of a data generation task is received, the configuration information of the iterator, generator, field processor, and writer of the data generation task can be searched in the memory by number.

[0047] S102. Create an iterator according to the configuration information of the iterator, and run the iterator to obtain iteration result data.

[0048] This embodiment can configure at least one iterator. An iterator can nest multiple sub-iterators of different types. Different types of iterators generate different iteration result data. For example, a digital iterator generates digital type iteration results, a time iterator generates time type iteration results, a database iterator generates row record iteration results of a query result set, an Excel file iterator generates row record iteration results, and a JSON file iterator generates JSON object iteration results. In addition to the iterator type, the configuration information of the iterator can also include information such as the start position, end position, iteration step, number of iterations, iteration mode, and termination condition of the iteration, such as Figure 2 As shown, in an optional embodiment, S102 may include the following sub-steps:

[0049] S1021. Create an iterator according to the configuration information of the iterator.

[0050] The server can determine the preset type of the iterator from the configuration information of the iterator, create an iterator of the preset type, and initialize the created iterator according to the iteration information described in the configuration information of the iterator, such as initializing the start position, end position, iteration step, number of iterations, iteration mode, and termination condition of the iterator. For example, for a digital iterator, the start value, end value, increment value (step) of each iteration, and termination condition can be initialized (termination condition must be set when the number of iterations is not limited), etc. The time iterator can initialize the start time and end time of the iteration, the increment value of each iteration and the corresponding unit, as well as the termination condition, etc. The database iterator can initialize the data source number, query SQL statement, query SQL parameter list, start page number, number of results per page, termination condition, etc.

[0051] S1022. Run the iterator, and iterate the data in the iterator to obtain the first iteration data.

[0052] For each iterator, an iteration data set may be determined according to the type and configuration information of the iterator, and the iteration data set may be looped and iterated in the iterator to generate data corresponding to the type of the iterator as the first iteration data.

[0053] Exemplarily, the server performs loop iteration processing according to the configuration information of the iterator and the iteration logic implemented by the iterator. The iterator may include two interfaces: 1) whether there is a next element; 2) obtaining the value of the next element. Different types of iterators implement the actual logic of the interface according to their own needs and configurations, wherein the interface "whether there is a next element" is used to determine whether the current iteration needs to continue, and the actual iteration result value is obtained when there is a next element.

[0054] Take the time iterator as an example: the configuration of the initialized time iterator is: datetime, defining the from time (2024-08-01 00:00:00) and to time (2024-08-03 00:00:00), defining the iterator's step value (1) and step unit (day). When running the iteration, first set the time value configured in from to the value of the first iteration. Then determine whether the iteration value is less than or equal to the time value configured in to. If it meets the condition, continue the iteration and return the current iteration value. Then add the step time to the current iteration value time. Continue the above determination and increase the step time until the increased iteration time is greater than the time value configured in to. Then, the iteration process is exited. Finally, the iterator generates the following time values: 2024-08-01 00:00:00, 2024-08-02 00:00:00, 2024-08-03 00:00:00.

[0055] S1023. Perform stage processing on the first iterative data to obtain second iterative data after stage processing.

[0056] In an optional embodiment, the configuration information of the iterator may also include a stage processing list, which includes zero or more stage processing processes. In this embodiment, the stage processing may include at least one of the following: 1) a reading stage, reading a data set from a specified data source; 2) a selection stage, selecting part or all of the data from the current input data set; 3) a script stage, performing script processing on the current input data set using the configured script processing engine and script content or expression; 4) a conditional stage, performing conditional filtering on the current input data set; 5) a mapping stage, mapping the current input data set according to data values; 6) a conversion stage, converting the current input data set to a numeric type; 7) a writing stage, writing a single field result set to the current input data set; 8) a log stage, printing out a log of the current input data set.

[0057] It should be noted that the stage processing listed in this embodiment is not limited to the above-mentioned stage processing examples. Technical personnel in this field can add more types of processing stages according to actual needs, and there is no restriction on the order and number of times each stage appears. Therefore, they can be freely combined according to actual business needs to adapt to various complex scenarios and business logic.

[0058] In this embodiment, when performing stage processing on the first iteration data generated by the iterator, it can be implemented by pipeline operation, wherein the pipeline stage processing steps are as follows:

[0059] S21. Create a pipeline operation instance according to the configuration information of the stage, and create an instance list of the stage according to the stage information list, and add them to the pipeline operation instance in sequence.

[0060] S22: Start the pipeline operation instance, and execute the stage processing in sequence according to the sorting order of the stage instances in the pipeline.

[0061] S23. End the pipeline operation instance and return the execution result of the pipeline operation instance.

[0062] Exemplarily, when the last stage in the pipeline is executed, the execution result of this stage is used as the final return result of the entire pipeline.

[0063] Specifically in this embodiment, S1023 may include the following sub-steps:

[0064] S1023-1. Create a phase processing instance for each phase processing in the iterative phase processing list.

[0065] Specifically, the corresponding phase processing instance can be created according to the iteration phase processing list in the iterator's configuration information. Taking the time iterator as an example, its phase processing list includes: 1) log printing processing phase, printing the time value of the current iteration; 2) data selection processing phase, selecting a time value from the iteration result set; 3) data script processing phase, adding 5 minutes to the current iteration time value; 4) log printing processing phase, printing the time value of the current iteration.

[0066] S1023-2. Create an iterative stage processing pipeline, add a stage processing instance to the iterative stage processing pipeline, and use the output data of the previous stage processing instance in the iterative stage pipeline as the input data of the next stage processing instance.

[0067] In this embodiment, the pipeline may include two objects: 1) pipeline; 2) stage, corresponding to which there are two interfaces: 1) pipeline interface; 2) stage interface. The pipeline interface may include two interfaces: 1) add next stage, which is used to add all the stages to be executed of the pipeline in sequence. An ordered linked list set can be maintained inside the pipeline object to ensure that all added stages can be executed in the expected order; 2) execute pipeline, which is used to start the pipeline and execute each stage in the pipeline in a preset stage sequence, and output the final pipeline result set, that is, the output result set of the last stage, wherein the stage interface definition includes a stage processing interface, which is used to process the given input data and then output the data. The pipeline is processed through the above-mentioned interface collaboration, and finally obtains the result after multiple stages of processing.

[0068] S1023-3, start the iterative stage processing pipeline, input the first iterative data into the first stage processing instance in the iterative stage processing pipeline, perform stage processing on the first iterative data in the iterative stage processing pipeline according to the order of the stage processing instances, and obtain second iterative data after stage processing.

[0069] Specifically, in the stage processing of the iterator of this embodiment, after creating an iterative stage processing pipeline and adding a stage processing instance to the iterative stage processing pipeline, the iterative stage processing pipeline can be started, the first iterative data is input into the first stage processing of the iterative stage processing pipeline, and the second iterative data is output in the last stage processing. Taking the time iterator as an example, its stage processing list includes: 1) log printing processing stage, printing the time value of the current iteration; 2) data selection processing stage, selecting a time value from the iteration result set; 3) data script processing stage, adding 5 minutes to the current iteration time value; 4) log printing processing stage, printing the time value of the current iteration. If the first iteration data is 2024-08-01 00:00:00, 2024-08-02 00:00:00, 2024-08-03 00:00:00, the first log printing stage outputs: 2024-08-01 00:00:00, 2024-08-02 00:00:00, 2024-08-03 00:00:00; the second log printing stage outputs: 2024-08-01 00:05:00, 2024-08-02 00:05:00, 2024-08-03 00:05:00, the data output in the second log printing phase is the second iteration data after the phase processing.

[0070] S1024. Perform conditional filtering on the second iteration data to obtain iteration data of the iterator.

[0071] This implementation can also perform zero or more conditional screening processes based on the iteration results of the iterator, and perform discarding, retaining, converting, etc. on the results generated by the iterator. Exemplarily, the conditional filtering process of the iterator can include two condition configurations: 1) a conditional branch list, including one or more conditional processing modules; 2) an exception branch, including a conditional processing module. Among them, the conditional processing module can include two interfaces: 1) a conditional judgment interface; 2) a conditional result processing interface. The conditional judgment interface uses the data set obtained by iteration as the input value, executes the conditional judgment of the branch, and outputs the logic of whether the branch condition is met; when the branch condition logic is met, it enters the conditional result processing interface, executes the conditional result processing, and decides whether the current iteration is skipped based on the output result value of the conditional result processing. Take the time iterator as an example: you can add a conditional branch when in the iterator configuration information and configure it to skip the iteration process when the iteration time value is equal to 2024-08-02 00:05:00. Otherwise, it is configured to add 5 minutes to the current iteration time value. The final iteration output results are: 2024-08-01 00:10:00, 2024-08-03 00:10:00.

[0072] In another optional embodiment, S102 may further include the following sub-steps:

[0073] S1025. Perform zero or more iterator nested loop iterations according to the nested iterator configuration information of the iterator.

[0074] Specifically, the configuration information can be used to configure the iterator to include zero or one sub-iterator, and the sub-iterator can also include zero or one sub-iterator, and so on, nesting of multiple iterators can be allowed to implement complex data generation logic. The processing method and processing logic of each sub-iterator can refer to S1021-S1024.

[0075] The type of the sub-iterator can be inconsistent with the type of the previous iterator. The iteration results of each iterator can be collected and finally merged into an iteration result set. Take the time iterator as an example: first configure the iterator to: datetime, define the from time (such as: 2024-08-01 00:00:00) and to time (such as: 2024-08-03 00:00:00), define the iterator step value (such as: 1) and step unit (such as: day), and add an iterator sub-iterator under the above time iterator according to the configuration information. The configuration method of this sub-iterator is the same as the configuration method of the iterator of the previous level. Configure the sub-iterator iterator to: number, define the from value (such as: 1) and to value (such as: 3), and define the iterator step value (such as: 1). According to the above configuration, the iterator will generate: [2024-08-0100:00:00,1], [2024-08-01 00:00:00,2]、[2024-08-01 00:00:00,3]、[2024-08-02 00:00:00,1]、[2024-08-02 00:00:00,2]、[2024-08-02 00:00:00,3]、[2024-08-03 00:00:00,1]、[2024-08-03 00:00:00,2]、[2024-08-03 00:00:00,3].

[0076] S103. Create a generator and a row data generation pipeline according to the configuration information of the generator, input the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors.

[0077] In this embodiment, when processing the iteration result data output by the iterator, a generator can be created according to the configuration information of the generator, and the generator starts the row data generation pipeline, and generates a row of data through the row data generation pipeline. The row data generation pipeline is used to make the field processor perform field processing on the iteration result data according to the sorting, and then assemble the field result sets obtained by each field processor into a row of test data and store them in the storage queue. In an optional embodiment, S103 includes the following sub-steps:

[0078] S1031. Create a generator and a row data generation pipeline according to the configuration information of the generator.

[0079] In this embodiment, the type of generator can be determined from the configuration information of the generator. From the perspective of the operation of the generator, it can be generally divided into two types of generators: 1) synchronous generators, where the data generation of each row generates a row data set in sequence according to the iteration order of the iterator; 2) asynchronous generators, where the data of each row does not depend on the iteration order of the iterator, and multiple rows of data sets are generated in parallel. From the perspective of the storage of the results generated by the generator, it includes: 1) internal message queues, which are usually memory message queues corresponding to the implemented application; 2) external message queues, which may be message queues of Kafka, RabbitMQ, etc.

[0080] In this embodiment, the configuration information of the generator can be used to describe how the generator generates data and how the data is stored. The server can create a generator according to the configuration information of the generator. If the operation type is configured as a synchronous generator, the generator's own execution thread is used without additional processing; if the operation type is configured as an asynchronous generator, the data generation thread pool is initialized using the thread pool configuration information attached to the asynchronous generator. If the storage type is configured as an internal message queue, the application's memory message queue is created and initialized; if the storage type is configured as an external message queue, the corresponding message queue's connection client is initialized and the client's connection with the message server is maintained.

[0081] Take the asynchronous generator and memory message queue as an example: configure the generator as: async, define the number of single batch submissions (such as: 10), the number of concurrent tasks (such as: 8), the maximum number of tasks (such as: 16), the number of task queues (such as: 8), etc., and create a generator instance according to the generator's configuration information, and create two thread pools: 1) Generate result collection thread pool, start a data collection task at the same time when creating the generator, collect row data set results in the memory message queue according to the number of batch submissions or by batch number, and then send the collected results to the writer; 2) Row data set generation thread pool, create a row data set generation pipeline task according to the iteration result, join the row data set generation thread pool and start it, and push the execution result to the memory message queue after each row data set generation pipeline is executed.

[0082] S1032. Determine the order of the field processors according to the field dependency list in the configuration information of the field processors, and determine the execution order of the field processors in the row data generation pipeline according to the order.

[0083] Specifically, the configuration information of the field processor includes a field dependency list, which includes the fields on which the fields depend. The execution order of each field processor is determined by analyzing the field dependency list in the configuration information of each field processor. The execution order of the field processor in the row data generation pipeline is determined by the execution order of the field processor, so as to control the execution order of each field processor through the row data generation pipeline. The row data generation pipeline can refer to S21-S23 in the iterator, which will not be described in detail here.

[0084] S104: Create a field processor according to the configuration information of the field processor, and the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline.

[0085] In this embodiment, the configuration information of the field processor may include a field number, a dependent field list, a stage processing list, etc. The field number is a unique identifier of the field, the dependent field list is whether the current field depends on other fields, and it is allowed to be configured as zero or more, and the stage processing list is the stage that the current field needs to perform, and it is allowed to be configured as one or more.

[0086] The field processor reorders the dependencies between the field lists and ensures that there are no circular dependencies between the fields, that is, the field dependency graph is a directed acyclic graph. After the fields are reordered, the field processor creates a field data generation pipeline in sequence according to the reordered field order, and the stage processing in the stage processing list is added to the field data generation pipeline in sequence, and then the field data generation pipeline is started and all stages are executed, such as Figure 3 As shown, in one embodiment, S104 may include the following sub-steps:

[0087] S1041. Create a field processor set according to the configuration information of the field processor.

[0088] This embodiment can configure multiple field processors, so that a set including multiple field processors can be created through the configuration information of the field processors. The field processor set can be returned to the row data generation pipeline to determine the processing logic of the field processor that needs to be executed in the row data generation pipeline.

[0089] S1042. Create a field data generation pipeline for each field processor in the field processor set.

[0090] For each field processor, a field data generation pipeline can be created according to its stage list, and the field data generation pipeline includes various stage processings that the field processor needs to perform.

[0091] In this embodiment, the stage processing may include:

[0092] The data reading phase is used to read data sets from different sources, including external input data sets (such as iterative result sets, dependent field result sets, row output result sets, etc.), constant data sets, randomly generated data sets, file data sets, relational database data sets, non-relational database data sets, etc.;

[0093] The data value selection stage is used to select a specific data set from the input data set;

[0094] In the data value script processing stage, the input data set is processed by the script and the processing result data set is returned;

[0095] In the data value conditional branch processing stage, conditional branch judgment is performed on the corresponding input data set, and then the data set is filtered or processed again to return the result data set;

[0096] In the data value mapping processing stage, the input data set is mapped and transformed, and the result data set is returned;

[0097] In the data numeric type conversion phase, the numeric type of the input data set is converted to conform to the field data type of the target data source configured in the writer, and then the result data set is returned;

[0098] In the special data processing stage, for special data processing scenarios and requirements, the processing logic of the data stage processor is customized to process the input data set and return the result data set.

[0099] In one embodiment, the stage processing of the field processor may include a reading stage, and the configuration information of the reading stage may include a data set number, whether the data set is reusable, zero or more reader parameter sets, a selection strategy for multiple readers, and one or more data readers, wherein:

[0100] Data source number: when the field data set is configured to be reusable, the unique data set number of the field data set;

[0101] Whether the dataset is reusable: Mark whether the dataset is reusable, that is, whether it is a shared dataset resource pool;

[0102] The reader parameter list can be used to configure zero or more parameter values ​​required by the reader. The parameter value is allowed to be a constant value or a computable variable value, such as the result value calculated by script calculation or other methods based on the input data set in the reading phase. The parameter result changes with the input data set in the reading phase.

[0103] Reader selection strategy: When there are two or more data readers, the reading phase is allowed to select the specified reader for data reading using the specified strategy. For example, it may include: equal selection strategy, using equal opportunities to select readers in the reader list; weighted selection strategy, selecting readers in the reader list based on the weight configuration information in the reader; special selection strategy, customizing the reader selection logic for special reader selection scenarios and requirements.

[0104] Data readers allow one or more different types of data readers to read the required data sets from different data sources, which can specifically include: constant data set readers, which read and parse constant data sets; relational data source readers, which read the required data sets from specified relational data sources; non-relational data source readers, which read the required data sets from specified non-relational data sources; script data set readers, which read the required data sets according to the specified script engine and script content; file readers, which parse the required data sets from a given file; message queue data source readers, which continuously read the required data sets from the specified message queue data source; AI model generation result readers, which obtain the user-described data sets that meet the requirements from the specified AI large model; special data readers, for special data sources, custom data readers read the required data sets from special data sources.

[0105] In one embodiment, the value selection stage of the field allows the use of different types of data selection strategies, including: a random repeatable selection strategy, which is a strategy for repeatable and random reading of the data set that has been read by the field; a repeatable sequential selection strategy, which is a strategy for repeatable and reading the data set that has been read by the field in the existing order of the data set; a one-time random selection strategy, which is a strategy for a one-time and random reading of the data set that has been read by the field; a one-time sequential selection strategy, which is a strategy for a one-time reading of the data set that has been read by the field and in the existing order of the data set; a multiple sequential selection strategy, which is a strategy for reading the data set that has been read by the field multiple times, that is, at least N times and at most M times, and in the existing order of the input data set; a special selection strategy, for special selection requirements, a customized data selection strategy selects the desired output data set from the input data set.

[0106] In this embodiment, the script processing stage allows the use of different types of data script processing languages, specifically, including: script language type, which describes the type definition of the script language; script content, which describes the script content that needs to be executed in the field and the output result set of the script; script language parsing engine, which parses and executes the script content in the field configuration and outputs the execution result.

[0107] In this embodiment, the conditional branch processing stage allows the use of zero or more conditional branches, including: a conditional branch list, which describes the data processing method for the input data set under specific conditions; a conditional branch judgment calculation, which performs conditional branch judgment calculation on the input data set in the script processing stage; a conditional branch result calculation, which performs conditional branch result calculation on the input data set or returns a fixed data set in the script processing stage; an unmatched condition result calculation, which performs conditional branch result calculation on the input data set in the script processing stage for a data set that does not match any conditional branch, or returns a fixed data set or returns the data set itself.

[0108] In this embodiment, the mapping processing stage allows the use of zero or more value mappings, including: a value mapping list, which describes the data processing method for the input data set under specific conditions; a mapping key value calculation, which performs a mapping key matching judgment calculation on the input data set in the script processing stage; a mapping value result value calculation, which performs a mapping value result calculation on the input data set in the script processing stage or returns a fixed data set; an unmatched mapping result calculation, which performs a default data set calculation on the input data set in the script processing stage for a data set that does not match any mapping, or returns a fixed data set or the data set itself.

[0109] In this embodiment, the numerical type conversion stage allows the use of numerical type converters of different types of data sets, including: a numerical converter type, which describes the type of the numerical converter of the data set; a source data set numerical type, which constrains the input numerical value of the data set numerical converter; a target data set numerical type, which constrains the output numerical value of the data set numerical converter; and a data set numerical converter, which matches the corresponding numerical converter type according to the source data set numerical type and the target data set numerical type, and then converts the input data set numerical value into an output data set numerical value of the target type.

[0110] This embodiment can select more than one stage processing to process the field according to actual conditions, and this embodiment does not limit this.

[0111] S1043. Add each stage processing in the stage processing list in the configuration information of the field processor to the field data generation pipeline.

[0112] After creating the field data generation pipeline, you can create corresponding stage processing instances according to the stage processing list of the field processor and add them to the field data generation pipeline according to the order of the stage processing.

[0113] S1044: Start the field data generation pipeline, and perform at least stage processing on the iteration result data in the field data generation pipeline to obtain a processed field result set.

[0114] After adding the stage processing to the field data generation pipeline, the iterative data generated by the iterator is input to the generator, the generator starts the row data generation pipeline, the iterative data input is input to each field processor, the field processor starts the field data generation pipeline, in the field data generation pipeline, the iterative data is obtained from other data sources for field reading, selection and other stage processing, and then the field result set is output and returned to the row data pipeline.

[0115] S105. The row data generation pipeline assembles the field result set into test data and returns it to the generator, and the generator stores the test data in the storage queue.

[0116] Each field processor in the row data generation pipeline executes the field processing logic to obtain a field result set, assembles the field result set into a row of test data and returns it to the generator, which stores the test data in a pre-configured storage queue.

[0117] For example, taking the above time iterator, value sub-iterator, asynchronous generator, etc. as examples, four field processor configurations can be created:

[0118] 1) The field code is: IT_DATETIME. The stage processing includes: (1) Selection stage (no selection strategy is configured), which means that a value is selected from the input value using the default selection strategy as the input value of the next field processing stage, such as: [2024-08-01 00:00:00,1]; (2) Script processing stage, the script type is configured as: SPEL, the expression content is to process the output value of the selection stage, obtain the first element value in the value of [2024-08-01 00:00:00,1], format the time, and the output format is: 20240801000000; (3) Selection stage, select a value again for the output value of the script processing stage, and the final output value is: 20240801000000;

[0119] 2) The field code is: IT_NUMBER. The stage processing includes: (1) Selection stage (no selection strategy is configured), which means that a value is selected from the input value using the default selection strategy as the input value of the next field processing stage, such as: [2024-08-01 00:00:00,1]; (2) Script processing stage, the script type is configured as: SPEL, and then the output value of the selection stage is processed using the expression language to obtain the second element value in the [2024-08-01 00:00:00,1] value, and then output the value; (3) Selection stage, the output value of the script processing stage is selected again, and the final output value is: 1;

[0120] 3) The field code is: CONCAT. The list of dependent fields includes: (1) IT_DATETIME; (2) IT_NUMBER. The stage processing includes: (1) Script processing stage. The script type is configured as: SPEL. Then, the expression language is used to process the output values ​​of the dependent fields IT_DATETIME and IT_NUMBER, that is, 20240801000000 and 1. The two values ​​are connected using "-" and then the value is output. The final output value is: 20240801000000-1.

[0121] 4) The field code is: CREATE_TIME. The stage processing includes: (1) Reading stage, including the reader list: a) The reader type is configured as: SPEL, the expression content is to output the current time and format it as: yyyy-MM-dd HH:mm:ss, and the final output value is similar to: 2024-10-2209:41:33.

[0122] The final generated row of data is as follows:

[0123] {"CREATE_TIME":"2024-10-21

[0124] 16:12:43","IT_NUMBER":1,"CONCAT":"20240801000000-1","IT_DATETIME":"20240801000000"}.

[0125] S106 , creating a writer according to the configuration information of the writer, and writing the test data in the storage queue into the target data source through the writer.

[0126] In an optional embodiment, when the data in the storage queue reaches the upper limit or the interval time expires, the server creates a writer to execute the data writing task and write the data in the storage queue to the target data source in a timely manner. In order to reprocess the collected data more flexibly, it is allowed to configure zero or more stage processors to reprocess the data to be written into the data source, such as Figure 4 As shown, in one embodiment, S106 includes the following sub-steps:

[0127] S1061. Create a writer and a write pipeline according to the configuration information of the writer.

[0128] Specifically, a writer and a corresponding write pipeline can be created according to the configuration information of the writer. In one embodiment, the writer may include: a standard device print writer, which outputs the generated result data set to a standard output device; a relational data source writer, which outputs the generated result data set to a relational data source with a specified data source number; a non-relational data source writer, which outputs the generated result data set to a non-relational data source with a specified data source number; a message queue data source writer, which outputs the generated result data set to a message queue data source with a specified data source number; a file writer, which outputs the generated result data set to a file of a specified type and target path; and a special data writer, for a special type of target data source, a custom data writer writes the generated result data set to a special type of target data source.

[0129] Similarly, you can refer to S21-S23 to create the write pipeline of each writer.

[0130] S1062. Add the stage processing in the stage processing list in the configuration information of the writer to the write pipeline.

[0131] After creating the write pipeline, you can create corresponding stage processing instances based on the stage list of the writer and add them to the created write pipeline.

[0132] S1063, start the write pipeline, collect the test data in the storage queue according to the preset write strategy, and perform stage processing on the test data in the write pipeline according to the order of stage processing.

[0133] Specifically, a batch of test data may be collected from the storage queue according to a pre-configured strategy. The strategy may be to collect data by batch quantity, time interval, etc. The collected test data may be packaged and then processed in stages through a write pipeline to obtain the test data after stage processing.

[0134] Taking the above time iterator, value sub-iterator, asynchronous generator, and field processor as examples, the writer's stage processing list includes: (1) Script processing stage, the script type is configured as: SPEL, and the CONCAT field value of each row of the result set is processed using expression language. The expression content is the value of the CONCAT field using "-" and the value formatted by the CREATE_TIME field (the CREATE_TIME field value is formatted as: yyyyMMddHHmmss) for splicing. The test data after the final stage processing is:

[0135] {"CREATE_TIME":"2024-10-21

[0136] 16:12:43","IT_NUMBER":1,"CONCAT":"20240801000000-1-20241021161243","IT_DATETIME":"20240801000000"}.

[0137] S1064. Write the test data after stage processing into a pre-configured target data source.

[0138] Specifically, the writer may configure one or more target data sources for writing, and write the test data into the target data sources.

[0139] The embodiment of the present invention creates an iterator, a generator, a field processor and a writer through configuration information, can generate iteration result data after data iteration through the iterator, and input the iteration result data into the generator, create a row data generation pipeline and determine the execution order of the field processor according to the configuration information of the field processor. After the generator starts the row data generation pipeline, the field processing logic of the field processor processes the iteration result data to obtain a field result set and returns it to the row data generation pipeline. The row data generation pipeline assembles the field result set into test data and returns it to the generator. The generator stores the test data in a storage queue and writes the test data into a target data source through the writer, so that the scale, type, format, distribution law and constraints between data of the expected test data to be generated can be described through the configuration information. The corresponding iterator, generator, field processor and writer can be configured through the configuration information to generate test data that meets the expectations, so that after the user configures the information of the generated test data on the client, the test data covering the user's expected scenarios can be generated on the server, and the generation efficiency of the test data is improved. In addition, the test data of different business needs and usage scenarios can be generated by modifying the configuration information, and the adaptability is strong.

[0140] Figure 5 A schematic diagram of the structure of a test data generating device provided by an embodiment of the present invention. Figure 5 As shown, the test data generating device comprises:

[0141] The configuration information acquisition module 501 is used to acquire the configuration information of the data generation task when receiving the execution instruction of the data generation task, wherein the configuration information includes the configuration information of the iterator, the generator, the field processor and the writer;

[0142] The data iteration module 502 is used to create an iterator according to the configuration information of the iterator, and run the iterator to obtain iteration result data;

[0143] A row data generation pipeline initialization module 503 is used to create a generator and a row data generation pipeline according to the configuration information of the generator, input the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors;

[0144] A field processing module 504 is used to create a field processor according to the configuration information of the field processor, and the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline;

[0145] A data generation module 505, used for the row data generation pipeline to assemble the field result set into test data and return it to the generator, and the generator stores the test data in a storage queue;

[0146] The data writing module 506 is used to create a writer according to the configuration information of the writer, and write the test data in the storage queue into the target data source through the writer.

[0147] Optionally, also include:

[0148] A generation request receiving module, used for receiving a data generation request from a client, wherein the data generation request includes a configuration file, which is a file generated after a user configures an iterator, a generator, a field processor, and a writer on the client;

[0149] A parsing module, used to create a data generation task, determine a parser and use the parser to parse the configuration file to obtain configuration information of the data generation task;

[0150] The configuration information storage module is used to store the configuration information in a database.

[0151] Optionally, the data iteration module 502 includes:

[0152] An iterator creation unit, used for creating an iterator according to the configuration information of the iterator;

[0153] An iteration unit, used for running the iterator, and performing data iteration in the iterator to obtain first iteration data;

[0154] An iterative stage processing unit, used for performing stage processing on the first iterative data to obtain second iterative data after stage processing;

[0155] An iterative filtering unit is used to perform conditional filtering on the second iterative data to obtain iterative data of the iterator.

[0156] Optionally, the iterator creation unit includes:

[0157] An iterator creation subunit, used to determine a preset type of iterator from the configuration information of the iterator, and create an iterator of the preset type;

[0158] The iterator initialization subunit is used to initialize the iterator according to the iteration information described in the configuration information of the iterator.

[0159] Optionally, the iteration unit includes:

[0160] An iterative data set determining subunit, used to determine an iterative data set according to the type and configuration information of the iterator;

[0161] The loop iteration subunit is used to loop iterate the iteration data set in the iterator to generate data corresponding to the type of the iterator as the first iteration data.

[0162] Optionally, the configuration information of the iterator includes an iteration phase processing list, and the iteration phase processing unit includes:

[0163] A phase processing instance creation subunit, used for creating a phase processing instance for each phase processing in the iterative phase processing list;

[0164] An iterative phase processing pipeline creation subunit, used to create an iterative phase processing pipeline, add the phase processing instance to the iterative phase processing pipeline, and use the output data of the previous phase processing instance in the iterative phase pipeline as the input data of the next phase processing instance;

[0165] The iterative stage processing pipeline startup subunit is used to start the iterative stage processing pipeline, input the first iterative data into the first stage processing instance in the iterative stage processing pipeline, perform stage processing on the first iterative data in the iterative stage processing pipeline according to the order of the stage processing instances, and obtain second iterative data after stage processing.

[0166] Optionally, the row data generation pipeline initialization module 503 includes:

[0167] A generator and row data generation pipeline creation unit, used to create a generator and a row data generation pipeline according to the configuration information of the generator;

[0168] The row data generation pipeline setting unit is used to determine the order of the field processors according to the field dependency list in the configuration information of the field processors, and determine the execution order of the field processors in the row data generation pipeline according to the order.

[0169] Optionally, the field processing module 504 includes:

[0170] A field processor set creation unit, used to create a field processor set according to the configuration information of the field processor;

[0171] A field data generation pipeline creation unit, used to create a field data generation pipeline for each field processor in the field processor set;

[0172] A field data generation pipeline setting unit, used to add each stage processing in the stage processing list in the configuration information of the field processor to the field data generation pipeline;

[0173] The field stage processing unit is used to start the field data generation pipeline, and in the field data generation pipeline at least perform stage processing on the iteration result data to obtain a processed field result set.

[0174] Optionally, the data writing module includes:

[0175] A writer and a write pipeline creation unit, used to create a writer and a write pipeline according to the configuration information of the writer;

[0176] A write pipeline setting unit, used to add the stage processing in the stage processing list in the configuration information of the writer to the write pipeline;

[0177] A write pipeline start unit, used to start the write pipeline, collect the test data in the storage queue according to a preset write strategy, and perform stage processing on the test data in the write pipeline according to the order of stage processing;

[0178] The writing unit is used to write the test data processed by the stage into a pre-configured target data source.

[0179] The test data generating device provided in the embodiment of the present invention can execute the test data generating method provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0180] Figure 6A schematic diagram of a test data generating device 40 that can be used to implement an embodiment of the present invention is shown. The test data generating device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The test data generating device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0181] like Figure 6 As shown, the test data generating device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the test data generating device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0182] A number of components in the test data generating device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the test data generating device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0183] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 41 performs the various methods and processes described above, such as a test data generation method.

[0184] In some embodiments, the test data generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the test data generation device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the test data generation method described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the test data generation method in any other suitable manner (e.g., by means of firmware).

[0185] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0187] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein may be implemented on a test data generating device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) through which a user can provide input to the test data generating device. Other types of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0189] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0190] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0191] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0192] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A test data generation method, characterized in that: include: When receiving an execution instruction of a data generation task, obtaining configuration information of the data generation task, the configuration information including configuration information of an iterator, a generator, a field processor, and a writer; Creating an iterator according to the configuration information of the iterator, and running the iterator to obtain iteration result data; Creating a generator and a row data generation pipeline according to the configuration information of the generator, inputting the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorting the execution order of the field processors according to the configuration information of the field processors; A field processor is created according to the configuration information of the field processor, and the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline; The row data generation pipeline assembles the field result set into test data and returns it to the generator, and the generator stores the test data in a storage queue; A writer is created according to the configuration information of the writer, and the test data in the storage queue is written into a target data source through the writer.

2. The test data generation method according to claim 1, characterized in that: Creating an iterator according to the configuration information of the iterator, and running the iterator to obtain iterative data, including: Create an iterator according to the configuration information of the iterator; Run the iterator, and iterate the data in the iterator to obtain the first iteration data; Performing stage processing on the first iterative data to obtain second iterative data after stage processing; Conditional filtering is performed on the second iterative data to obtain iterative data of the iterator.

3. The test data generating method according to claim 2, characterized in that: Creating an iterator according to the configuration information of the iterator, including: Determine a preset type of an iterator from configuration information of the iterator, and create an iterator of the preset type; Initialize the iterator according to the iteration information described in the configuration information of the iterator; Running the iterator, and performing data iteration in the iterator to obtain first iteration data, including: Determine an iterative data set according to the type and configuration information of the iterator; The iteration data set is iterated in a loop in the iterator to generate data corresponding to the type of the iterator as first iteration data.

4. The test data generating method according to claim 2, characterized in that: The configuration information of the iterator includes an iteration stage processing list, and performing stage processing on the first iteration data to obtain second iteration data after stage processing includes: Create a phase processing instance for each phase processing in the iterative phase processing list; Creating an iterative phase processing pipeline, adding the phase processing instance to the iterative phase processing pipeline, and using the output data of the previous phase processing instance in the iterative phase processing pipeline as the input data of the next phase processing instance; Start the iterative stage processing pipeline, input the first iterative data into the first stage processing instance in the iterative stage processing pipeline, perform stage processing on the first iterative data in the iterative stage processing pipeline according to the order of the stage processing instances, and obtain second iterative data after stage processing.

5. The test data generating method according to any one of claims 1 to 4, characterized in that: A generator and a row data generation pipeline are created according to the configuration information of the generator, the iteration result data is input into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors, including: Create a generator and row data generation pipeline based on the generator's configuration information; The order of the field processors is determined according to the field dependency list in the configuration information of the field processors, and the execution order of the field processors in the row data generation pipeline is determined according to the order.

6. The test data generating method according to any one of claims 1 to 4, characterized in that: A field processor is created according to the configuration information of the field processor, and the field processor at least performs stage processing on the iteration result data to obtain a processed field result set, including: Creating a field processor set according to the configuration information of the field processor; Creating a field data generation pipeline for each field processor in the field processor set; Adding each stage processing in the stage processing list in the configuration information of the field processor to the field data generation pipeline; The field data generation pipeline is started, and in the field data generation pipeline, at least the iteration result data is subjected to stage processing to obtain a processed field result set.

7. The test data generating method according to any one of claims 1 to 4, characterized in that: Creating a writer according to the configuration information of the writer, and writing the test data in the storage queue into the memory through the writer, comprises: Creating a writer and a writing pipeline according to the configuration information of the writer; Adding the stage processing in the stage processing list in the configuration information of the writer to the write pipeline; Starting the write pipeline, collecting the test data in the storage queue according to a preset write strategy, and performing stage processing on the test data in the write pipeline according to the order of stage processing; Write the processed test data to the pre-configured target data source.

8. A test data generating device, characterized in that: include: A configuration information acquisition module, used to acquire configuration information of the data generation task when receiving an execution instruction of the data generation task, wherein the configuration information includes configuration information of an iterator, a generator, a field processor, and a writer; A data iteration module, used to create an iterator according to the configuration information of the iterator, and run the iterator to obtain iteration result data; A row data generation pipeline initialization module, used to create a generator and a row data generation pipeline according to the configuration information of the generator, input the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors; A field processing module, used to create a field processor according to the configuration information of the field processor, wherein the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline; A data generation module, used for the row data generation pipeline to assemble the field result set into test data and return it to the generator, and the generator stores the test data in a storage queue; The data writing module is used to create a writer according to the configuration information of the writer, and write the test data in the storage queue into the target data source through the writer.

9. A test data generating device, characterized in that: The test data generating device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the test data generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the test data generating method according to any one of claims 1 to 7 when executed.

Images