Test data generation method, device, test data generation equipment and storage medium

Abstract

The present invention discloses a test data generation method, apparatus, test data generation equipment and storage medium, including: creating an iterator, a generator, a field processor and a writer through configuration information, which can realize the description of the scale, type, format, distribution law and constraints between data of the expected test data to be generated through the configuration information, and the corresponding iterator, generator, field processor and writer can be configured through the configuration information to generate test data that meets the expectations, so that after the user configures the information of the generated test data on the client, the test data covering the user's expected scenarios can be generated on the server, and the test data generation efficiency is improved. In addition, by modifying the configuration information, test data for different business needs and usage scenarios can be generated, and the adaptability is strong.

Description

Technical Field

[0001] The present invention relates to the technical field of software testing, and in particular to a test data generating method, apparatus, test data generating device and storage medium. Background Art

[0002] After software-related products are developed, they need to be tested to test their performance and discover vulnerabilities in the software. When testing the software, test data is usually required to simulate the software's execution environment.

[0003] Taking the development of urban transportation vehicle big data products as an example, testing requires massive amounts of captured data on motor vehicles, non-motor vehicles, and driver and passenger faces. This data must be correlated and have strong temporal and locational sequencing requirements. Certain tactics and strategies require that the test data be distributed in a regular pattern, such as for hidden vehicles, vehicles that are active at night, frequent entries and exits, first-time urban residents, destinations, and accompanying vehicles. These data requirements and constraints make test data generation difficult. Conventional random data generation tools cannot generate data that covers these scenarios, and manually generating data piece by piece is too laborious. Summary of the Invention

[0004] The present invention provides a test data generation method, apparatus, test data generation device and storage medium to solve the problem that randomly generated test data cannot cover product scenarios and manually generated test data is inefficient and low-cost.

[0005] In a first aspect, the present invention provides a test data generating method, comprising:

[0006] Upon receiving an execution instruction for a data generation task, obtaining configuration information of the data generation task, the configuration information including configuration information of an iterator, a generator, a field processor, and a writer;

[0007] Creating an iterator according to the configuration information of the iterator, and running the iterator to obtain iteration result data;

[0008] Creating a generator and a row data generation pipeline according to the configuration information of the generator, inputting the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorting the execution order of the field processors according to the configuration information of the field processors;

[0009] Creating a field processor according to the configuration information of the field processor, wherein the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline;

[0010] The row data generation pipeline assembles the field result set into test data and returns it to the generator, and the generator stores the test data in a storage queue;

[0011] A writer is created according to the configuration information of the writer, and the test data in the storage queue is written into a target data source through the writer.

[0012] In a second aspect, the present invention provides a test data generating device, comprising:

[0013] A configuration information acquisition module is used to acquire the configuration information of the data generation task when receiving the execution instruction of the data generation task, wherein the configuration information includes the configuration information of the iterator, generator, field processor and writer;

[0014] A data iteration module, configured to create an iterator according to the configuration information of the iterator, and run the iterator to obtain iteration result data;

[0015] A row data generation pipeline initialization module is used to create a generator and a row data generation pipeline according to the configuration information of the generator, input the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors;

[0016] A field processing module, configured to create a field processor according to the configuration information of the field processor, wherein the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline;

[0017] A data generation module, configured for the row data generation pipeline to assemble the field result set into test data and return it to the generator, and the generator to store the test data into a storage queue;

[0018] The data writing module is used to create a writer according to the configuration information of the writer, and write the test data in the storage queue into the target data source through the writer.

[0019] In a third aspect, the present invention provides a test data generating device, the test data generating device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the test data generating method according to the first aspect of the present invention.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the test data generation method described in the first aspect of the present invention when executed.

[0024] In a fifth aspect, the present invention provides a computer program product, characterized in that it includes a computer program, and when the computer program is executed by a processor, it implements the test data generation method according to the first aspect.

[0025] The embodiment of the present invention creates an iterator, a generator, a field processor, and a writer through configuration information. After data iteration through the iterator, it can generate iteration result data, input the iteration result data into the generator, create a row data generation pipeline, and determine the execution order of the field processors according to the configuration information of the field processors. After the generator starts the row data generation pipeline, the field processing logic of the field processor processes the iteration result data to obtain a field result set and returns it to the row data generation pipeline. The row data generation pipeline assembles the field result set into test data and returns it to the generator. The generator stores the test data in a storage queue and writes the test data to the target data source through the writer. In this way, the scale, type, format, distribution pattern, and constraints between data of the expected test data can be described through the configuration information. The corresponding iterator, generator, field processor, and writer can be configured through the configuration information to generate test data that meets the expectations. After the user configures the information of the generated test data on the client, test data covering the user's expected scenarios can be generated on the server, while improving the efficiency of test data generation. In addition, by modifying the configuration information, test data for different business needs and usage scenarios can be generated, which has strong adaptability.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 is a flow chart of a test data generating method provided by an embodiment of the present invention;

[0029] Figure 2 It is a flowchart of iterative data obtained by iterative iteration;

[0030] Figure 3 It is a flowchart of the field processor performing field processing to generate a field result set;

[0031] Figure 4 It is the flow chart of writing data by the writer;

[0032] Figure 5 It is a structural diagram of a test data generating device provided by an embodiment of the present invention;

[0033] Figure 6 It is a structural diagram of a test data generating device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0035] The test data generation method provided in this embodiment can be executed by a test data generation device. The test data generation device can be implemented in software and / or hardware. The test data generation device can be composed of two or more physical entities, or a single physical entity. For example, the test data generation device can be a server, or alternatively, a server cluster.

[0036] The test data generating device is installed with at least one type of operating system, wherein the operating system includes but is not limited to Android system, Linux system and Windows system. The test data generating device can install at least one application based on the operating system. The application can be an application that comes with the operating system, or it can be an application downloaded from a third-party device or server. In an embodiment of the present invention, the test data generating device has at least an application that can execute the test data generation method. For ease of understanding, the embodiment of the present invention is described by taking the server as the execution subject of the test data generation method as an example.

[0037] Figure 1This is a flow chart of a test data generation method provided by an embodiment of the present invention. This embodiment is applicable to the case of generating test data. The method can be executed by a test data generation device. The test data generation device can be implemented in the form of hardware and / or software and can be configured in a test data generation device, such as a server. Figure 1 As shown, the test data generation method includes:

[0038] S101. When receiving an execution instruction of a data generation task, obtain configuration information of the data generation task, where the configuration information includes configuration information of an iterator, a generator, a field processor, and a writer.

[0039] In this embodiment, the test data generation task is a task that the server runs and needs to batch generate data that meets the requirements of expected scale, format and data constraints. Taking the generation of vehicle capture data to test the application as an example, the test data generation task can be to generate capture data of 100,000 vehicles. The capture data is distributed in chronological order in the past month, with at least 1 and at most 100 capture data for each vehicle per day, and the capture positions of all vehicle capture data are generated in the order of a given checkpoint list. Of course, it can also be used to generate test data for other business scenarios. This embodiment does not limit the business scenarios of the generated test data.

[0040] Among them, the iterator can be a module that generates data through iterative operations, and the configuration information of the iterator can be information that describes the iterative behavior of the iterator. For example, the configuration information of the iterator can include information such as the iterator type, the starting position of the iteration, the ending position, the iteration step, the number of iterations, the iteration method, and the termination condition. It can also include information such as the data processing stage and the filtering conditions.

[0041] A generator may be a module that generates test data using the iterative data output by an iterator as input. The configuration information of the generator may be information that describes how the iterator generates data. For example, the configuration information of the generator may include information such as the type of the generator, the data storage location, the method of generating data, and the method of storing data.

[0042] A field processor can be a module that processes fields in iterated data and obtains a field result set and returns it to the generator. The configuration information of a field processor can be information that describes the logic of the field processor in performing field processing. For example, the configuration information of a field processor can include information such as a field list, a field dependency list, and a stage processing list.

[0043] The writer can be a module that writes the test data in the storage queue to the target data source. The configuration information of the writer can be information that describes the strategy of the writer to write data. For example, the configuration information of the writer can include writing strategy, stage processing list, target data source and other information.

[0044] In one embodiment, the user can fill in the configuration information of the test data generation task such as iterator (one or more nested), generator, field processor (one or more), writer (one or more) in the interactive interface of the client. The client generates a configuration file in yaml / json / xml format or other allowed parseable format according to the configuration information filled in by the user, and sends the configuration file to the server in the form of a data generation request. The server can receive the data generation request from the client and create a data generation task, determine the parser and use the parser to parse the configuration file to obtain the configuration information of the data generation task, and store the configuration information in the database. This embodiment greatly simplifies the configuration operation of the test data generation task by providing the client with the configuration information for the user to fill in the test data generation task, which is conducive to improving the user experience.

[0045] Specifically, before configuring the test data generation task, a repository for the test data generation task configuration information can be created first. The server responds to the test data generation task creation instruction, creates a test data generation task and assigns a number to the test data generation task. After receiving the configuration file, the server searches for a match in the existing parser list based on the data source of the configuration file (such as local files, network files, configuration strings, etc.) and the configuration content format, automatically scans the parser interface and attempts to parse the configuration file in turn. If the parsing is successful, the configuration information and the assigned number obtained by parsing the configuration file are stored. Otherwise, the server returns a message to the client indicating that there is no matching parser and the parsing configuration failed.

[0046] When an execution instruction of a data generation task is received, the configuration information of the iterator, generator, field processor, and writer of the data generation task can be found in the memory by number.

[0047] S102. Create an iterator according to the configuration information of the iterator, and run the iterator to obtain iteration result data.

[0048] This embodiment can configure at least one iterator. An iterator can nest multiple sub-iterators of different types. Different types of iterators generate different iteration result data. For example, a digital iterator generates iteration results of digital type, a time iterator generates iteration results of time type, a database iterator generates iteration results of rows and records of a query result set, an Excel file iterator generates iteration results of rows and records, and a JSON file iterator generates iteration results of JSON objects. In addition to the iterator type, the configuration information of the iterator can also include information such as the start position, end position, iteration step, number of iterations, iteration method, and termination condition of the iteration, such as Figure 2 As shown, in an optional embodiment, S102 may include the following sub-steps:

[0049] S1021. Create an iterator according to the configuration information of the iterator.

[0050] The server can determine the preset type of the iterator from the configuration information of the iterator, create an iterator of the preset type, and initialize the created iterator according to the iteration information described in the configuration information of the iterator, such as initializing the start position, end position, iteration step, number of iterations, iteration method and termination condition of the iterator. For example, for a digital iterator, the start value, end value, increment value (step) of each iteration and termination condition can be initialized (termination condition must be set when the number of iterations is not limited). A time iterator can initialize the start time and end time of the iteration, the increment value of each iteration and the corresponding unit and termination condition. A database iterator can initialize the data source number, query SQL statement, query SQL parameter list, start page number, number of results per page, termination condition and so on.

[0051] S1022. Run the iterator, and iterate the data in the iterator to obtain the first iteration data.

[0052] For each iterator, an iterative data set may be determined according to the type and configuration information of the iterator, and the iterative data set may be iterated in a loop in the iterator to generate data corresponding to the iterator type as first iterative data.

[0053] For example, the server executes a loop iteration process based on the iterator's configuration information and the iteration logic implemented by the iterator. The iterator may include two interfaces: 1) whether there is a next element; and 2) how to get the value of the next element. Different iterator types implement the actual logic of the interfaces based on their own needs and configuration. The "whether there is a next element" interface determines whether to continue iteration and obtains the actual iteration result value when the next element is found.

[0054] Take the time iterator as an example: the configuration of the initialized time iterator is: datetime, defining the from time (2024-08-01 00:00:00) and to time (2024-08-03 00:00:00), defining the iterator's step value (1) and step unit (days), and first setting the time value of the from configuration to the value of the first iteration when running the iteration. Secondly, judging whether the iteration value is less than or equal to the time value of the to configuration. If it meets the conditions, the iteration continues and returns the current iteration value. Then, the step time is added to the current iteration value time, and the above judgment and increase of the step time are continued until the increased iteration time is greater than the time value of the to configuration, then the iteration process is jumped out, and finally the iterator generates the following time values: 2024-08-01 00:00:00, 2024-08-02 00:00:00, 2024-08-03 00:00:00.

[0055] S1023. Perform stage processing on the first iterative data to obtain second iterative data after stage processing.

[0056] In an optional embodiment, the configuration information of the iterator may also include a stage processing list, which includes zero or more stage processing processes. In this embodiment, the stage processing may include at least one of the following: 1) reading stage, reading the data set from the specified data source; 2) selection stage, selecting part or all of the data from the current input data set; 3) script stage, using the configured script processing engine and script content or expression to perform script processing on the current input data set; 4) conditional stage, conditional filtering on the current input data set; 5) mapping stage, mapping the current input data set according to the data value; 6) conversion stage, converting the current input data set to a numeric type; 7) writing stage, writing a single field result set to the current input data set; 8) logging stage, printing and outputting the log of the current input data set.

[0057] It should be noted that the stage processing listed in this embodiment is not limited to the examples of the above-mentioned stage processing. Technical personnel in this field can add more types of processing stages according to actual needs, and there is no restriction on the order and number of times each stage appears. Therefore, it can be freely combined according to actual business needs, so as to adapt to various complex scenarios and business logic.

[0058] In this embodiment, when performing stage processing on the first iteration data generated by the iterator, it can be implemented by pipeline operation, wherein the pipeline stage processing steps are as follows:

[0059] S21. Create a pipeline operation instance according to the configuration information of the stage, and create a stage instance list according to the stage information list, and add them to the pipeline operation instance in sequence.

[0060] S22: Start the pipeline operation instance and execute the stage processing in sequence according to the sorting order of the stage instances in the pipeline.

[0061] S23. End the pipeline operation instance and return the execution result of the pipeline operation instance.

[0062] For example, when the last stage in the pipeline is completed, the execution result of this stage is used as the final return result of the entire pipeline.

[0063] Specifically in this embodiment, S1023 may include the following sub-steps:

[0064] S1023-1. Create a phase processing instance for each phase processing in the iterative phase processing list.

[0065] Specifically, the corresponding stage processing instance can be created according to the iteration stage processing list in the iterator's configuration information. Taking the time iterator as an example, its stage processing list includes: 1) log printing processing stage, printing the time value of the current iteration; 2) data selection processing stage, selecting a time value from the iteration result set; 3) data script processing stage, adding 5 minutes to the current iteration time value; 4) log printing processing stage, printing the time value of the current iteration.

[0066] S1023-2. Create an iterative stage processing pipeline, add a stage processing instance to the iterative stage processing pipeline, and use the output data of the previous stage processing instance in the iterative stage pipeline as the input data of the next stage processing instance.

[0067] In this embodiment, the pipeline can include two objects: 1) pipeline; 2) stage, which corresponds to two interfaces: 1) pipeline interface; 2) stage interface. The pipeline interface can include two interfaces: 1) add next stage, which is used to add all the stages to be executed in the pipeline in sequence. An ordered linked list set can be maintained inside the pipeline object to ensure that all added stages can be executed in the expected order; 2) execute pipeline, which is used to start the pipeline and execute each stage in the pipeline in a preset stage order, and output the final pipeline result set, that is, the output result set of the last stage. The stage interface definition includes a stage processing interface, which is used to process the given input data and then output the data. The pipeline is processed collaboratively through the above interfaces and finally obtains the result after multiple stages of processing.

[0068] S1023-3. Start the iterative stage processing pipeline, input the first iterative data into the first stage processing instance in the iterative stage processing pipeline, perform stage processing on the first iterative data in the iterative stage processing pipeline according to the order of the stage processing instances, and obtain the second iterative data after stage processing.

[0069] Specifically in the stage processing of the iterator of this embodiment, after creating an iterative stage processing pipeline and adding a stage processing instance to the iterative stage processing pipeline, the iterative stage processing pipeline can be started, the first iterative data can be input into the first stage processing of the iterative stage processing pipeline, and the second iterative data can be output in the last stage processing. Taking the time iterator as an example, its stage processing list includes: 1) log printing processing stage, printing the time value of the current iteration; 2) data selection processing stage, selecting a time value from the iteration result set; 3) data script processing stage, adding 5 minutes to the current iteration time value; 4) log printing processing stage, printing the time value of the current iteration. If the first iteration data is 2024-08-01 00:00:00, 2024-08-02 00:00:00, 2024-08-03 00:00:00, then the first log printing stage outputs: 2024-08-01 00:00:00, 2024-08-02 00:00:00, 2024-08-03 00:00:00; the second log printing stage outputs: 2024-08-01 00:05:00, 2024-08-02 00:05:00, 2024-08-03 00:05:00, the data output in the second log printing stage is the second iteration data after stage processing.

[0070] S1024. Perform conditional filtering on the second iterative data to obtain iterative data of the iterator.

[0071] This implementation can also perform zero or more conditional screening processes based on the iterative results of the iterator, and perform discarding, retaining, converting, etc. on the results generated by the iterator. Exemplarily, the conditional filtering process of the iterator may include two conditional configurations: 1) a conditional branch list, which includes one or more conditional processing modules; 2) an exception branch, which includes a conditional processing module. The conditional processing module may include two interfaces: 1) a conditional judgment interface; 2) a conditional result processing interface. The conditional judgment interface uses the data set obtained by the iteration as the input value, executes the conditional judgment of the branch, and outputs the logic of whether the branch condition is met; when the branch condition logic is met, it enters the conditional result processing interface, executes the conditional result processing, and decides whether the current iteration should be skipped based on the output result value of the conditional result processing. Taking the time iterator as an example, you can add a conditional branch "when" to the iterator configuration information and configure it to skip iteration processing when the iteration time value is equal to 2024-08-02 00:05:00. Otherwise, configure it to add 5 minutes to the current iteration time value. The final iteration output results are: 2024-08-01 00:10:00 and 2024-08-03 00:10:00.

[0072] In another optional embodiment, S102 may further include the following sub-steps:

[0073] S1025. Perform zero or more iterator nested loop iterations according to the nested iterator configuration information of the iterator.

[0074] Specifically, through configuration information, an iterator can be configured to include zero or one sub-iterators, and a sub-iterator can also include zero or one sub-iterators. This allows nesting of multiple iterators to implement complex data generation logic. The processing method and processing logic of each sub-iterator can be found in S1021-S1024.

[0075] The type of the child iterator can be inconsistent with the type of the upper-level iterator. The iteration results of each iterator can be collected and finally merged into an iteration result set. Take the time iterator as an example: first configure the iterator to: datetime, define the from time (such as: 2024-08-01 00:00:00) and to time (such as: 2024-08-03 00:00:00), define the iterator step value (such as: 1) and step unit (such as: day), and add an iterator child iterator under the above time iterator according to the configuration information. The configuration method of this child iterator is the same as the configuration method of the upper-level iterator. Configure the child iterator iterator to: number, define the from value (such as: 1) and to value (such as: 3), and define the iterator step value (such as: 1). According to the above configuration, the iterator will generate: [2024-08-0100:00:00,1], [2024-08-01 00:00:00,2]、[2024-08-01 00:00:00,3]、[2024-08-02 00:00:00,1]、[2024-08-02 00:00:00,2]、[2024-08-02 00:00:00,3]、[2024-08-03 00:00:00,1]、[2024-08-03 00:00:00,2]、[2024-08-03 00:00:00,3].

[0076] S103. Create a generator and a row data generation pipeline according to the configuration information of the generator, input the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors.

[0077] In this embodiment, when processing the iteration result data output by the iterator, a generator can be created according to the generator's configuration information. The generator starts the row data generation pipeline, and generates a row of data through the row data generation pipeline. The row data generation pipeline is used to cause the field processors to perform field processing on the iteration result data according to the sort order, and then assemble the field result sets obtained by each field processor into a row of test data and store it in the storage queue. In an optional embodiment, S103 includes the following sub-steps:

[0078] S1031. Create a generator and a row data generation pipeline according to the configuration information of the generator.

[0079] In this embodiment, the type of generator can be determined from the generator's configuration information. From the perspective of the generator's operation, it can generally be divided into two categories: 1) synchronous generators, in which the data generation of each row is based on the iteration order of the iterator to generate a row data set; 2) asynchronous generators, in which the data of each row is independent of the iteration order of the iterator and multiple rows of data sets are generated in parallel. From the perspective of the storage of the results generated by the generator, the following are distinguished: 1) internal message queues, which are usually the memory message queues corresponding to the implemented application; 2) external message queues, which may be message queues such as Kafka and RabbitMQ.

[0080] In this embodiment, the generator's configuration information can be used to describe how the generator generates data and how the data is stored. The server can create a generator based on the generator's configuration information. If the running type is configured as a synchronous generator, the generator's own execution thread is used without additional processing; if the running type is configured as an asynchronous generator, the data generation thread pool is initialized using the thread pool configuration information attached to the asynchronous generator. If the storage type is configured as an internal message queue, the application's memory message queue is created and initialized; if the storage type is configured as an external message queue, the connection client of the corresponding message queue is initialized and the connection between the client and the message server is maintained.

[0081] Take the asynchronous generator and memory message queue as an example: configure the generator to: async, define the number of single batch submissions (such as: 10), the number of concurrent tasks (such as: 8), the maximum number of tasks (such as: 16), the number of task queues (such as: 8), etc., and create a generator instance according to the generator's configuration information, and create two thread pools: 1) Generate result collection thread pool, start a data collection task at the same time when creating the generator, collect row data set results in the memory message queue according to the batch submission number or the batch number, and then send the collected results to the writer; 2) Row data set generation thread pool, create a row data set generation pipeline task according to the iteration result, join the row data set generation thread pool and start it, and push the execution result to the memory message queue after each row data set generation pipeline is executed.

[0082] S1032. Determine the order of the field processors according to the field dependency list in the configuration information of the field processors, and determine the execution order of the field processors in the row data generation pipeline according to the order.

[0083] Specifically, the configuration information of the field processor includes a field dependency list, which includes the fields on which the fields depend. By analyzing the field dependency lists in the configuration information of each field processor, the execution order of each field processor is determined. The execution order of the field processors in the row data generation pipeline is determined by the execution order of the field processors, so as to control the execution order of each field processor through the row data generation pipeline. The row data generation pipeline can refer to S21-S23 in the iterator and will not be described in detail here.

[0084] S104: Create a field processor according to the configuration information of the field processor, and the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline.

[0085] In this embodiment, the configuration information of the field processor may include a field number, a list of dependent fields, a stage processing list, etc. The field number is the unique identifier of the field, the dependent field list indicates whether the current field depends on other fields, and can be configured as zero or more, and the stage processing list indicates the stages that the current field needs to perform, and can be configured as one or more.

[0086] The field processor reorders the dependencies between the field lists and ensures that there are no circular dependencies between the fields, that is, the field dependency graph is a directed acyclic graph. After the fields are reordered, the field processor creates a field data generation pipeline in sequence according to the reordered field order, and adds the stage processing in the stage processing list to the field data generation pipeline in sequence, and then starts the field data generation pipeline and executes all stages, such as Figure 3 As shown, in one embodiment, S104 may include the following sub-steps:

[0087] S1041. Create a field processor set according to the configuration information of the field processor.

[0088] This embodiment can configure multiple field processors, so that a set including multiple field processors can be created through the configuration information of the field processors. The field processor set can be returned to the row data generation pipeline to determine the processing logic of the field processor that needs to be executed in the row data generation pipeline.

[0089] S1042. Create a field data generation pipeline for each field processor in the field processor set.

[0090] For each field processor, a field data generation pipeline can be created according to its stage list, and the field data generation pipeline includes various stage processings that the field processor needs to perform.

[0091] In this embodiment, the stage processing may include:

[0092] The data reading phase is used to read data sets from different sources, including external input data sets (such as iterative result sets, dependent field result sets, row output result sets, etc.), constant data sets, randomly generated data sets, file data sets, relational database data sets, non-relational database data sets, etc.;

[0093] The data value selection stage is used to select a specific data set from the input data set;

[0094] In the data value script processing stage, the script processes the input data set and returns the processing result data set;

[0095] In the data value conditional branch processing stage, conditional branch judgment is performed on the corresponding input data set, and then the data set is filtered or reprocessed before returning the result data set;

[0096] In the data value mapping processing stage, the input data set is mapped and transformed, and the result data set is returned;

[0097] During the data value conversion phase, the data type of the input dataset is converted to match the field data type of the target data source configured in the writer, and then the result dataset is returned.

[0098] In the special data processing stage, for special data processing scenarios and requirements, the processing logic of the data stage processor is customized to process the input data set and return the result data set.

[0099] In one embodiment, the stage processing of the field processor may include a reading stage, and the configuration information of the reading stage may include a dataset number, whether the dataset is reusable, zero or more reader parameter sets, a selection strategy for multiple readers, and one or more data readers, wherein:

[0100] Data source number: When the field dataset is configured to be reusable, the unique dataset number of the field dataset;

[0101] Whether the dataset is reusable: Mark whether the dataset is reusable, that is, whether it is a shared dataset resource pool;

[0102] The reader parameter list can be used to configure zero or more parameter values ​​required by the reader. The parameter values ​​can be constant values ​​or computable variable values, such as the result values ​​calculated by script or other methods based on the input data set in the reading phase. The parameter results change with the changes in the input data set in the reading phase.

[0103] Reader selection strategy: When there are two or more data readers, the reading phase is allowed to select a specified reader for data reading using a specified strategy. For example, it may include: equal selection strategy, which uses equal opportunities to select readers in the reader list; weighted selection strategy, which selects readers in the reader list based on the weight configuration information in the reader; special selection strategy, which customizes the reader selection logic for special reader selection scenarios and requirements.

[0104] Data readers allow one or more different types of data readers to read the required data sets from different data sources, including: constant data set readers, which read and parse constant data sets; relational data source readers, which read the required data sets from specified relational data sources; non-relational data source readers, which read the required data sets from specified non-relational data sources; script data set readers, which read the required data sets according to the specified script engine and script content; file readers, which parse the required data sets from a given file; message queue data source readers, which continuously read the required data sets from the specified message queue data source; AI model generation result readers, which obtain the user-described data sets that meet the requirements from the specified AI large model; special data readers, for special data sources, custom data readers read the required data sets from special data sources.

[0105] In one embodiment, the field value selection stage allows the use of different types of data selection strategies, including: a random repeatable selection strategy, which is a strategy for repeatable and random reading of the data set that has been read for the field; a repeatable sequential selection strategy, which is a strategy for repeatable and reading the data set that has been read for the field in the existing order of the data set; a one-time random selection strategy, which is a strategy for a one-time and random reading of the data set that has been read for the field; a one-time sequential selection strategy, which is a strategy for a one-time reading of the data set that has been read for the field in the existing order of the data set; a multiple sequential selection strategy, which is a strategy for reading the data set that has been read for the field multiple times, that is, at least N times and at most M times, and in the existing order of the input data set; a special selection strategy, which is a customized data selection strategy for special selection requirements to select the desired output data set from the input data set.

[0106] In this embodiment, the script processing stage allows the use of different types of data script processing languages, specifically including: script language type, which describes the type definition of the script language; script content, which describes the script content to be executed in the field and the output result set of the script; and a script language parsing engine, which parses and executes the script content in the field configuration and outputs the execution result.

[0107] In this embodiment, the conditional branch processing stage allows the use of zero or more conditional branches, including: a conditional branch list, which describes the data processing method for the input data set under specific conditions; a conditional branch judgment calculation, which performs conditional branch judgment calculation on the input data set in the script processing stage; a conditional branch result calculation, which performs conditional branch result calculation on the input data set or returns a fixed data set in the script processing stage; and an unmatched condition result calculation, which performs conditional branch result calculation on the input data set in the script processing stage for a data set that does not match any conditional branch, or returns a fixed data set or the data set itself.

[0108] In this embodiment, the mapping processing stage allows the use of zero or more value mappings, including: a numerical mapping list, which describes the data processing method for the input data set under specific conditions; numerical calculation of the mapping key, which performs mapping key matching judgment calculation on the input data set in the script processing stage; result value calculation of the mapping value, which performs mapping value result calculation on the input data set or returns a fixed data set in the script processing stage; unmatched mapping result calculation, which performs default data set calculation on the input data set or returns a fixed data set or returns the data set itself for the data set that does not match any mapping in the script processing stage.

[0109] In this embodiment, the numerical type conversion stage allows the use of numerical type converters of different types of data sets, including: numerical converter type, which describes the type of the numerical converter of the data set; source data set numerical type, which constrains the input numerical value of the data set numerical converter; target data set numerical type, which constrains the output numerical value of the data set numerical converter; data set numerical converter, which matches the corresponding numerical converter type according to the source data set numerical type and the target data set numerical type, and then converts the input data set numerical value into the output data set numerical value of the target type.

[0110] This embodiment can select more than one stage to process the field according to actual conditions, and this embodiment does not impose any limitation on this.

[0111] S1043. Add each stage processing in the stage processing list in the configuration information of the field processor to the field data generation pipeline.

[0112] After creating the field data generation pipeline, you can create corresponding stage processing instances based on the stage processing list of the field processor and add them to the field data generation pipeline according to the order of the stage processing.

[0113] S1044: Start the field data generation pipeline, and perform at least stage processing on the iteration result data in the field data generation pipeline to obtain a processed field result set.

[0114] After adding stage processing to the field data generation pipeline, the iterative data generated by the iterator is input to the generator, the generator starts the row data generation pipeline, and the iterative data input is input to each field processor. The field processor starts the field data generation pipeline. In the field data generation pipeline, the iterative data is obtained from other data sources for field reading, selection and other stage processing, and then the field result set is output and returned to the row data pipeline.

[0115] S105. The row data generation pipeline assembles the field result set into test data and returns it to the generator. The generator stores the test data in the storage queue.

[0116] After executing the field processing logic, each field processor in the row data generation pipeline obtains a field result set, assembles the field result set into a row of test data and returns it to the generator, which stores the test data in a pre-configured storage queue.

[0117] For example, taking the above time iterator, value sub-iterator, asynchronous generator, etc. as examples, four field processor configurations can be created:

[0118] 1) The field code is: IT_DATETIME. The stage processing includes: (1) Selection stage (no selection strategy is configured), which means using the default selection strategy to select a value from the input value as the input value of the next field processing stage, such as: [2024-08-01 00:00:00,1]; (2) Script processing stage, the script type is configured as: SPEL, the expression content is to process the output value of the selection stage, obtain the first element value in the value of [2024-08-01 00:00:00,1], format the time, and the output format is: 20240801000000; (3) Selection stage, select another value for the output value of the script processing stage, and the final output value is: 20240801000000;

[0119] 2) The field code is: IT_NUMBER. The stage processing includes: (1) Selection stage (no selection strategy is configured), which means using the default selection strategy to select a value from the input value as the input value of the next field processing stage, such as: [2024-08-01 00:00:00, 1]; (2) Script processing stage, the script type is configured as: SPEL, and then the output value of the selection stage is processed using the expression language to obtain the second element value in the value [2024-08-01 00:00:00, 1], and then output the value; (3) Selection stage, the output value of the script processing stage is selected again, and the final output value is: 1;

[0120] 3) The field code is: CONCAT, and the dependent field list includes: (1) IT_DATETIME; (2) IT_NUMBER. The stage processing includes: (1) script processing stage, the script type is configured as: SPEL, and then the expression language is used to process the output values ​​of the dependent fields IT_DATETIME and IT_NUMBER, that is, 20240801000000 and 1, and the two values ​​are connected with "-", and then the value is output. The final output value is: 20240801000000-1;

[0121] 4) The field code is: CREATE_TIME. The stage processing includes: (1) Reading stage, including the reader list: a) Reader type configuration is: SPEL, the expression content is to output the current time and format it as: yyyy-MM-dd HH:mm:ss, and the final output value is similar to: 2024-10-2209:41:33.

[0122] The final generated row of data example is:

[0123] {"CREATE_TIME":"2024-10-21

[0124] 16:12:43","IT_NUMBER":1,"CONCAT":"20240801000000-1","IT_DATETIME":"20240801000000"}.

[0125] S106 , creating a writer according to the configuration information of the writer, and writing the test data in the storage queue into the target data source through the writer.

[0126] In an optional embodiment, when the data in the storage queue reaches the upper limit or the interval time expires, the server creates a writer to execute the data writing task and write the data in the storage queue to the target data source in a timely manner. In order to more flexibly reprocess the collected data, it is allowed to configure zero or more stage processors to reprocess the data to be written to the data source, such as Figure 4 As shown, in one embodiment, S106 includes the following sub-steps:

[0127] S1061. Create a writer and a write pipeline according to the configuration information of the writer.

[0128] Specifically, a writer and a corresponding writing pipeline can be created according to the configuration information of the writer. In one embodiment, the writer may include: a standard device print writer, which outputs the generated result data set to a standard output device; a relational data source writer, which outputs the generated result data set to a relational data source with a specified data source number; a non-relational data source writer, which outputs the generated result data set to a non-relational data source with a specified data source number; a message queue data source writer, which outputs the generated result data set to a message queue data source with a specified data source number; a file writer, which outputs the generated result data set to a file of a specified type and target path; a special data writer, for a special type of target data source, a custom data writer writes the generated result data set to the special type of target data source.

[0129] Similarly, you can refer to S21-S23 to create the write pipeline of each writer.

[0130] S1062: Add the stage processing in the stage processing list in the configuration information of the writer to the write pipeline.

[0131] After creating a write pipeline, you can create corresponding stage processing instances based on the writer's stage list and add them to the created write pipeline.

[0132] S1063: Start the write pipeline, collect the test data in the storage queue according to the preset write strategy, and perform stage processing on the test data in the write pipeline according to the order of stage processing.

[0133] Specifically, a batch of test data can be collected from the storage queue according to a pre-configured strategy. The strategy can be the batch quantity, time interval, etc. for collecting data. The collected test data is packaged and then written into the pipeline for stage processing to obtain the test data after stage processing.

[0134] Taking the above-mentioned time iterator, value sub-iterator, asynchronous generator, and field processor as examples, the writer's stage processing list includes: (1) Script processing stage, the script type is configured as: SPEL, and the CONCAT field value of each row of the result set is processed using expression language. The expression content is to use "-" in the CONCAT field value and the formatted value of the CREATE_TIME field (the CREATE_TIME field value is formatted as: yyyyMMddHHmmss) for splicing. The test data after the final stage processing is:

[0135] {"CREATE_TIME":"2024-10-21

[0136] 16:12:43","IT_NUMBER":1,"CONCAT":"20240801000000-1-20241021161243","IT_DATETIME":"20240801000000"}.

[0137] S1064. Write the test data after stage processing into a pre-configured target data source.

[0138] Specifically, the writer may configure one or more target data sources for writing and write the test data into the target data sources.

[0139] The embodiment of the present invention creates an iterator, a generator, a field processor, and a writer through configuration information. After data iteration through the iterator, it can generate iteration result data, input the iteration result data into the generator, create a row data generation pipeline, and determine the execution order of the field processors according to the configuration information of the field processors. After the generator starts the row data generation pipeline, the field processing logic of the field processor processes the iteration result data to obtain a field result set and returns it to the row data generation pipeline. The row data generation pipeline assembles the field result set into test data and returns it to the generator. The generator stores the test data in a storage queue and writes the test data to the target data source through the writer. In this way, the scale, type, format, distribution pattern, and constraints between data of the expected test data can be described through the configuration information. The corresponding iterator, generator, field processor, and writer can be configured through the configuration information to generate test data that meets the expectations. After the user configures the information of the generated test data on the client, test data covering the user's expected scenarios can be generated on the server, while improving the efficiency of test data generation. In addition, by modifying the configuration information, test data for different business needs and usage scenarios can be generated, which has strong adaptability.

[0140] Figure 5 A schematic diagram of the structure of a test data generating device provided by an embodiment of the present invention. Figure 5 As shown, the test data generating device includes:

[0141] The configuration information acquisition module 501 is used to acquire the configuration information of the data generation task when receiving the execution instruction of the data generation task, wherein the configuration information includes the configuration information of the iterator, generator, field processor and writer;

[0142] The data iteration module 502 is used to create an iterator according to the configuration information of the iterator, and run the iterator to obtain iteration result data;

[0143] A row data generation pipeline initialization module 503 is configured to create a generator and a row data generation pipeline according to the configuration information of the generator, input the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors;

[0144] A field processing module 504 is configured to create a field processor according to the configuration information of the field processor, wherein the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline;

[0145] A data generation module 505 is used for the row data generation pipeline to assemble the field result set into test data and return it to the generator, and the generator stores the test data in a storage queue;

[0146] The data writing module 506 is configured to create a writer according to the configuration information of the writer, and write the test data in the storage queue into a target data source through the writer.

[0147] Optionally, also include:

[0148] A generation request receiving module is used to receive a data generation request from a client, wherein the data generation request includes a configuration file, which is a file generated after the user configures the iterator, generator, field processor, and writer on the client;

[0149] A parsing module, configured to create a data generation task, determine a parser, and use the parser to parse the configuration file to obtain configuration information of the data generation task;

[0150] The configuration information storage module is used to store the configuration information in a database.

[0151] Optionally, the data iteration module 502 includes:

[0152] An iterator creation unit, configured to create an iterator according to the configuration information of the iterator;

[0153] An iteration unit, configured to run the iterator, and iterate data in the iterator to obtain first iteration data;

[0154] an iterative stage processing unit, configured to perform stage processing on the first iterative data to obtain second iterative data after stage processing;

[0155] The iterative filtering unit is used to perform conditional filtering on the second iterative data to obtain the iterative data of the iterator.

[0156] Optionally, the iterator creation unit includes:

[0157] An iterator creation subunit, configured to determine a preset type of an iterator from configuration information of the iterator and create an iterator of the preset type;

[0158] The iterator initialization subunit is used to initialize the iterator according to the iteration information described in the configuration information of the iterator.

[0159] Optionally, the iteration unit includes:

[0160] an iterative data set determining subunit, configured to determine an iterative data set according to the type and configuration information of the iterator;

[0161] The loop iteration subunit is used to loop iterate the iterative data set in the iterator to generate data corresponding to the type of the iterator as first iterative data.

[0162] Optionally, the configuration information of the iterator includes an iteration phase processing list, and the iteration phase processing unit includes:

[0163] A phase processing instance creation subunit, configured to create a phase processing instance for each phase processing in the iterative phase processing list;

[0164] an iterative stage processing pipeline creation subunit, configured to create an iterative stage processing pipeline, add the stage processing instance to the iterative stage processing pipeline, and use the output data of the previous stage processing instance in the iterative stage processing pipeline as the input data of the next stage processing instance;

[0165] The iterative stage processing pipeline startup subunit is used to start the iterative stage processing pipeline, input the first iterative data into the first stage processing instance in the iterative stage processing pipeline, perform stage processing on the first iterative data in the iterative stage processing pipeline in the order of the stage processing instances, and obtain second iterative data after stage processing.

[0166] Optionally, the row data generation pipeline initialization module 503 includes:

[0167] A generator and row data generation pipeline creation unit, used to create a generator and a row data generation pipeline according to the configuration information of the generator;

[0168] The row data generation pipeline setting unit is used to determine the order of field processors according to the field dependency list in the configuration information of the field processors, and determine the execution order of the field processors in the row data generation pipeline according to the order.

[0169] Optionally, the field processing module 504 includes:

[0170] A field processor set creation unit, configured to create a field processor set according to the configuration information of the field processor;

[0171] A field data generation pipeline creation unit, configured to create a field data generation pipeline for each field processor in the field processor set;

[0172] A field data generation pipeline setting unit, configured to add each stage processing in the stage processing list in the configuration information of the field processor to the field data generation pipeline;

[0173] The field stage processing unit is used to start the field data generation pipeline, and perform stage processing on at least the iterative result data in the field data generation pipeline to obtain a processed field result set.

[0174] Optionally, the data writing module includes:

[0175] A writer and write pipeline creation unit, configured to create a writer and a write pipeline according to configuration information of the writer;

[0176] A write pipeline setting unit, configured to add the stage processing in the stage processing list in the configuration information of the writer to the write pipeline;

[0177] A write pipeline starting unit, configured to start the write pipeline, collect test data in the storage queue according to a preset write strategy, and perform stage processing on the test data in the write pipeline according to a stage processing order;

[0178] The writing unit is used to write the test data processed by the stage into the pre-configured target data source.

[0179] The test data generating device provided in the embodiment of the present invention can execute the test data generating method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0180] Figure 6A schematic diagram of a test data generation device 40 that can be used to implement an embodiment of the present invention is shown. The test data generation device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The test data generation device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0181] like Figure 6 As shown, the test data generation device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., which is communicatively connected to the at least one processor 41. The memory stores a computer program that can be executed by the at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 into the random access memory (RAM) 43. Various programs and data required for the operation of the test data generation device 40 can also be stored in the RAM 43. The processor 41, ROM 42, and RAM 43 are connected to each other via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0182] Multiple components in the test data generation device 40 are connected to an I / O interface 45, including an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a magnetic disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the test data generation device 40 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0183] The processor 41 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors for running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 41 executes the various methods and processes described above, such as the test data generation method.

[0184] In some embodiments, the test data generation method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program can be loaded and / or installed on the test data generation device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the test data generation method described above can be performed. Alternatively, in other embodiments, processor 41 can be configured to perform the test data generation method in any other suitable manner (e.g., by means of firmware).

[0185] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0187] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0188] To provide for user interaction, the systems and techniques described herein can be implemented on a test data generation device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the test data generation device. Other types of devices can also be used to provide for user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0189] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0190] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0191] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0192] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A test data generation method, characterized in that: include: Upon receiving an execution instruction for a data generation task, obtaining configuration information of the data generation task, the configuration information including configuration information of an iterator, a generator, a field processor, and a writer, the configuration information of the iterator including an iterator type, an iteration start position, an iteration end position, an iteration step, an iteration number, an iteration mode, and a termination condition, the configuration information of the generator including a generator type, a data storage location, a data generation mode, and a data storage mode, the configuration information of the field processor including a field list, a field dependency list, and a stage processing list, and the configuration information of the writer including a write strategy, a stage processing list, and a target data source; Creating an iterator according to the configuration information of the iterator, and running the iterator to obtain iteration result data; Creating a generator and a row data generation pipeline according to the configuration information of the generator, inputting the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorting the execution order of the field processors according to the configuration information of the field processors; Creating a field processor according to the configuration information of the field processor, wherein the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline; The row data generation pipeline assembles the field result set into test data and returns it to the generator, and the generator stores the test data in a storage queue; Creating a writer according to the configuration information of the writer, and writing the test data in the storage queue into a target data source through the writer; The step of creating a writer according to the configuration information of the writer and writing the test data in the storage queue to the target data source through the writer includes: Creating a writer and a writing pipeline according to the configuration information of the writer; Adding the stage processing in the stage processing list in the configuration information of the writer to the write pipeline; Starting the write pipeline, collecting the test data in the storage queue according to a preset write strategy, and performing stage processing on the test data in the write pipeline according to the order of stage processing; Write the processed test data into the pre-configured target data source.

2. The test data generation method according to claim 1, wherein: An iterator is created according to the configuration information of the iterator, and the iterator is run to obtain iteration result data, including: Create an iterator according to the configuration information of the iterator; Running the iterator, and performing data iteration in the iterator to obtain first iteration data; Performing stage processing on the first iterative data to obtain stage-processed second iterative data; Conditional filtering is performed on the second iterative data to obtain iterative data of the iterator.

3. The test data generation method according to claim 2, wherein: Creating an iterator according to the configuration information of the iterator, including: Determining a preset type of an iterator from configuration information of the iterator, and creating an iterator of the preset type; Initialize the iterator according to the iteration information described in the configuration information of the iterator; Running the iterator and iterating data in the iterator to obtain first iteration data includes: Determine an iterative data set according to the type and configuration information of the iterator; The iterative data set is iterated in a loop in the iterator to generate data corresponding to the type of the iterator as first iterative data.

4. The test data generation method according to claim 2, wherein: The configuration information of the iterator includes an iterative stage processing list, and performing stage processing on the first iterative data to obtain second iterative data after stage processing includes: Create a phase processing instance for each phase processing in the iterative phase processing list; Creating an iterative stage processing pipeline, adding the stage processing instance to the iterative stage processing pipeline, and using the output data of the previous stage processing instance in the iterative stage processing pipeline as the input data of the next stage processing instance; Start the iterative stage processing pipeline, input the first iterative data into the first stage processing instance in the iterative stage processing pipeline, perform stage processing on the first iterative data in the iterative stage processing pipeline according to the order of the stage processing instances, and obtain second iterative data after stage processing.

5. The test data generation method according to any one of claims 1 to 4, characterized in that: A generator and a row data generation pipeline are created according to the configuration information of the generator, the iteration result data is input into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors, including: Create a generator and row data generation pipeline based on the generator's configuration information; The order of the field processors is determined according to the field dependency list in the configuration information of the field processors, and the execution order of the field processors in the row data generation pipeline is determined according to the order.

6. The test data generating method according to any one of claims 1 to 4, characterized in that: A field processor is created according to the configuration information of the field processor, and the field processor performs at least stage processing on the iteration result data to obtain a processed field result set, including: Create a field processor set according to the configuration information of the field processor; Creating a field data generation pipeline for each field processor in the field processor set; Adding each stage processing in the stage processing list in the configuration information of the field processor to the field data generation pipeline; The field data generation pipeline is started, and in the field data generation pipeline, at least stage processing is performed on the iteration result data to obtain a processed field result set.

7. A test data generating device, characterized in that: include: a configuration information acquisition module, configured to acquire configuration information of the data generation task upon receiving an execution instruction of the data generation task, wherein the configuration information includes configuration information of the iterator, generator, field processor, and writer; the configuration information of the iterator includes the iterator type, the starting position of the iteration, the ending position, the iteration step, the number of iterations, the iteration method, and the termination condition; the configuration information of the generator includes the generator type, the data storage location, the method for generating data, and the method for storing data; the configuration information of the field processor includes the field list, the field dependency list, and the stage processing list; the configuration information of the writer includes the write strategy, the stage processing list, and the target data source; A data iteration module, configured to create an iterator according to the configuration information of the iterator, and run the iterator to obtain iteration result data; A row data generation pipeline initialization module is used to create a generator and a row data generation pipeline according to the configuration information of the generator, input the iteration result data into the generator to start the row data generation pipeline, and the row data generation pipeline sorts the execution order of the field processors according to the configuration information of the field processors; A field processing module, configured to create a field processor according to the configuration information of the field processor, wherein the field processor performs at least stage processing on the iteration result data to obtain a processed field result set and returns it to the row data generation pipeline; A data generation module, configured for the row data generation pipeline to assemble the field result set into test data and return it to the generator, and the generator to store the test data into a storage queue; a data writing module, configured to create a writer according to the configuration information of the writer, and write the test data in the storage queue into a target data source through the writer; Data writing module, including: A writer and write pipeline creation unit, configured to create a writer and a write pipeline according to configuration information of the writer; A write pipeline setting unit, configured to add the stage processing in the stage processing list in the configuration information of the writer to the write pipeline; A write pipeline starting unit, configured to start the write pipeline, collect test data in the storage queue according to a preset write strategy, and perform stage processing on the test data in the write pipeline according to a stage processing order; The writing unit is used to write the test data processed by the stage into the pre-configured target data source.

8. A test data generating device, characterized in that: The test data generating device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the test data generating method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the test data generating method according to any one of claims 1 to 6 when executed.

Images