A test data generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a test data generation method and device, electronic equipment and storage medium. The embodiments of the present application can generate a script file based on user-defined configuration information. The configuration information at least includes data fields, logical relationships between fields and data distribution. Based on the script file, simulated structured test data is generated. Based on the structured test data, a SQL script conforming to the syntax of a target database is generated. The SQL script is executed in the target database to obtain test data. Thus, through the end-to-end automation process of "configuration, script generation, data generation, SQL conversion and database execution", an automation pipeline from parameter definition to data landing is formed, which significantly improves the generation efficiency of test data and can provide high-quality test data for subsequent software testing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of software testing, in particular to a test data generation method and device, electronic equipment and storage medium. BACKGROUND

[0002] In the field of software testing, the preparation of test data is a key link to ensure the quality and efficiency of testing. Currently, the generation of test data mainly relies on two mainstream methods: manual construction and semi-automatic tool assistance. However, both methods have significant technical limitations and cannot meet the needs of high-frequency iteration, large-scale testing and complex business scenarios in modern software development.

[0003] Among them, manual creation of test data is usually constructed by test engineers according to requirement documents and business logic. This method allows testers to customize personalized data according to specific needs, but has problems such as low efficiency, high error rate, poor dynamic adaptability, and high labor cost.

[0004] Semi-automatic test data generation tools usually provide graphical configuration interfaces or simple script definition functions, allowing users to generate test data through visual parameter settings (such as field type selection, data level input) or basic scripts (such as regular expression templates, simple logic rules), which reduces the workload of manual operation to some extent. However, it also has problems such as high configuration complexity, which limits the generation speed, and poor flexibility. SUMMARY

[0005] The embodiments of the present application provide a test data generation method, device, electronic equipment and storage medium, which can improve the generation efficiency of test data.

[0006] The embodiments of the present application provide a test data generation method, which comprises:

[0007] Based on the user-defined configuration information, a script file is generated; the configuration information at least includes data fields, logical relationships between fields, and data distribution;

[0008] Based on the script file, simulated structured test data is generated;

[0009] Based on the structured test data, a SQL script conforming to the target database syntax is generated;

[0010] The SQL script is executed in the target database to obtain test data.

[0011] Optionally, based on the script file, simulated structured test data is generated, comprising:

[0012] The script file is parsed by a script parser to convert the script file into an intermediate representation format;

[0013] The simulated structured test data is generated by a data generation engine based on the intermediate representation format.

[0014] Optionally, the data fields include static fields, and the script file is parsed by a script parser to convert the script file into an intermediate representation format, including:

[0015] The script content is analyzed line by line by the script parser to extract key configuration parameters;

[0016] Based on the key configuration parameters, a metadata model of table structure is constructed, and the field composition and constraint rules of each table are determined;

[0017] The simulated structured test data is generated by a data generation engine based on the intermediate representation format, including:

[0018] For the static fields in the metadata model, a random data generation algorithm is called to generate corresponding simulation values;

[0019] The simulation values corresponding to each static field are assembled in the order of the table structure to obtain the simulated structured test data.

[0020] Optionally, after the random data generation algorithm is called to generate the simulation values corresponding to the static fields in the metadata model, the method further includes:

[0021] Based on the logical relationship between the fields, the association between the simulation values of each static field is verified;

[0022] If the simulation values of the static fields violate the logical relationship between the fields, the simulation values of the static fields are processed based on a preset hierarchical processing mechanism.

[0023] Optionally, the simulation values of the static fields are processed based on the preset hierarchical processing mechanism, including:

[0024] If the violation level is level one, the simulation values of the static fields are corrected, and a correction log is recorded;

[0025] If the violation level is level two, the simulation values of the static fields are marked as abnormal data, and a warning report containing abnormal statistics is generated.

[0026] Optionally, based on the structured test data, a SQL script conforming to the syntax of a target database is generated, including:

[0027] Based on the type of the target database, a syntax adaptation rule set corresponding to the target database is obtained;

[0028] mapping table, convert a general type definition of each field in the structured test data into a corresponding specific field type of the target database;

[0029] generate a DDL creation statement and a DML insertion statement based on the structured test data of the specific field type and the syntax adaptation rule set;

[0030] integrate the DDL creation statement and the DML insertion statement in a table structure order to obtain a SQL script conforming to the syntax of the target database.

[0031] Optionally, the data field further includes a dynamic field, and the simulated structured test data is generated based on the script file, including:

[0032] parsing the dynamic field in the script file and obtaining a real-time dynamic parameter value through a predefined interface;

[0033] combining the real-time dynamic parameter value with simulated values of the remaining data fields in a table structure order to generate the simulated structured test data.

[0034] Embodiments of the present application further provide a test data generation device, including:

[0035] a configuration module configured to generate a script file based on user-defined configuration information; the configuration information at least includes data fields, logical relationships between the fields, and data distribution;

[0036] a parsing module configured to generate simulated structured test data based on the script file;

[0037] a script generation module configured to generate a SQL script conforming to the syntax of the target database based on the structured test data;

[0038] a script execution module configured to execute the SQL script in the target database to obtain test data.

[0039] Embodiments of the present application further provide an electronic device including a memory storing a plurality of instructions; a processor loads the instructions from the memory to execute steps in any of the test data generation methods provided by embodiments of the present application.

[0040] Embodiments of the present application further provide a computer readable storage medium storing a plurality of instructions, the instructions being adapted to be loaded by a processor to execute steps in any of the test data generation methods provided by embodiments of the present application.

[0041] The embodiment of the application can generate a script file based on user-defined configuration information, the configuration information at least including data fields, logical relationships between fields and data distribution, generate simulated structured test data based on the script file, generate a SQL script conforming to a target database syntax based on the structured test data, and execute the SQL script in the target database to obtain test data. Thus, through the end-to-end automatic process of "configuration->script generation->data generation->SQL conversion->database execution", an automatic pipeline from parameter definition to data landing is formed, the generation efficiency of test data is significantly improved, and high-quality test data can be provided for subsequent software testing. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0043] Figure 1 FIG. 1 is a flowchart of a test data generation method provided by the embodiment of the application;

[0044] Figure 2 FIG. 2 is a flowchart of generating structured test data provided by the embodiment of the application;

[0045] Figure 3 FIG. 3 is a flowchart of generating a SQL script provided by the embodiment of the application;

[0046] Figure 4 FIG. 4 is a structural diagram of a test data generation device provided by the embodiment of the application;

[0047] Figure 5 FIG. 5 is a structural diagram of an electronic device provided by the embodiment of the application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, but not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative effort fall within the scope of the application.

[0049] The main technical terms related to the embodiments of the application are as follows:

[0050] Test data: Test data is data used to test a certain function of software, such as a piece of data used by an administrator to test the function of a website. In many test scenarios, different test data needs to be used to test the same function or feature multiple times.

[0051] Python, a widely used interpreted, high-level and general-purpose computer programming language. Python was created by Guido van Rossum of the Netherlands Mathematical and Computer Science Research Institute, and the first version was released in 1991. It is the successor of the ABC language and can also be regarded as a LISP dialect using traditional infix expressions. Python provides efficient high-level data structures and simple and effective object-oriented programming.

[0052] SQL, Structured Query Language, is a special-purpose programming language, a database query and programming language for accessing data and querying, updating and managing relational database systems, and is also the extension name of database script files.

[0053] SQL script: An executable script written for automated testing. SQL scripts generally refer to a series of instructions for a specific test that can be executed by automated testing tools. In order to improve the maintainability and reusability of test scripts, SQL scripts must be built before they are executed. Some operations may occur in several test processes, so the target of these operations should be determined purposefully so that their implementation can be reused. SQL scripts are computer-readable instructions for automatically executing test processes, which can be created (recorded) or automatically generated using automated testing tools, or programmed in a programming language.

[0054] The embodiment of the present application considers that in the related art, manual creation of test data has the following problems: 1. Low efficiency: In a big data environment, manually creating data is very time-consuming and not suitable for large-scale testing; 2. Error-prone: Human error is difficult to avoid, which may result in inaccurate test data; 3. Inconsistent: The consistency and completeness of the data are difficult to guarantee, which affects the effectiveness of the test results; 4. Poor flexibility: It is difficult to quickly adapt to changing needs; 5. High labor cost: A large amount of manpower is required, especially when the data volume is large.

[0055] The semi-automatic test data generation tool has the following problems: 1. Complex configuration: users need to have certain technical knowledge to effectively use the tool, increasing the difficulty of getting started; 2. Limited flexibility: although efficiency is improved, support for complex business logic is insufficient; 3. Weak dynamic adaptation: difficult to adapt to rapidly changing requirements; 4. Partial manual intervention: still requires some human intervention, reducing overall efficiency; 5. Slow generation speed: the configuration process is tedious and may affect the generation speed.

[0056] Therefore, the embodiments of the present application provide a test data generation method, through an end-to-end automated process of "configuration → script generation → data generation → SQL conversion → database execution", forming an automated pipeline from parameter definition to data landing, to improve the generation efficiency of test data, thereby providing high-quality test data for subsequent software testing.

[0057] Specifically, please refer to Figure 1 The specific process of the test data generation method can be as follows:

[0058] S110, generating a script file based on user-defined configuration information; the configuration information at least includes data fields, logical relationships between fields, and data distribution.

[0059] In this embodiment, the configuration information is a set of parameters defined by the user for generating test data, and is the input basis for the entire data generation process. The configuration information at least includes data fields, logical relationships between fields, and data distribution. Among them, the data field is the basic unit of table structure, that is, the column definition of the data to be generated (such as order ID, customer ID, order amount, order time in the order table). The logical relationship between fields refers to the business rule constraint between fields to ensure that the generated data conforms to the real business logic, including: numerical constraint (such as freight ≤ order amount × 0.3), time constraint (such as delivery time > order time), association constraint (such as the customer ID of the order table must exist in the customer ID primary key of the customer table), distribution constraint (such as the order amount of VIP customers needs to be higher than that of ordinary customers). Data distribution refers to the distribution characteristics of field values, which is used to describe the statistical characteristics and business distribution rules of field values to generate data closer to the real scene, including: numerical distribution (such as the age field is normally distributed and concentrated in 20-40 years old, and the price field is long-tailed distributed and concentrated in the low price interval), category distribution (such as electronic products account for 30% of the product category, and clothes account for 50%), geographical distribution (such as user location is divided into 40% in first-tier cities, 30% in second-tier cities, and 30% in other cities). In addition, the configuration information may also include some other parameters, such as the number of tables, the number of fields per table, the number of records per table, field types (integer / character string / date), etc.

[0060] Script files are intermediate carriers generated based on user configuration information, containing executable logical instructions and rule definitions.

[0061] In this embodiment, by providing users with an interactive graphical configuration interface, including functions such as selecting data fields, defining logical relationships between fields, and setting data distribution characteristics, users can configure data generation rules through an intuitive graphical interface. Subsequently, the configuration tool automatically converts the user's operations and settings on the graphical configuration interface into script files. These script files contain all the necessary information configured by the user and can be directly read and executed by the subsequent data generation engine to generate the required data. The script files can be written using common scripting languages ​​that are easy to understand and write (such as Python, JavaScript, etc.), which allows users to easily define complex data generation rules.

[0062] For example, the generated script file may contain user-defined parameters such as table prefix (table_), field prefix (field_), number of tables (100), number of fields (50), number of rows per table (1 million rows), and field type (integer or string), as well as the logic for generating table data records, such as how to generate corresponding random values ​​or strings based on field types and organize them into a structured data set according to preset rules. This script file acts as an intermediate carrier, encapsulating all data processing details for easy subsequent calling and expansion.

[0063] Thus, through the graphical configuration interface, users can configure data generation rules more conveniently, simplifying the entire configuration process. At the same time, the interface can also provide real-time feedback to help users discover and correct any possible errors in a timely manner during the configuration process, thereby improving the accuracy and efficiency of the configuration.

[0064] S120. Generate simulated structured test data based on script files.

[0065] Structured test data refers to a set of in-memory data generated based on script files, which is simulated data conforming to user configuration. Its format is as follows: In-memory structure: a mapping between table names and records (e.g., {"Order Table":[{"Order ID":1001,"Customer ID":205,"Order Amount":150},...],...}); Content characteristics: includes random values ​​for static fields (e.g., randomly generated order IDs), real-time values ​​for dynamic fields (e.g., the order time in the current system time), and all field values ​​are validated through logical relationships (e.g., the order amount does not exceed the associated inventory value).

[0066] In this embodiment, by parsing and running the script file generated in step S110 above, a multi-table related dataset that meets the configuration requirements is constructed in memory. Each table contains a specified number of fields (such as integers, strings, dates, etc.), and is filled with random values ​​according to business rules (such as ID card numbers following a check digit algorithm and timestamps maintaining continuity). It should be noted that the structured test data generated at this stage has not yet been persistently stored, but it already has complete field relationships (such as primary and foreign key relationships, where the primary key is the field in the table used to uniquely identify each record, and the foreign key is the field in the table used to associate with the primary key of another table) and business logic consistency, and can be used as a raw data pool for subsequent processing.

[0067] Specifically, the process mainly includes: parsing the script file using a script parser to convert it into an intermediate representation format; and generating simulated structured test data based on the intermediate representation format using a data generation engine.

[0068] In this embodiment, the script parser is responsible for converting script files (typically in Python, JSON, or YAML format) written by the user through a graphical configuration interface or text editor into structured data that the system can understand. The intermediate representation format serves as a standardized data bridge between the script parser and the data generation engine. The data generation engine is the core processor that actually generates simulated data based on the intermediate representation format; its function is to transform abstract configuration rules into concrete sets of in-memory data.

[0069] Specifically, users define the configuration rules for test data through a graphical configuration interface or a text editor (such as "generate 100 tables, each with 50 fields, where field_1 is an integer primary key, field_2 is a 50-character string, and field_3 needs to reference field_1 from other tables") and generate script files.

[0070] After receiving the script file, the script parser first performs syntax analysis and lexical analysis. For example, when it recognizes a configuration fragment such as table table_1{field_1:integer(1..1000000),field_2:string(50),field_3:integerreferencestable_2.field_1}, the script parser will convert keywords such as table and field into tokens (numeric identifiers), extract the table name table_1, field names field_1 to field_3, as well as field types (integer, string) and constraint rules (primary key, foreign key references).

[0071] Subsequently, the script parser performs semantic validation: checking whether field types are within the predefined range (e.g., custom types xyz are not allowed), whether table names conform to naming conventions (e.g., whether they begin with table_), and whether constraint rules are logically consistent (e.g., whether the foreign key referenced table_2 is defined). If the validation passes, the script parser converts the extracted configuration parameters into an intermediate representation format, such as a JSON structure. This intermediate representation format is the core basis for subsequent data generation, preserving the user-configured business logic (e.g., primary keys, foreign key constraints) while unifying the data structure, facilitating standardized processing by the data generation engine.

[0072] Afterwards, the data generation engine reads the intermediate representation format and generates data table by table, including: 1. Table structure instantiation: Create an empty list `records` for each table (e.g., `table_1`) to store all records of the table (each record is a dictionary of field names and values). 2. Field value generation: For each field, generate random values ​​according to the type and rules defined in the intermediate representation: If the field is an integer (e.g., `field_1`), the engine calls `random.randint(1, 1000000)` to generate a random number between 1 and 1 million, ensuring its uniqueness (by recording generated values ​​using a hash table to avoid primary key duplication); if the field is a string (e.g., `field_2`), the engine generates a combination of 50 random letters using `random.choices(string.ascii_letters, k=50)`; if the field is a foreign key (e.g., `field_3`), the engine first queries all generated valid values ​​(primary key set) in the `field_1` column of the related table (e.g., `table_2`), and then randomly selects one as the value of `field_3` for the current record, ensuring that the foreign key constraint is met. 3. Record Assembly and Batch Generation: The values ​​generated from single fields are combined into complete records according to the table structure (e.g., {'field_1':123,'field_2':'abc...','field_3':456}), and a configured number of records (e.g., 1 million) are generated through a loop and stored in an in-memory list. 4. Data Validation and Output: After generation, the engine checks the uniqueness of the primary key (e.g., no duplicate values ​​in field_1) and the validity of foreign keys in each table (e.g., all values ​​in field_3 exist in the field_1 column of the related table), and outputs structured test data (e.g., a Python dictionary {"table_1":records_1,"table_2":records_2,...}) for subsequent conversion into SQL scripts or direct storage.

[0073] Please refer to Figure 2 Step S120 generates simulated structured test data based on the script file, including:

[0074] S120-1: Analyze the script content line by line using a script parser to extract key configuration parameters.

[0075] In this embodiment, key configuration parameters include: basic table information (such as table name prefix table_, number of tables 100), field definitions (such as field name prefix field_, number of fields 50, field type integer / string and corresponding range or length), number of data rows (such as 1 million records per table), and logical relationships between fields (such as primary key constraint field_1 is unique, foreign key association field_2 references fields in other tables).

[0076] The script is analyzed line by line by line by the script parser. After extracting key configuration parameters, the script's legality is ensured through lexical analysis and syntax verification. If parameter conflicts (such as undefined field types) or syntax errors (such as missing closing brackets) are found, error messages are immediately fed back and execution is terminated to ensure the reliability of subsequent data generation.

[0077] S120-2: Based on key configuration parameters, construct a metadata model of the table structure and determine the field composition and constraint rules of each table.

[0078] Specifically, iterate through all tables to be generated (e.g., table_1 to table_100), generate a field list for each table (e.g., field_1 to field_50), and set the data type and length according to the field type (e.g., integers correspond to INT type, strings correspond to VARCHAR(50)). Simultaneously, mark the constraints of special fields (e.g., set field_1 as the primary key to ensure its uniqueness; if foreign key relationships exist, record the source table and target field of the related field). This metadata model serves as the foundation for subsequent data generation, guiding the logic of field value generation and the maintenance of data relationships.

[0079] S120-3: For static fields in the metadata model, call the random data generation algorithm to generate corresponding simulated values.

[0080] In this embodiment, the data fields in the user configuration information include static fields whose values ​​are generated by a random algorithm, such as the order ID of integer type and the product name of string type.

[0081] Specifically, for each static field defined in the metadata model, the data generation engine calls the corresponding random data generation algorithm based on the field's type to generate simulated values ​​that conform to business logic. For example, for integer fields (such as field_1), a random number generator generates non-repeating or allowed-repeating integer values ​​within a configured range (such as 1 to 1,000,000). For string fields (such as field_2), a random combination of letters of a specified length (such as 50 characters) is generated based on the string.ascii_letters character set, or words that conform to the business scenario (such as city names, product categories) are generated based on a preset dictionary. If a field has special distribution requirements (such as a 10% probability of generating an emergency flag True, and the rest False), the generation ratio is adjusted through a probability control algorithm to ensure that the generated data conforms to the type definition and business common sense at the single-field dimension.

[0082] S120-4: Assemble the simulated values ​​corresponding to each static field according to the table structure order to obtain the simulated structured test data.

[0083] Specifically, the random values ​​generated by each static field are assembled into a complete single record according to the table structure (e.g., {'field_1':123,'field_2':'abc...',...,'field_50':'xyz...'}), and a configured number of records are generated through a loop (e.g., 1 million records per table). For tables with logical relationships between fields (e.g., field_3 needs to reference the value of field_1 in table_2), the system uses a join key mapping mechanism to dynamically query the valid values ​​already generated in the join tables when generating records in the current table, ensuring that foreign key constraints are met (e.g., the value of field_3 must exist in the field_1 column of the join table). If it is a join within a single table (e.g., a parent-child table relationship), data consistency is maintained through logical keys such as record ID or timestamp.

[0084] Finally, the generated simulated data is temporarily stored in memory or a temporary file using an efficient data structure (such as a nested list of Python dictionaries, with the table name as the key and a list of all records in the table as the value), thus fully preserving the multi-field relationships and business logic constraints of the table.

[0085] Optionally, the data fields also include dynamic fields. In the process of generating structured test data, the above method includes: parsing the dynamic fields in the script file and obtaining real-time dynamic parameter values ​​through a predefined interface; combining the real-time dynamic parameter values ​​with the simulated values ​​of the remaining data fields in the order of the table structure to generate simulated structured test data.

[0086] In this embodiment, the data fields in the user configuration information may include not only static fields, but also dynamic fields, whose values ​​are obtained through real-time interfaces. For example, the order time of the timestamp type is taken from the current system time, and the total inventory value associated with the external system is taken from the real-time inventory data.

[0087] Specifically, users can explicitly declare variables (such as current_timestamp, external_source_value) in the configuration script file to represent information that needs to be dynamically obtained at runtime (such as the current timestamp, real-time metrics from external data sources). These dynamic parameters, together with regular static parameters (such as fixed field types, preset data ranges), constitute complete configuration information. However, unlike the direct definition of static values, the values ​​of dynamic parameters are only obtained through external interfaces or real-time queries at script runtime, allowing the generated data rules to flexibly adapt to changes in business scenarios (such as generating time-sensitive data based on the current date, or referencing the latest configuration parameters from external systems).

[0088] Subsequently, when the data generation engine executes the script file to generate structured test data, it prioritizes parsing the dynamic parameters declared in the script (such as timestamps and external data source metrics) and obtains the latest values ​​in real time before each data generation through predefined interfaces (such as the system clock API and external database query services). For example, if the script file declares the dynamic parameter `current_date`, the data generation engine will call the system time interface to obtain the current date when generating each record; if the parameter is associated with an external data source (such as the real-time inventory of the business system), the data generation engine will query the latest value of that data source through the API.

[0089] Finally, the data generation engine combines the dynamically acquired parameters in real time with the statically generated field values ​​(such as random integers and fixed-length strings) according to the table structure order to generate a complete record that conforms to the business logic. This process ensures that the basic data structure conforms to the user configuration (such as field types and relationships), and keeps the data content synchronized with the actual business scenario through real-time updates of dynamic parameters, ultimately generating simulated data that combines static rule constraints with dynamic business adaptability.

[0090] Optionally, after generating corresponding simulated values ​​by calling a random data generation algorithm for static fields in the metadata model, the above method further includes:

[0091] Based on the logical relationships between fields, the correlation between the simulated values ​​of each static field is verified; if the simulated value of a static field violates the logical relationships between fields, the simulated value of that static field is processed based on a preset hierarchical processing mechanism.

[0092] In this embodiment, when the data generation engine executes the script file to generate structured test data, it parses and executes the logical relationship verification module embedded in the script in real time. For each generated data record (or data batch), after the engine completes the basic field value filling, it immediately triggers the logical relationship detection process: according to user-defined logical rules (such as numerical comparison, time series constraints, and related field matching), it verifies the correlation between fields, for example, checking whether "order amount" is indeed less than or equal to the value of the associated "total inventory value" field. Furthermore, this detection process runs through every step of data generation, ensuring that each record meets the pre-configured business logic constraints at the time of generation.

[0093] If the generated mock value is detected to violate the logical relationship between fields, such as "end time is earlier than start time" or "foreign key value does not exist in the associated table", the mock value of the static field will be processed based on the preset hierarchical processing mechanism.

[0094] Specifically, if the violation level is Level 1, the simulated value of the static field will be corrected and a correction log will be recorded; if the violation level is Level 2, the simulated value of the static field will be marked as abnormal data and an alert report containing anomaly statistics will be generated.

[0095] Violation level 1 refers to simulated values ​​that violate logical relationships but can be automatically corrected and recovered, with the corrected data still conforming to business rules and constraints. These violations are typically caused by random value generation deviations or boundary condition conflicts. While they affect data accuracy, they do not disrupt the overall logical structure of the data or cause system processing interruptions. For example, a numeric field might slightly exceed the user-defined range (e.g., an order amount configured as 1-100 yuan, but 101 yuan is generated), or the logical order of a time field and its associated time field might be slightly reversed (e.g., the shipping time is only 1 second earlier than the order placement time, but business requirements dictate it must be later). In such cases, the field values ​​can be adjusted using an automatic correction mechanism (e.g., reducing the order amount proportionally to 90% of the total inventory value, or fine-tuning the timestamp to after the start time) to obtain data conforming to logical relationships, and a correction log can be recorded.

[0096] A Level 2 violation indicates a simulated value that violates logical relationships and cannot be recovered through simple correction, or whose correction may trigger other related problems. These violations are typically caused by missing related fields, hard constraint conflicts, or unmet complex business rules. Forced correction may break the data logic chain or cause system anomalies. For example, a foreign key field value may not exist in the primary key set of the related tables (e.g., "Customer ID" in the order table is not defined in the customer table), or a hard constraint conflict (e.g., "Order amount must be ≤ total inventory value," but the order amount is still positive when the total inventory value is 0). In such cases, the simulated value is marked as abnormal data, error details are retained (e.g., specific field, violated logical rule, generation batch), and an alert report containing error statistics is generated (e.g., "12 records were detected violating inventory constraints, 5 records had missing foreign keys," etc.) to ensure that the final structured test dataset conforms to the user-defined business logic constraints as a whole, maintaining the logical integrity and business authenticity of the data.

[0097] S130. Based on the structured test data, generate an SQL script that conforms to the target database syntax.

[0098] The SQL script is a standardized set of instructions that converts structured test data into a syntax that conforms to the target database. It includes two categories: DDL (Data Definition Language), which contains SQL statements used to create table structures (e.g., `CREATE TABLE order_table(orderID INT, customerID INT, order amount DECIMAL(10,2), PRIMARY KEY (orderID), FOREIGN KEY (customerID) REFERENCES customer_table(customerID)`) and DML (Data Manipulation Language), which contains SQL statements used to insert data (e.g., `INSERT INTO order_table VALUES(1001,205,150.00,'2025-02-11 14:30:00')`).

[0099] The target database refers to the database management system (DBMS) where the test data is ultimately stored. It supports multiple types (such as relational databases MySQL, Oracle, PostgreSQL, or distributed databases TiDB), must be compatible with the generated SQL script syntax, and provide data storage and query capabilities.

[0100] Please refer to Figure 3 Step S130 generates an SQL script that conforms to the target database syntax based on the structured test data, including:

[0101] S130-1: Based on the type of the target database, obtain the syntax adaptation rule set corresponding to the target database.

[0102] Among them, the syntax adaptation rule set is a set of predefined SQL syntax specifications for different database management systems (DBMS) to address the differences in basic syntax structure, field type definitions, constraint rules and statement formats between different databases.

[0103] Specifically, based on the target database type (such as MySQL, Oracle, PostgreSQL) specified by the user during the configuration phase, the system loads the database-specific SQL syntax specification set from the built-in rule base to provide an adaptation basis for subsequent field type conversion and statement generation.

[0104] For example, when the user selects MySQL as the target database, the MySQL rule set is loaded to obtain field type mappings (such as STRING→VARCHAR(255)), identifier reference characters (backticks `), and auto-incrementing primary key syntax (AUTO_INCREMENT); if Oracle is selected, the Oracle rule set is loaded (such as STRING→VARCHAR2(255), double quote identifiers, and sequence mechanism).

[0105] S130-2: Based on a preset field type mapping table, the general type definition of each field in the structured test data is converted into the specific field type corresponding to the target database.

[0106] The field type mapping table is a built-in table that maps general types to database-specific types. It is used to automatically convert general field types (such as INTEGER, STRING, DATETIME) in user configuration or structured test data into specific field types supported by the target database.

[0107] Specifically, by traversing the metadata model of the structured test data (including table names, field names, general types, and constraints), for each field, the specific field type corresponding to the target database is found through the field type mapping table. For example, if the field order_amount (general type INTEGER) is defined in the structured test data, it is converted to INT when the target database is MySQL; and converted to NUMBER (10) when the target database is Oracle.

[0108] S130-3: Generate DDL creation statements and DML insert statements based on structured test data and syntax adaptation rule sets of specific field types.

[0109] DDL creation statements are used to define table structures (field names, specific field types, constraint rules) and conform to the syntax of the target database. DML insert statements are used to insert specific records into the structured test data and conform to the insert syntax of the target database (such as batch insert or single row insert).

[0110] S130-4: Integrate DDL creation statements and DML insert statements in the order of table structure to obtain an SQL script that conforms to the target database syntax.

[0111] Specifically, by integrating all DDL creation statements and DML insert statements generated in step S130-3 in logical order to form a complete SQL script file, the script is ensured to be directly executable in the target database. Integration rules include:

[0112] Execution order: First, execute all DDL creation statements to create the table structure, then execute all DML insert statements to insert the actual data, avoiding insertion failure due to the table not existing;

[0113] Table order: Arrange DDL creation statements and DML insert statements according to the table order configured by the user or alphabetical order. For example, first generate the table creation and insert statements for the customers table, and then generate the orders table.

[0114] Script formatting: Add comment markers (such as --=====DDL section=====), separators, and error handling prompts to improve script readability.

[0115] Therefore, this solution addresses the syntax differences between different databases through a syntax adaptation rule set, achieves precise conversion from general types to specific types through a field type mapping table, generates and integrates data through DDL creation statements and DML insert statements, and finally outputs standardized SQL scripts that conform to the target database syntax. This approach ensures that structured test data can be realistically implemented in mainstream databases such as MySQL, Oracle, and PostgreSQL, providing a reliable data foundation for database testing, performance verification, and feature development.

[0116] S140. Execute the SQL script in the target database to obtain test data.

[0117] The test data is a real data set generated by executing SQL scripts in the target database, representing the physical storage format of structured test data. Its characteristics include: authenticity (field values ​​conform to business logic, such as order amount ≤ inventory value) and realistic distribution (such as a high proportion of VIP customers); availability (it can be directly used for database performance testing, synchronization tool verification (such as DATAX), functional regression testing, and other scenarios); and scalability (supporting complex scenarios such as millions of records in a single table and multi-table joins, such as 100 tables with 1 million records each).

[0118] Specifically, by directly executing SQL scripts (DDL+DML) in the target database, users can quickly build a test environment, such as a real test environment containing 100 tables, 50 fields per table, and 1 million records. These table structures and data fully conform to user-configured rules (such as field types, primary key constraints, and data distribution), and possess characteristics of large scale, multi-table joins, and high complexity. They can be directly used for synchronization capability testing, performance stress testing, and functional verification of big data platforms (such as DATAX), ultimately providing a solid testing foundation for the system's stability and reliability.

[0119] Traditional test data preparation requires manual writing of SQL scripts or the use of semi-automated tools for step-by-step operations (such as creating tables first and then inserting data). However, the embodiments of this application form an automated pipeline from parameter definition to data storage through an end-to-end automated process of "configuration → script generation → data generation → SQL conversion → database execution", which significantly improves the efficiency of test data generation and can provide high-quality test data for subsequent software testing.

[0120] The following example, using the e-commerce order system test data generation scenario, illustrates the process of generating the aforementioned test data:

[0121] 1. Configuration Information: Users define three tables (Customer Table, Product Table, and Order Table) through a graphical configuration interface. The Order Table contains fields such as Order ID (integer), Customer ID (foreign key related to the Customer Table), Product ID (foreign key related to the Product Table), Order Amount (numerical), Order Time (dynamic timestamp), and Shipping Address (string). The logical relationships between the fields include: Order Amount ≤ Total Inventory Value of the Product Table, Order Time > Registration Time of the Customer Table, and foreign key constraints (Customer ID and Product ID must exist in the primary key of the related table).

[0122] 2. Parse the configuration information, generate script files and create simulated data (such as 1 million order records, with order amounts generated by a random algorithm but automatically checked to be less than or equal to the inventory value of the associated products, and the order time taken as the current system time and ensured to be later than the customer registration time).

[0123] 3. Convert the data into an SQL script that conforms to MySQL syntax (including CREATE TABLE statement and INSERTINTO statement).

[0124] 4. Execute SQL scripts in the MySQL database to generate a real data environment that can be directly used for testing order synchronization tools (such as DATAX).

[0125] Therefore, the generated test data conforms to e-commerce business logic (such as reasonable order amounts and correct time order), and verifies the data synchronization tool's ability to handle multi-table joins, large number of rows (1 million rows in a single table), and dynamic time fields, avoiding synchronization failures caused by dirty data.

[0126] To better implement the above methods, this application also provides a test data generation device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0127] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the test data generation device as specifically integrated into the server.

[0128] For example, such as Figure 4 As shown, the test data generation device may include a configuration module 410, a parsing module 420, a script generation module 430, and a script execution module 440, as follows:

[0129] Configuration module 410 is used to generate script files based on user-defined configuration information; the configuration information includes at least data fields, logical relationships between fields, and data distribution;

[0130] The parsing module 420 is used to generate simulated structured test data based on script files;

[0131] The script generation module 430 is used to generate SQL scripts that conform to the syntax of the target database based on structured test data.

[0132] The script execution module 440 is used to execute SQL scripts in the target database to obtain test data.

[0133] Optionally, the parsing module 420 includes:

[0134] The conversion unit is used to parse the script file through the script parser and convert the script file into an intermediate representation format;

[0135] The generation unit is used to generate simulated structured test data based on an intermediate representation format through the data generation engine.

[0136] Optionally, the data fields include static fields, and the above transformation unit includes:

[0137] Extract sub-units, used to analyze the script content line by line through the script parser and extract key configuration parameters;

[0138] Construct sub-units to build a metadata model of the table structure based on key configuration parameters, and determine the field composition and constraint rules of each table;

[0139] The above-mentioned generation unit includes:

[0140] The simulated value generation subunit is used to call a random data generation algorithm to generate corresponding simulated values ​​for static fields in the metadata model;

[0141] The simulated value assembly subunit is used to assemble the simulated values ​​corresponding to each static field according to the table structure order to obtain simulated structured test data.

[0142] Optionally, the analog value generation sub-unit is also used for:

[0143] Based on the logical relationships between fields, the correlation between the simulated values ​​of each static field is verified;

[0144] If the simulated value of a static field violates the logical relationship between fields, the simulated value of the static field will be processed based on a preset hierarchical processing mechanism.

[0145] Optionally, the analog value generation sub-unit is also used for:

[0146] If the violation level is Level 1, the simulated value of the static field will be corrected and the correction log will be recorded.

[0147] If the violation level is level two, the simulated value of the static field will be marked as abnormal data, and an alert report containing abnormal statistics will be generated.

[0148] Optionally, the script generation module 430 includes:

[0149] The syntax rule acquisition unit is used to acquire the syntax adaptation rule set corresponding to the target database based on the type of the target database;

[0150] The field mapping unit is used to convert the general type definition of each field in the structured test data into the specific field type corresponding to the target database based on a preset field type mapping table.

[0151] The statement generation unit is used to generate DDL creation statements and DML insert statements based on structured test data of specific field types and syntax adaptation rule sets.

[0152] The statement integration unit is used to integrate DDL creation statements and DML insert statements in the order of table structure to obtain an SQL script that conforms to the target database syntax.

[0153] Optionally, the parsing module 420 is also used for:

[0154] Parse the dynamic fields in the script file and obtain the real-time dynamic parameter values ​​through a predefined interface;

[0155] The real-time dynamic parameter values ​​are combined with the simulated values ​​of the remaining data fields in the order of the table structure to generate simulated structured test data.

[0156] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0157] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0158] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0159] In some embodiments, the test data generation device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the test data generation method of this application.

[0160] In this embodiment, as Figure 5 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0161] The electronic device may include components such as a processor 510 with one or more processing cores, a memory 520 with one or more computer-readable storage media, a power supply 530, an input module 540, and a communication module 550. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0162] The processor 510 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby performing overall detection of the electronic device. In some embodiments, the processor 510 may include one or more processing cores; in some embodiments, the processor 510 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 510.

[0163] The memory 520 can be used to store software programs and modules. The processor 510 executes various functional applications and data processing by running the software programs and modules stored in the memory 520. The memory 520 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 520 may also include a memory controller to provide the processor 510 with access to the memory 520.

[0164] The electronic device also includes a power supply 530 that supplies power to the various components. In some embodiments, the power supply 530 can be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 530 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0165] The electronic device may also include an input module 540, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0166] The electronic device may also include a communication module 550. In some embodiments, the communication module 550 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 550 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0167] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 510 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 520 according to the following instructions, and the processor 510 runs the applications stored in the memory 520, thereby realizing the various functions in the above-described test data generation method.

[0168] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0169] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the test data generation methods provided in embodiments of this application.

[0170] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0171] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the test data generation method provided in the above embodiments.

[0172] Since the instructions stored in the storage medium can execute the steps in any of the test data generation methods provided in the embodiments of this application, the beneficial effects that any of the test data generation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0173] The above provides a detailed description of a test data generation method, apparatus, electronic device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating test data, characterized in that, The method includes: A script file is generated based on user-defined configuration information; the configuration information includes at least data fields, logical relationships between fields, and data distribution. The script content is analyzed line by line by the script parser to extract key configuration parameters, which include the basic information of the table, field definitions, number of data rows and logical relationships between fields. Based on the key configuration parameters, a metadata model of the table structure is constructed, and the field composition and constraint rules of each table are determined. The metadata model is used to guide the generation logic of field values ​​and the maintenance of data association relationships. For the static fields in the metadata model, a random data generation algorithm is invoked to generate corresponding simulated values; The simulated values ​​corresponding to each static field are assembled in the order of the table structure to obtain the simulated structured test data; Based on the type of the target database, obtain the syntax adaptation rule set corresponding to the target database; Based on a preset field type mapping table, the general type definition of each field in the structured test data is converted into the specific field type corresponding to the target database; Based on the structured test data of specific field types and the syntax adaptation rule set, generate DDL creation statements and DML insert statements; The DDL creation statement and the DML insert statement are integrated in the order of the table structure to obtain an SQL script that conforms to the syntax of the target database. The SQL script is executed in the target database to obtain test data; It also includes: verifying the correlation between the simulated values ​​of each static field based on the logical relationship between the fields; if the simulated value of a static field violates the logical relationship between the fields, if the violation level is level one, then the simulated value of the static field is corrected and a correction log is recorded; if the violation level is level two, then the simulated value of the static field is marked as abnormal data and an alert report containing abnormal statistics is generated.

2. The test data generation method as described in claim 1, characterized in that, Based on the script file, simulated structured test data is generated, including: The script file is parsed by a script parser and converted into an intermediate representation format; Based on the intermediate representation format, the data generation engine generates simulated structured test data.

3. The test data generation method as described in claim 1, characterized in that, The data fields also include dynamic fields, which generate simulated structured test data based on the script file, including: The dynamic fields in the script file are parsed, and real-time dynamic parameter values ​​are obtained through a predefined interface; The real-time dynamic parameter values ​​are combined with the simulated values ​​of the remaining data fields in the order of the table structure to generate simulated structured test data.

4. A test data generation device, characterized in that, The device includes: The configuration module is used to generate script files based on user-defined configuration information; the configuration information includes at least data fields, logical relationships between fields, and data distribution. The parsing module is used to analyze the script content line by line through the script parser and extract key configuration parameters, including the basic information of the table, field definitions, number of data rows and logical relationships between fields. Based on the key configuration parameters, a metadata model of the table structure is constructed, and the field composition and constraint rules of each table are determined. The metadata model is used to guide the generation logic of field values ​​and the maintenance of data association relationships. For the static fields in the metadata model, a random data generation algorithm is invoked to generate corresponding simulated values; The simulated values ​​corresponding to each static field are assembled in the order of the table structure to obtain the simulated structured test data; The script generation module is used to obtain the syntax adaptation rule set corresponding to the target database based on the target database type; based on a preset field type mapping table, convert the general type definition of each field in the structured test data into the specific field type corresponding to the target database; based on the structured test data of the specific field type and the syntax adaptation rule set, generate DDL creation statements and DML insert statements; integrate the DDL creation statements and the DML insert statements in the order of the table structure to obtain an SQL script that conforms to the syntax of the target database. The script execution module is used to execute the SQL script in the target database to obtain test data; The parsing module is also used to verify the correlation between the simulated values ​​of each static field based on the logical relationship between the fields; if the simulated value of a static field violates the logical relationship between the fields, and the violation level is level one, the simulated value of the static field is corrected and a correction log is recorded; if the violation level is level two, the simulated value of the static field is marked as abnormal data and an alert report containing abnormal statistics is generated.

5. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the test data generation method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the test data generation method as described in any one of claims 1 to 3.

Images