Test data generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a test data generation method and device, electronic equipment and a storage medium. According to the embodiment of the invention, a script file can be generated based on configuration information defined by a user; the configuration information at least comprises data fields, logic relations among the fields and data distribution; generating simulated structured test data based on the script file; based on the structured test data, generating an SQL script conforming to the grammar of the target database; and executing the SQL script in the target database to obtain test data. Therefore, through the end-to-end automatic process of configuration, script generation, data generation, SQL conversion and database execution, an automatic assembly line from parameter definition to data landing is formed, the generation efficiency of the test data is remarkably improved, and high-quality test data can be provided for subsequent software testing.

Description

Technical Field

[0001] This application relates to the field of software testing technology, specifically to a test data generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] In the field of software testing, the preparation of test data is a crucial step in ensuring test quality and efficiency. Currently, test data generation mainly relies on two mainstream methods: manual construction and semi-automated tool assistance. However, both have significant technical limitations and are difficult to meet the needs of high-frequency iteration, large-scale testing, and complex business scenarios in modern software development.

[0003] Manually creating test data typically involves test engineers manually constructing test datasets based on requirements documents and business logic. While this method allows testers to customize data to meet specific needs, it suffers from inefficiency, high error rates, poor dynamic adaptability, and high labor costs.

[0004] Semi-automated test data generation tools typically offer graphical configuration interfaces or simple script definition capabilities, allowing users to generate test data through visual parameter settings (such as field type selection and data volume input) or basic scripts (such as regular expression templates and simple logic rules), thus reducing the workload of direct manual operation to some extent. However, they also suffer from high configuration complexity, resulting in limited generation speed and poor flexibility. Summary of the Invention

[0005] This application provides a test data generation method, apparatus, electronic device, and storage medium, which can improve the efficiency of test data generation.

[0006] This application provides a test data generation method, including: A script file is generated based on user-defined configuration information; the configuration information includes at least data fields, logical relationships between fields, and data distribution. Based on the script file, simulated structured test data is generated; Based on the structured test data, generate an SQL script that conforms to the syntax of the target database; The SQL script is executed in the target database to obtain test data.

[0007] Optionally, based on the script file, simulated structured test data is generated, including: The script file is parsed by a script parser and converted into an intermediate representation format; Based on the intermediate representation format, the data generation engine generates simulated structured test data.

[0008] Optionally, the data fields include static fields. The script file is parsed by a script parser, converting the script file into an intermediate representation format, including: The script content is analyzed line by line by a script parser to extract key configuration parameters; Based on the aforementioned key configuration parameters, a metadata model of the table structure is constructed, and the field composition and constraint rules of each table are determined. Based on the intermediate representation format, the data generation engine generates simulated structured test data, including: For the static fields in the metadata model, a random data generation algorithm is invoked to generate corresponding simulated values; The simulated values ​​corresponding to each static field are assembled in the order of the table structure to obtain the simulated structured test data.

[0009] Optionally, after generating corresponding simulated values ​​by calling a random data generation algorithm for the static fields in the metadata model, the method further includes: Based on the logical relationships between the fields, the correlation between the simulated values ​​of each static field is verified; If the simulated value of the static field violates the logical relationship between the fields, the simulated value of the static field will be processed based on a preset hierarchical processing mechanism.

[0010] Optionally, the simulated value of the static field is processed based on a preset hierarchical processing mechanism, including: If the violation level is Level 1, the simulated value of the static field will be corrected and the correction log will be recorded. If the violation level is level two, the simulated value of the static field will be marked as abnormal data, and an alert report containing abnormal statistics will be generated.

[0011] Optionally, based on the structured test data, an SQL script conforming to the target database syntax is generated, including: Based on the type of the target database, obtain the syntax adaptation rule set corresponding to the target database; Based on a preset field type mapping table, the general type definition of each field in the structured test data is converted into the specific field type corresponding to the target database; Based on the structured test data of specific field types and the syntax adaptation rule set, generate DDL creation statements and DML insert statements; The DDL creation statement and the DML insert statement are integrated in the order of the table structure to obtain an SQL script that conforms to the target database syntax.

[0012] Optionally, the data fields also include dynamic fields, which generate simulated structured test data based on the script file, including: The dynamic fields in the script file are parsed, and real-time dynamic parameter values ​​are obtained through a predefined interface; The real-time dynamic parameter values ​​are combined with the simulated values ​​of the remaining data fields in the order of the table structure to generate simulated structured test data.

[0013] This application embodiment also provides a test data generation apparatus, including: The configuration module is used to generate script files based on user-defined configuration information; the configuration information includes at least data fields, logical relationships between fields, and data distribution. The parsing module is used to generate simulated structured test data based on the script file; The script generation module is used to generate an SQL script that conforms to the target database syntax based on the structured test data. The script execution module is used to execute the SQL script in the target database to obtain test data.

[0014] This application also provides an electronic device, including a memory storing multiple instructions; the processor loads the instructions from the memory to execute the steps in any of the test data generation methods provided in this application.

[0015] This application also provides a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the test data generation methods provided in this application.

[0016] This application embodiment can generate a script file based on user-defined configuration information. The configuration information includes at least data fields, logical relationships between fields, and data distribution. Based on the script file, simulated structured test data is generated. Based on the structured test data, an SQL script conforming to the target database syntax is generated. The SQL script is executed in the target database to obtain test data. Thus, through an end-to-end automated process of "configuration → script generation → data generation → SQL conversion → database execution," an automated pipeline from parameter definition to data storage is formed, significantly improving the efficiency of test data generation and providing high-quality test data for subsequent software testing. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the test data generation method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the process for generating structured test data provided in an embodiment of this application; Figure 3 This is a schematic diagram of the process for generating SQL scripts provided in an embodiment of this application; Figure 4 This is a schematic diagram of the test data generation device provided in the embodiments of this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Key technical terms involved in the embodiments of this invention: Test data: Test data is used to test a specific function of software. For example, a piece of data might be used by an administrator to test website functionality. In many testing scenarios, it's necessary to use different test data to test the same function or feature multiple times.

[0021] Python is a widely used interpreted, high-level, and general-purpose computer programming language. Created by Guido van Rossum of the Netherlands Organization for Mathematical and Computer Science, the first version of Python was released in 1991. It is the successor to the ABC language and can also be considered a dialect of LISP that uses traditional infix expressions. Python provides efficient high-level data structures and enables simple and effective object-oriented programming.

[0022] SQL, Structured Query Language, is a special-purpose programming language used for database querying and programming. It is used to access, query, update, and manage relational database systems, and is also the file extension for database script files.

[0023] SQL scripts are executable scripts written for automated testing. An SQL script generally refers to a series of instructions for a specific test, which can be executed by automated testing tools. To improve the maintainability and reusability of test scripts, they must be constructed before execution. Some operations may appear in several test processes; therefore, the goals of these operations should be purposefully defined so that their implementation can be reused. SQL scripts are computer-readable instructions for automating test processes; they can be created (recorded), automatically generated using automated testing tools, or programmed using a programming language.

[0024] The embodiments of the present invention take into account the following problems in the related technologies of manually creating test data: 1. Low efficiency: In a big data environment, manually creating data is very time-consuming and not suitable for large-scale testing; 2. Prone to errors: Human error is difficult to avoid, which may lead to inaccurate test data; 3. Inconsistency: The consistency and integrity of data are difficult to guarantee, affecting the validity of test results; 4. Poor flexibility: It is difficult to quickly adapt to changing needs; 5. High labor costs: A large amount of manpower is required, especially when the amount of data is large.

[0025] Semi-automated test data generation tools have the following problems: 1. Complex configuration: Users need to have certain technical knowledge to use the tool effectively, which increases the difficulty of getting started; 2. Limited flexibility: Although it improves efficiency, it does not support complex business logic; 3. Weak dynamic adaptability: It is difficult to cope with rapidly changing requirements; 4. Partial manual intervention: Some manual intervention is still required, which reduces the overall efficiency; 5. Slow generation speed: The configuration process is cumbersome, which may affect the generation speed.

[0026] Therefore, this application provides a test data generation method that forms an automated pipeline from parameter definition to data storage through an end-to-end automated process of "configuration → script generation → data generation → SQL conversion → database execution", thereby improving the efficiency of test data generation and providing high-quality test data for subsequent software testing.

[0027] For details, please refer to Figure 1 The specific process for generating this test data can be summarized as follows: S110. Generate a script file based on user-defined configuration information; the configuration information includes at least data fields, logical relationships between fields, and data distribution.

[0028] In this embodiment, the configuration information is a set of parameters defined by the user to generate test data, serving as the input foundation for the entire data generation process. The configuration information includes at least data fields, the logical relationships between fields, and data distribution. Data fields are the basic units constituting the table structure, i.e., the column definitions of the data to be generated (such as order ID, customer ID, order amount, and order time in the order table). The logical relationships between fields refer to the business rule constraints between fields, ensuring that the generated data conforms to real business logic, including: numerical constraints (e.g., shipping cost ≤ order amount × 0.3), time constraints (e.g., shipping time > order time), association constraints (e.g., the customer ID in the order table must exist in the customer ID primary key in the customer table), and distribution constraints (e.g., the order amount percentage for VIP customers must be higher than that for ordinary customers). Data distribution refers to the distribution characteristics of field values, used to describe the statistical characteristics and business distribution patterns of field values ​​to generate data that more closely resembles real-world scenarios. This includes: numerical distribution (e.g., age fields showing a normal distribution concentrated in the 20-40 age group, price fields showing a long-tail distribution concentrated in the low-price range), category distribution (e.g., in product categories, electronics account for 30%, and clothing for 50%), and geographical distribution (e.g., user locations are divided into first-tier cities 40%, second-tier cities 30%, and other 30%). Additionally, configuration information may include other parameters such as the number of tables, the number of fields per table, the number of records per table, and field types (integer / string / date).

[0029] Script files are intermediate carriers generated based on user configuration information, containing executable logical instructions and rule definitions.

[0030] In this embodiment, by providing users with an interactive graphical configuration interface, including functions such as selecting data fields, defining logical relationships between fields, and setting data distribution characteristics, users can configure data generation rules through an intuitive graphical interface. Subsequently, the configuration tool automatically converts the user's operations and settings on the graphical configuration interface into script files. These script files contain all the necessary information configured by the user and can be directly read and executed by the subsequent data generation engine to generate the required data. The script files can be written using common scripting languages ​​that are easy to understand and write (such as Python, JavaScript, etc.), which allows users to easily define complex data generation rules.

[0031] For example, the generated script file may contain user-defined parameters such as table prefix (table_), field prefix (field_), number of tables (100), number of fields (50), number of rows per table (1 million rows), and field type (integer or string), as well as the logic for generating table data records, such as how to generate corresponding random values ​​or strings based on field types and organize them into a structured data set according to preset rules. This script file acts as an intermediate carrier, encapsulating all data processing details for easy subsequent calling and expansion.

[0032] Thus, through the graphical configuration interface, users can configure data generation rules more conveniently, simplifying the entire configuration process. At the same time, the interface can also provide real-time feedback to help users discover and correct any possible errors in a timely manner during the configuration process, thereby improving the accuracy and efficiency of the configuration.

[0033] S120. Generate simulated structured test data based on script files.

[0034] Structured test data refers to a set of in-memory data generated based on script files, which is simulated data conforming to user configuration. Its format is as follows: In-memory structure: a mapping between table names and records (e.g., {"Order Table":[{"Order ID":1001,"Customer ID":205,"Order Amount":150},...],...}); Content characteristics: includes random values ​​for static fields (e.g., randomly generated order IDs), real-time values ​​for dynamic fields (e.g., the order time in the current system time), and all field values ​​are validated through logical relationships (e.g., the order amount does not exceed the associated inventory value).

[0035] In this embodiment, by parsing and running the script file generated in step S110 above, a multi-table related dataset that meets the configuration requirements is constructed in memory. Each table contains a specified number of fields (such as integers, strings, dates, etc.), and is filled with random values ​​according to business rules (such as ID card numbers following a check digit algorithm and timestamps maintaining continuity). It should be noted that the structured test data generated at this stage has not yet been persistently stored, but it already has complete field relationships (such as primary and foreign key relationships, where the primary key is the field in the table used to uniquely identify each record, and the foreign key is the field in the table used to associate with the primary key of another table) and business logic consistency, and can be used as a raw data pool for subsequent processing.

[0036] Specifically, the process mainly includes: parsing the script file using a script parser to convert it into an intermediate representation format; and generating simulated structured test data based on the intermediate representation format using a data generation engine.

[0037] In this embodiment, the script parser is responsible for converting script files (typically in Python, JSON, or YAML format) written by the user through a graphical configuration interface or text editor into structured data that the system can understand. The intermediate representation format serves as a standardized data bridge between the script parser and the data generation engine. The data generation engine is the core processor that actually generates simulated data based on the intermediate representation format; its function is to transform abstract configuration rules into concrete sets of in-memory data.

[0038] Specifically, users define the configuration rules for test data through a graphical configuration interface or a text editor (such as "generate 100 tables, each with 50 fields, where field_1 is an integer primary key, field_2 is a 50-character string, and field_3 needs to reference field_1 from other tables") and generate script files.

[0039] After receiving the script file, the script parser first performs syntax analysis and lexical analysis. For example, when it recognizes a configuration fragment such as table table_1{field_1:integer(1..1000000),field_2:string(50),field_3:integerreferencestable_2.field_1}, the script parser will convert keywords such as table and field into tokens (numeric identifiers), extract the table name table_1, field names field_1 to field_3, as well as field types (integer, string) and constraint rules (primary key, foreign key references).

[0040] Subsequently, the script parser performs semantic validation: checking whether field types are within the predefined range (e.g., custom types xyz are not allowed), whether table names conform to naming conventions (e.g., whether they begin with table_), and whether constraint rules are logically consistent (e.g., whether the foreign key referenced table_2 is defined). If the validation passes, the script parser converts the extracted configuration parameters into an intermediate representation format, such as a JSON structure. This intermediate representation format is the core basis for subsequent data generation, preserving the user-configured business logic (e.g., primary keys, foreign key constraints) while unifying the data structure, facilitating standardized processing by the data generation engine.

[0041] Afterwards, the data generation engine reads the intermediate representation format and generates data table by table, including: 1. Table structure instantiation: Create an empty list `records` for each table (e.g., `table_1`) to store all records of the table (each record is a dictionary of field names and values). 2. Field value generation: For each field, generate random values ​​according to the type and rules defined in the intermediate representation: If the field is an integer (e.g., `field_1`), the engine calls `random.randint(1, 1000000)` to generate a random number between 1 and 1 million, ensuring its uniqueness (by recording generated values ​​using a hash table to avoid primary key duplication); if the field is a string (e.g., `field_2`), the engine generates a combination of 50 random letters using `random.choices(string.ascii_letters, k=50)`; if the field is a foreign key (e.g., `field_3`), the engine first queries all generated valid values ​​(primary key set) in the `field_1` column of the related table (e.g., `table_2`), and then randomly selects one as the value of `field_3` for the current record, ensuring that the foreign key constraint is met. 3. Record Assembly and Batch Generation: The values ​​generated from single fields are combined into complete records according to the table structure (e.g., {'field_1':123,'field_2':'abc...','field_3':456}), and a configured number of records (e.g., 1 million) are generated through a loop and stored in an in-memory list. 4. Data Validation and Output: After generation, the engine checks the uniqueness of the primary key (e.g., no duplicate values ​​in field_1) and the validity of foreign keys in each table (e.g., all values ​​in field_3 exist in the field_1 column of the related table), and outputs structured test data (e.g., a Python dictionary {"table_1":records_1,"table_2":records_2,...}) for subsequent conversion into SQL scripts or direct storage.

[0042] Please refer to Figure 2 Step S120 generates simulated structured test data based on the script file, including: S120-1: Analyze the script content line by line using a script parser to extract key configuration parameters.

[0043] In this embodiment, key configuration parameters include: basic table information (such as table name prefix table_, number of tables 100), field definitions (such as field name prefix field_, number of fields 50, field type integer / string and corresponding range or length), number of data rows (such as 1 million records per table), and logical relationships between fields (such as primary key constraint field_1 is unique, foreign key association field_2 references fields in other tables).

[0044] The script is analyzed line by line by line by the script parser. After extracting key configuration parameters, the script's legality is ensured through lexical analysis and syntax verification. If parameter conflicts (such as undefined field types) or syntax errors (such as missing closing brackets) are found, error messages are immediately fed back and execution is terminated to ensure the reliability of subsequent data generation.

[0045] S120-2: Based on key configuration parameters, construct a metadata model of the table structure and determine the field composition and constraint rules of each table.

[0046] Specifically, iterate through all tables to be generated (e.g., table_1 to table_100), generate a field list for each table (e.g., field_1 to field_50), and set the data type and length according to the field type (e.g., integers correspond to INT type, strings correspond to VARCHAR(50)). Simultaneously, mark the constraints of special fields (e.g., set field_1 as the primary key to ensure its uniqueness; if foreign key relationships exist, record the source table and target field of the related field). This metadata model serves as the foundation for subsequent data generation, guiding the logic of field value generation and the maintenance of data relationships.

[0047] S120-3: For static fields in the metadata model, call the random data generation algorithm to generate corresponding simulated values.

[0048] In this embodiment, the data fields in the user configuration information include static fields whose values ​​are generated by a random algorithm, such as the order ID of integer type and the product name of string type.

[0049] Specifically, for each static field defined in the metadata model, the data generation engine calls the corresponding random data generation algorithm based on the field's type to generate simulated values ​​that conform to business logic. For example, for integer fields (such as field_1), a random number generator generates non-repeating or allowed-repeating integer values ​​within a configured range (such as 1 to 1,000,000). For string fields (such as field_2), a random combination of letters of a specified length (such as 50 characters) is generated based on the string.ascii_letters character set, or words that conform to the business scenario (such as city names, product categories) are generated based on a preset dictionary. If a field has special distribution requirements (such as a 10% probability of generating an emergency flag True, and the rest False), the generation ratio is adjusted through a probability control algorithm to ensure that the generated data conforms to the type definition and business common sense at the single-field dimension.

[0050] S120-4: Assemble the simulated values ​​corresponding to each static field according to the table structure order to obtain the simulated structured test data.

[0051] Specifically, the random values ​​generated by each static field are assembled into a complete single record according to the table structure (e.g., {'field_1':123,'field_2':'abc...',...,'field_50':'xyz...'}), and a configured number of records are generated through a loop (e.g., 1 million records per table). For tables with logical relationships between fields (e.g., field_3 needs to reference the value of field_1 in table_2), the system uses a join key mapping mechanism to dynamically query the valid values ​​already generated in the join tables when generating records in the current table, ensuring that foreign key constraints are met (e.g., the value of field_3 must exist in the field_1 column of the join table). If it is a join within a single table (e.g., a parent-child table relationship), data consistency is maintained through logical keys such as record ID or timestamp.

[0052] Finally, the generated simulated data is temporarily stored in memory or a temporary file using an efficient data structure (such as a nested list of Python dictionaries, with the table name as the key and a list of all records in the table as the value), thus fully preserving the multi-field relationships and business logic constraints of the table.

[0053] Optionally, the data fields also include dynamic fields. In the process of generating structured test data, the above method includes: parsing the dynamic fields in the script file and obtaining real-time dynamic parameter values ​​through a predefined interface; combining the real-time dynamic parameter values ​​with the simulated values ​​of the remaining data fields in the order of the table structure to generate simulated structured test data.

[0054] In this embodiment, the data fields in the user configuration information may include not only static fields, but also dynamic fields, whose values ​​are obtained through real-time interfaces. For example, the order time of the timestamp type is taken from the current system time, and the total inventory value associated with the external system is taken from the real-time inventory data.

[0055] Specifically, users can explicitly declare variables (such as current_timestamp, external_source_value) in the configuration script file to represent information that needs to be dynamically obtained at runtime (such as the current timestamp, real-time metrics from external data sources). These dynamic parameters, together with regular static parameters (such as fixed field types, preset data ranges), constitute complete configuration information. However, unlike the direct definition of static values, the values ​​of dynamic parameters are only obtained through external interfaces or real-time queries at script runtime, allowing the generated data rules to flexibly adapt to changes in business scenarios (such as generating time-sensitive data based on the current date, or referencing the latest configuration parameters from external systems).

[0056] Subsequently, when the data generation engine executes the script file to generate structured test data, it prioritizes parsing the dynamic parameters declared in the script (such as timestamps and external data source metrics) and obtains the latest values ​​in real time before each data generation through predefined interfaces (such as the system clock API and external database query services). For example, if the script file declares the dynamic parameter `current_date`, the data generation engine will call the system time interface to obtain the current date when generating each record; if the parameter is associated with an external data source (such as the real-time inventory of the business system), the data generation engine will query the latest value of that data source through the API.

[0057] Finally, the data generation engine combines the dynamically acquired parameters in real time with the statically generated field values ​​(such as random integers and fixed-length strings) according to the table structure order to generate a complete record that conforms to the business logic. This process ensures that the basic data structure conforms to the user configuration (such as field types and relationships), and keeps the data content synchronized with the actual business scenario through real-time updates of dynamic parameters, ultimately generating simulated data that combines static rule constraints with dynamic business adaptability.

[0058] Optionally, after generating corresponding simulated values ​​by calling a random data generation algorithm for static fields in the metadata model, the above method further includes: Based on the logical relationships between fields, the correlation between the simulated values ​​of each static field is verified; if the simulated value of a static field violates the logical relationships between fields, the simulated value of that static field is processed based on a preset hierarchical processing mechanism.

[0059] In this embodiment, when the data generation engine executes the script file to generate structured test data, it parses and executes the logical relationship verification module embedded in the script in real time. For each generated data record (or data batch), after the engine completes the basic field value filling, it immediately triggers the logical relationship detection process: according to user-defined logical rules (such as numerical comparison, time series constraints, and related field matching), it verifies the correlation between fields, for example, checking whether "order amount" is indeed less than or equal to the value of the associated "total inventory value" field. Furthermore, this detection process runs through every step of data generation, ensuring that each record meets the pre-configured business logic constraints at the time of generation.

[0060] If the generated mock value is detected to violate the logical relationship between fields, such as "end time is earlier than start time" or "foreign key value does not exist in the associated table", the mock value of the static field will be processed based on the preset hierarchical processing mechanism.

[0061] Specifically, if the violation level is Level 1, the simulated value of the static field will be corrected and a correction log will be recorded; if the violation level is Level 2, the simulated value of the static field will be marked as abnormal data and an alert report containing anomaly statistics will be generated.

[0062] Violation level 1 refers to simulated values ​​that violate logical relationships but can be automatically corrected and recovered, with the corrected data still conforming to business rules and constraints. These violations are typically caused by random value generation deviations or boundary condition conflicts. While they affect data accuracy, they do not disrupt the overall logical structure of the data or cause system processing interruptions. For example, a numeric field might slightly exceed the user-defined range (e.g., an order amount configured as 1-100 yuan, but 101 yuan is generated), or the logical order of a time field and its associated time field might be slightly reversed (e.g., the shipping time is only 1 second earlier than the order placement time, but business requirements dictate it must be later). In such cases, the field values ​​can be adjusted using an automatic correction mechanism (e.g., reducing the order amount proportionally to 90% of the total inventory value, or fine-tuning the timestamp to after the start time) to obtain data conforming to logical relationships, and a correction log can be recorded. A Level 2 violation indicates a simulated value that violates logical relationships and cannot be recovered through simple correction, or whose correction may trigger other related problems. These violations are typically caused by missing related fields, hard constraint conflicts, or unmet complex business rules. Forced correction may break the data logic chain or cause system anomalies. For example, a foreign key field value may not exist in the primary key set of the related tables (e.g., "Customer ID" in the order table is not defined in the customer table), or a hard constraint conflict (e.g., "Order amount must be ≤ total inventory value," but the order amount is still positive when the total inventory value is 0). In such cases, the simulated value is marked as abnormal data, error details are retained (e.g., specific field, violated logical rule, generation batch), and an alert report containing error statistics is generated (e.g., "12 records were detected violating inventory constraints, 5 records had missing foreign keys," etc.) to ensure that the final structured test dataset conforms to the user-defined business logic constraints as a whole, maintaining the logical integrity and business authenticity of the data.

[0063] S130. Based on structured test data, generate an SQL script that conforms to the target database syntax.

[0064] The SQL script is a standardized set of instructions that converts structured test data into a syntax that conforms to the target database. It includes two categories: DDL (Data Definition Language), which contains SQL statements used to create table structures (e.g., `CREATE TABLE order_table(orderID INT, customerID INT, order amount DECIMAL(10,2), PRIMARY KEY (orderID), FOREIGN KEY (customerID) REFERENCES customer_table(customerID)`) and DML (Data Manipulation Language), which contains SQL statements used to insert data (e.g., `INSERT INTO order_table VALUES(1001,205,150.00,'2025-02-11 14:30:00')`).

[0065] The target database refers to the database management system (DBMS) where the test data is ultimately stored. It supports multiple types (such as relational databases MySQL, Oracle, PostgreSQL, or distributed databases TiDB), must be compatible with the generated SQL script syntax, and provide data storage and query capabilities.

[0066] Please refer to Figure 3 Step S130 generates an SQL script that conforms to the target database syntax based on the structured test data, including: S130-1: Based on the type of the target database, obtain the syntax adaptation rule set corresponding to the target database.

[0067] Among them, the syntax adaptation rule set is a set of predefined SQL syntax specifications for different database management systems (DBMS) to address the differences in basic syntax structure, field type definitions, constraint rules and statement formats between different databases.

[0068] Specifically, based on the target database type (such as MySQL, Oracle, PostgreSQL) specified by the user during the configuration phase, the system loads the database-specific SQL syntax specification set from the built-in rule base to provide an adaptation basis for subsequent field type conversion and statement generation.

[0069] For example, when the user selects MySQL as the target database, the MySQL rule set is loaded to obtain field type mappings (such as STRING→VARCHAR(255)), identifier reference characters (backticks `), and auto-incrementing primary key syntax (AUTO_INCREMENT); if Oracle is selected, the Oracle rule set is loaded (such as STRING→VARCHAR2(255), double quote identifiers, and sequence mechanism).

[0070] S130-2: Based on a preset field type mapping table, the general type definition of each field in the structured test data is converted into the specific field type corresponding to the target database.

[0071] The field type mapping table is a built-in table that maps general types to database-specific types. It is used to automatically convert general field types (such as INTEGER, STRING, DATETIME) in user configuration or structured test data into specific field types supported by the target database.

[0072] Specifically, by traversing the metadata model of the structured test data (including table names, field names, general types, and constraints), for each field, the specific field type corresponding to the target database is found through the field type mapping table. For example, if the field order_amount (general type INTEGER) is defined in the structured test data, it is converted to INT when the target database is MySQL; and converted to NUMBER (10) when the target database is Oracle.

[0073] S130-3: Generate DDL creation statements and DML insert statements based on structured test data and syntax adaptation rule sets of specific field types.

[0074] DDL creation statements are used to define table structures (field names, specific field types, constraint rules) and conform to the syntax of the target database. DML insert statements are used to insert specific records into the structured test data and conform to the insert syntax of the target database (such as batch insert or single row insert).

[0075] S130-4: Integrate DDL creation statements and DML insert statements in the order of table structure to obtain an SQL script that conforms to the target database syntax.

[0076] Specifically, by integrating all DDL creation statements and DML insert statements generated in step S130-3 in logical order to form a complete SQL script file, the script is ensured to be directly executable in the target database. Integration rules include: Execution order: First, execute all DDL creation statements to create the table structure, then execute all DML insert statements to insert the actual data, avoiding insertion failure due to the table not existing; Table order: Arrange DDL creation statements and DML insert statements according to the table order configured by the user or alphabetical order. For example, first generate the table creation and insert statements for the customers table, and then generate the orders table. Script formatting: Add comment markers (such as --=====DDL section=====), separators, and error handling prompts to improve script readability.

[0077] Therefore, this solution addresses the syntax differences between different databases through a syntax adaptation rule set, achieves precise conversion from general types to specific types through a field type mapping table, generates and integrates data through DDL creation statements and DML insert statements, and finally outputs standardized SQL scripts that conform to the target database syntax. This approach ensures that structured test data can be realistically implemented in mainstream databases such as MySQL, Oracle, and PostgreSQL, providing a reliable data foundation for database testing, performance verification, and feature development.

[0078] S140. Execute the SQL script in the target database to obtain test data.

[0079] The test data is a real data set generated by executing SQL scripts in the target database, representing the physical storage format of structured test data. Its characteristics include: authenticity (field values ​​conform to business logic, such as order amount ≤ inventory value) and realistic distribution (such as a high proportion of VIP customers); availability (it can be directly used for database performance testing, synchronization tool verification (such as DATAX), functional regression testing, and other scenarios); and scalability (supporting complex scenarios such as millions of records in a single table and multi-table joins, such as 100 tables with 1 million records each).

[0080] Specifically, by directly executing SQL scripts (DDL+DML) in the target database, users can quickly build a test environment, such as a real test environment containing 100 tables, 50 fields per table, and 1 million records. These table structures and data fully conform to user-configured rules (such as field types, primary key constraints, and data distribution), and possess characteristics of large scale, multi-table joins, and high complexity. They can be directly used for synchronization capability testing, performance stress testing, and functional verification of big data platforms (such as DATAX), ultimately providing a solid testing foundation for the system's stability and reliability.

[0081] Traditional test data preparation requires manual writing of SQL scripts or the use of semi-automated tools for step-by-step operations (such as creating tables first and then inserting data). However, the embodiments of this application form an automated pipeline from parameter definition to data storage through an end-to-end automated process of "configuration → script generation → data generation → SQL conversion → database execution", which significantly improves the efficiency of test data generation and can provide high-quality test data for subsequent software testing.

[0082] The following example, using the e-commerce order system test data generation scenario, illustrates the process of generating the aforementioned test data: 1. Configuration Information: Users define three tables (Customer Table, Product Table, and Order Table) through a graphical configuration interface. The Order Table contains fields such as Order ID (integer), Customer ID (foreign key related to the Customer Table), Product ID (foreign key related to the Product Table), Order Amount (numerical), Order Time (dynamic timestamp), and Shipping Address (string). The logical relationships between the fields include: Order Amount ≤ Total Inventory Value of the Product Table, Order Time > Registration Time of the Customer Table, and foreign key constraints (Customer ID and Product ID must exist in the primary key of the related table).

[0083] 2. Parse the configuration information, generate script files and create simulated data (such as 1 million order records, with order amounts generated by a random algorithm but automatically checked to be less than or equal to the inventory value of the associated products, and the order time taken as the current system time and ensured to be later than the customer registration time).

[0084] 3. Convert the data into an SQL script that conforms to MySQL syntax (including CREATE TABLE statement and INSERTINTO statement).

[0085] 4. Execute SQL scripts in the MySQL database to generate a real data environment that can be directly used for testing order synchronization tools (such as DATAX).

[0086] Therefore, the generated test data conforms to e-commerce business logic (such as reasonable order amounts and correct time order), and verifies the data synchronization tool's ability to handle multi-table joins, large number of rows (1 million rows in a single table), and dynamic time fields, avoiding synchronization failures caused by dirty data.

[0087] To better implement the above methods, this application also provides a test data generation device, which can be integrated into an electronic device, such as a terminal or server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, or personal computer; the server can be a single server or a server cluster composed of multiple servers.

[0088] For example, in this embodiment, the method of this application embodiment will be described in detail by taking the test data generation device as specifically integrated into the server.

[0089] For example, such as Figure 4 As shown, the test data generation device may include a configuration module 410, a parsing module 420, a script generation module 430, and a script execution module 440, as follows: Configuration module 410 is used to generate script files based on user-defined configuration information; the configuration information includes at least data fields, logical relationships between fields, and data distribution; The parsing module 420 is used to generate simulated structured test data based on script files; The script generation module 430 is used to generate SQL scripts that conform to the syntax of the target database based on structured test data. The script execution module 440 is used to execute SQL scripts in the target database to obtain test data.

[0090] Optionally, the parsing module 420 includes: The conversion unit is used to parse the script file through the script parser and convert the script file into an intermediate representation format; The generation unit is used to generate simulated structured test data based on an intermediate representation format through the data generation engine.

[0091] Optionally, the data fields include static fields, and the above transformation unit includes: Extract sub-units, used to analyze the script content line by line through the script parser and extract key configuration parameters; Construct sub-units to build a metadata model of the table structure based on key configuration parameters, and determine the field composition and constraint rules of each table; The above-mentioned generation unit includes: The simulated value generation subunit is used to call a random data generation algorithm to generate corresponding simulated values ​​for static fields in the metadata model; The simulated value assembly subunit is used to assemble the simulated values ​​corresponding to each static field according to the table structure order to obtain simulated structured test data.

[0092] Optionally, the analog value generation sub-unit is also used for: Based on the logical relationships between fields, the correlation between the simulated values ​​of each static field is verified; If the simulated value of a static field violates the logical relationship between fields, the simulated value of the static field will be processed based on a preset hierarchical processing mechanism.

[0093] Optionally, the analog value generation sub-unit is also used for: If the violation level is Level 1, the simulated value of the static field will be corrected and the correction log will be recorded. If the violation level is level two, the simulated value of the static field will be marked as abnormal data, and an alert report containing abnormal statistics will be generated.

[0094] Optionally, the script generation module 430 includes: The syntax rule acquisition unit is used to acquire the syntax adaptation rule set corresponding to the target database based on the type of the target database; The field mapping unit is used to convert the general type definition of each field in the structured test data into the specific field type corresponding to the target database based on a preset field type mapping table. The statement generation unit is used to generate DDL creation statements and DML insert statements based on structured test data of specific field types and syntax adaptation rule sets. The statement integration unit is used to integrate DDL creation statements and DML insert statements in the order of table structure to obtain an SQL script that conforms to the target database syntax.

[0095] Optionally, the parsing module 420 is also used for: Parse the dynamic fields in the script file and obtain the real-time dynamic parameter values ​​through a predefined interface; The real-time dynamic parameter values ​​are combined with the simulated values ​​of the remaining data fields in the order of the table structure to generate simulated structured test data.

[0096] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0097] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0098] This application also provides an electronic device, which can be a terminal, a server, or other similar device. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.

[0099] In some embodiments, the test data generation device may also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the test data generation method of this application.

[0100] In this embodiment, as Figure 5 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically: The electronic device may include components such as a processor 510 with one or more processing cores, a memory 520 with one or more computer-readable storage media, a power supply 530, an input module 540, and a communication module 550. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 510 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines, and performs various functions and processes data by running or executing software programs and / or modules stored in the memory 520, and by calling data stored in the memory 520, thereby performing overall detection of the electronic device. In some embodiments, the processor 510 may include one or more processing cores; in some embodiments, the processor 510 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 510.

[0101] The memory 520 can be used to store software programs and modules. The processor 510 executes various functional applications and data processing by running the software programs and modules stored in the memory 520. The memory 520 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 520 may also include a memory controller to provide the processor 510 with access to the memory 520.

[0102] The electronic device also includes a power supply 530 that supplies power to the various components. In some embodiments, the power supply 530 can be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. The power supply 530 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0103] The electronic device may also include an input module 540, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0104] The electronic device may also include a communication module 550. In some embodiments, the communication module 550 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 550 can be used to help users send and receive emails, browse web pages, and access streaming media.

[0105] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 510 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 520 according to the following instructions, and the processor 510 runs the applications stored in the memory 520, thereby realizing the various functions in the above-described test data generation method.

[0106] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0107] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the test data generation methods provided in embodiments of this application.

[0108] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0109] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the test data generation method provided in the above embodiments.

[0110] Since the instructions stored in the storage medium can execute the steps in any of the test data generation methods provided in the embodiments of this application, the beneficial effects that any of the test data generation methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0111] The above provides a detailed description of a test data generation method, apparatus, electronic device, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A test data generation method, characterized by, The method comprises: generating a script file based on user-defined configuration information; the configuration information at least includes data fields, logical relationships between fields, and data distribution; generating simulated structured test data based on the script file; generating SQL scripts conforming to the syntax of a target database based on the structured test data; executing the SQL scripts in the target database to obtain test data.

2. The test data generation method of claim 1, wherein, Generating simulated structured test data based on the script file comprises: parsing the script file through a script parser to convert the script file into an intermediate representation format; generating simulated structured test data based on the intermediate representation format through a data generation engine.

3. The test data generation method of claim 2, wherein, The data fields include static fields, and parsing the script file through a script parser to convert the script file into an intermediate representation format comprises: analyzing script content line by line through a script parser to extract key configuration parameters; based on the key configuration parameters, constructing a metadata model of table structure and determining the field composition and constraint rules of each table; generating simulated structured test data based on the intermediate representation format through a data generation engine comprises: for the static fields in the metadata model, calling a random data generation algorithm to generate corresponding simulation values; assembling the simulation values of each static field in the order of table structure to obtain simulated structured test data.

4. The test data generation method of claim 3, wherein, After calling a random data generation algorithm to generate corresponding simulation values for the static fields in the metadata model, the method further comprises: verifying the association between the simulation values of each static field based on the logical relationships between the fields; if the simulation values of the static fields violate the logical relationships between the fields, processing the simulation values of the static fields based on a preset hierarchical processing mechanism.

5. The test data generation method of claim 4, wherein, Processing the simulation values of the static fields based on a preset hierarchical processing mechanism comprises: if the violation level is level one, correcting the simulation values of the static fields and recording correction logs; if the violation level is level two, marking the simulation values of the static fields as abnormal data and generating a reminder report containing abnormal statistics.

6. The test data generation method of claim 1, wherein, Generating SQL scripts conforming to the syntax of a target database based on the structured test data comprises: based on the type of the target database, obtaining a syntax adaptation rule set corresponding to the target database; based on a preset field type mapping table, converting the general type definition of each field in the structured test data into a specific field type corresponding to the target database; based on the structured test data of the specific field type and the syntax adaptation rule set, generating DDL creation statements and DML insertion statements; integrating the DDL creation statements and the DML insertion statements in the order of table structure to obtain SQL scripts conforming to the syntax of the target database.

7. The test data generation method of claim 1, wherein, The data fields also include dynamic fields, and generating simulated structured test data based on the script file comprises: parsing the dynamic fields in the script file and obtaining real-time dynamic parameter values through a predefined interface; The real-time dynamic parameter value is combined with analog values of the rest of the data fields in a table structure order to generate analog structured test data.

8. A test data generation apparatus characterized by comprising: The device comprises: A configuration module configured to generate a script file based on user-defined configuration information, wherein the configuration information at least includes data fields, logical relationships between the fields, and data distribution; An analysis module configured to generate analog structured test data based on the script file; A script generation module configured to generate an SQL script conforming to a target database syntax based on the structured test data; A script execution module configured to execute the SQL script in the target database to obtain test data.

9. An electronic device, comprising: The device comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads the instructions from the memory to execute steps in the test data generation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute steps in the test data generation method according to any one of claims 1-7.

Images