Method and apparatus for generating test data for mass database tables based on configuration

Abstract

The application discloses a method and device for generating test data of a mass database table based on configuration, and a method for implementing the same, and the method comprises the following steps: obtaining table building information, extracting table metadata from the table building information, and performing standardization processing on the table metadata to obtain standardized table metadata; obtaining a data generation template based on the standardized table metadata; modifying the data generation template according to target requirements to generate a target data generation template; extracting global variable information in a global configuration node of the target data generation template; analyzing data generation rules corresponding to each field based on the global variable information, and generating test data based on the field generation rules; and generating an insertion statement based on the test data, writing the insertion statement into a preset file, and obtaining a final test data file. According to the embodiment of the application, the whole process from table metadata extraction to test data file can be automatically realized, the data real correlation is guaranteed, and the test effectiveness is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of test data construction, and in particular relates to a method and device for generating test data for a mass database table based on configuration. BACKGROUND

[0002] In software development, especially in database-intensive application development, the quality and scale of test data are crucial for system correctness verification and performance testing. However, the traditional test data generation method has significant defects and cannot meet the efficient and flexible testing requirements.

[0003] Traditionally, test data generation relies on manual writing, stored procedures or customized code development. Manual writing is extremely inefficient and time-consuming when dealing with massive data. Stored procedures and code solutions require high technical requirements, which are difficult for ordinary testers to master. Moreover, the generation logic is fixed and can only be applied to specific tables, which lacks universality and cannot adapt to complex business scenarios.

[0004] Existing test data generation based on databases can generate basic data using database syntax, but still has many shortcomings: generating massive data requires a lot of time and cannot quickly meet large-scale testing requirements; it relies on stored procedures or code, which requires personnel to have deep technical expertise, making it difficult for testers to get started; it is developed for specific tables and lacks universal generation capabilities, requiring repeated development and adaptation for different tables or business scenarios; the generation logic is limited by fixed rules and cannot handle diverse and complex business testing scenarios, such as simulating real user behavior links and special business rule data. These problems make the test data generation process a bottleneck in the development process, affecting test coverage and efficiency, and may delay project progress and increase quality risks. SUMMARY

[0005] Therefore, it is necessary to provide a method and device for generating test data for a mass database table based on configuration to solve at least one of the problems existing in the prior art.

[0006] In a first aspect, a method for generating test data for a mass database table based on configuration is provided, comprising:

[0007] Obtaining table creation information, extracting table metadata from the table creation information, and performing standardized processing on the table metadata to obtain standardized table metadata;

[0008] Generating a data generation template based on the standardized table metadata;

[0009] Modifying the data generation template based on target requirements to obtain a target data generation template;

[0010] extract global variable information in a global configuration node of the target data generation template;

[0011] generate test data based on the data generation rule corresponding to each field according to the global variable information;

[0012] generate an insert statement based on the test data, write the insert statement into a preset file, and obtain a final test data file, wherein the insert statement refers to an operation statement in a structured query language for adding a new data record to a database table.

[0013] In a second aspect, a device for generating test data for a mass database table based on configuration is provided, comprising:

[0014] a table building information acquisition unit configured to acquire table building information, extract table metadata from the table building information, and perform standardized processing on the table metadata to obtain standardized table metadata;

[0015] a template generation unit configured to generate a data generation template based on the standardized table metadata;

[0016] a template modification unit configured to modify the data generation template based on target requirements to obtain a target data generation template;

[0017] a global variable processing unit configured to extract global variable information in a global configuration node of the target data generation template;

[0018] a test data generation unit configured to parse a data generation rule corresponding to each field based on the global variable information, and generate test data based on the field generation rule;

[0019] a final test data file generation unit configured to generate an insert statement based on the test data, write the insert statement into a preset file, and obtain a final test data file, wherein the insert statement refers to an operation statement in a structured query language for adding a new data record to a database table.

[0020] The above-mentioned method and device for generating massive database table test data based on configuration, its method implementation includes: obtaining table creation information, extracting table metadata from the table creation information, and standardizing the table metadata to obtain standardized table metadata; generating a data generation template based on the standardized table metadata; modifying the data generation template based on target requirements to obtain a target data generation template; extracting global variable information in the global configuration node of the target data generation template; parsing the data generation rules corresponding to each field based on the global variable information, and generating test data based on the field generation rules; generating an insert statement based on the test data, writing the insert statement into a preset file to obtain a final test data file, wherein the insert statement refers to an operation statement in structured query language for adding new data records to a database table. In the embodiment of the present application, the entire process from table metadata extraction to test data file generation is automated, and the efficiency is particularly significantly improved in massive data generation scenarios, which greatly reduces the time spent on test data generation; there is no need to rely on complex stored procedures or manual code writing, and relying on templated configuration and extensible visual modification, the technical entry threshold is lowered, and ordinary testers can get started quickly, reducing manpower and learning costs; based on standardized table metadata and customizable templates, it is adapted to different database types and business scenarios, has general generation capabilities, and avoids repeated development; supports flexible configuration of business rules, data ranges and associations, accurately simulates complex business scenarios, and the generated data can effectively verify the correctness and robustness of the system, and broaden the test boundary coverage; with the help of preset constraints and association processing, the authenticity and relevance of the test data are guaranteed, meeting the database integrity requirements and improving the effectiveness of the test. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 This is an application environment diagram of a method for generating massive database table test data based on configuration in an embodiment of the present application;

[0023] Figure 2 This is a flow chart of a method for generating massive database table test data based on configuration in one embodiment of the present application;

[0024] Figure 3 This is a flow chart of a method for generating a data template in one embodiment of the present application. Figure 1 ;

[0025] Figure 4 is a flowchart of a data generation template generation method in an embodiment of the present application Figure 2

[0026] Figure 5 is a flowchart of a data template perfecting method in an embodiment of the present application

[0027] Figure 6 is a flowchart of a data generation method extension method in an embodiment of the present application

[0028] Figure 7 is a flowchart of a global variable analysis method in an embodiment of the present application

[0029] Figure 8 is a flowchart of a test data generation method in an embodiment of the present application Figure 1

[0030] Figure 9 is a flowchart of a test data generation method in an embodiment of the present application Figure 2

[0031] Figure 10 is a data generation process example diagram in an embodiment of the present application

[0032] Figure 11 is a test data file sample diagram in an embodiment of the present application

[0033] Figure 12 is a structure diagram of an apparatus for generating test data for a mass database table based on configuration in an embodiment of the present application

[0034] Figure 13 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0036] The method for generating test data for a mass database table based on configuration provided by the present embodiment can be applied in, for example Figure 1 ​​​In the system environment, the system comprises a database statement processing module, a yaml file processing module, a template processing module, a faker extension module, a database type data generation module, and a data file generation module. The database statement processing module is responsible for parsing the CREATE TABLE statement, obtaining the table

field name

field type

field annotation

[0037] Specifically, the table structure standardized processing is performed on the table structure after the user input table structure information is input into the database statement processing module, and the standardized processing table structure is transmitted to the yaml file processing module to generate a data generation template according to a default rule. The user can specify the data generation rule as needed, modify the content of the generated template according to the requirement, and obtain a target data generation template. Then, the target data generation template is transmitted to the yaml file processing module, the env node information in the target data generation template is extracted and transmitted to the template processing module, the loading and generation of global variables are completed, and the tables in the template file are transmitted to the template processing module. If tables.table.field.genRule has a value, the constants, python system functions, built-in object functions (Faker objects), jinja2 templates, and custom functions in the genRule node will generate corresponding test data; if tables.table.field.genRule has no value and the type of the same level node has a value, the type is transmitted to the database type data generation module, which will randomly generate corresponding test data according to the corresponding type. The generated test data is transmitted to the database statement processing module, which will finally generate corresponding insert statements. The generated insert statements are transmitted to the data file generation module to complete the generation of the final test data file.

[0038] In an embodiment, as shown in Figure 2 A method for generating test data for a large database table based on configuration is provided, comprising the following steps:

[0039] In step S110, table building information is acquired, table metadata is extracted from the table building information, and the table metadata is standardized to obtain standardized table metadata.

[0040] The table building information includes at least one of a table building statement, a file path, database connection, and a table name.

[0041] Optionally, the fields of the table, the data type, whether it is a primary key, and the like can be acquired by using a function provided by the database system, such as querying the information_schema.COLUMNS table in MySQL; the table building statement can be directly obtained, such as obtaining the creation statement of the table by using the show create table command, or the table building statement can be obtained by reading a text file, and the like. Through these operations, the structure information of the table can be collected, including the table name, the detailed definition of each field, the constraint condition, and the like.

[0042] Then, the key table metadata, such as the field name, the data type of the field, the field length, the field default value, the primary key, and the like, is extracted from the acquired table building information. The extracted table metadata is uniformly processed, and the table structure information of different databases is uniformly standardized. For example, similar types in different databases are uniformly named, the default value of the field is standardized, and the like, so as to obtain the standardized table metadata, and provide a uniform basis for subsequent data generation templates and the like.

[0043] In step S120, a data generation template is generated based on the standardized table metadata.

[0044] Optionally, the table data after the standardization, such as the field name, the type, the constraint, and the like, is organized into a hierarchical structure in the yaml format, each table corresponds to a node, the field can be used as a subnode, including the type, the length, and the like, and the basic attribute, and the default generation rule can be automatically filled, such as the primary key self-increment and the random value generation of the string field, so as to form the standardized data generation template.

[0045] The standardized data generation template can be as follows:

[0046]

[0047] The env is a global variable configuration node, and stores the global variable defined by the user for the subsequent genRule node of the tables node. The tables is a root node of the table information, and all table information is hung under the node.

[0048] In step S130, the data generation template is modified based on the target requirement to obtain a target data generation template;

[0049] Optionally, the genRule node in the data generation template can be perfected according to user requirements. The node supports system functions (such as random_int () for generating random numbers), built-in object functions (fake.name () of the Faker library for generating real names), jinja2 templates (reference to global variables and calculation through {{env.base_id+1}}), and constants (such as the fixed value "completed"), and the like. By flexibly combining, data generation from basic numerical values to complex business logic can be realized, and the diversified requirements of different test scenarios for data can be met. The perfected data generation template can be as follows:

[0050]

[0051] Among them, the genRule node defines the generation logic of field data, supports flexible combination of multiple rule types, and realizes generation of basic data to complex business data. System functions refer to general functions predefined by the template engine, which are used to realize basic data generation. The built-in object function is used to generate realistic simulation data. The jinja2 template is used to embed variables and logic through the template syntax to realize dynamic calculation. The constant directly specifies the fixed generated value of the field.

[0052] In step S140, global variable information is extracted from a global configuration node of the target data generation template;

[0053] Optionally, the global configuration node (env node) in the template is extracted, the user-defined global variables are stored in the env node, then the original values (which can be constants, expressions or functions) are parsed into actual available values, and finally the values are stored in the global variable dictionary for shared use by all fields in subsequent data generation.

[0054] For example, in the yaml template, there can be variable definitions such as base_id: 1000 and calc_expr: 100+5 under the env node. Extract both the variable names (such as base_id) and the corresponding original values (such as 1000, 100+5). If the variable value is a fixed value, directly use the fixed value as the actual value. If it is an expression such as fake.date(), the corresponding tool can be called to generate the real date. If it is a template syntax such as {{base_id+1}}, the global variable (such as base_id=1000) that has been parsed can be combined to calculate the real value. Store the processed variable name and actual value in a shared dictionary and store it in the program global shared memory. When generating data for each field, the required variable value can be obtained from here to ensure that the same set of parameters is used for different fields.

[0055] In step S150, the data generation rule corresponding to each field is parsed based on the global variable information, and the test data is generated based on the field generation rule.

[0056] Optionally, the table structure root node (tables node) in the target data generation template is extracted to obtain the structure information of each table: first, the table name and the included fields are determined, then the generation rule corresponding to each field (such as self-increment, generating a random name, etc.) is determined, and the dependency relationship between different table fields (such as the user ID of the order table referring to the ID of the user table) is identified. Then, the actual value can be calculated according to the generation rule of the field: if the genRule node has no data generation rule, the field data type is obtained, and a default fixed value is used based on the field data type; if it is a function rule (such as generating a random number, a real name), the corresponding tool is called to generate it; if it is an expression rule (such as calculating based on global variables), the result is calculated in combination with the global variable value; if it is a correlation rule (such as referring to the field value of another table), it is obtained from the generated associated table data. After generation, it is verified whether the data type matches the field definition (such as a numerical field cannot be filled with a string), to ensure that the generated value meets the field type constraint and the preset business rule, so as to obtain test data that meets the preset constraint condition.

[0057] In step S160, based on the test data, an insert statement is generated, and the insert statement is written into a preset file to obtain a final test data file, wherein the insert statement is an operation statement in the structured query language for adding new data records to a database table.

[0058] Optionally, according to the table structure and the generated field values, an insert statement (insert) is assembled according to the SQL syntax. Each statement explicitly indicates the table name, the field list and the corresponding values (such as INSERT INTO table name (field 1, field 2) VALUES ('value 1', 'value 2');). To improve efficiency, multiple statements are batched (such as 1000 statements per batch), reducing the number of interactions with the database. By defining the field order in the standardized table metadata, the field list of the insert statement is consistent with the table structure, avoiding syntax errors. To prevent excessive memory usage, a batch writing method can be used, such as assembling 1000 data into a batch SQL statement, writing to a file and clearing the temporary data. After all batch writing is completed, a test data file that conforms to the SQL format specification is generated, and the final test data file can be generated.

[0059] The embodiment of the present application provides a method for generating test data for a mass database table based on configuration, including: obtaining table creation information, extracting table metadata from the table creation information, and performing standardized processing on the table metadata to obtain standardized table metadata; generating a data generation template based on the standardized table metadata; modifying the data generation template based on target requirements to obtain a target data generation template; extracting global variable information in the global configuration node of the target data generation template; analyzing the data generation rules corresponding to each field based on the global variable information, and generating test data based on the field generation rules; based on the test data, generating an insert statement, and writing the insert statement into a preset file to obtain a final test data file, wherein the insert statement refers to an operation statement in a structured query language for adding new data records to a database table. In the embodiment of the present application, the whole process from extraction of table metadata to generation of test data file is automatically implemented, which is particularly efficient in the mass data generation scenario, greatly reducing the time consumption of test data generation; without relying on complex storage procedures or manual code writing, relying on template configuration and extensible visual modification, reducing the technical access threshold, ordinary test personnel can quickly start, reducing manpower and learning cost; based on standardized table metadata and customizable templates, adapting to different database types and business scenarios, having general generation ability, avoiding repeated development; supporting flexible configuration of business rules, data range and association, accurately simulating complex business scenarios, and the generated data can effectively verify the correctness and robustness of the system, widening the test boundary coverage; relying on preset constraint conditions and association processing, ensuring the authenticity and association of test data, meeting the database integrity requirements, and improving the test effectiveness.

[0060] In an embodiment of the present application, the table creation information includes a table creation statement, and the obtaining of the table creation information and the extraction of the table metadata therefrom include:

[0061] Obtaining a key parameter;

[0062] Determining a table creation statement obtaining mode based on the key parameter;

[0063] Reading the table creation statement from a text file or obtaining the table creation statement from a database based on the table creation statement obtaining mode;

[0064] Analyzing the table creation statement to obtain a field name and a data type.

[0065] As shown in Figure 3 , Figure 4 shown, a branch judgment is entered according to a type parameter passed in, to determine whether a template is generated in a text mode. If it is determined that the template is generated in the text mode, the system will locate and read a text file under a configuration path, to extract a table creation statement from the text file. The text file usually pre-stores standardized or customized table structure definition content. If it is determined that the template is not generated in the text mode, a show create table statement is called by means of database interaction capability, to dynamically obtain the table creation statement from a target database, to ensure that the table structure obtained is the latest state actually in effect in the database. After the table creation statement is obtained, a syntax structure of the table creation statement is analyzed by means of a SQL syntax parser or a regular expression, to accurately extract key meta information such as a field name, a data type, and a constraint condition (such as a primary key, non-empty, etc.), to parse the SQL text into a structured table metadata object and a table structure description that can be recognized by a subsequent process.

[0066] In an embodiment of the present application, the standardized table metadata includes a field name and a data type, and the generating a data generation template based on the standardized table metadata includes:

[0067] The field name and the data type are arranged into content in a yaml format according to a preset rule;

[0068] The content in the yaml format is written into a yaml file, to obtain the data generation template.

[0069] As shown in Figure 3 , Figure 4 shown, the parsed table name, field name, type, and other information are arranged into a yaml format according to a preset rule, and then written into a yaml file, to output a data generation template. The preset rule can be a template default rule, and can be used to define a structure of the yaml, such as a table name level, a field list format, and extended information such as an annotation and a generation rule.

[0070] In an embodiment of the present application, the modifying the data generation template based on a target requirement to obtain a target data generation template includes:

[0071] determine the database type associated with the data generation template, generate base data conforming to the database type;

[0072] generate an auto-increment id;

[0073] generate a snowflake id;

[0074] According to the field data type, randomly select type-matched data from the preset candidate value list;

[0075] Call the Chinese name generation logic to generate the corresponding Chinese name for the semantic field;

[0076] Execute business logic through a custom Python function to generate corresponding data;

[0077] Query the generated data in the associated table and reuse / associate it to fill in the current table field.

[0078] As shown in Figure 5 When the data generation template is obtained, in order to make the template generate more business-conforming and flexible data, the data generation template can be improved, and seven types of data generation strategies (database adaptation, ID generation, random selection, Chinese name, custom function, cross-table association) can be added layer by layer around the data generation template, so that the template can generate more rich and real-scene-conforming data to meet complex test / business requirements.

[0079] Optionally, if the target field requirement is to generate corresponding data according to the database type: no modification is required at this point. If the target field requirement is an auto-increment id: modify the target field genRule to "next (iterator variable name defined by the template env node)". If the target field requirement is a snowflake id: modify the target field genRule to "fake.snow_id()". If the target field requirement is a uuid: modify the target field genRule to "fake.uuid()". If the target field requirement is to randomly select one: modify the target field genRule to "fake.random_choice(['Zhang San', 'Li Si', 'Wang Wu'])". If the target field requirement is to generate a Chinese name: modify the target field genRule to "fake.name()". If the target field requirement is to be implemented through a custom Python function: modify the target field genRule to "custom package name::function name (parameters)". If the target field requirement is the primary key of the previous table: modify the target field genRule to "{{sword_order_info_202410.id}}".

[0080] Specifically, by identifying the database type associated with the template (such as MySQL, Oracle), the corresponding type generator (such as mysql_datetime() to generate YYYY-MM-DD HH:ii:ss format, oracle_timestamp() to generate time with time zone) is called to ensure the legality of the data base type. Based on the type legality of the base data (such as confirming that the id is of BIGINT type), a global / table-level counter is maintained for the table primary key unique identification field (such as user_id), and each time it is generated, a continuous, ordered and incremental unique id (such as 1, 2, 3...) is output. For business identification fields, the Snowflake algorithm is called to generate a distributed unique ID (such as 147258369123456789) through timestamp, machine ID, sequence number, etc. For discrete fields (such as gender, status), type-compatible values (such as string type fields selecting string values) are randomly selected from the preset candidate list for filling. For semantic fields, Faker library fake.name(), fake.address() or custom Chinese name library can be called to generate corresponding Chinese names for Chinese semantic fields. For business-specific scenarios, Python functions can be called to execute, covering customized needs that Faker / base rules cannot meet. Query the associated table of other generated data, cache its field values, and select and fill them into the current table fields in order to form real business association for multi-table data.

[0081] In an embodiment of the present application, after generating the data generation template based on the standardized table metadata, the method comprises:

[0082] Implementing a subclass FakerExt of faker.Faker;

[0083] Implementing the FakerExt.uuid method;

[0084] Implementing the FakerExt.auto_increment method;

[0085] Implementing the FakerExt.random_choice method;

[0086] Implementing the FakerExt.snow_id method;

[0087] Instantiating the FakerExt object and storing the object in the global dictionary.

[0088] For example, Figure 6As shown, due to the fact that Faker itself does not support the above-mentioned ordered self-increment id distributed unique snowflake id, etc., through the faker.Faker extension class, some common data generation capabilities are extended, such as snowflake id, auto-increment ID, random selection, etc.

[0089] Specifically, by creating a faker.Faker subclass FakerExt and implementing uuid (generating UUID with business identification), auto_increment (constructing ordered self-increment sequence based on global / table level counter), random_choice (supporting random selection with weight and exclusion logic), snow_id (generating distributed unique snowflake ID) and other exclusive methods, the limitations of Faker's original functions are broken through, the needs of businesses for customized and refined data generation are met, and the complex architecture scenarios such as monomer and distributed are adapted; At the same time, the FakerExt object is stored in the global dictionary, providing a rich and controllable set of generation methods for data templates, ensuring the consistency and reusability of multi-link data generation, and giving the entire data generation system (covering template configuration to real business data output) customized and landable capabilities from the bottom, so that the generated data can accurately fit the business rules, support testing, initialization and other practical applications.

[0090] It should be noted that FakerExt is the basis of template generation and modification, and the method preset rules are relied on during template generation, and the method implementation customization is relied on during template modification. Finally, the global variable is called and executed during the data generation stage.

[0091] In an embodiment of the present application, the global variable information of the global configuration node in the target data generation template is extracted, and the global variable information is loaded and preprocessed, including:

[0092] Based on the template structure of the target data generation template, the global configuration node is determined;

[0093] Extract all variables and variable expressions corresponding to the global configuration node;

[0094] Add all the variables and the variable expressions to the global variable dictionary and store them on the global memory context of the current program for direct calling in the subsequent data generation step.

[0095] As Figure 7As shown, the data generation parameters such as the template file path, the generated data file path, and the number of generated data groups are obtained, a target data generation template is loaded based on the data generation parameters using a template engine, in the data generation template, a global configuration is found and parsed, the global configuration node is usually a preset fixed node (such as an env node) in the template, which can be directly located through template structure definition, the env node stores global variables (such as in yaml format), and then the variables and variable expressions in the node are traversed, which are temporarily stored in a global variable dictionary, and the global variables are injected into a local execution context (locals()) to build an execution context.

[0096] In an embodiment of the present application, after adding all the variables and variable expressions to the global variable dictionary, the following steps are included:

[0097] If the variable expression is a custom function expression, the custom function expression is parsed and executed to obtain an actual variable value;

[0098] If the variable expression is a system function or built-in function expression, the system function or built-in function expression is executed to obtain an actual variable value;

[0099] The variable and the corresponding actual variable value are associated and stored in the global variable dictionary again.

[0100] As shown, Figure 7 After adding all the variables and variable expressions to the global variable dictionary, the variable expressions are processed, if the variable expression is a custom function, the package and function part are parsed, and then the actual variable value is obtained by using eval; if it is a system or built-in function, the actual variable value is directly obtained by using eval, and finally the processed variable and the corresponding actual variable value are stored in the global variable dictionary again, so as to realize the conversion from template static configuration to dynamic available global variable, provide unified, flexible and safe parameter support for data generation links, make complex data generation requirements be completed through template configuration, decouple configuration and logic, and improve maintainability and expansibility.

[0101] Wherein, the actual value code returned by the expression is as follows:

[0102] def get_expr_actual_value(expr, env):

[0103] """

[0104] Get the value after executing the py code

[0105] : param expr:

[0106] :return:

[0107] """

[0108] locals().update(**env)

[0109] if isinstance(expr, str):

[0110] # If it does not contain ::, it is considered a py system function, otherwise it is considered a custom function

[0111] if expr.find('::') == -1:

[0112] if "(" in expr and ")" in expr:# If this condition is met, it is considered a function

[0113] expr_value = eval(expr)

[0114] else:

[0115] expr_value = expr

[0116] else:

[0117] func_info = expr.split('::')

[0118] if len(func_info) == 2:

[0119] expr_value = evaluate_expr(func_info[1], func_info[0])

[0120] else:

[0121] expr_value = evaluate_expr(func_info[0])

[0122] return expr_value

[0123] else:

[0124] return expr

[0125] def evaluate_expr(expression, modules=None):

[0126] """

[0127] Performs an eval operation on the passed expression

[0128] :param expression:

[0129] :param modules:

[0130] :return:

[0131] """

[0132] if modules:

[0133] current_g_vars = globals()

[0134] # Import a\ab\abc, otherwise an error will be reported when evaling the abc method of the module

[0135] if modules.find('.')>-1:

[0136] subs = modules.split('.')

[0137] current = ''

[0138] for sub in subs:

[0139] current += sub if current == '' else '.' + sub

[0140] if not current_g_vars.get(current):

[0141] current_g_vars[current] = importlib.import_module(current)

[0142] cmd = f'''{modules}.{expression}'''

[0143] else:

[0144] cmd = f'''{expression}'''

[0145] try:

[0146] result = eval(cmd)

[0147] except Exception as e:

[0148] raise Exception(f'modules={modules},expression={expression},cmd={cmd},failure reason:{str(e)}')

[0149] return result

[0150] In one embodiment of the present application, the global variable information includes actual values ​​of global variables, and the data generation rules corresponding to each field are parsed based on the global variable information, and test data is generated based on the field generation rules, including:

[0151] Determine the root node and field definition of the table structure in the target data generation template;

[0152] Based on the field definition, determine whether each field in each table is configured with a data generation rule;

[0153] If so, generating test data based on the data generation rule;

[0154] If not, generate test data based on the field data type.

[0155] like Figure 8 、 Figure 9 As shown, when processing the root node (tables node) of the table structure in the data generation template file, different data generation strategies are executed based on whether tables.table.fieldname.genRule has a value. If genRule has a value, it indicates that a data generation rule exists; otherwise, no data generation rule exists. If genRule has a value, it parses constants, Python system functions, built-in object functions (such as Faker objects), Jinja2 templates, and custom functions within it to generate test data that conforms to business rules (such as prefixed order numbers and random amounts). If genRule has no value but the same-level node type has a value, the type value is passed as a parameter to the database type data generation module. This module generates basic test data that conforms to field type constraints based on the database type's default rules (such as generating random integers using INT and random strings using VARCHAR). This ensures data generation flexibility in complex business scenarios while reducing configuration costs for basic scenarios. Standardized test data is output through a unified interface, providing consistent data input for subsequent storage and verification, forming a complete template-driven data generation chain.

[0156] It should be noted that if Figure 9As shown, when processing the generation rule containing Jinja2 template syntax, the specific method is to load the global variable dictionary as the context environment through the Jinja2 template engine, and replace the template variables in the rule with actual values: first, initialize the template engine and import global variables, then compile the genRule rule string into a template object, and when calling the render() method, the engine will automatically replace the variables with the values in the global variable dictionary. If the rule contains function calls, the Faker object needs to be registered in the global variable dictionary in advance, and the function is executed directly to generate the result during rendering; for complex expressions, the engine will parse the operators and filters according to the Jinja2 syntax, and finally return the calculated string or numerical value as the test data of the field.

[0157] wherein the field value acquisition code is as follows:

[0158] def get_field_value(col_value_info: dict):

[0159] """Get the value corresponding to the field in the template"""

[0160] gen_rule = col_value_info.get('genRule')

[0161] if gen_rule:

[0162] gen_rule = template_util.template_render(gen_rule, env) # Replace variables in the template, follow jinja2

[0163] val = template_util.get_expr_actual_value(gen_rule, env)

[0164] else:

[0165] data_type = col_value_info.get('type')

[0166] start = data_type.find('(')

[0167] end = data_type.find(')')

[0168] if start>-1 and end>-1:

[0169] # Extract type and length

[0170] type_part = data_type[:start]# Type part: from the beginning to the position before the left bracket

[0171] length_part = data_type[start + 1:end]# Length part: remove the left and right brackets

[0172] val = gen_data_util.generate_data(type_part, length_part)

[0173] else:

[0174] val = gen_data_util.generate_data(data_type)

[0175] return val

[0176] The corresponding data code generated according to the database field type is as follows:

[0177] def generate_data(field_type, field_length=None, field_range=None):

[0178] """Generate random data based on field type and possible length or range"""

[0179] if field_type.lower() in ('int', 'bigint', 'tinyint'):

[0180] min_val = field_range[0] if field_range else 0

[0181] max_val = field_range[1] if field_range else 2 ** 31 - 1

[0182] return random.randint(min_val, max_val)

[0183] elif field_type.lower() == 'float':

[0184] min_val = field_range[0] if field_range else 0.0

[0185] max_val = field_range[1] if field_range else 1.0

[0186] return round(random.uniform(min_val, max_val), 2)# 假设保留两位小数

[0187] elif field_type.lower() in ['varchar', 'char']:

[0188] if field_length is None:

[0189] raise ValueError("Length must be specified for VARCHAR or CHARtypes.")

[0190] letters = string.ascii_letters + string.digits

[0191] return ''.join(random.choice(letters) for i in range(int(field_length)))

[0192] elif field_type.lower() in ('date', 'datetime'):

[0193] start_date = field_range[0] if field_range else datetime(1970, 1, 1)

[0194] end_date = field_range[1] if field_range else datetime.now()

[0195] delta = end_date - start_date

[0196] int_delta = (delta.days * 24 * 60 * 60) + delta.seconds

[0197] random_second = random.randrange(int_delta)

[0198] return str(start_date + timedelta(seconds=random_second))

[0199] # elif field_type.lower() == 'boolean':

[0200] #return random.choice([True, False])

[0201] elif field_type.lower() == 'decimal':

[0202] x, y = (int(item) for item in field_length.split(','))

[0203] # Set the decimal precision

[0204] getcontext().prec = x

[0205] # Generate the integer part

[0206] integer_part = random.randint(0, 10 ** (x - y) - 1)

[0207] # Generate the decimal part

[0208] fractional_part = random.randint(0, 10 ** y - 1)

[0209] # Combine integer and fractional parts

[0210] decimal_value = Decimal(f"{integer_part}.{fractional_part:0{y}d}")

[0211] return str(decimal_value)

[0212] else:

[0213] raise ValueError(f"Unsupported field type: {field_type}")

[0214] In one embodiment of the present application, generating an insert statement based on the test data, writing the insert statement into a preset file, and obtaining a final test data file includes:

[0215] Associating the table, the field, and the test data to form a table-field value relationship;

[0216] The table field value relationship is spliced ​​to generate an insert statement according to the preset grammatical rules;

[0217] The insert statement is written into a file of a preset format to form the final test data file.

[0218] like Figure 8 As shown, the tables, fields and their corresponding test data are associated to build a structured relationship between table, field and value, clarify the table and field to which each piece of data belongs, and ensure the strict correspondence between the data and the table structure; according to the preset grammatical rules (such as SQL insert statement grammar or CSV format specification), these structured table field value relationships are spliced ​​into the corresponding insert statement insert, so that the data has a format that can be recognized by the database or stored in the file; the spliced ​​insert statement insert is written into a file in the preset format (such as .sql file, .csv file, etc.) to form the final test data file, as shown. Figure 10 shows an example of the data generation process, Figure 11 The following figure shows an example of a test data file, which provides a standardized data source for subsequent operations such as database import and test case execution, thus achieving a complete closed loop from test data generation to implementation.

[0219] In the embodiment of the present application, the entire process from table metadata extraction to test data file generation is automated, and the efficiency is particularly significantly improved in massive data generation scenarios, which greatly reduces the time spent on test data generation; there is no need to rely on complex stored procedures or manual code writing, and relying on templated configuration and extensible visual modification, the technical entry threshold is lowered, and ordinary testers can get started quickly, reducing manpower and learning costs; based on standardized table metadata and customizable templates, it is adapted to different database types and business scenarios, has general generation capabilities, and avoids repeated development; supports flexible configuration of business rules, data ranges and associations, accurately simulates complex business scenarios, and the generated data can effectively verify the correctness and robustness of the system, and broaden the test boundary coverage; with the help of preset constraints and association processing, the authenticity and relevance of the test data are guaranteed, meeting the database integrity requirements and improving the effectiveness of the test.

[0220] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0221] In an embodiment, an apparatus for generating test data for mass database tables based on configuration is provided, which corresponds to the method for generating test data for mass database tables based on configuration in the above embodiments. As shown in the figure, the apparatus for generating test data for mass database tables based on configuration includes a table creation information acquisition unit 10, a template generation unit 20, a template modification unit 30, a global variable processing unit 40, a test data generation unit 50, and a final test data file generation unit 60. The functions of each module are described in detail as follows. Figure 12

[0222] The table creation information acquisition unit 10 is configured to acquire table creation information, extract table metadata from the table creation information, and perform standardization processing on the table metadata to obtain standardized table metadata.

[0223] The template generation unit 20 is configured to generate a data generation template based on the standardized table metadata.

[0224] The template modification unit 30 is configured to modify the data generation template based on target requirements to obtain a target data generation template.

[0225] The global variable processing unit 40 is configured to extract global variable information from a global configuration node of the target data generation template.

[0226] The test data generation unit 50 is configured to parse data generation rules corresponding to each field based on the global variable information, and generate test data based on the field generation rules.

[0227] The final test data file generation unit 60 is configured to generate an insert statement based on the test data, write the insert statement into a preset file, and obtain a final test data file, wherein the insert statement refers to an operation statement in a structured query language for adding new data records to a database table.

[0228] In an embodiment of the present application, the table creation information acquisition unit 10 is further configured to:

[0229] acquire key parameters;

[0230] determine the table creation statement acquisition method based on the key parameters;

[0231] read the table creation statement from a text file or acquire the table creation statement from a database based on the table creation statement acquisition method;

[0232] parse the table creation statement to obtain field names and data types.

[0233] In an embodiment of the present application, the template modification unit 30 is further configured to:​

[0234] The field name and data type are sorted into a yaml format content according to a preset rule;

[0235] The yaml format content is written into a yaml file to obtain the data generation template.

[0236] In an embodiment of the present application, the template modification unit 30 is further configured to:

[0237] Determine the database type associated with the data generation template, and generate basic data conforming to the database type;

[0238] Generate an auto-increment id;

[0239] Generate a snowflake id;

[0240] According to the field data type, randomly select type-matched data from a preset candidate value list;

[0241] Call the Chinese name generation logic to generate corresponding Chinese names for semantic fields;

[0242] Execute business logic through a custom Python function to generate corresponding data;

[0243] Query the generated data in the associated table, and reuse / associate it to fill the current table field.

[0244] In an embodiment of the present application, the device further comprises an extension module configured to:

[0245] Implement a subclass FakerExt of faker.Faker;

[0246] Implement the FakerExt.uuid method;

[0247] Implement the FakerExt.auto_increment method;

[0248] Implement the FakerExt.random_choice method;

[0249] Implement the FakerExt.snow_id method;

[0250] Instantiate the FakerExt object and store the object in a global variable dictionary.

[0251] In an embodiment of the present application, the global variable processing unit 40 is further configured to:

[0252] Determine the global configuration node based on the template structure of the target data generation template;

[0253] extracting all variables and variable expressions corresponding to the global configuration node;

[0254] adding all the variables and the variable expressions to a global variable dictionary and storing to a global memory context of a current program for direct calling by a subsequent data generation step.

[0255] In an embodiment of the present application, the global variable processing unit 40 is further configured to:

[0256] if the variable expression is a self-defined function expression, parsing and executing the self-defined function expression to obtain an actual variable value;

[0257] if the variable expression is a system function or a built-in function expression, executing the system function or the built-in function expression to obtain an actual variable value;

[0258] associating the variable with the corresponding actual variable value and re-storing to the global variable dictionary.

[0259] In an embodiment of the present application, the test data generation unit 50 is further configured to:

[0260] determining a root node and a field definition of a table structure in a target data generation template;

[0261] judging whether each field of each table is configured with a data generation rule based on the field definition;

[0262] if yes, generating test data based on the data generation rule;

[0263] if no, generating test data based on a field data type.

[0264] In an embodiment of the present application, the final test data file generation unit 60 is further configured to:

[0265] associating a table, a field and the test data to form a table-field-value relationship;

[0266] splicing the table-field-value relationship to generate an insert statement according to a preset syntax rule;

[0267] writing the insert statement into a file in a preset format to form the final test data file.

[0268] In the embodiments of the present application, the whole process automation from table metadata extraction to test data file generation is realized, which is particularly significant in the mass data generation scenario, greatly compresses the test data generation time; without relying on complex stored procedures or manually writing codes, relying on the templated configuration and the extensible visual modification, the technical access threshold is reduced, and ordinary test personnel can quickly get started, reducing the manpower and learning cost; based on the standardized table metadata and the customizable templates, different database types and business scenarios are adapted, and the universal generation capability is provided, avoiding repeated development; the business rules, data range and association relationship are flexibly configured, the complex business scenarios are accurately simulated, the generated data can effectively verify the correctness and robustness of the system, and the test boundary coverage is widened; the preset constraint conditions and the association relationship processing are used to guarantee the authenticity and association of the test data, meet the database integrity requirement, and improve the test effectiveness.

[0269] The specific limitations of the device for generating test data of a mass database table based on configuration can be referred to the limitations of the method for generating test data of a mass database table based on configuration in the above, which will not be repeated here. Each module in the device for generating test data of a mass database table based on configuration can be realized by software, hardware and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0270] In one embodiment, a computer device is provided, which can be a terminal device, and an internal structure diagram thereof can be as shown in Figure 13 The computer device includes a processor, a memory and a network interface connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer readable instructions. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer readable instructions are executed by the processor to implement a method for generating test data of a mass database table based on configuration. The readable storage medium provided in the embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0271] In the embodiments of the present application, a computer device is provided, which includes a memory, a processor and computer readable instructions stored in the memory and executable on the processor. When the processor executes the computer readable instructions, the steps of the method for generating test data of a mass database table based on configuration are implemented.

[0272] In the application embodiment, a readable storage medium is provided, and the readable storage medium stores computer readable instructions. When the computer readable instructions are executed by a processor, the steps of the method for generating test data of a mass database table based on configuration are implemented.

[0273] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by computer readable instructions instructing related hardware. The computer readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer readable instructions are executed, the processes of the above-mentioned embodiments can be included. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. The non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. The volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0274] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified. In actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-described functions.

[0275] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for generating massive database table test data based on configuration, characterized in that: The method comprises: Acquire table creation information, extract table metadata from the table creation information, and standardize the table metadata to obtain standardized table metadata; Generate a data generation template based on the standardized table metadata; Modify the data generation template based on the target requirements to obtain the target data generation template, including: determining the database type associated with the data generation template and generating basic data that conforms to the database type; generating an auto-increment ID; generating a snowflake ID; randomly selecting data of matching type from a preset candidate value list based on the field data type; calling the Chinese name generation logic to generate corresponding Chinese names for semantic fields; executing business logic through a custom Python function to generate corresponding data; querying the generated data in the associated table and reusing / associating it to fill the current table field; Extracting global variable information from the global configuration node of the target data generation template, and loading and preprocessing the global variable information, including: determining the global configuration node based on the template structure of the target data generation template; extracting all variables and variable expressions corresponding to the global configuration node; adding all the variables and variable expressions to a global variable dictionary, and storing them in the global memory context of the current program for direct call by subsequent data generation steps; wherein, after adding all the variables and variable expressions to the global variable dictionary, the steps include: if the variable expression is a custom function expression, parsing and executing the custom function expression to obtain an actual variable value; if the variable expression is a system function or a built-in function expression, executing the system function or the built-in function expression to obtain an actual variable value; associating the variables and the corresponding actual variable values, and re-storing them in the global variable dictionary; Parsing data generation rules corresponding to each field based on global variable information in the global memory context, and generating test data based on the data generation rules; Based on the test data, an insert statement is generated, wherein the insert statement refers to an operation statement in structured query language for adding new data records to a database table, the table and field are associated with the test data to form a table-field value relationship, and the table-field value relationship is spliced ​​to generate an insert statement according to preset grammatical rules; the insert statement is written into a preset file to obtain a final test data file.

2. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: The table creation information includes a table creation statement, and the acquiring of the table creation information and the extraction of table metadata from the table creation information include: Get key parameters; Determine the table creation statement acquisition method based on the key parameters; Based on the table creation statement acquisition method, read the table creation statement from a text file, or obtain the table creation statement from a database; Parse the table creation statement to obtain field names and data types.

3. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: The standardized table metadata includes field names and data types. Generating a data generation template based on the standardized table metadata includes: Arrange the field names and data types into YAML format according to preset rules; Write the content in the YAML format into a YAML file to obtain the data generation template.

4. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: After generating the data generation template based on the standardized table metadata, the method includes: Implement the FakerExt subclass of faker.Faker; Implement the FakerExt.uuid method; Implement the FakerExt.auto_increment method; Implement the FakerExt.random_choice method; Implement the FakerExt.snow_id method; Instantiate the FakerExt object and store it in the global variable dictionary.

5. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: The global variable information includes actual values ​​of global variables, and the data generation rules corresponding to each field are parsed based on the global variable information, and test data is generated based on the data generation rules, including: Determine the root node and field definition of the table structure in the target data generation template; Based on the field definition, determine whether each field in each table is configured with a data generation rule; If so, generating test data based on the data generation rule; If not, generate test data based on the field data type.

6. A device for generating massive database table test data based on configuration, characterized in that: The device comprises: a table creation information acquisition unit, configured to acquire table creation information, extract table metadata from the table creation information, and perform standardization processing on the table metadata to obtain standardized table metadata; A template generating unit, configured to generate a data generation template based on the standardized table metadata; The template modification unit is used to modify the data generation template based on target requirements to obtain a target data generation template, including: determining the database type associated with the data generation template and generating basic data that conforms to the database type; generating an auto-increment ID; generating a snowflake ID; randomly selecting data of matching type from a preset candidate value list based on the field data type; calling the Chinese name generation logic to generate corresponding Chinese names for semantic fields; executing business logic through a custom Python function to generate corresponding data; querying the generated data in the associated table and reusing / associating it to fill the current table field; a global variable processing unit, configured to extract global variable information from a global configuration node of the target data generation template, and load and preprocess the global variable information, including: determining the global configuration node based on the template structure of the target data generation template; extracting all variables and variable expressions corresponding to the global configuration node; adding all the variables and variable expressions to a global variable dictionary, and storing them in the global memory context of the current program for direct call by subsequent data generation steps; The global variable processing unit is further configured to: if the variable expression is a user-defined function expression, parse and execute the user-defined function expression to obtain an actual variable value; if the variable expression is a system function or a built-in function expression, execute the system function or the built-in function expression to obtain an actual variable value; associate the variable with the corresponding actual variable value, and store the resultant value in the global variable dictionary; A test data generating unit, configured to parse data generation rules corresponding to each field based on global variable information in a global memory context, and generate test data based on the data generation rules; The final test data file generation unit is used to generate an insert statement based on the test data, wherein the insert statement refers to an operation statement in the structured query language for adding a new data record to a database table, associates the table and field with the test data to form a table-field value relationship, and splices the table-field value relationship according to preset grammatical rules to generate an insert statement; writes the insert statement into a preset file to obtain a final test data file.

Images