Method and device for generating mass database table test data based on configuration

Abstract

The invention discloses a method and device for generating mass database table test data based on configuration, and the method comprises the steps: obtaining table building information, extracting table metadata from the table building information, and carrying out the standardization processing of the table metadata, and obtaining the standardized table metadata; obtaining a data generation template based on the standardized table metadata; modifying the data generation template according to a target demand to generate a target data generation template; extracting global variable information from a global configuration node of the target data generation template; analyzing a data generation rule corresponding to each field based on the global variable information, and generating test data based on the field generation rule; and based on the test data, generating an insertion statement, and writing the insertion statement into a preset file to obtain a final test data file. According to the embodiment of the invention, full-process automation from table metadata extraction to test data files can be realized, real association of data is guaranteed, and test effectiveness is improved.

Description

Technical Field

[0001] The present application relates to the technical field of test data construction, and in particular to a method and device for generating massive database table test data based on configuration. Background Art

[0002] In software development, especially for database-intensive applications, the quality and scale of test data are crucial for system correctness verification and performance testing. However, traditional test data generation methods have significant flaws and cannot meet the requirements of efficient and flexible testing.

[0003] Traditionally, test data generation relies on manual coding, stored procedures, or custom code development. Manual coding is extremely inefficient and time-consuming when dealing with massive amounts of data. Stored procedures and code solutions require high technical skills, making them difficult for ordinary testers to master. Furthermore, the generated logic is rigid and can only be applied to specific tables, resulting in poor versatility and inability to flexibly adapt to complex business scenarios.

[0004] While existing database-based test data generation can leverage the database's native syntax to generate basic data, it still suffers from numerous shortcomings: Generating massive amounts of data is time-consuming and difficult to rapidly meet large-scale testing requirements; It relies on stored procedures or code, requiring deep technical expertise and making it difficult for testers to master; It is often developed for specific tables and lacks universal generation capabilities, requiring repeated development and adaptation for different tables or business scenarios; Generation logic is constrained by fixed rules, making it difficult to address diverse and complex business testing scenarios, such as simulating real user behavior links and using specialized business rule data. These issues make test data generation a bottleneck in the development process, impacting test coverage and efficiency, which can delay project progress and increase quality risks. Summary of the Invention

[0005] Based on this, it is necessary to provide a method and device for generating massive database table test data based on configuration in order to solve at least one problem existing in the above-mentioned prior art.

[0006] In a first aspect, a method for generating massive database table test data based on configuration is provided, comprising: Acquire table creation information, extract table metadata from the table creation information, and standardize the table metadata to obtain standardized table metadata; Generate a data generation template based on the standardized table metadata; Modify the data generation template based on target requirements to obtain a target data generation template; Extracting global variable information from a global configuration node of the target data generation template; Parsing data generation rules corresponding to each field based on global variable information, and generating test data based on the field generation rules; Based on the test data, an insert statement is generated, and the insert statement is written into a preset file to obtain a final test data file, wherein the insert statement refers to an operation statement in structured query language for adding new data records to a database table.

[0007] In a second aspect, a device for generating massive database table test data based on configuration is provided, comprising: a table creation information acquisition unit, configured to acquire table creation information, extract table metadata from the table creation information, and perform standardization processing on the table metadata to obtain standardized table metadata; A template generating unit, configured to generate a data generation template based on the standardized table metadata; A template modification unit, configured to modify the data generation template based on target requirements to obtain a target data generation template; A global variable processing unit, configured to extract global variable information from a global configuration node of the target data generation template; A test data generating unit, configured to parse data generation rules corresponding to each field based on global variable information, and generate test data based on the field generation rules; The final test data file generation unit is used to generate an insert statement based on the test data, write the insert statement into a preset file, and obtain the final test data file, wherein the insert statement refers to an operation statement in the structured query language for adding new data records to a database table.

[0008] The above-mentioned method and device for generating massive database table test data based on configuration, its method implementation includes: obtaining table creation information, extracting table metadata from the table creation information, and standardizing the table metadata to obtain standardized table metadata; generating a data generation template based on the standardized table metadata; modifying the data generation template based on target requirements to obtain a target data generation template; extracting global variable information in the global configuration node of the target data generation template; parsing the data generation rules corresponding to each field based on the global variable information, and generating test data based on the field generation rules; generating an insert statement based on the test data, writing the insert statement into a preset file to obtain a final test data file, wherein the insert statement refers to an operation statement in structured query language for adding new data records to a database table. In the embodiment of the present application, the entire process from table metadata extraction to test data file generation is automated, and the efficiency is particularly significantly improved in massive data generation scenarios, which greatly reduces the time spent on test data generation; there is no need to rely on complex stored procedures or manual code writing, and relying on templated configuration and extensible visual modification, the technical entry threshold is lowered, and ordinary testers can get started quickly, reducing manpower and learning costs; based on standardized table metadata and customizable templates, it is adapted to different database types and business scenarios, has general generation capabilities, and avoids repeated development; supports flexible configuration of business rules, data ranges and associations, accurately simulates complex business scenarios, and the generated data can effectively verify the correctness and robustness of the system, and broaden the test boundary coverage; with the help of preset constraints and association processing, the authenticity and relevance of the test data are guaranteed, meeting the database integrity requirements and improving the effectiveness of the test. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0010] Figure 1 This is an application environment diagram of a method for generating massive database table test data based on configuration in an embodiment of the present application; Figure 2 This is a flow chart of a method for generating massive database table test data based on configuration in one embodiment of the present application; Figure 3 This is a flow chart of a method for generating a data template in one embodiment of the present application. Figure 1 ; Figure 4 This is a flow chart of a method for generating a data generation template in an embodiment of the present application. Figure 2 ; Figure 5 This is a flow chart of a data template improvement method in one embodiment of the present application; Figure 6 This is a flow chart of a method for expanding a data generation method in an embodiment of the present application; Figure 7 This is a flowchart of a global variable parsing method in one embodiment of the present application; Figure 8 This is a flow chart of a test data generation method in one embodiment of the present application. Figure 1 ; Figure 9 This is a flow chart of a test data generation method in one embodiment of the present application. Figure 2 ; Figure 10 This is a sample diagram of the data generation process in one embodiment of the present application; Figure 11 This is a sample diagram of a test data file in an embodiment of the present application; Figure 12 This is a structural diagram of an apparatus for generating massive database table test data based on configuration in one embodiment of the present application; Figure 13 Schematic diagram of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0012] The method for generating massive database table test data based on configuration provided in this embodiment can be applied in Figure 1The system environment includes a database statement processing module, a YAML file processing module, a template processing module, a faker extension module, a database type data generation module, and a data file generation module. The database statement processing module is responsible for parsing the CREATE TABLE statement to obtain the table's [field name], [field type], and [field comment]; and is responsible for generating INSERT statements. The YAML file processing module is responsible for parsing YAML files to read structured data and writing structured data to YAML files. The template processing module is responsible for replacing calls to Python system functions, built-in object functions (Faker objects), Jinja2 templates, and custom functions in YAML files with executed values. The faker extension module is used to extend the faker.Faker class. In addition to providing the functions included in Faker itself, it also adds some common data generation capabilities, such as snowflake ID, auto-increment ID, and random selection. The database type data generation module generates corresponding data based on common database types. The data file generation module is responsible for generating template files, generating corresponding values for table fields, and finally generating data files.

[0013] Specifically, the table creation information entered by the user is fed into the database statement processing module, where the table structure is standardized. This standardized table structure is then transferred to the YAML file processing module, where a data generation template is generated according to default rules. Users can specify data generation rules as needed and modify the template content to obtain the target data generation template. The target data generation template is then transferred to the YAML file processing module, where the env node information in the target data generation template is extracted and transferred to the template processing module to load and generate global variables. The tables in the template file are transferred to the template processing module. If tables.table.fieldname.genRule has a value, the constants, Python system functions, built-in object functions (Faker objects), Jinja2 templates, and custom functions in the genRule node are generated accordingly. If tables.table.fieldname.genRule does not have a value, and the corresponding node type has a value, type is transferred to the database type data generation module, which randomly generates corresponding test data based on the corresponding type. The generated test data is then transferred to the database statement processing module, which ultimately generates the corresponding insert statement. The generated insert statement will be transferred to the data file generation module to complete the generation of the final test data file.

[0014] In one embodiment, if Figure 2 As shown, a method for generating massive database table test data based on configuration is provided, comprising the following steps: In step S110, table creation information is acquired, table metadata is extracted from the table creation information, and the table metadata is standardized to obtain standardized table metadata; The table creation information includes at least one of a table creation statement, a file path, a database connection, and a table name.

[0015] Alternatively, you can use the database system's built-in functions, such as querying the information_schema.COLUMNS table in MySQL to obtain information such as the table field name, data type, and whether it is a primary key. You can also directly obtain the table creation statement, such as by using the show create table command or by reading a text file. These operations can collect the table's structural information, including the table name, detailed definition of each field, constraints, and other information related to table creation.

[0016] Next, from the acquired table creation information, key table metadata is extracted, such as field names, field data types, field lengths, field default values, primary keys, and other information. This extracted table metadata is then processed uniformly, unifying table structure information from different databases into a standardized representation. For example, similar types across different databases are given uniform names, and field default values are standardized, distinguishing between string default values and keyword default values. This results in standardized table metadata, providing a unified foundation for subsequent operations such as data template generation.

[0017] In step S120, a data generation template is generated based on the standardized table metadata; Optionally, the standardized table data, such as field names, types, and constraints, can be organized into a hierarchical structure in YAML format. Each table corresponds to a node, and fields can be used as child nodes, including basic attributes such as type and length. At the same time, default generation rules can be automatically filled in, such as auto-increment of primary keys and generation of random values for string fields, to form a standardized data generation template.

[0018] The standardized data generation template may be as follows:

[0019] Among them, env is the global variable configuration node, which stores user-defined global variables for use by the genRule node of the subsequent tables node. Tables is the root node of table information, and all table information is attached to this node.

[0020] In step S130, the data generation template is modified based on the target requirements to obtain a target data generation template; Optionally, the genRule node in the data generation template can be improved based on user needs. This node supports rules such as system functions (such as random_int() for generating random numbers), built-in object functions (such as fake.name() in the Faker library for generating real names), Jinja2 templates (using {{env.base_id+1}} to reference global variables and perform calculations), and constants (such as the fixed value "Completed"). Through flexible combination, data generation can be achieved from basic numerical values to complex business logic, meeting the diverse data requirements of different test scenarios. The improved data generation template can be shown below:

[0021] The genRule node defines the logic for generating field data, supporting the flexible combination of multiple rule types to achieve the generation of data from basic to complex business data. System functions are general functions predefined by the template engine and are used to generate basic data. Built-in object functions are used to generate realistic simulation data. Jinja2 templates are used to embed variables and logic through template syntax to achieve dynamic calculations. Constants directly specify fixed generated values for fields.

[0022] In step S140, global variable information is extracted from the global configuration node of the target data generation template; Optionally, extract the global configuration node (env node) in the template. The env node stores user-defined global variables. Then parse its original value (which may be a constant, expression, or function) into an actual usable value. Finally, store these values in the global variable dictionary for sharing by all fields during subsequent data generation.

[0023] For example, in a YAML template, the env node might contain variable definitions such as base_id:1000 and calc_expr:100+5. Extract these variable names (such as base_id) and their corresponding original values (such as 1000 and 100+5). If the variable value is a fixed value, use the fixed value as the actual value. If it is an expression, such as fake.date(), use the corresponding tool to generate a real date. If the template syntax is {{base_id+1}}, combine it with the parsed global variable (for example, base_id=1000) to calculate the actual value. Save the processed variable names and actual values in a shared dictionary and store it in the program's global shared memory. When generating data for each field later, you can retrieve the required variable values from this dictionary, ensuring that different fields use the same set of parameters.

[0024] In step S150, the data generation rules corresponding to each field are parsed based on the global variable information, and test data is generated based on the field generation rules; Optionally, extract the structural information of each table from the table structure root node (tables node) in the target data generation template. First, determine the table name and fields it contains. Then, determine the generation rules for each field (such as auto-increment or random name generation). Also, identify dependencies between fields in different tables (for example, a user ID in the order table must reference the ID in the user table). Then, calculate the actual value according to the field generation rules. If the genRule node does not have a data generation rule, obtain the field data type and use a default fixed value based on the field data type. If it is a function rule (such as generating random numbers or real names), call the corresponding tool to generate it. If it is an expression rule (such as calculating based on global variables), calculate the result based on the global variable value. If it is an association rule (such as referencing field values from other tables), obtain the result from the generated associated table data. After generation, verify that the data type matches the field definition (for example, numeric fields cannot be filled with strings). This ensures that the generated value complies with the field type constraints and pre-set business rules, thereby obtaining test data that meets the pre-set constraints.

[0025] In step S160, an insert statement is generated based on the test data, and the insert statement is written into a preset file to obtain a final test data file, wherein the insert statement refers to an operation statement in structured query language for adding new data records to a database table.

[0026] Optionally, insert statements are assembled using SQL syntax based on the table structure and generated field values. Each statement specifies the table name, field list, and corresponding values (e.g., INSERT INTO tablename(field1, field2) VALUES ('value1', 'value2');). To improve efficiency, multiple statements are batched together (for example, in batches of 1,000 records) to reduce database interactions. By standardizing the field order in table metadata, the field list in the insert statement is consistent with the table structure, avoiding syntax errors. To prevent excessive memory usage, batch writing can be used. For example, each batch of 1,000 records is assembled into a SQL statement, written to a file, and the temporary data is cleared. After all batches are written, a test data file conforming to SQL format specifications is generated, and the final test data file is generated.

[0027] An embodiment of the present application provides a method for generating massive database table test data based on configuration, including: obtaining table creation information, extracting table metadata from the table creation information, and standardizing the table metadata to obtain standardized table metadata; generating a data generation template based on the standardized table metadata; modifying the data generation template based on target requirements to obtain a target data generation template; extracting global variable information in a global configuration node of the target data generation template; parsing data generation rules corresponding to each field based on the global variable information, and generating test data based on the field generation rules; generating an insert statement based on the test data, and writing the insert statement to a preset file to obtain a final test data file, wherein the insert statement refers to an operation statement in a structured query language for adding new data records to a database table. In the embodiment of the present application, the entire process from table metadata extraction to test data file generation is automated, and the efficiency is particularly significantly improved in massive data generation scenarios, which greatly reduces the time spent on test data generation; there is no need to rely on complex stored procedures or manual code writing, and relying on templated configuration and extensible visual modification, the technical entry threshold is lowered, and ordinary testers can get started quickly, reducing manpower and learning costs; based on standardized table metadata and customizable templates, it is adapted to different database types and business scenarios, has general generation capabilities, and avoids repeated development; supports flexible configuration of business rules, data ranges and associations, accurately simulates complex business scenarios, and the generated data can effectively verify the correctness and robustness of the system, and broaden the test boundary coverage; with the help of preset constraints and association processing, the authenticity and relevance of the test data are guaranteed, meeting the database integrity requirements and improving the effectiveness of the test.

[0028] In an embodiment of the present application, the table creation information includes a table creation statement, and obtaining the table creation information and extracting table metadata from the table creation information includes: Get key parameters; Determine the table creation statement acquisition method based on the key parameters; Based on the table creation statement acquisition method, read the table creation statement from a text file, or obtain the table creation statement from a database; Parse the table creation statement to obtain field names and data types.

[0029] like Figure 3 、 Figure 4As shown, based on the type parameter passed in, the system enters a branch to determine whether to use text-based template generation. If the system determines that the template should be generated using text-based generation, it locates and reads the text files in the configuration path and extracts the table creation statements from them. These text files typically contain pre-stored standardized or customized table structure definitions. If the system determines that the text-based generation method is not appropriate, it leverages database interaction capabilities to call the "show create table" statement and dynamically retrieve the table creation statements directly from the target database, ensuring that the retrieved table structure is the latest and effective database state. After obtaining the table creation statements, the system uses technologies such as SQL parsers or regular expressions to parse their syntax, accurately extracting key metadata such as field names, data types, and constraints (such as primary keys and not null). The system then parses the SQL text into structured table metadata objects, a table structure description that can be recognized by subsequent processes.

[0030] In one embodiment of the present application, the standardized table metadata includes field names and data types. Generating a data generation template based on the standardized table metadata includes: Arrange the field names and data types into YAML format according to preset rules; Write the content in the YAML format into a YAML file to obtain the data generation template.

[0031] like Figure 3 、 Figure 4 As shown in the figure, the parsed table name, field name, type and other information are organized into YAML format according to preset rules, then written to the YAML file, and the output data is generated into a template. Among them, the preset rules can be the template default rules, which can be used to define the YAML structure, such as the table name hierarchy, field list format, and extended information such as comments and generation rules.

[0032] In an embodiment of the present application, modifying the data generation template based on target requirements to obtain a target data generation template includes: Determine the database type associated with the data generation template and generate basic data that conforms to the database type; Generate auto-increment id; Generate snowflake id; According to the field data type, randomly select data with matching type from the preset candidate value list; Call the Chinese name generation logic to generate corresponding Chinese names for semantic fields; Execute business logic through custom Python functions to generate corresponding data; Query the generated data in the associated table and reuse / associate it to fill the current table fields.

[0033] As Figure 5 shown, after obtaining the data generation template, in order to make the template generate more business - compliant and flexible data, the data generation template can be improved. Around the data generation template, 7 types of data generation strategies are layered and superimposed (database adaptation, ID generation, random selection, Chinese name, custom function, cross - table association), enabling the template to generate richer and more realistic - scenario - compliant data to meet complex testing / business requirements.

[0034] Optionally, if the target field requirement is to generate corresponding data according to the database type: no modification is required here. If the target field requirement is an auto - incrementing id: then modify the target field genRule to "next(iterator variable name defined in the template env node)". If the target field requirement is a snowflake id, then modify the target field genRule to "fake.snow_id()". If the target field requirement is a uuid, then modify the target field genRule to "fake.uuid()". If the target field requirement is to randomly select one, then modify the target field genRule to "fake.random_choice(['Zhang San', 'Li Si', 'Wang Wu'])". If the target field requirement is to generate a Chinese name, then modify the target field genRule to "fake.name()". If the target field requirement is to be implemented through a custom python function, then modify the target field genRule to "custom package name::function name(parameters)". If the target field requirement is the primary key of the previous table, then modify the target field genRule to "{{sword_order_info_202410.id}}".

[0035] Specifically, by identifying the database type associated with the template (such as MySQL or Oracle), the corresponding type generator is invoked (e.g., mysql_datetime() generates a YYYY-MM-DD HH:ii:ss format, or oracle_timestamp() generates a time zone-inclusive time) to ensure the validity of the underlying data type. Based on the validity of the underlying data type (e.g., confirming that the id type is a BIGINT), a global / table-level counter is maintained for the table's primary key unique identifier field (e.g., user_id). Each counter generates a sequentially increasing unique ID (e.g., 1, 2, 3, etc.). For business identifier fields, the Snowflake algorithm is invoked to generate distributed unique IDs (e.g., 147258369123456789) using timestamps, machine IDs, and serial numbers. For discrete fields (e.g., gender and status), type-compatible values are randomly selected from a preset candidate list (e.g., string values for string fields). For semantic fields, you can call the Faker library's fake.name() and fake.address() functions, or customize the Chinese name library, to generate corresponding Chinese names for Chinese semantic fields. For business-specific scenarios, you can call Python functions to implement them, covering customized requirements that Faker or basic rules cannot meet. Query related tables for other generated data, cache their field values, and select and populate them into the current table fields in sequence, allowing multi-table data to form real business connections.

[0036] In one embodiment of the present application, after generating the data generation template based on the standardized table metadata, the following steps are included: Implement the FakerExt subclass of faker.Faker; Implement the FakerExt.uuid method; Implement the FakerExt.auto_increment method; Implement the FakerExt.random_choice method; Implement the FakerExt.snow_id method; Instantiate the FakerExt object and store it in the global dictionary.

[0037] like Figure 6 As shown, since Faker itself does not support the above-mentioned ordered auto-incrementing IDs, distributed unique snowflake IDs, etc., the faker.Faker extension class can be used to extend some common data generation capabilities, such as snowflake IDs, auto-incrementing IDs, random selection, etc.

[0038] Specifically, by creating the faker.Faker subclass FakerExt and implementing exclusive methods such as uuid (generates a UUID with a business identifier), auto_increment (builds an ordered auto-increment sequence based on a global / table-level counter), random_choice (supports random selection of values with weights and exclusion logic), and snow_id (generates a distributed unique snowflake ID), we can break through the limitations of Faker's native functions, meet the business needs for customized and refined data generation, and adapt to complex architectural scenarios such as single-body and distributed architectures. At the same time, FakerExt objects are stored in the global dictionary, providing a rich and controllable set of generation methods for data templates, ensuring the consistency and reusability of multi-link data generation, and giving the entire data generation system (covering template configuration to real business data output) customized and implementable capabilities from the bottom up, so that the generated data can accurately fit business rules and support practical applications such as testing and initialization.

[0039] It should be noted that FakerExt is the basis for template generation and modification. When generating a template, it relies on its method to preset rules. When modifying a template, it relies on its method to implement customization. Finally, it is executed through global variable calls during the data generation stage.

[0040] In one embodiment of the present application, extracting global variable information of the global configuration node in the target data generation template, and loading and preprocessing the global variable information, includes: Determining the global configuration node based on a template structure of the target data generation template; Extract all variables and variable expressions corresponding to the global configuration node; Add all the variables and the variable expressions to the global variable dictionary and store them in the global memory context of the current program for direct calling by subsequent data generation steps.

[0041] like Figure 7 As shown, data generation parameters are obtained, such as the template file path, the generated data file path, and the number of generated data groups. Based on the data generation parameters, the target data generation template is loaded using the template engine. In the data generation template, the global configuration is searched and parsed. The global configuration node is usually a fixed node preset in the template (such as the env node), which can be directly located through the template structure definition. The env node stores global variables (such as YAML format). Then, the variables and variable expressions in the node are traversed and temporarily stored in the global variable dictionary. At the same time, the global variables are injected into the local execution context (locals()) to build the execution context.

[0042] In one embodiment of the present application, after adding all the variables and variable expressions to the global variable dictionary, the steps include: If the variable expression is a user-defined function expression, the user-defined function expression is parsed and executed to obtain the actual variable value; If the variable expression is a system function or a built-in function expression, executing the system function or the built-in function expression to obtain the actual variable value; The variables and the corresponding actual variable values are associated and stored back into the global variable dictionary.

[0043] like Figure 7 As shown, after all the variables and variable expressions are added to the global variable dictionary, the variable expressions are classified and processed. If the variable expression is a custom function, its package and function parts are parsed, and then eval is used to evaluate it to get the actual variable value; if it is a system or built-in function, the actual variable value is directly obtained through eval. Finally, the processed variables and the corresponding actual variable values are saved back to the global variable dictionary, thereby realizing the transformation from static template configuration to dynamically available global variables, providing unified, flexible and secure parameter support for all aspects of data generation, so that complex data generation requirements can be completed through template configuration, decoupling configuration and logic, and improving maintainability and scalability.

[0044] The code to obtain the actual value returned by the expression is as follows: def get_expr_actual_value(expr, env): """ Get the value after the py code is executed :param expr: :return: """ locals().update(**env) if isinstance(expr, str): # If it does not contain ::, it is considered a py system function, otherwise it is considered a custom function if expr.find('::') == -1: if "(" in expr and ")" in expr:# If this condition is met, it is considered a function expr_value = eval(expr) else: expr_value = expr else: func_info = expr.split('::') if len(func_info) == 2: expr_value = evaluate_expr(func_info[1], func_info[0]) else: expr_value = evaluate_expr(func_info[0]) return expr_value else: return expr def evaluate_expr(expression, modules=None): """ Execute the eval operation on the passed-in expression :param expression: :param modules: :return: """ if modules: current_g_vars = globals() # Import a\a.b\a.b.c, otherwise an error will occur when evaluating the method of module a.b.c if modules.find('.')>-1: subs = modules.split('.') current = '' for sub in subs: current += sub if current == '' else '.' + sub if not current_g_vars.get(current): current_g_vars[current] = importlib.import_module(current) cmd = f'''{modules}.{expression}''' else: cmd = f'''{expression}''' try: result = eval(cmd) except Exception as e: raise Exception(f'modules={modules},expression={expression},cmd={cmd},failure reason:{str(e)}') return result In one embodiment of the present application, the global variable information includes actual values of global variables, and the data generation rules corresponding to each field are parsed based on the global variable information, and test data is generated based on the field generation rules, including: Determine the root node and field definition of the table structure in the target data generation template; Based on the field definition, determine whether each field in each table is configured with a data generation rule; If so, generating test data based on the data generation rule; If not, generate test data based on the field data type.

[0045] like Figure 8 、 Figure 9 As shown, when processing the root node (tables node) of the table structure in the data generation template file, different data generation strategies are executed based on whether tables.table.fieldname.genRule has a value. If genRule has a value, it indicates that a data generation rule exists; otherwise, no data generation rule exists. If genRule has a value, it parses constants, Python system functions, built-in object functions (such as Faker objects), Jinja2 templates, and custom functions within it to generate test data that conforms to business rules (such as prefixed order numbers and random amounts). If genRule has no value but the same-level node type has a value, the type value is passed as a parameter to the database type data generation module. This module generates basic test data that conforms to field type constraints based on the database type's default rules (such as generating random integers using INT and random strings using VARCHAR). This ensures data generation flexibility in complex business scenarios while reducing configuration costs for basic scenarios. Standardized test data is output through a unified interface, providing consistent data input for subsequent storage and verification, forming a complete template-driven data generation chain.

[0046] It should be noted that if Figure 9As shown, when processing generation rules containing Jinja2 template syntax, the Jinja2 template engine loads the global variable dictionary as the context and replaces template variables in the rule with actual values. First, the template engine is initialized and global variables are imported. Then, the genRule rule string is compiled into a template object. When the render() method is called, the engine automatically replaces the variables with the values in the global variable dictionary. If the rule contains function calls, the global variable dictionary must be pre-registered with a Faker object, so that the function can be directly executed to generate the result during rendering. For complex expressions, the engine parses operators and filters according to Jinja2 syntax, ultimately returning the calculated string or value as the test data for the field.

[0047] The code for getting the field value is as follows: def get_field_value(col_value_info: dict): """Get the value corresponding to the field in the template""" gen_rule = col_value_info.get('genRule') if gen_rule: gen_rule = template_util.template_render(gen_rule, env)# Replace variables in the template, follow jinja2 val = template_util.get_expr_actual_value(gen_rule, env) else: data_type = col_value_info.get('type') start = data_type.find('(') end = data_type.find(')') if start>-1 and end>-1: # Interception type and length type_part = data_type[:start]# Type part: from the beginning to the position before the left bracket length_part = data_type[start + 1:end]# Length part: remove the left and right brackets val = gen_data_util.generate_data(type_part, length_part) else: val = gen_data_util.generate_data(data_type) return val The corresponding data code generated according to the database field type is as follows: def generate_data(field_type, field_length=None, field_range=None): """Generate random data based on field type and possible length or range""" if field_type.lower() in ('int', 'bigint', 'tinyint'): min_val = field_range[0] if field_range else 0 max_val = field_range[1] if field_range else 2 ** 31 - 1 return random.randint(min_val, max_val) elif field_type.lower() == 'float': min_val = field_range[0] if field_range else 0.0 max_val = field_range[1] if field_range else 1.0 return round(random.uniform(min_val, max_val), 2) # Assume two decimal places are retained elif field_type.lower() in ['varchar', 'char']: if field_length is None: raise ValueError("Length must be specified for VARCHAR or CHARtypes.") letters = string.ascii_letters + string.digits return ''.join(random.choice(letters) for i in range(int(field_length))) elif field_type.lower() in ('date', 'datetime'): start_date = field_range[0] if field_range else datetime(1970, 1, 1) end_date = field_range[1] if field_range else datetime.now() delta = end_date - start_date int_delta = (delta.days * 24 * 60 * 60) + delta.seconds random_second = random.randrange(int_delta) return str(start_date + timedelta(seconds=random_second)) # elif field_type.lower() == 'boolean': #return random.choice([True, False]) elif field_type.lower() == 'decimal': x, y = (int(item) for item in field_length.split(',')) # 设置decimal的精度 getcontext().prec = x # 生成整数部分 integer_part = random.randint(0, 10 ** (x - y) - 1) # 生成小数部分 fractional_part = random.randint(0, 10 ** y - 1) # Combine integer and fractional parts decimal_value = Decimal(f"{integer_part}.{fractional_part:0{y}d}") return str(decimal_value) else: raise ValueError(f"Unsupported field type: {field_type}") In one embodiment of the present application, generating an insert statement based on the test data, writing the insert statement into a preset file, and obtaining a final test data file includes: Associating the table, the field, and the test data to form a table-field value relationship; The table field value relationship is spliced to generate an insert statement according to the preset grammatical rules; The insert statement is written into a file of a preset format to form the final test data file.

[0048] like Figure 8 As shown, the tables, fields and their corresponding test data are associated to build a structured relationship between table, field and value, clarify the table and field to which each piece of data belongs, and ensure the strict correspondence between the data and the table structure; according to the preset grammatical rules (such as SQL insert statement grammar or CSV format specification), these structured table field value relationships are spliced into the corresponding insert statement insert, so that the data has a format that can be recognized by the database or stored in the file; the spliced insert statement insert is written into a file in the preset format (such as .sql file, .csv file, etc.) to form the final test data file, as shown. Figure 10 shows an example of the data generation process, Figure 11 The following figure shows an example of a test data file, which provides a standardized data source for subsequent operations such as database import and test case execution, thus achieving a complete closed loop from test data generation to implementation.

[0049] In the embodiment of the present application, the entire process from table metadata extraction to test data file generation is automated, and the efficiency is particularly significantly improved in massive data generation scenarios, which greatly reduces the time spent on test data generation; there is no need to rely on complex stored procedures or manual code writing, and relying on templated configuration and extensible visual modification, the technical entry threshold is lowered, and ordinary testers can get started quickly, reducing manpower and learning costs; based on standardized table metadata and customizable templates, it is adapted to different database types and business scenarios, has general generation capabilities, and avoids repeated development; supports flexible configuration of business rules, data ranges and associations, accurately simulates complex business scenarios, and the generated data can effectively verify the correctness and robustness of the system, and broaden the test boundary coverage; with the help of preset constraints and association processing, the authenticity and relevance of the test data are guaranteed, meeting the database integrity requirements and improving the effectiveness of the test.

[0050] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0051] In one embodiment, a device for generating massive database table test data based on configuration is provided. The device for generating massive database table test data based on configuration corresponds to the method for generating massive database table test data based on configuration in the above embodiment. Figure 12 As shown, the device for generating massive database table test data based on configuration includes a table building information acquisition unit 10, a template generation unit 20, a template modification unit 30, a global variable processing unit 40, a test data generation unit 50 and a final test data file generation unit 60. The functional modules are described in detail as follows: The table building information acquisition unit 10 is used to acquire table building information, extract table metadata from the table building information, and standardize the table metadata to obtain standardized table metadata; A template generating unit 20 is configured to generate a data generation template based on the standardized table metadata; The template modification unit 30 is used to modify the data generation template based on target requirements to obtain a target data generation template; A global variable processing unit 40 is used to extract global variable information from the global configuration node of the target data generation template; A test data generating unit 50 is configured to parse data generation rules corresponding to each field based on global variable information, and generate test data based on the field generation rules; The final test data file generating unit 60 is used to generate an insert statement based on the test data, write the insert statement into a preset file, and obtain a final test data file, wherein the insert statement refers to an operation statement in the structured query language for adding new data records to a database table.

[0052] In one embodiment of the present application, the table building information acquisition unit 10 is further configured to: Get key parameters; Determine the table creation statement acquisition method based on the key parameters; Based on the table creation statement acquisition method, read the table creation statement from a text file, or obtain the table creation statement from a database; Parse the table creation statement to obtain field names and data types.

[0053] In one embodiment of the present application, the template modification unit 30 is further configured to: Arrange the field names and data types into YAML format according to preset rules; Write the content in the YAML format into a YAML file to obtain the data generation template.

[0054] In one embodiment of the present application, the template modification unit 30 is further configured to: Determine the database type associated with the data generation template and generate basic data that conforms to the database type; Generate auto-increment id; Generate snowflake id; According to the field data type, randomly select data with matching type from the preset candidate value list; Call the Chinese name generation logic to generate corresponding Chinese names for semantic fields; Execute business logic through custom Python functions to generate corresponding data; Query the generated data in the associated table and reuse / associate it to fill the current table fields.

[0055] In one embodiment of the present application, the device further includes an expansion module configured to: Implement the FakerExt subclass of faker.Faker; Implement the FakerExt.uuid method; Implement the FakerExt.auto_increment method; Implement the FakerExt.random_choice method; Implement the FakerExt.snow_id method; Instantiate the FakerExt object and store it in the global variable dictionary.

[0056] In one embodiment of the present application, the global variable processing unit 40 is further configured to: Determining the global configuration node based on a template structure of the target data generation template; Extract all variables and variable expressions corresponding to the global configuration node; Add all the variables and the variable expressions to the global variable dictionary and store them in the global memory context of the current program for direct calling by subsequent data generation steps.

[0057] In one embodiment of the present application, the global variable processing unit 40 is further configured to: If the variable expression is a user-defined function expression, the user-defined function expression is parsed and executed to obtain the actual variable value; If the variable expression is a system function or a built-in function expression, executing the system function or the built-in function expression to obtain the actual variable value; The variables and the corresponding actual variable values are associated and stored back into the global variable dictionary.

[0058] In one embodiment of the present application, the test data generating unit 50 is further configured to: Determine the root node and field definition of the table structure in the target data generation template; Based on the field definition, determine whether each field in each table is configured with a data generation rule; If so, generating test data based on the data generation rule; If not, generate test data based on the field data type.

[0059] In one embodiment of the present application, the final test data file generating unit 60 is further configured to: Associating the table, the field, and the test data to form a table-field value relationship; The table field value relationship is spliced to generate an insert statement according to the preset grammatical rules; The insert statement is written into a file of a preset format to form the final test data file.

[0060] In the embodiment of the present application, the entire process from table metadata extraction to test data file generation is automated, and the efficiency is particularly significantly improved in massive data generation scenarios, which greatly reduces the time spent on test data generation; there is no need to rely on complex stored procedures or manual code writing, and relying on templated configuration and extensible visual modification, the technical entry threshold is lowered, and ordinary testers can get started quickly, reducing manpower and learning costs; based on standardized table metadata and customizable templates, it is adapted to different database types and business scenarios, has general generation capabilities, and avoids repeated development; supports flexible configuration of business rules, data ranges and associations, accurately simulates complex business scenarios, and the generated data can effectively verify the correctness and robustness of the system, and broaden the test boundary coverage; with the help of preset constraints and association processing, the authenticity and relevance of the test data are guaranteed, meeting the database integrity requirements and improving the effectiveness of the test.

[0061] Regarding the specific definition of the device for generating massive database table test data based on configuration, please refer to the definition of the method for generating massive database table test data based on configuration above, and will not be repeated here. Each module in the above-mentioned device for generating massive database table test data based on configuration can be implemented in whole or in part by software, hardware, and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0062] In one embodiment, a computer device is provided. The computer device may be a terminal device, and its internal structure diagram may be as follows: Figure 13 As shown. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium. The readable storage medium stores computer-readable instructions. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer-readable instructions are executed by the processor, a method for generating massive database table test data based on a configuration is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0063] In an embodiment of the present application, a computer device is provided, including a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the steps of the method for generating massive database table test data based on the configuration as described above are implemented.

[0064] In an embodiment of the application, a readable storage medium is provided, which stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the steps of the method for generating massive database table test data based on configuration as described above are implemented.

[0065] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through computer-readable instructions. The computer-readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer-readable instructions are executed, they can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0066] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0067] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for generating massive database table test data based on configuration, characterized in that: The method comprises: Acquire table creation information, extract table metadata from the table creation information, and standardize the table metadata to obtain standardized table metadata; Generate a data generation template based on the standardized table metadata; Modify the data generation template based on target requirements to obtain a target data generation template; Extracting global variable information from a global configuration node of the target data generation template; Parsing data generation rules corresponding to each field based on global variable information, and generating test data based on the field generation rules; Based on the test data, an insert statement is generated, and the insert statement is written into a preset file to obtain a final test data file, wherein the insert statement refers to an operation statement in structured query language for adding new data records to a database table.

2. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: The table creation information includes a table creation statement, and the acquiring of the table creation information and the extraction of table metadata from the table creation information include: Get key parameters; Determine the table creation statement acquisition method based on the key parameters; Based on the table creation statement acquisition method, read the table creation statement from a text file, or obtain the table creation statement from a database; Parse the table creation statement to obtain field names and data types.

3. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: The standardized table metadata includes field names and data types. Generating a data generation template based on the standardized table metadata includes: Arrange the field names and data types into YAML format according to preset rules; Write the content in the YAML format into a YAML file to obtain the data generation template.

4. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: The step of modifying the data generation template based on target requirements to obtain a target data generation template includes: Determine the database type associated with the data generation template and generate basic data that conforms to the database type; Generate auto-increment id; Generate snowflake id; According to the field data type, randomly select data with matching type from the preset candidate value list; Call the Chinese name generation logic to generate corresponding Chinese names for semantic fields; Execute business logic through custom Python functions to generate corresponding data; Query the generated data in the associated table and reuse / associate it to fill the current table fields.

5. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: After generating the data generation template based on the standardized table metadata, the method includes: Implement the FakerExt subclass of faker.Faker; Implement the FakerExt.uuid method; Implement the FakerExt.auto_increment method; Implement the FakerExt.random_choice method; Implement the FakerExt.snow_id method; Instantiate the FakerExt object and store it in the global variable dictionary.

6. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: The extracting global variable information of the global configuration node in the target data generation template and loading and preprocessing the global variable information includes: Determining the global configuration node based on a template structure of the target data generation template; Extract all variables and variable expressions corresponding to the global configuration node; Add all the variables and the variable expressions to the global variable dictionary and store them in the global memory context of the current program for direct calling by subsequent data generation steps.

7. The method for generating massive database table test data based on configuration according to claim 6, characterized in that: After adding all the variables and variable expressions to the global variable dictionary, the following steps are included: If the variable expression is a user-defined function expression, the user-defined function expression is parsed and executed to obtain the actual variable value; If the variable expression is a system function or a built-in function expression, executing the system function or the built-in function expression to obtain the actual variable value; The variables and the corresponding actual variable values are associated and stored back into the global variable dictionary.

8. The method for generating massive database table test data based on configuration according to claim 1, characterized in that: The global variable information includes actual values of global variables, and the data generation rules corresponding to each field are parsed based on the global variable information, and test data is generated based on the field generation rules, including: Determine the root node and field definition of the table structure in the target data generation template; Based on the field definition, determine whether each field in each table is configured with a data generation rule; If so, generating test data based on the data generation rule; If not, generate test data based on the field data type.

9. The method for generating massive database table test data based on configuration according to claim 8, characterized in that: The method generates an insert statement based on the test data, writes the insert statement into a preset file, and obtains a final test data file, including: Associating the table, the field, and the test data to form a table-field value relationship; The table field value relationship is spliced to generate an insert statement according to the preset grammatical rules; The insert statement is written into a file of a preset format to form the final test data file.

10. A device for generating massive database table test data based on configuration, characterized in that: The device comprises: a table creation information acquisition unit, configured to acquire table creation information, extract table metadata from the table creation information, and perform standardization processing on the table metadata to obtain standardized table metadata; A template generating unit, configured to generate a data generation template based on the standardized table metadata; A template modification unit, configured to modify the data generation template based on target requirements to obtain a target data generation template; A global variable processing unit, configured to extract global variable information from a global configuration node of the target data generation template; A test data generating unit, configured to parse data generation rules corresponding to each field based on global variable information, and generate test data based on the field generation rules; The final test data file generation unit is used to generate an insert statement based on the test data, write the insert statement into a preset file, and obtain the final test data file, wherein the insert statement refers to an operation statement in the structured query language for adding new data records to a database table.

Images