Method and system for automatically generating and importing data based on database interface coding

Abstract

The present invention provides a method and system for automatically generating and importing data based on database interface coding. The method includes the following steps: determining whether a table creation file, a data feature configuration file, a character set file, an import parameter configuration file, and a data import module relationship configuration file have been successfully established; traversing and parsing the data import module relationship configuration file, automatically generating a table import SQL file, and placing the file in an import SQL directory; determining whether all data import module relationship configuration files have been parsed; and calling the SQL file in the import SQL directory to perform the import operation, with the SQL files being executed in parallel. The present invention has the beneficial effects of automating data preparation and import, transforming a large amount of repetitive data generation and import work into a standard process, freeing up manpower.

Description

Technical Field

[0001] The present invention belongs to the field of database automated testing, and in particular relates to a method and system for automatically generating and importing data based on database interface coding. Background Art

[0002] As database functionality increases, initial data needs to be stored during testing to prepare for subsequent functional tests. The current approach involves manually preparing test data based on test requirements and importing it into the database before each functional test. This is especially true when large amounts of random data that meet expectations are required, requiring individual scripts to generate the data. Different people use different methods, making unified management difficult and potentially leading to duplication and waste of manpower.

[0003] The above testing method mainly faces the following problems: a large amount of repetitive data generation work takes up a lot of manpower and wastes resources; manual data preparation focuses on how to generate and import data, and may miss out on controlling the data's own characteristics; manual data preparation in the early stage is too time-consuming and lacks understanding of the combination of data and functions; the import process is fragmented, and manual table import can easily cause uneven load on distributed nodes, and existing manual operations lack consideration for concurrent data writing. Summary of the Invention

[0004] In view of this, the present invention aims to propose a method and system for automatically generating and importing data based on database interface coding, so as to solve the problems of complex operation, waste of resources and uneven load of distributed nodes in the existing technology.

[0005] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0006] In a first aspect, the present invention provides a method for automatically generating and importing data based on database interface coding, comprising the following steps:

[0007] S1. Create table;

[0008] S2, data feature configuration;

[0009] S3. Import parameter configuration;

[0010] S4, data import module relationship configuration;

[0011] S5. Generate import operation statement;

[0012] S6. Determine whether the table creation file, data feature configuration file, character set file, import parameter configuration file, and data import module relationship configuration file are successfully established. If yes, jump to the next step S7; otherwise, report an error and exit;

[0013] S7. Traverse and parse the data import module relationship configuration file, automatically generate a table import SQL file, and place it in the import SQL placement directory;

[0014] S8, determine whether the data import module relationship configuration file has been fully parsed, if so, jump to the next step S9, otherwise, return to the previous step S7 to continue;

[0015] S9. Call the SQL file in the directory where the SQL is placed to perform the import operation. The SQL files are executed in parallel.

[0016] Furthermore, in step S1, a table is created, including:

[0017] Write SQL files based on the function: Place the table creation SQL statements required for batch import. Save each table creation statement as a separate SQL file and place the SQL file in a specified directory.

[0018] Furthermore, in step S2, data feature configuration includes:

[0019] According to the type characteristics, function value characteristics and test tendency characteristics of different columns in the database table, different feature attributes are configured for each database table, and a data feature configuration file is generated in the form of a txt file and placed separately in a specified directory.

[0020] Furthermore, in step S3, parameter configuration is imported, including:

[0021] Configure the parameters of the import module according to the table creation statement and data file characteristics. Each import operation corresponds to an import parameter configuration file, which is placed separately in the specified directory.

[0022] Furthermore, in step S4, the data import module relationship configuration includes:

[0023] The files generated in steps S1 to S3 are matched and set, and a data import module relationship configuration file is generated.

[0024] Furthermore, in step S5, an import operation statement is generated, including:

[0025] According to the data import module relationship configuration file information generated in step S4, batches of import operation statements are generated in the order of information, and are saved into the import statement sql file by table.

[0026] In a second aspect, based on the same concept, the present invention further provides a system for automatically generating and importing data based on database interface coding, comprising:

[0027] The table creation module is used to store the table creation SQL statements required for batch import;

[0028] The data feature configuration module is used to configure different feature attributes for each database table based on the type characteristics, function value characteristics, and test tendency characteristics of different columns in the database table;

[0029] Import parameter configuration module, used to configure the parameters required for the import module according to the table creation statement and data file characteristics;

[0030] The data import module relationship configuration module is used to match and set the files of the above table building module, data feature configuration module and import parameter configuration module;

[0031] The automated distributed import module is used to configure the module's configuration file information based on the data import module relationship, batch generate import operation statements in the order of information, and save them into the import statement SQL file by table.

[0032] Compared with the prior art, the method and system for automatically generating and importing data based on database interface coding according to the present invention have the following beneficial effects:

[0033] (1) Transform a large amount of repetitive data generation and import work into standard process work, freeing up manpower.

[0034] (2) Data preparation is achieved by users through the configuration of relevant configuration files based on functional needs. Users focus on controlling the characteristics of the data itself to avoid unnecessary energy consumption.

[0035] (3) By automating data preparation and import, personnel can focus more on understanding the combination of data and functions, and concentrate more on functional testing itself, providing a guarantee for faster problem discovery and resolution.

[0036] (4) Through automatically planned concurrent imports, better load balancing of database distributed nodes can be achieved, thereby improving efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which constitute part of the present invention, are provided to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are provided to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 This is a schematic diagram of the relationship between the directory files of each module according to an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the overall process described in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other.

[0041] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, features defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0042] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0043] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0044] like Figures 1 to 2 As shown, the method and system for automatically generating and importing data based on database interface coding include the following steps:

[0045] S1. Create table;

[0046] S2, data feature configuration;

[0047] S3. Import parameter configuration;

[0048] S4, data import module relationship configuration;

[0049] S5. Generate import operation statement;

[0050] S6. Determine whether the table creation file, data feature configuration file, character set file, import parameter configuration file, and data import module relationship configuration file are successfully established. If yes, jump to the next step S7; otherwise, report an error and exit;

[0051] S7. Traverse and parse the data import module relationship configuration file, automatically generate a table import SQL file, and place it in the import SQL placement directory;

[0052] S8, determine whether the data import module relationship configuration file has been fully parsed, if so, jump to the next step S9, otherwise, return to the previous step S7 to continue;

[0053] S9. Call the SQL file in the directory where the SQL is placed to perform the import operation. The SQL files are executed in parallel.

[0054] The specific implementation is as follows:

[0055] 1. Table Creation Module: This module stores the table creation SQL statements required for batch import. Each table creation statement is saved as a separate SQL file for subsequent automated import. This module is placed in a specific directory to facilitate scripts or tools to locate and retrieve content. Each SQL file is written by the tester based on the current functionality.

[0056] 2. Data Feature Configuration Module: This module configures different feature attributes for each database table based on the type, function, and testability of each column. This module is placed in a dedicated directory to facilitate scripts or tools in locating and retrieving content. It can be written by testers in the form of a .txt file (each column has a .txt tag, which must be in the same order as the corresponding table creation statement column, and the corresponding values ​​for each feature are listed below).

[0057] 3. Import parameter configuration module: This module configures the parameters required for the import module based on the table creation statement and data file characteristics. Each import operation corresponds to a configuration file. The module is placed separately in a specific directory to facilitate scripts or tools to locate and obtain content.

[0058] 4. Data import module relationship configuration module: This module will match and set the files in the above three modules as the basis for the final automated process.

[0059] 5. Automated Distributed Import Module: This module generates all import statements in batches, sequentially, based on the configuration file information of the data import module relationship configuration module. It then saves these statements into SQL files for each table. These SQL import operations are then executed serially or in parallel. [A SQL file is generated for each table, and the import statements within it are executed serially (this optimizes transmission efficiency by leveraging GBase 8a's native batch loading protocol); import statements in each SQL file for different tables can be executed in parallel (leveraging GBase 8a's distributed node load balancing, making the import more efficient and preventing deadlocks), completing the automated import of all data files. Different import tools generate different configuration information and implement different import methods.

[0060] Example 1: Data feature configuration:

[0061] Data file character set characteristics - define the corresponding character set of the generated data file as needed:

[0062] [File_Character_set];

[0063] Char_set_value="utf-8"——》Specify the file character set as utf-8;

[0064] Data separator features - set the field separator, row separator, etc. between data as needed

[0065] [Data_Separate];

[0066] Fields_terminated=","——》Specify the field separator;

[0067] Lines_terminated="\n"——》Specify the line separator;

[0068] Row characteristics - set the total number of rows in the file, the range of the number of special rows (such as those containing error data, null data, etc.), and other row characteristic information as needed:

[0069] [Lines_Info];

[0070] All_lines_num=10000——》The total number of lines in the file is 10,000;

[0071] [Special_Lines]——》The first set of special data line block information;

[0072] Special_id=1——》corresponds to the Special_id in the special data column type characteristics below;

[0073] Lines_begin=1——》The current special data part starts from line 1;

[0074] Lines_end=100——》The current special data part ends at line 100;

[0075] [Normal_Lines]——》Each group of normal data line block information;

[0076] Normal_id=1——》corresponds to the Normal_id in the normal data column type characteristics below;

[0077] Lines_begin=1001——》The current normal data part starts from line 1001;

[0078] Lines_end=9999——》The current normal data part ends at line 9999;

[0079] [Special_Lines]——》The second set of special data line block information;

[0080] Special_id=2——》corresponds to the Special_id in the special data column type characteristics below;

[0081] Lines_begin=10000——》The current special data part starts from line 10000;

[0082] Lines_end=10000——》The current special data part ends at line 10000;

[0083] Column type characteristics - Under the [Normal_Lines_Col_Type] and [Special_Lines_Col_Type] tags, write the corresponding type label attribute information based on the type characteristics of each column in the table (the label attributes of each type are different based on the characteristics), such as: string type, integer type, floating point type, date type, etc. These correspond to the types of scripts or tools and are used when generating data. Among them, all data under the [Normal_Lines_Col_Type] tag is normal data; all data under the [Special_Lines_Col_Type] tag is special data. You can select one or more of these two tags based on the data characteristics to generate the required:

[0084] [Normal_Lines_Info]——》The following are the type characteristics of all columns of normal data;

[0085] Normal_id=1——》Normal_id in the corresponding row feature;

[0086] [Col_Type]——》The following are the type characteristics of all columns. Each type corresponds to a column in the table, and the order must be consistent with the table;

[0087] [Char_Type]——》First column, character type;

[0088] Char_list=". / Character set 1.txt"——》A list file of all generated characters. Each character is separated by a separator, or placed line by line. This can increase flexibility. Various contents such as English, Chinese, numbers, and rare characters can be placed in this list. When generating data, characters will be randomly extracted from here for content splicing;

[0089] Char_min_num=10——》minimum number of characters;

[0090] Char_max_num=20——>maximum number of characters;

[0091] Char_is_null=false——"Whether null is allowed;

[0092] [Special_Lines_Info]——》The following are the type characteristics of all columns of special data;

[0093] Special_id=1;

[0094] [Col_Type]——》The following are the type characteristics of all columns. Each type corresponds to a column in the table, and the order must be consistent with the table;

[0095] [Char_Type]——》First column, character type;

[0096] Char_list=". / Character set 1.txt"——》A list file of all generated characters. Each character is separated by a separator, or placed line by line. This can increase flexibility. Various contents such as English, Chinese, numbers, and rare characters can be placed in this list. When generating data, characters will be randomly extracted from here for content splicing;

[0097] Char_min_num=0——》The minimum number of characters, the empty character, can be used to correspond to the situation where the column does not allow null and becomes an abnormal value;

[0098] Char_max_num=30——>The maximum number of characters, which exceeds the character definition length of this column (for example: 20);

[0099] Char_is_null=true——"Whether null is allowed;

[0100] Char_null_value="NN"——》null value;

[0101] [Special_Lines_Info]——》The following are the type characteristics of all columns of special data;

[0102] Special_id=2;

[0103] [Char_Type]——》First column, character type;

[0104] Char_list=". / Character set 2.txt"——》A list file of all generated characters. Each character is separated by a separator, or placed line by line. This can increase flexibility. Various contents such as English, Chinese, numbers, and rare characters can be placed in this list. When generating data, characters will be randomly extracted from here for content splicing;

[0105] Char_min_num=0——》Minimum number of characters Char_max_num=0——》Maximum number of characters. When Char_min_num and Char_min_num are both null, the value must be the following Char_null_value value;

[0106] Char_is_null=true——"Whether null is allowed;

[0107] Char_null_value="NN"——》null value;

[0108] Example 2: Import parameter configuration:

[0109] Data file placement information - sets the final location of the file, which is used to automatically transfer the generated data file to the specified location and perform path splicing when generating the import statement:

[0110] [Data_File_Location];

[0111] URL="ftp: / / test:test@10.10.1.1 / test_data / "——》According to the needs of the imported module, it can be set to the path format of the import statement;

[0112] IP="10.10.1.1";

[0113] User="test";

[0114] Passwd="test";

[0115] Location=" / opt / test_data / ";

[0116] Import statement parameters - If you use a load statement to import, based on the characteristics of a data file, to successfully load it, you can create a configuration file and write the subsequent parameters of the load statement in it:

[0117] [Import_Arguments];

[0118] Value="fields terminated by ',' null_value='\N'";

[0119] Example 3: Data import module relationship configuration:

[0120] [Table_Info];

[0121] Table_file_name="Create table 1.sql";

[0122] [Data_File_Info];

[0123] Import_id=1——》A table can have multiple data files corresponding to it, which are distinguished by the id. The id value will also be included in the name of the automatically generated import SQL to facilitate searching;

[0124] Data_file="data feature 1.txt"——》corresponding to the file in the data feature configuration module, the corresponding data file will be generated;

[0125] Data_file_name="data file name";

[0126] Import_Arg_file="Import parameter 1.txt"——》The file corresponding to the import parameter configuration module is used to generate the import statement;

[0127] [Database_Info];

[0128] Database_ip="10.10.1.2"——》Import database ip;

[0129] Database_user="test"——"Import database user name;

[0130] Database_passwd="123456"——"Import database user password;

[0131] Advantages and beneficial effects of the present invention:

[0132] (1) Transform a large amount of repetitive data generation and import work into standard process work, freeing up manpower.

[0133] (2) Data preparation is achieved by users through the configuration of relevant configuration files based on functional needs. Users focus on controlling the characteristics of the data itself to avoid unnecessary energy consumption.

[0134] (3) By automating data preparation and import, personnel can focus more on understanding the combination of data and functions, and concentrate more on functional testing itself, providing a guarantee for faster problem discovery and resolution.

[0135] (4) Through automatically planned concurrent imports, better load balancing of database distributed nodes can be achieved, thereby improving efficiency.

[0136] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for automatically generating and importing data based on database interface coding, characterized in that: The following steps are involved: S1. Create table; S2, data feature configuration; S3. Import parameter configuration; S4, data import module relationship configuration; S5. Generate import operation statement; S6. Determine whether the table creation file, data feature configuration file, character set file, import parameter configuration file, and data import module relationship configuration file are successfully established. If yes, jump to the next step S7; otherwise, report an error and exit; S7. Traverse and parse the data import module relationship configuration file, automatically generate a table import SQL file, and place it in the import SQL placement directory; S8, determine whether the data import module relationship configuration file has been fully parsed, if so, jump to the next step S9, otherwise, return to the previous step S7 to continue; S9. Call the SQL file in the directory where the SQL is placed to perform the import operation. The SQL files are executed in parallel. In step S2, data feature configuration includes: According to the type characteristics, function value characteristics and test tendency characteristics of different columns in the database table, different feature attributes are configured for each database table, and a data feature configuration file is generated in the form of a txt file and placed separately in a designated directory; In step S3, parameter configuration is imported, including: Configure the parameters of the import module according to the table creation statement and data file characteristics. Each import operation corresponds to an import parameter configuration file, which is placed separately in the specified directory.

2. The method for automatically generating and importing data based on database interface code according to claim 1, characterized in that: In step S1, a table is created, including: Write SQL files based on the function: Place the table creation SQL statements required for batch import. Save each table creation statement as a separate SQL file and place the SQL file in a specified directory.

3. The method for automatically generating and importing data based on database interface coding according to claim 1, characterized in that: In step S4, the data is imported into the module relationship configuration, including: The files generated in steps S1 to S3 are matched and set, and a data import module relationship configuration file is generated.

4. The method for automatically generating and importing data based on database interface code according to claim 1, characterized in that: In step S5, an import operation statement is generated, including: According to the data import module relationship configuration file information generated in step S4, batches of import operation statements are generated in the order of information, and are saved into the import statement sql file by table.

5. A system for automatically generating and importing data based on database interface coding, characterized by: The method for automatically generating and importing data based on database interface coding as described in any one of claims 1 to 4, the system comprising: The table creation module is used to store the table creation SQL statements required for batch import; The data feature configuration module is used to configure different feature attributes for each database table based on the type characteristics, function value characteristics, and test tendency characteristics of different columns in the database table; Import parameter configuration module, used to configure the parameters required for the import module according to the table creation statement and data file characteristics; The data import module relationship configuration module is used to match and set the files of the above table building module, data feature configuration module and import parameter configuration module; The automated distributed import module is used to configure the module's configuration file information based on the data import module relationship, batch generate import operation statements in the order of information, and save them into the import statement SQL file by table.

6. The system for automatically generating and importing data based on database interface code according to claim 5, characterized in that: In the data feature configuration module, each column in the txt file format has a txt tag, and the order is consistent with the column order of its corresponding table creation statement, and the corresponding values ​​of each feature are listed below it.

Images