Kettle conversion task generation method and device applied to government affair field, equipment and medium

Abstract

The invention discloses a Kettle conversion task generation method and device applied to the government affair field, equipment and a medium, and relates to the technical field of databases, and the method comprises the following steps: analyzing a historical government affair conversion file of Kettle by using a preset analysis tool, and determining an initial task template corresponding to a conversion task based on an analysis result, a preset coordinate rule and a business scene; establishing each database connection configuration template based on different government affair databases, performing field information matching on the source database and the target database, and determining field mapping information by utilizing a matching result; a first database connection configuration template corresponding to the source database and a second database connection configuration template corresponding to the target database are determined, a preset coordinate rule and the initial task template are integrated to determine a target task template, and a target conversion task file used for being recognized by Kettle is determined based on the target task template. And manual configuration required by migration among different types of databases is avoided.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a Kettle conversion task generation method, device, equipment and medium applied to the government affairs field. Background Art

[0002] With the rapid development of technologies such as big data, cloud computing, and the Internet of Things, the amount of data processed in government affairs has increased dramatically, and the data types are becoming increasingly diverse. To effectively utilize these data resources, it is necessary to build efficient and flexible data processing systems to support key processes such as data extraction, transformation, and loading. Kettle (primarily used for data extraction, transformation, and loading, supporting multi-source data integration and processing) is an open-source ETL (Extraction-Transformation-Loading) tool. Its powerful data processing capabilities, rich transformation components, and user-friendly graphical interface make it a standout among many ETL tools. However, despite its many advantages, Kettle still faces some challenges in practical application. Data migration between different database types cannot be handled using the database's built-in full-database migration scripts; instead, the migration task must be manually configured. If hundreds of tables are involved, the configuration workload is significantly increased, adding to the workload.

[0003] As can be seen from the above, how to avoid manual configuration when migrating between different types of databases is an urgent problem that needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a Kettle conversion task generation method, device, equipment and medium for use in the government affairs field, which can avoid the need for manual configuration when migrating between different types of databases. The specific solution is as follows:

[0005] In the first aspect, the present application provides a Kettle conversion task generation method applied in the government affairs field, including:

[0006] Use a preset parsing tool to parse Kettle's historical government conversion files to obtain corresponding parsing results, determine the target variable based on the parsing results, and use the target variable, preset coordinate rules, and business scenarios to determine the initial task template corresponding to the conversion task; the preset coordinate rules are rules for controlling the display position of the conversion task;

[0007] Creating database connection configuration templates based on different government databases, converting each database connection configuration template into a template file, and determining a template name corresponding to the template file; the government databases and the template names are in a one-to-one correspondence;

[0008] Designing and laying out interface components using a preset development tool to obtain a visual interactive interface, determining a source database and a target database corresponding to the user terminal based on the visual interactive interface, and matching field information of the source database and the target database to determine field mapping information using the matching results;

[0009] Determine respectively a first database connection configuration template corresponding to the source database and a second database connection configuration template corresponding to the target database, determine coordinate parameters of each conversion task based on the preset coordinate rule, utilize the coordinate parameters, the field mapping information, the source database and the target database and determine a target task template based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determine a target conversion task file for being recognized by Kettle based on the target task template.

[0010] Optionally, the preset parsing tool is used to parse Kettle's historical government affairs conversion file to obtain corresponding parsing results, the target variable is determined based on the parsing results, and the initial task template corresponding to the conversion task is determined using the target variable, preset coordinate rules and business scenarios, including:

[0011] Parsing the file structure corresponding to Kettle's historical government affairs conversion file using an XML parsing tool, and determining the target variable in the file structure based on the obtained parsing result;

[0012] The target variable is replaced based on the preset template variable to obtain a replaced variable, and the target coordinate value is increased by using the task quantity of the conversion task and the preset increment value to obtain a preset coordinate rule;

[0013] An initial task template corresponding to the conversion task is determined based on the business scenario and in combination with the replaced variables and the preset coordinate rules.

[0014] Optionally, the creation of database connection configuration templates based on different government databases includes:

[0015] Determining whether the government affairs database is a target database;

[0016] If the government database is the target database, the target parameters corresponding to the target database are configured using the general database type of the Kettle, and after the configuration is completed, it is determined whether the Kettle can access the target database; the target parameters include the JDBC connection information, user name and password corresponding to the target database;

[0017] If the target database can be accessed, a connection file corresponding to the target parameters is generated, and the connection file is extracted to obtain target tag content, and a database connection configuration template corresponding to the target database is determined based on the target tag content.

[0018] Optionally, the step of designing and laying out interface components using a preset development tool to obtain a visual interactive interface, determining a source database and a target database corresponding to the user terminal based on the visual interactive interface, and matching field information of the source database and the target database to determine field mapping information using the matching results includes:

[0019] Use preset development tools to design and layout table selection areas, information display areas, prompt areas, and buttons to obtain a visual interactive interface;

[0020] Acquire a source database and a target database corresponding to the user terminal based on the visual interface, and respectively read first field information corresponding to a first target table in the source database and second field information corresponding to a second target table in the target database using a preset interface;

[0021] matching a first field name in the first field information with a second field name in the second field information;

[0022] If the match is successful, field mapping information is determined based on the field pair including the first field name and the second field name.

[0023] Optionally, after matching the first field name in the first field information with the second field name in the second field information, the method further includes:

[0024] If the match is unsuccessful, the preset mapping rule or regular expression is used and the process jumps to the step of matching the first field name in the first field information with the second field name in the second field information until the first field name and the second field name are matched successfully to obtain the field mapping information.

[0025] Optionally, the steps of respectively determining a first database connection configuration template corresponding to the source database and a second database connection configuration template corresponding to the target database, determining coordinate parameters of each conversion task based on the preset coordinate rule, determining a target task template based on the initial task template, the first database connection configuration template, and the second database connection configuration template using the coordinate parameters, the field mapping information, the source database, and the target database, and determining a target conversion task file for recognition by Kettle based on the target task template include:

[0026] respectively determining a first database connection configuration template corresponding to the source database and a second database connection configuration template corresponding to the target database;

[0027] Determine the target task template based on user needs, determine the initialization coordinates, and determine the corresponding conversion tasks using the number of tables selected by the user terminal;

[0028] Determining coordinate parameters corresponding to each of the conversion tasks based on the initialization coordinates and the preset coordinate rules;

[0029] Determine a target task template using the coordinate parameters, the field mapping information, the source database and the target database based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determine a target conversion task file for being recognized by Kettle based on the target task template.

[0030] Optionally, the determining a target task template based on the coordinate parameters, the field mapping information, the source database, and the target database and based on the initial task template, the first database connection configuration template, and the second database connection configuration template, and determining a target conversion task file for being recognized by Kettle based on the target task template includes:

[0031] Based on a preset template engine, first database parameters corresponding to the source database and second database parameters corresponding to the target database are written into the first database connection configuration template and the second database connection configuration template, respectively, so as to determine target connection configuration information using the obtained first populated configuration template and the obtained second populated configuration template;

[0032] The target connection configuration information, the coordinate parameters and the field mapping information are written into the initial task template to obtain a target task template, and a target conversion task file for being recognized by Kettle is generated based on the target task template.

[0033] In a second aspect, the present application provides a Kettle conversion task generation device for use in the government affairs field, including:

[0034] The task template determination module is used to parse Kettle's historical government conversion files using a preset parsing tool to obtain corresponding parsing results, determine the target variable based on the parsing results, and determine the initial task template corresponding to the conversion task using the target variable, preset coordinate rules, and business scenarios; the preset coordinate rules are rules for controlling the display position of the conversion task;

[0035] A template name determination module, configured to create database connection configuration templates based on different government databases, convert each database connection configuration template into a template file, and determine a template name corresponding to the template file; the government databases and the template names are in a one-to-one correspondence;

[0036] An information matching module is used to design and layout interface components using a preset development tool to obtain a visual interactive interface, determine a source database and a target database corresponding to the user terminal based on the visual interactive interface, and match field information of the source database and the target database to determine field mapping information using the matching results;

[0037] A target file determination module is used to respectively determine the first database connection configuration template corresponding to the source database and the second database connection configuration template corresponding to the target database, determine the coordinate parameters of each conversion task based on the preset coordinate rules, use the coordinate parameters, the field mapping information, the source database and the target database and determine the target task template based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determine the target conversion task file for being recognized by Kettle based on the target task template.

[0038] In a third aspect, the present application provides an electronic device, comprising:

[0039] Memory, used to store computer programs;

[0040] A processor is used to execute the computer program to implement the aforementioned Kettle conversion task generation method applied to the government affairs field.

[0041] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned Kettle conversion task generation method applied in the government affairs field.

[0042] This application uses a preset parsing tool to parse Kettle's historical government conversion files to obtain corresponding parsing results, determines the target variable based on the parsing results, and uses the target variable, preset coordinate rules and business scenarios to determine the initial task template corresponding to the conversion task; the preset coordinate rules are rules for controlling the display position of the conversion task; creates each database connection configuration template based on different government databases, and converts each database connection configuration template into a template file, and determines the template name corresponding to the template file; the government database and the template name are in a one-to-one correspondence; uses preset development tools to design and layout interface components to obtain a visual interactive interface, based on the visual interactive The interface determines the source database and target database corresponding to the user terminal, and matches the field information of the source database and the target database to determine the field mapping information using the matching results; respectively determines the first database connection configuration template corresponding to the source database and the second database connection configuration template corresponding to the target database, determines the coordinate parameters of each conversion task based on the preset coordinate rule, uses the coordinate parameters, the field mapping information, the source database and the target database and determines the target task template based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determines the target conversion task file for being recognized by Kettle based on the target task template.

[0043] As can be seen from the above, this application first parses the Kettle government conversion file, which can quickly extract key variables, and generate standardized task templates based on preset coordinate rules and business scenario requirements to ensure that the task structure is highly matched with business requirements; create database connection configuration templates based on different government databases, and make connection templates for various databases in advance. You only need to select the template name to complete the connection configuration, which greatly reduces the configuration error rate; through the visual interface designed by the preset development tool, users can intuitively select the source database and the target database, match the mapping information determined based on the field information, and combine the task template, the source database, and the connection template of the target database to obtain a target conversion task file that Kettle can recognize. In this way, through template reuse and automatic filling, all conversion tasks can be generated in batches, and the automation of Kettle's data conversion task configuration in government scenarios can be realized. It is especially suitable for scenarios with multiple departments, multiple databases, and large-scale data migration in the government field, and can significantly improve data processing efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 This is a flowchart of a Kettle conversion task generation method for government affairs disclosed in this application;

[0046] Figure 2 This is a schematic diagram of the structure of a Kettle conversion task generation device for use in the government affairs field disclosed in this application;

[0047] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] At present, the amount of data processed in the government sector has increased dramatically, and it is necessary to build an efficient and flexible data processing system to support key links such as data extraction, conversion, and loading. However, for the data migration needs between different types of databases, Kettle cannot be processed through the database's own whole database migration script, and can only manually configure the migration task. If hundreds of tables are involved, the workload of the configuration task will increase greatly, which increases the workload. To this end, the present application provides a Kettle conversion task generation method applied to the government sector. Through template reuse and automatic filling, all conversion tasks can be generated in batches, realizing the automation of Kettle's data conversion task configuration in government scenarios. It is especially suitable for scenarios with multiple departments, multiple databases, and large-scale data migration in the government sector, and can significantly improve data processing efficiency and accuracy.

[0050] See also Figure 1 As shown, the embodiment of the present invention discloses a Kettle conversion task generation method applied to the government affairs field, including:

[0051] Step S11: Use the preset parsing tool to parse Kettle's historical government affairs conversion files to obtain corresponding parsing results, determine the target variable based on the parsing results, and use the target variable, preset coordinate rules and business scenarios to determine the initial task template corresponding to the conversion task; the preset coordinate rules are rules for controlling the display position of the conversion task.

[0052] In this embodiment, an XML parsing tool is used to parse the .ktr file structure corresponding to Kettle's historical government conversion file to obtain variable variables in the file structure, and the variable variables are determined as target variables in the file structure. The target variables are replaced with preset template variables to obtain replaced variables. For example, ${sourceTable} is used to represent the source table name, and the target coordinate value is increased by the task number and preset increment value of the conversion task to obtain the preset coordinate rule, and the preset coordinate rule is replaced with the initial task template. <gui>In the tag, the initial task template corresponding to the conversion task is determined in combination with the business scenario and the replaced variable. In addition, if the business scenario is a full migration scenario, Kettle's Table Input component can be used to read the data to be migrated corresponding to the source table in the source database, and then the target table in the target database can be cleared based on Kettle's Truncate Table component to obtain the cleared target table, and the data to be migrated can be migrated to the cleared target table to obtain the corresponding full migration task, and the full migration task can be exported as a .ktr file to obtain a new historical government conversion file, and jump to the step of using the XML parsing tool to parse the .ktr file structure corresponding to Kettle's historical government conversion file to obtain preset coordinate rules and replaced variables, and determine the first initial task template corresponding to the full migration based on the replaced variables and the preset coordinate rules; if the business scenario is a batch migration scenario, the corresponding target data is obtained from the source table based on the preset filtering conditions; the target data is the new data or modified data relative to the last migration operation; and then the target data is written into the target table to obtain the corresponding incremental migration task, and then the same as the full migration scenario is followed to obtain the second initial task template corresponding to the incremental migration.

[0053] Specifically, the preset parsing tool is used to parse Kettle's historical government affairs conversion file to obtain corresponding parsing results, the target variable is determined based on the parsing result, and the initial task template corresponding to the conversion task is determined using the target variable, preset coordinate rules and business scenarios, including: using an XML parsing tool to parse the file structure corresponding to Kettle's historical government affairs conversion file, and determining the target variable in the file structure based on the obtained parsing result; replacing the target variable based on the preset template variable to obtain a replaced variable, and increasing the target coordinate value using the task quantity and preset increment value of the conversion task to obtain the preset coordinate rule; determining the initial task template corresponding to the conversion task based on the business scenario and in combination with the replaced variable and the preset coordinate rule.

[0054] Step S12: Create database connection configuration templates based on different government databases, convert each database connection configuration template into a template file, and determine the template name corresponding to the template file; the government database and the template name are in a one-to-one correspondence.

[0055] In this embodiment, it is determined whether the government database is a target domestic database. If so, the target parameters corresponding to the target domestic database are configured using the generic type of the Kettle. The target parameters include database connection parameters, i.e., JDBC (Java Database Connectivity) connection information, user name, and password corresponding to the target database. After the configuration is completed, it is determined whether the Kettle can access the target database. If the Kettle can access the target database, a connection file corresponding to the target parameters is generated, and the connection file is extracted to obtain the target tag content, such as the content in the connection file. <connection>The tag content is then used to determine the database connection configuration template corresponding to the target database based on the target tag content. Each database connection configuration template is stored in a file format to obtain each template file, and a template name corresponding to each template file is determined. It is worth mentioning that there is a one-to-one correspondence between the government database and the template name, and the template name is unique.

[0056] Specifically, the creation of each database connection configuration template based on different government databases includes: determining whether the government database is the target database; if the government database is the target database, configuring the target parameters corresponding to the target database using the general database type of the Kettle, and determining whether the Kettle can access the target database after the configuration is completed; the target parameters include the JDBC connection information, user name and password corresponding to the target database; if the target database can be accessed, generating a connection file corresponding to the target parameters, and extracting the connection file to obtain the target tag content, and determining the database connection configuration template corresponding to the target database based on the target tag content.

[0057] Step S13: Use a preset development tool to design and layout interface components to obtain a visual interactive interface, determine the source database and target database corresponding to the user end based on the visual interactive interface, and match the field information of the source database and the target database to determine the field mapping information using the matching results.

[0058] In this embodiment, Java Swing (a Java GUI (Graphical User Interface) development component) is used to construct an interface consisting of a table selection area, an information display area, a prompt area, and buttons to obtain a visual interactive interface. The table selection area includes a checkbox, a panel for holding the checkbox, and a scroll panel; the information display area includes a label for displaying information; and the prompt area includes a description of the display scenario, such as supporting only migrations where the source and target tables have consistent structures. In one specific embodiment, to achieve a full table swap, a drop-down box can be set up to query all tables in the database. Select all and unselect all checkboxes are added to determine which tables need to be converted. A scroll bar should be added to the drop-down box content to ensure readability, and a button should be added to trigger the conversion field generation instruction.

[0059] It can be understood that after obtaining the visual interface, the source database and target database corresponding to the user terminal are obtained through the visual interface, and the database structure information is read using Java's DataSource and DatabaseMetaData interfaces; the connection object is obtained through the getConnection() method, the database metadata is obtained through the getMetaData() method, and the getColumns() method is called to obtain the first field information corresponding to the source table in the source database and the second field information corresponding to the target table in the target database respectively, and the field names of the source table and the target table are compared through equals(). If the field names of the source table and the target table are the same, they are automatically mapped.

[0060] Specifically, the method uses a preset development tool to design and layout interface components to obtain a visual interactive interface, determines the source database and target database corresponding to the user terminal based on the visual interactive interface, and matches the field information of the source database and the target database to determine the field mapping information using the matching results, including: using the preset development tool to design and layout the table selection area, information display area, prompt area and buttons to obtain a visual interactive interface; based on the visual interface, obtains the source database and target database corresponding to the user terminal, and uses a preset interface to read the first field information corresponding to the first target table in the source database and the second field information corresponding to the second target table in the target database respectively; matches the first field name in the first field information with the second field name in the second field information; if the match is successful, determines the field mapping information based on the field pair including the first field name and the second field name.

[0061] Furthermore, if the field names of the source table and the target table are different but have the same meaning, the field names are matched using a preset mapping rule or regular expression, wherein the preset mapping rule includes ignoring underscores in the field names; then the field pairs corresponding to the successfully matched field names are stored in a collection to obtain field mapping information. Specifically, after matching the first field name in the first field information with the second field name in the second field information, the method further includes: if the match is unsuccessful, using a preset mapping rule or regular expression and jumping to the step of matching the first field name in the first field information with the second field name in the second field information, until the first field name and the second field name are successfully matched to obtain field mapping information.

[0062] Step S14: respectively determine the first database connection configuration template corresponding to the source database and the second database connection configuration template corresponding to the target database, determine the coordinate parameters of each conversion task based on the preset coordinate rule, use the coordinate parameters, the field mapping information, the source database and the target database and determine the target task template based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determine the target conversion task file for being recognized by Kettle based on the target task template.

[0063] In this embodiment, a first database connection configuration template corresponding to the source database and a second database connection configuration template corresponding to the target database are determined respectively. Then, based on user needs, a target task template is determined, initialization coordinates are determined, and the corresponding number of conversion tasks is determined using the number of tables selected by the user end. In a specific embodiment, if the initialization coordinates are x=100, y=200, and the increment step=200, the coordinates of each conversion task are determined, and the coordinates corresponding to the i-th conversion task are (x, y+i*step). The x-axis coordinates remain unchanged, and the y-axis increments sequentially to avoid task overlap. The target task template is then determined based on the coordinates of each conversion task, the field mapping information, the source database, and the target database, and based on the initial task template, the first database connection configuration template, and the second database connection configuration template. Based on the target task template, a target conversion task file for Kettle recognition is determined.

[0064] Specifically, the first database connection configuration template corresponding to the source database and the second database connection configuration template corresponding to the target database are determined respectively, the coordinate parameters of each conversion task are determined based on the preset coordinate rule, the target task template is determined based on the coordinate parameters, the field mapping information, the source database and the target database and based on the initial task template, the first database connection configuration template and the second database connection configuration template, and the target conversion task file for being recognized by Kettle is determined based on the target task template, including: respectively determining the first database connection configuration template corresponding to the source database and the second database connection configuration template corresponding to the target database; determining the target task template based on user needs, determining the initialization coordinates, and determining the corresponding conversion tasks based on the number of tables selected by the user end; determining the coordinate parameters corresponding to each conversion task based on the initialization coordinates and the preset coordinate rule; determining the target task template based on the coordinate parameters, the field mapping information, the source database and the target database and based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determining the target conversion task file for being recognized by Kettle based on the target task template.

[0065] It is understood that FreeMarker (i.e., a template engine) is used to write the first database parameters corresponding to the source database into the first database connection configuration template, and the second database parameters corresponding to the target database into the second database connection configuration template. The target connection configuration information is determined using the obtained first and second filled configuration templates. The target connection configuration information, the coordinate parameters, and the field mapping information are then written into the initial task template to obtain the target task template. The target task template is then converted into a .ktr format file to obtain a target conversion task file for Kettle recognition. The target conversion task file can be written to a specified path using FileWriter (i.e., a convenient class for writing character files). If there are too many target conversion task files, the target conversion files can be compressed and packaged to reduce storage occupancy and transmission costs.

[0066] Specifically, the target task template is determined based on the initial task template, the first database connection configuration template and the second database connection configuration template using the coordinate parameters, the field mapping information, the source database and the target database, and the target conversion task file for being recognized by Kettle is determined based on the target task template, including: writing the first database parameters corresponding to the source database and the second database parameters corresponding to the target database into the first database connection configuration template and the second database connection configuration template respectively based on a preset template engine, so as to determine the target connection configuration information using the obtained first post-fill configuration template and the second post-fill configuration template; writing the target connection configuration information, the coordinate parameters and the field mapping information into the initial task template to obtain the target task template, and generating a target conversion task file for being recognized by Kettle based on the target task template.

[0067] As can be seen from the above, this application first parses the Kettle government conversion file, which can quickly extract key variables, and generate standardized task templates based on preset coordinate rules and business scenario requirements to ensure that the task structure is highly matched with business requirements; create database connection configuration templates based on different government databases, and make connection templates for various databases in advance. You only need to select the template name to complete the connection configuration, which greatly reduces the configuration error rate; through the visual interface designed by the preset development tool, users can intuitively select the source database and the target database, match the mapping information determined based on the field information, and combine the task template, the source database, and the connection template of the target database to obtain a target conversion task file that Kettle can recognize. In this way, through template reuse and automatic filling, all conversion tasks can be generated in batches, and the automation of Kettle's data conversion task configuration in government scenarios can be realized. It is especially suitable for scenarios with multiple departments, multiple databases, and large-scale data migration in the government field, and can significantly improve data processing efficiency and accuracy.

[0068] Accordingly, see Figure 2 As shown, the present application also provides a Kettle conversion task generation device applied to the government affairs field, including:

[0069] The task template determination module 11 is used to parse Kettle's historical government conversion files using a preset parsing tool to obtain corresponding parsing results, determine target variables based on the parsing results, and determine the initial task template corresponding to the conversion task using the target variables, preset coordinate rules, and business scenarios; the preset coordinate rules are rules for controlling the display position of the conversion task;

[0070] The template name determination module 12 is used to create database connection configuration templates based on different government databases, convert each database connection configuration template into a template file, and determine the template name corresponding to the template file; the government database and the template name have a one-to-one correspondence;

[0071] An information matching module 13 is configured to design and layout interface components using a preset development tool to obtain a visual interactive interface, determine a source database and a target database corresponding to the user terminal based on the visual interactive interface, and match field information of the source database and the target database to determine field mapping information using the matching results;

[0072] The target file determination module 14 is used to respectively determine the first database connection configuration template corresponding to the source database and the second database connection configuration template corresponding to the target database, determine the coordinate parameters of each conversion task based on the preset coordinate rules, use the coordinate parameters, the field mapping information, the source database and the target database and determine the target task template based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determine the target conversion task file for being recognized by Kettle based on the target task template.

[0073] As can be seen from the above, this application first parses the Kettle government conversion file, which can quickly extract key variables, and generate standardized task templates based on preset coordinate rules and business scenario requirements to ensure that the task structure is highly matched with business requirements; create database connection configuration templates based on different government databases, and make connection templates for various databases in advance. You only need to select the template name to complete the connection configuration, which greatly reduces the configuration error rate; through the visual interface designed by the preset development tool, users can intuitively select the source database and the target database, match the mapping information determined based on the field information, and combine the task template, the source database, and the connection template of the target database to obtain a target conversion task file that Kettle can recognize. In this way, through template reuse and automatic filling, all conversion tasks can be generated in batches, and the automation of Kettle's data conversion task configuration in government scenarios can be realized. It is especially suitable for scenarios with multiple departments, multiple databases, and large-scale data migration in the government field, and can significantly improve data processing efficiency and accuracy.

[0074] In some specific implementations, the task template determination module 11 may specifically include:

[0075] A file structure parsing unit, configured to parse the file structure corresponding to Kettle's historical government affairs conversion file using an XML parsing tool, and determine a target variable in the file structure based on the obtained parsing result;

[0076] a variable replacement unit, configured to replace the target variable based on a preset template variable to obtain a replaced variable, and to increase the target coordinate value by using the task quantity of the conversion task and a preset increment value to obtain a preset coordinate rule;

[0077] An initial template determining unit is configured to determine an initial task template corresponding to the conversion task based on a business scenario and in combination with the replaced variables and the preset coordinate rules.

[0078] In some specific implementations, the template name determination module 12 may specifically include:

[0079] A database determination unit, configured to determine whether the government affairs database is a target database;

[0080] a target parameter configuration unit, configured to, if the government database is the target database, configure target parameters corresponding to the target database using the general database type of the Kettle, and determine whether the Kettle can access the target database after the configuration is completed; the target parameters include JDBC connection information, user name, and password corresponding to the target database;

[0081] The file extraction unit is used to generate a connection file corresponding to the target parameters if the target database can be accessed, extract the connection file to obtain target tag content, and determine a database connection configuration template corresponding to the target database based on the target tag content.

[0082] In some specific implementations, the information matching module 13 may specifically include:

[0083] The interface design unit is used to design and layout the table selection area, information display area, prompt area and buttons using preset development tools to obtain a visual interactive interface;

[0084] an information reading unit, configured to obtain a source database and a target database corresponding to the user terminal based on the visual interface, and read first field information corresponding to a first target table in the source database and second field information corresponding to a second target table in the target database using a preset interface;

[0085] a first name matching unit, configured to match a first field name in the first field information with a second field name in the second field information;

[0086] A mapping information determining unit is configured to determine field mapping information based on a field pair including the first field name and the second field name if the match is successful.

[0087] In some specific implementations, the Kettle conversion task generation device applied to the government affairs field may further include:

[0088] The second name matching unit is used to use a preset mapping rule or regular expression and jump to the step of matching the first field name in the first field information with the second field name in the second field information if the match is unsuccessful, until the first field name and the second field name are successfully matched to obtain field mapping information.

[0089] In some specific implementations, the target file determination module 14 may specifically include:

[0090] a template determining unit, configured to respectively determine a first database connection configuration template corresponding to the source database and a second database connection configuration template corresponding to the target database;

[0091] A conversion task determination unit, configured to determine a target task template based on user requirements, determine initialization coordinates, and determine corresponding conversion tasks using the number of tables selected by the user terminal;

[0092] a coordinate parameter determining unit, configured to determine a coordinate parameter corresponding to each of the conversion tasks based on the initialization coordinates and the preset coordinate rule;

[0093] The first target template determination unit is used to determine the target task template based on the coordinate parameters, the field mapping information, the source database and the target database and based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determine the target conversion task file for being recognized by Kettle based on the target task template.

[0094] In some specific implementations, the target file determination module 14 may specifically include:

[0095] a target information determining unit, configured to write, based on a preset template engine, first database parameters corresponding to the source database and second database parameters corresponding to the target database into the first database connection configuration template and the second database connection configuration template, respectively, so as to determine target connection configuration information using the obtained first populated configuration template and second populated configuration template;

[0096] The second target template determination unit is used to write the target connection configuration information, the coordinate parameters and the field mapping information into the initial task template to obtain a target task template, and generate a target conversion task file for being recognized by Kettle based on the target task template.

[0097] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 3 This is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the Kettle conversion task generation method applied to the government affairs field disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment can specifically be an electronic computer.

[0098] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0099] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the Kettle conversion task generation method for the government affairs field executed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0101] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned disclosed method for generating Kettle conversion tasks for use in the government affairs field is implemented. The specific steps of this method can be referred to the corresponding contents disclosed in the aforementioned embodiments and will not be repeated here.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0103] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0105] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise," "include," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a set of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0106] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.< / connection> < / gui>

Claims

1. A Kettle conversion task generation method applied to the government affairs field, characterized in that: include: Use a preset parsing tool to parse Kettle's historical government conversion files to obtain corresponding parsing results, determine the target variable based on the parsing results, and use the target variable, preset coordinate rules, and business scenarios to determine the initial task template corresponding to the conversion task; the preset coordinate rules are rules for controlling the display position of the conversion task; Creating database connection configuration templates based on different government databases, converting each database connection configuration template into a template file, and determining a template name corresponding to the template file; There is a one-to-one correspondence between the government affairs database and the template name; Designing and laying out interface components using a preset development tool to obtain a visual interactive interface, determining a source database and a target database corresponding to the user terminal based on the visual interactive interface, and matching field information of the source database and the target database to determine field mapping information using the matching results; Determine respectively a first database connection configuration template corresponding to the source database and a second database connection configuration template corresponding to the target database, determine coordinate parameters of each conversion task based on the preset coordinate rule, utilize the coordinate parameters, the field mapping information, the source database and the target database and determine a target task template based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determine a target conversion task file for being recognized by Kettle based on the target task template.

2. The Kettle conversion task generation method applied to the government affairs field according to claim 1 is characterized in that: The preset parsing tool is used to parse Kettle's historical government affairs conversion files to obtain corresponding parsing results, and the target variable is determined based on the parsing results. The target variable, preset coordinate rules and business scenarios are used to determine the initial task template corresponding to the conversion task, including: Parsing the file structure corresponding to Kettle's historical government affairs conversion file using an XML parsing tool, and determining the target variable in the file structure based on the obtained parsing result; The target variable is replaced based on the preset template variable to obtain a replaced variable, and the target coordinate value is increased by using the task quantity of the conversion task and the preset increment value to obtain a preset coordinate rule; An initial task template corresponding to the conversion task is determined based on the business scenario and in combination with the replaced variables and the preset coordinate rules.

3. The Kettle conversion task generation method applied to the government affairs field according to claim 1 is characterized in that: The creation of database connection configuration templates based on different government databases includes: Determining whether the government affairs database is a target database; If the government database is the target database, the target parameters corresponding to the target database are configured using the general database type of the Kettle, and after the configuration is completed, it is determined whether the Kettle can access the target database; the target parameters include the JDBC connection information, user name and password corresponding to the target database; If the target database can be accessed, a connection file corresponding to the target parameters is generated, and the connection file is extracted to obtain target tag content, and a database connection configuration template corresponding to the target database is determined based on the target tag content.

4. The Kettle conversion task generation method applied to the government affairs field according to claim 1 is characterized in that: The method includes: designing and laying out interface components using a preset development tool to obtain a visual interactive interface; determining a source database and a target database corresponding to a user terminal based on the visual interactive interface; matching field information of the source database and the target database to determine field mapping information using the matching result; and Use preset development tools to design and layout table selection areas, information display areas, prompt areas, and buttons to obtain a visual interactive interface; Acquire a source database and a target database corresponding to the user terminal based on the visual interface, and respectively read first field information corresponding to a first target table in the source database and second field information corresponding to a second target table in the target database using a preset interface; matching a first field name in the first field information with a second field name in the second field information; If the match is successful, field mapping information is determined based on the field pair including the first field name and the second field name.

5. The Kettle conversion task generation method applied to the government affairs field according to claim 4 is characterized in that: After matching the first field name in the first field information with the second field name in the second field information, the method further includes: If the match is unsuccessful, the preset mapping rule or regular expression is used and the process jumps to the step of matching the first field name in the first field information with the second field name in the second field information until the first field name and the second field name are matched successfully to obtain the field mapping information.

6. The Kettle conversion task generation method applied to the government affairs field according to claim 5 is characterized in that: The steps of respectively determining a first database connection configuration template corresponding to the source database and a second database connection configuration template corresponding to the target database, determining coordinate parameters of each conversion task based on the preset coordinate rule, determining a target task template based on the initial task template, the first database connection configuration template, and the second database connection configuration template using the coordinate parameters, the field mapping information, the source database, and the target database, and determining a target conversion task file for being recognized by Kettle based on the target task template include: respectively determining a first database connection configuration template corresponding to the source database and a second database connection configuration template corresponding to the target database; Determine the target task template based on user needs, determine the initialization coordinates, and determine the corresponding conversion tasks using the number of tables selected by the user terminal; Determining coordinate parameters corresponding to each of the conversion tasks based on the initialization coordinates and the preset coordinate rules; Determine a target task template using the coordinate parameters, the field mapping information, the source database and the target database based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determine a target conversion task file for being recognized by Kettle based on the target task template.

7. The Kettle conversion task generation method applied to the government affairs field according to claim 6 is characterized in that: The step of determining a target task template based on the coordinate parameters, the field mapping information, the source database, and the target database and based on the initial task template, the first database connection configuration template, and the second database connection configuration template, and determining a target conversion task file for being recognized by Kettle based on the target task template includes: Based on a preset template engine, first database parameters corresponding to the source database and second database parameters corresponding to the target database are written into the first database connection configuration template and the second database connection configuration template, respectively, so as to determine target connection configuration information using the obtained first populated configuration template and the obtained second populated configuration template; The target connection configuration information, the coordinate parameters and the field mapping information are written into the initial task template to obtain a target task template, and a target conversion task file for being recognized by Kettle is generated based on the target task template.

8. A Kettle conversion task generation device used in the government affairs field, characterized in that: include: The task template determination module is used to parse Kettle's historical government conversion files using a preset parsing tool to obtain corresponding parsing results, determine the target variable based on the parsing results, and determine the initial task template corresponding to the conversion task using the target variable, preset coordinate rules, and business scenarios; the preset coordinate rules are rules for controlling the display position of the conversion task; A template name determination module, configured to create database connection configuration templates based on different government databases, convert each database connection configuration template into a template file, and determine a template name corresponding to the template file; the government databases and the template names are in a one-to-one correspondence; An information matching module is used to design and layout interface components using a preset development tool to obtain a visual interactive interface, determine a source database and a target database corresponding to the user terminal based on the visual interactive interface, and match field information of the source database and the target database to determine field mapping information using the matching results; A target file determination module is used to respectively determine the first database connection configuration template corresponding to the source database and the second database connection configuration template corresponding to the target database, determine the coordinate parameters of each conversion task based on the preset coordinate rules, use the coordinate parameters, the field mapping information, the source database and the target database and determine the target task template based on the initial task template, the first database connection configuration template and the second database connection configuration template, and determine the target conversion task file for being recognized by Kettle based on the target task template.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is used to execute the computer program to implement the Kettle conversion task generation method applied to the government affairs field as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, it implements the Kettle conversion task generation method applied to the government affairs field as described in any one of claims 1 to 7.

Images