Visual data integration development method and system

Abstract

The invention provides a visual data integration development method and system. The method comprises the steps that a deployment directory is created, and a development program is created; according to a data development result and an existing component function, customizing component development, and designing a program link; performing program development; testing is executed, and related logs are printed; and issuing and configuring scheduling. According to the method, data integration development is completed through a visual interface, data development work such as data collection, cleaning and conversion is completed on one interface, page switching is reduced, and implementation logic is visually displayed through a graphical interface; each function is provided in a component mode, a complex coding process is converted into visual graphical operation, and the development difficulty is reduced; when the data source changes, only relevant data source configuration needs to be modified, scripts do not need to be updated, when service expansion needs to be carried out, only certain nodes need to be added, deleted and modified, the current data development function can be expanded, the data integration development cost is reduced, and the data integration development efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data integration development; in particular, to a visual data integration development method and system. Background Art

[0002] With the advent of the big data era, the amount of data collection, storage, processing and dissemination is also increasing day by day, which provides a broad space for the development of data integration technology. In recent decades, the rapid development of science and technology and the advancement of informatization have provided strong technical support for data integration technology. For example, the rise of cloud computing, big data processing, artificial intelligence and other technologies have brought more possibilities and innovations to data integration technology.

[0003] However, traditional data integration usually relies on manually written scripts or the use of ETL (Extract, Transform, Load) tools to complete data extraction, transformation and loading. This process is time-consuming, and when the data source or business requirements change, these scripts need to be updated frequently, which is prone to problems such as script change errors and script loss, increasing the time cost burden; it also lacks flexibility and cannot be expanded quickly, and cannot adapt to the growing data integration development needs.

[0004] In addition, maintaining these manually written scripts and processes requires specialized skills, and when developers leave or technology is upgraded, it may be necessary to retrain or hire new professionals, increasing the human resource costs of data integration development.

[0005] In addition, traditional data integration development, including data collection, cleaning, conversion and other data processing and analysis, are scattered across various menus and pages, making it impossible to display and process them in a centralized manner, and unable to provide users with an intuitive and clear processing interface and result explanation. Summary of the invention

[0006] In view of this, the purpose of the present invention is to propose a method and system for visual data integration development, which completes data integration development through a visual interface, completes various data development tasks such as data collection, cleaning, and conversion in one interface, reduces page switching, and presents the implementation logic more intuitively with a graphical interface; each function is provided in the form of a component, and the complex coding process is converted into an intuitive graphical operation to reduce the difficulty of development; when the data source changes, only the relevant data source configuration needs to be modified without updating the script; when business expansion is required, only certain nodes need to be added, deleted, and modified to expand the current data development function, so as to reduce the cost of data integration development and improve the efficiency of data integration development.

[0007] The present invention provides a method for visual data integration development, comprising the following steps: Step 1. Create and deploy different directories according to different categories of business needs (creating different directories is to better classify the development programs and avoid too chaotic distribution of programs); create programs in the corresponding directories and fill in program information; the program information includes: program name, program Chinese name, program remarks; Step 2: Based on the data development results to be achieved and the functions that can be achieved by existing components, if the existing components do not meet the data development requirements, develop customized components to determine the components to be used and the overall program chain; Step 3. Start the program created in Step 1, enter the integrated development canvas interface, drag related components from the component pool to the canvas according to the program design, connect the components according to the logic of the program design; configure the components used; Exemplarily, the configuration information for configuring the used components is as follows: (1) The mysqlreader component contains the following configuration information: The step name of the input box, the default is step number + component name; The radio button indicates whether to use SQL. The options are Yes and No. In the database drop-down box, select the database information for which data needs to be read; The import table of the input box is displayed only when the radio button "Whether to use SQL" is selected as "No". Click the button on the right side of the input box to open a pop-up box, which displays the tables of the database in the drop-down box in a list. Select a table and click OK. The selected table name will be filled in the input box. This configuration is used to obtain the field information of the table to be read. The entity table name of the input box. This configuration item is displayed only when "No" is selected in the radio button "Whether to use SQL"; this configuration item is used to configure the table name actually read. The export column of the multiple-select box is displayed only when the "Use SQL" option in the radio button is selected "No". This configuration item is used to configure the columns of the database to be read; Export SQL, this configuration item is displayed only when the "Use SQL" radio button is selected. This configuration item is used to directly fill in the SQL for querying data, rather than configuring the columns of the database to be read; (2) The mysqlwriter component contains the following configuration information: The step name of the input box, the default is step number + component name; In the drop-down box database, select the database information where data needs to be written; Import table in the input box. Click the button on the right side of the input box to open a pop-up window, which displays the tables in the database in the drop-down box in a list. Select a table and click OK. The selected table name will be filled in the input box. This configuration is used to obtain the field information of the table to be written. The entity table name of the input box. This configuration item is used to configure the actual table name to be written. The writing mode of the drop-down box has three options: insert, replace, and update, which are used to write configuration data; The number of concurrent operations in the drop-down box has multiple options from 1 to 6, which are used to configure the number of data written simultaneously each time; (3) The replacement character component of the cleaning conversion includes the following configuration information: The step name of the input box, the default is step number + component name; The original characters of the input box are used to set the characters to be replaced; The new character of the input box is used to set the replaced character; (4) The simplified and traditional Chinese conversion component of the cleaning conversion includes the following configuration information: The step name of the input box, the default is step number + component name; Conversion format, there are two options: Traditional to Simplified and Simplified to Traditional; For example, the left side of the canvas is a directory tree, where you can create multiple levels of directories, and you can add development programs under each level of the directory. You can enter text above the directory tree for fuzzy matching to search for directories or programs. You can check both the left side of the directory or program name, and after selecting, you can click the button below to publish or deactivate it.

[0008] The middle of the canvas is the main body of the canvas. All component configuration links are carried out in the main body of the canvas. There is a thumbnail at the bottom left of the canvas for global thumbnail viewing. There are some action buttons at the top of the canvas, including the following: To undo, press CTRL+Z on the keyboard to perform the same operation; To redo, press CTRL+Y on the keyboard to perform the same operation; Delete: When you click a node in the canvas, this button is available. Click it to delete the current node. Pressing DELETE on the keyboard can also perform the same operation. Version management: each time you save a program, a version will be generated. Click this button to open a version management page, where you can compare versions, roll back versions, view changes, etc. Import, click the button to open an input box, which will be filled with the overall configuration information of the current program, including the nodes, segments, node positions, etc. used, saved in XML format. You can copy the content from other programs, paste it into this input box, and then click Import to import other programs into the current program as is; Save, save the current program. Each save will generate a version. Press CTRL+S on the keyboard to perform the same operation. Test, after clicking the button, a test interface will open. There are several input boxes at the top to set the test task time, start step, end step, and execution parameters. Click "Execute Test" to run the program, and relevant logs will be printed at the bottom of the interface; On the right side of the canvas is the component pool, which displays all components by category. The search box at the top can be used for fuzzy matching to search for components.

[0009] Step 4. After the program is developed, test the program, execute the current link, and print relevant logs; Step 5. After the program test passes, enter the configuration interface of the program release and configure the scheduling information in the configuration interface of the program release.

[0010] Specifically, multiple programs may be selected for publishing at the same time.

[0011] Furthermore, the method of determining the components to be used and the overall program link in step 2 includes: The existing components used are connected in series according to the business logic, combined with branches, loop components, etc. to form a step diagram; the existing components include: Logical components, including the following components: branch, loop, infinite loop, loop break, etc.; General functional components, including the following components: variable assignment, printing message, sleep, executing terminal commands, program alarm, etc.; preferably, the configuration item variable type of the variable assignment component can be configured as a constant, JSON object, SQL object, SQLList data set, file object, etc.; Remote call components, including the following components: http request, webservice reception, etc.; File operation components, including the following components: file movement, file scanning, file deletion, file upload, etc.; Cleaning and conversion components, including the following components: field splitting, character replacement, character deletion, simplified and traditional conversion, full-width and half-width conversion, encryption and decryption, simulation, etc. Data standard check components, including the following components: duplicate data check, fluctuation check, value range check, specification check, null value check, etc.; Data security components, including the following components: table security classification, sensitive discovery, sensitive metadata discovery, sensitive field classification, sensitive data classification, etc.; Database operation components, including the following components: SQL statement execution, table creation, table deletion, etc.; Data exchange components include the following components: database read-write components Oracle, MySql, MongoDB, Postgres, Doris, OpenGauss, Hive, Hbase, etc., data format read-write components xml, json, excel, etc., third-party middleware read-write components Kafka, HDFS, Es, etc., Ftp file read-write components, etc.

[0012] Furthermore, in the step 2, the development of the customized component includes the following steps: S21. Obtain the jar package where the component abstract class and the component metadata abstract class are located; S22. Create a Java project and add the obtained jar package as a dependency in Java; use Java code to implement the component abstract class and component metadata abstract class provided in the jar package to complete the required functions; wherein the component abstract class implements the run method of the logic of executing specific steps, and the component metadata abstract class contains the configuration information of the steps; S23. Package the created Java project into a jar package and put it in the plugins directory under the directory deployed in Step 1; S24. Configure the relevant information of the custom component on the component management page, add the component's own information on the left side of the component management page, and add the component's configuration item information on the right side of the component management page; S25. Use the customized component in the canvas interface of data development.

[0013] The present invention can be adapted to different databases. If the database used is not adapted, relevant components can be added for adaptation and use.

[0014] Furthermore, the configuration item information of step S24 is added using a dynamic form, including: Dynamically set the type of configuration item, the types of configuration items include: text box, text field, drop-down box, cascade selector; Set the dependency conditions of the configuration item; for example, this configuration item will only be displayed when an option is selected in another drop-down box configuration item; Configure the input type of the configuration item; if the configuration item is a drop-down box, the data source can be configured as static data. For example, if it is configured as "1, Male|0, Female|2, Unknown", the drop-down box will display 3 options, namely Male, Female, and Unknown, with values ​​of 1, 0, and 2 respectively; the data source can be configured as a SQL statement, and the data is the result of the SQL query; the data source can be configured as an interface address, and the data is the content returned by the interface; Configure the default value of the configuration item. The default value can be a static string or number filled in directly, or SQL can be filled in. The default value is the result of the SQL query.

[0015] Furthermore, the method for performing program testing in step 4 includes: Click the test button above the canvas to enter the test interface, set the start and end steps, add custom parameters, click to execute the test, and view detailed test log information at the bottom of the test interface. The test log information includes the default output information, the information output when each component is executed, the information in the print message component added by the user, and the error stack information printed after the error is reported. The relevant logs of the test log information can be used to determine whether the program is executed correctly, and problems can be solved according to the relevant logs.

[0016] Furthermore, the configuration scheduling information in step Step5 includes: Select the trigger type, including: Time trigger: Generate tasks based on the running cycle and execution time; Event triggering: automatically start the task when the predecessor task and input table in the dependency relationship pass the detection; Select the startup type, including: Sequential start: The next batch of tasks can be executed only after a batch of tasks is completed; Multi-start: can execute multiple batches of tasks at the same time; Unique start: Only one batch of tasks can be executed at the current time; Only start within a cycle. Only one batch of tasks can be executed within a cycle (such as a month). The failure to complete the tasks of the previous cycle does not affect the start of the tasks of the next cycle. Configure the operation cycle (year, month, day, hour, minute), execution time, priority, maximum operation time, failure redo interval, failure redo count, effective date, etc.; Configure the prerequisite tasks.

[0017] The present invention also provides a system for visual data integration development, which executes the visual data integration development method as described above, including: Create a development program module: used to create and deploy different directories corresponding to different categories of business needs; create a program in the corresponding directory and fill in the program information; the program information includes: program name, program Chinese name, program remarks; Design program link module: It is used to develop customized components according to the data development results to be achieved and the functions that can be achieved by existing components. If the existing components do not meet the data development requirements, the components to be used and the overall program link are determined; Program development module: used to start the program created in Step 1, enter the canvas interface of integrated development, drag related components from the component pool to the canvas according to the program design, connect the components according to the logic of the program design; configure the components used; Execution test module: used to test the program after the program development is completed, execute the current link, and print relevant logs; Release and configure the scheduling module: After the program test is passed, enter the configuration interface of the program release and configure the scheduling information in the configuration interface of the program release.

[0018] Further, the design program link module includes a customized component development submodule, and the customized component development submodule includes: Get jar package unit: used to get the jar package where the component abstract class and component metadata abstract class are located; Create a Java project unit: used to create a Java project, add the obtained jar package as a dependency in Java; use Java code to implement the component abstract class and component metadata abstract class provided in the jar package to complete the required functions; wherein the component abstract class implements the run method of the logic of executing specific steps, and the component metadata abstract class contains the configuration information of the steps; Packaging and deployment unit: used to package the created Java project into a jar package and put it in the plugins directory under the directory deployed in Step 1; Configure custom component unit: used to configure the relevant information of custom components on the component management page, add the component's own information on the left side of the component management page, and add the component's configuration item information on the right side of the component management page; Interface usage unit: used to use the customized component in the canvas interface of data development.

[0019] The present invention provides a visual and easy-to-operate data integration development mode. In most scenarios, various data development needs can be met by only relying on built-in components to write relevant logic. There is no need to develop code for the business, nor is there any need to perform various configurations on each page. Various unique development components are provided, including variable assignment components (constants, JSON objects, Sql objects, SqlList, files and other variable assignment methods), logic components (loops, branches), database operation components (SQL execution, data reading and writing), external requests (http, webservice), script components, etc.; the extensibility of the components, on the one hand, can use js script components to write some scripts to achieve simple extension functions. If js scripts cannot meet the needs, java developers can write codes to implement the relevant interfaces and methods of this solution, which can be used on the development page after configuration and registration.

[0020] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of the method for visual data integration development as described above are implemented.

[0021] The present invention also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method for visual data integration development as described above are implemented.

[0022] Compared with the prior art, the present invention has the following beneficial effects: The visualized data integration development method and system provided by the present invention completes data integration development through a visualized interface, completes various data development tasks such as data collection, cleaning, and conversion in one interface, reduces page switching, and more intuitively displays the implementation logic with a graphical interface; each function is provided in the form of a component, converts the complex coding process into an intuitive graphical operation, and reduces the development difficulty; when the data source changes, only the relevant data source configuration needs to be modified without updating the script; when business expansion is required, only certain nodes need to be added, deleted, and modified to expand the current data development function, which reduces the data integration development cost and effectively improves the data integration development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0024] In the attached picture: Figure 1 A basic process architecture diagram for visual data integration development in accordance with an embodiment of the present invention; Figure 2 A basic process architecture diagram for developing a customized component according to an embodiment of the present invention; Figure 3 A flow chart of a method for visual data integration development of the present invention; Figure 4 A flow chart for the development of a customized component for the present invention; Figure 5 The figure is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of systems and products consistent with some aspects of the present disclosure as detailed in the appended claims.

[0026] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] The embodiments of the present invention are described in further detail below.

[0029] The present invention provides a method for visualizing data integration development. Figure 3 As shown, the following steps are included: Step 1. Create and deploy different directories according to different categories of business needs; create programs in the corresponding directories and fill in program information; the program information includes: program name, program Chinese name, and program notes; Step 2: Based on the data development results to be achieved and the functions that can be achieved by existing components, if the existing components do not meet the data development requirements, develop customized components to determine the components to be used and the overall program chain; Methods for determining the components to be used and the overall program chain include: The existing components used are connected in series according to the business logic, combined with branch and loop components to form a step diagram; the existing components include: Logical components, including the following components: branch, loop, infinite loop, and loop break; General functional components, including the following components: variable assignment, message printing, sleep, terminal command execution, program alarm; In this embodiment, the configuration item variable type of the variable assignment component is configured as constant, JSON object, SQL object, SQLList data set, file object; Remote call components, including the following components: http request, webservice reception; File operation components, including the following components: file movement, file scanning, file deletion, and file upload; Cleaning and conversion components, including the following components: field splitting, character replacement, character deletion, simplified and traditional conversion, full-width and half-width conversion, encryption and decryption, and simulation; Data standard check component, including the following components: duplicate data check, fluctuation check, value range check, specification check, and null value check; Data security components, including the following components: table security classification, sensitive discovery, sensitive metadata discovery, sensitive field classification, and sensitive data classification; Database operation components, including the following components: SQL statement execution, table creation, and table deletion; Data exchange components include the following components: database read and write components Oracle, MySql, MongoDB, Postgres, Doris, OpenGauss, Hive, Hbase, data format read and write components xml, json, excel, third-party middleware read and write components Kafka, HDFS, Es, and Ftp file read and write components.

[0030] See also Figure 4 As shown, the development of a customized component includes the following steps: S21. Obtain the jar package where the component abstract class and the component metadata abstract class are located; S22. Create a Java project and add the obtained jar package as a dependency in Java; use Java code to implement the component abstract class and component metadata abstract class provided in the jar package to complete the required functions; wherein the component abstract class implements the run method of the logic of executing specific steps, and the component metadata abstract class contains the configuration information of the steps; S23. Package the created Java project into a jar package and put it in the plugins directory under the directory deployed in Step 1; S24. Configure the relevant information of the custom component on the component management page, add the component's own information on the left side of the component management page, and add the component's configuration item information on the right side of the component management page; Configuration item information is added using dynamic forms, including: Dynamically set the type of configuration item, the types of configuration items include: text box, text field, drop-down box, cascade selector; Set the dependency condition of the configuration item; in this embodiment, the dependency condition is set to: this configuration item is displayed only when an option is selected in another drop-down box configuration item; The input type of the configuration item; in this embodiment, for the configuration item is a drop-down box, including: configuring the data source as static data, such as configuring "1, male | 0, female | 2, unknown", the drop-down box displays 3 options, namely male, female, and unknown, with values ​​of 1, 0, and 2 respectively; configuring the data source as a sql statement, and the data is the result of the sql query; configuring the data source as an interface address, and the data is the content returned by the interface; Configure the default value of the configuration item. The default value is a static string, number, or SQL directly filled in. The default value of SQL is the result of the SQL query.

[0031] Figure 2 The basic process architecture for developing customized components of this embodiment is shown.

[0032] S25. Use the customized component in the canvas interface of data development.

[0033] Step 3. Start the program created in Step 1, enter the integrated development canvas interface, drag related components from the component pool to the canvas according to the program design, connect the components according to the logic of the program design; configure the components used; The configuration information for configuring the components used is: (1) The mysqlreader component contains the following configuration information: Step name of the input box; whether to use sql for the radio button; database information of the drop-down box; import table of the input box; entity table name of the input box; export column of the multiple-choice box; export sql.

[0034] (2) The mysqlwriter component contains the following configuration information: Step name of the input box; database of the drop-down box; import table of the input box; entity table name of the input box; writing method of the drop-down box; number of concurrent calls of the drop-down box; (3) The replacement character component of the cleaning conversion includes the following configuration information: The step name of the input box; the original character of the input box; the new character of the input box; (4) The simplified and traditional Chinese conversion component of the cleaning conversion includes the following configuration information: Input box step name; conversion format; The left side of the canvas is a directory tree, where multi-level directories are established, and development programs can be added to each directory. Text can be entered above the directory tree for fuzzy matching to search for directories or programs.

[0035] The middle of the canvas is the main body of the canvas. All component configuration links are carried out in the main body of the canvas. There is a thumbnail at the bottom left of the canvas for global thumbnail viewing. The action buttons at the top of the canvas include the following: Undo; Redo; Delete; Version management; Import; Save; Test; On the right side of the canvas is the component pool, which displays all components by category. The search box at the top searches for components through fuzzy matching.

[0036] Step 4. After the program is developed, test the program, execute the current link, and print relevant logs; Click the test button above the canvas to enter the test interface, set the start and end steps, add custom parameters, click to execute the test, and view detailed test log information at the bottom of the test interface. The test log information includes the default output information, the information output when each component is executed, the information in the print message component added by the user, and the error stack information printed after the error is reported. The relevant logs of the test log information can be used to determine whether the program is executed correctly, and problems can be solved according to the relevant logs.

[0037] Step 5. After the program test passes, enter the configuration interface of the program release and configure the scheduling information in the configuration interface of the program release.

[0038] Configuration scheduling information includes: Select the trigger type, including: Time trigger: Generate tasks based on the running cycle and execution time; Event triggering: automatically start the task when the predecessor task and input table in the dependency relationship pass the detection; Select the startup type, including: Sequential start: The next batch of tasks can be executed only after a batch of tasks is completed; Multi-start: can execute multiple batches of tasks at the same time; Unique start: Only one batch of tasks can be executed at the current time; Only one batch of tasks can be executed in a certain cycle (month). The failure to complete the tasks in the previous cycle will not affect the start of the tasks in the next cycle. Configure the operation cycle (year, month, day, hour, minute), execution time, priority, maximum operation time, failure redo interval, failure redo count, and effective date; Configure the prerequisite tasks.

[0039] Figure 1 The basic process architecture of the visualized data integration development of this embodiment is shown.

[0040] The present invention also provides a system for visual data integration development, which executes the visual data integration development method as described above, including: Create a development program module: used to create and deploy different directories corresponding to different categories of business needs; create a program in the corresponding directory and fill in the program information; the program information includes: program name, program Chinese name, program remarks; Design program link module: It is used to develop customized components according to the data development results to be achieved and the functions that can be achieved by existing components. If the existing components do not meet the data development requirements, the components to be used and the overall program link are determined; Program development module: used to start the program created in Step 1, enter the canvas interface of integrated development, drag related components from the component pool to the canvas according to the program design, connect the components according to the logic of the program design; configure the components used; Execution test module: used to test the program after the program development is completed, execute the current link, and print relevant logs; Release and configure the scheduling module: After the program test is passed, enter the configuration interface of the program release and configure the scheduling information in the configuration interface of the program release.

[0041] Further, the design program link module includes a customized component development submodule, and the customized component development submodule includes: Get jar package unit: used to get the jar package where the component abstract class and component metadata abstract class are located; Create a Java project unit: used to create a Java project, add the obtained jar package as a dependency in Java; use Java code to implement the component abstract class and component metadata abstract class provided in the jar package to complete the required functions; wherein the component abstract class implements the run method of the logic of executing specific steps, and the component metadata abstract class contains the configuration information of the steps; Packaging and deployment unit: used to package the created Java project into a jar package and put it in the plugins directory under the directory deployed in Step 1; Configure custom component unit: used to configure the relevant information of custom components on the component management page, add the component's own information on the left side of the component management page, and add the component's configuration item information on the right side of the component management page; Interface usage unit: used to use the customized component in the canvas interface of data development.

[0042] The visual data integration development method and system of this embodiment completes data integration development through a visual interface, completing various data development tasks such as data collection, cleaning, and conversion in one interface, reducing page switching, and more intuitively presenting the implementation logic with a graphical interface; each function is provided in the form of a component, converting the complex coding process into an intuitive graphical operation, reducing the development difficulty; when the data source changes, only the relevant data source configuration needs to be modified without updating the script; when business expansion is required, only certain nodes need to be added, deleted, or modified to expand the current data development function, reducing the data integration development cost and improving the data integration development efficiency.

[0043] An embodiment of the present invention further provides a computer device, Figure 5 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention; see the accompanying drawings Figure 5 As shown, the computer device includes: an input system 23, an output system 24, a memory 22 and a processor 21; the memory 22 is used to store one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the visual data integration development method provided by the above embodiment; wherein the input system 23, the output system 24, the memory 22 and the processor 21 can be connected by a bus or other means, Figure 5 The example of connecting through bus is taken in the following.

[0044] The memory 22 is a readable and writable storage medium of a computing device, which can be used to store software programs, computer executable programs, such as program instructions corresponding to the method for visual data integration development described in the embodiment of the present invention; the memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the device, etc.; in addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device; in some instances, the memory 22 can further include a memory remotely arranged relative to the processor 21, and these remote memories can be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0045] The input system 23 may be used to receive input digital or character information, and to generate key signal input related to user settings and function control of the device; the output system 24 may include display devices such as display screens.

[0046] The processor 21 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 22, that is, realizes the above-mentioned visualized data integration development method.

[0047] The computer device provided above can be used to execute the visual data integration development method provided in the above embodiment, and has corresponding functions and beneficial effects.

[0048] The embodiment of the present invention also provides a storage medium containing computer executable instructions, which are used to execute the visual data integration development method provided by the above embodiment when executed by a computer processor. The storage medium is any of various types of memory devices or storage devices, and the storage medium includes: installation media, such as CD-ROM, floppy disk or tape system; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc.; the storage medium may also include other types of memory or a combination thereof; in addition, the storage medium may be located in a first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet); the second computer system may provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (for example, in different computer systems connected via a network). The storage medium can store program instructions (for example, specifically implemented as a computer program) that can be executed by one or more processors.

[0049] Of course, the storage medium containing computer executable instructions provided by an embodiment of the present invention is not limited to the visual data integration development method described in the above embodiment, and can also execute related operations in the visual data integration development method provided by any embodiment of the present invention.

[0050] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0051] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A visual data integration development method, characterized in that: The following steps are involved: Step 1. Create and deploy different directories according to different categories of business needs; create programs in the corresponding directories and fill in program information; the program information includes: program name, program Chinese name, and program notes; Step 2: Based on the data development results to be achieved and the functions that can be achieved by existing components, if the existing components do not meet the data development requirements, develop customized components to determine the components to be used and the overall program chain; Step 3. Start the program created in Step 1, enter the integrated development canvas interface, drag related components from the component pool to the canvas according to the program design, connect the components according to the logic of the program design; configure the components used; Step 4: After the program is developed, test the program, execute the current link, and print relevant logs; Step 5. After the program test passes, enter the configuration interface of the program release and configure the scheduling information in the configuration interface of the program release.

2. The method for visual data integration development according to claim 1, characterized in that: The method of determining the components to be used and the overall program link in step 2 includes: The existing components used are connected in series according to the business logic, combined with branch and loop components to form a step diagram; the existing components include: Logical components, including the following components: branch, loop, infinite loop, and loop break; General functional components, including the following components: variable assignment, message printing, sleep, terminal command execution, program alarm; Remote call components, including the following components: http request, webservice reception; File operation components, including the following components: file movement, file scanning, file deletion, and file upload; Cleaning and conversion components, including the following components: field splitting, character replacement, character deletion, simplified and traditional conversion, full-width and half-width conversion, encryption and decryption, and simulation; Data standard check component, including the following components: duplicate data check, fluctuation check, value range check, specification check, and null value check; Data security components, including the following components: table security classification, sensitive discovery, sensitive metadata discovery, sensitive field classification, and sensitive data classification; Database operation components, including the following components: SQL statement execution, table creation, and table deletion; Data exchange components include the following components: database read and write components Oracle, MySql, MongoDB, Postgres, Doris, OpenGauss, Hive, Hbase, data format read and write components xml, json, excel, third-party middleware read and write components Kafka, HDFS, Es, and Ftp file read and write components.

3. The method for visual data integration development according to claim 1, characterized in that: In the step 2, the development of the customized component includes the following steps: S21. Obtain the jar package where the component abstract class and the component metadata abstract class are located; S22. Create a Java project and add the obtained jar package as a dependency in Java; use Java code to implement the component abstract class and component metadata abstract class provided in the jar package to complete the required functions; wherein the component abstract class implements the run method of the logic of executing specific steps, and the component metadata abstract class contains the configuration information of the steps; S23. Package the created Java project into a jar package and put it in the plugins directory under the directory deployed in Step 1; S24. Configure the relevant information of the custom component on the component management page, add the component's own information on the left side of the component management page, and add the component's configuration item information on the right side of the component management page; S25. Use the customized component in the canvas interface of data development.

4. The method for visual data integration development according to claim 3, characterized in that: The configuration item information of step S24 is added using a dynamic form, including: Dynamically set the type of configuration item, the types of configuration items include: text box, text field, drop-down box, cascade selector; Set the dependency conditions of the configuration items; Configure the input type of the configuration item; Configure the default value of the configuration item.

5. The visual data integration development method according to claim 1, characterized in that: The method for performing program testing in step 4 includes: Click the test button above the canvas to enter the test interface, set the start and end steps, add custom parameters, click to execute the test, and view detailed test log information at the bottom of the test interface. The test log information includes the default output information, the information output when each component is executed, the information in the print message component added by the user, and the error stack information printed after the error is reported. The relevant logs of the test log information can be used to determine whether the program is executed correctly, and problems can be solved according to the relevant logs.

6. The visual data integration development method according to claim 1, characterized in that: The configuration scheduling information in step 5 includes: Select the trigger type, including: Time trigger: Generate tasks based on the running cycle and execution time; Event triggering: automatically start the task when the predecessor task and input table in the dependency relationship pass the detection; Select the startup type, including: Sequential start: The next batch of tasks can be executed only after a batch of tasks is completed; Multi-start: can execute multiple batches of tasks at the same time; Unique start: Only one batch of tasks can be executed at the current time; Only one batch of tasks can be executed in a cycle. The failure to complete the tasks in the previous cycle will not affect the start of the tasks in the next cycle. Configure the operation cycle, execution time, priority, maximum operation time, failure redo interval, failure redo times, and effective date; Configure the prerequisite tasks.

7. A visual data integration development system, which executes the visual data integration development method according to any one of claims 1 to 6, characterized in that: include: Create a development program module: used to create and deploy different directories according to different categories of business needs; Create a program in the corresponding directory and fill in the program information; The program information includes: program name, program Chinese name, and program notes; Design program link module: It is used to develop customized components according to the data development results to be achieved and the functions that can be achieved by existing components. If the existing components do not meet the data development requirements, the components to be used and the overall program link are determined; Program development module: used to start the program created in Step 1, enter the canvas interface of integrated development, drag related components from the component pool to the canvas according to the program design, connect the components according to the logic of the program design; configure the components used; Execution test module: used to test the program after the program development is completed, execute the current link, and print relevant logs; Release and configure the scheduling module: After the program test is passed, enter the configuration interface of the program release and configure the scheduling information in the configuration interface of the program release.

8. The visual data integration development system according to claim 7 is characterized in that: The design program link module includes a customized component development submodule, and the customized component development submodule includes: Get jar package unit: used to get the jar package where the component abstract class and component metadata abstract class are located; Create a Java project unit: used to create a Java project, add the obtained jar package as a dependency in Java; use Java code to implement the component abstract class and component metadata abstract class provided in the jar package to complete the required functions; wherein the component abstract class implements the run method of the logic of executing specific steps, and the component metadata abstract class contains the configuration information of the steps; Packaging and deployment unit: used to package the created Java project into a jar package and put it in the plugins directory under the directory deployed in Step 1; Configure custom component unit: used to configure the relevant information of custom components on the component management page, add the component's own information on the left side of the component management page, and add the component's configuration item information on the right side of the component management page; Interface usage unit: used to use the customized component in the canvas interface of data development.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for visual data integration development described in any one of claims 1 to 6 are implemented.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method for visual data integration development as described in any one of claims 1 to 6 are implemented.

Images