Kettle-based web data processing method, device, system and electronic device

Abstract

The present application provides a web data processing method, device, system and electronic device based on Kettle, which builds an intermediate layer system and Kettle application in the server, so that when the user terminal issues a processing request based on the operation of the web browser, the intermediate layer system is started, and the data processing task and the configuration scheduling rules are obtained. The communication protocol between the foreground and background of the Kettle application is simulated by the intermediate layer system, so that the interface of the application is called to access the function of the application, so as to perform corresponding data processing based on the data processing task and the configuration scheduling rules in the intermediate layer system to obtain the processing result. In this solution, it is only necessary to build an intermediate layer system and Kettle application in the server, and use the intermediate layer system to access the Kettle application function. The user terminal only needs to access the intermediate layer system based on the web browser operation to use all the functions of Kettle, saving the performance resources of the terminal.

Description

Technical Field

[0001] The present application relates to the field of computer software technology, and in particular to a kettle-based web data processing method, device, system, and electronic device. Background Art

[0002] Kettle is desktop software and an open-source ETL tool that builds data pipelines by dragging components, connecting them, and configuring them. It can connect to data sources including traditional databases, files, big data platforms, interfaces, and streaming data. As an end-to-end data integration platform, Kettle can extract, load, clean, transform, and blend data from a variety of data sources.

[0003] However, the current application of Kettle is to download, install and use it on the terminal device. The installation package of Kettle itself is large, and the configuration requirements for the terminal device are relatively high. In addition, the software interface and interaction logic are not satisfactory, the learning cost is high, and the window response speed is slow, which brings many inconveniences to users. Summary of the Invention

[0004] The purpose of this application includes, for example, providing a kettle-based web data processing method, device, system and electronic device, which can save terminal performance resources by building an intermediate layer system and kettle application in the server.

[0005] The embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides a Kettle-based web data processing method, which is applied to a server that communicates with multiple user terminals and includes an intermediate layer system and a Kettle application. The method includes:

[0007] Upon obtaining a processing request from any user terminal to access the intermediate layer system, starting the intermediate layer system, wherein the processing request is issued based on an operation of a web browser of the user terminal;

[0008] Obtaining data processing tasks and configuring scheduling rules based on the processing request;

[0009] The communication protocol between the foreground and background of the kettle application is simulated by the middle layer system to call the interface of the kettle application;

[0010] Based on the interface of the called kettle application, the function of the kettle application is accessed to perform corresponding data processing in the middle layer system based on the data processing task and the configuration scheduling rules to obtain the processing result.

[0011] In an optional embodiment, the step of starting the intermediate layer system upon obtaining a processing request from any user terminal to access the intermediate layer system includes:

[0012] When receiving a processing request from any user terminal to access the intermediate layer system, obtaining a user account corresponding to the user terminal and determining user rights based on the user account;

[0013] After the middle layer system is started, the corresponding data resources are fed back to the user terminal based on the user authority, so that the user terminal displays the data resources in the web browser.

[0014] In an optional embodiment, the step of obtaining a data processing task based on the processing request and configuring a scheduling rule includes:

[0015] Obtaining a job and an executable task in the data processing task based on the processing request, wherein the job includes at least one executable task, and the executable task is obtained by operating a component based on a web browser of the user terminal;

[0016] The trigger information of the job and the trigger information of the at least one executable task in the configuration scheduling rule are obtained.

[0017] In an optional embodiment, the step of obtaining jobs and executable tasks in the data processing task based on the processing request includes:

[0018] Obtaining a job created in the data processing task and an executable task created under the job based on the processing request;

[0019] Obtaining the component corresponding to the executable task, and obtaining the source database, source data table, target database, and target data table of the executable task;

[0020] A query statement for the source data table and the target data table is obtained to determine matching fields of the source data table and the target data table.

[0021] In an optional implementation manner, the query statement is a query statement for a single field of the source data table and the target data table, or the query statement is a query statement for multiple fields of the source data table and the target data table.

[0022] In an optional embodiment, the step of performing corresponding data processing in the intermediate layer system based on the data processing task and the configured scheduling rule includes:

[0023] In the middle-layer system, if the configuration scheduling rule indicates that the trigger information of each executable task is consistent and consistent with the trigger information of the job, then data processing of each executable task under the job is synchronously triggered based on the trigger information of the job;

[0024] If the configuration scheduling rule indicates that the trigger information of each executable task is inconsistent, then the data processing of each executable task under the job is triggered in sequence according to the trigger information of each executable task.

[0025] In an optional embodiment, the method further comprises:

[0026] Analyzing the processing results to obtain analysis results, wherein the analysis results include the number of successfully processed items, the number of failed processed items, and the data processing speed;

[0027] The short message platform is accessed through the middle layer system to send the analysis result in the form of a short message through the short message platform.

[0028] In a second aspect, the present application provides a Kettle-based web data processing device, which is applied to a server that communicates with multiple user terminals. The server includes an intermediate layer system and a Kettle application. The device includes:

[0029] a startup module, configured to start the intermediate layer system upon obtaining a processing request from any user terminal to access the intermediate layer system, wherein the processing request is issued based on an operation of a web browser of the user terminal;

[0030] An acquisition module, configured to obtain data processing tasks based on the processing request and configure scheduling rules;

[0031] A calling module, configured to simulate the communication protocol between the foreground and backend of the kettle application through the middle-layer system to call the interface of the kettle application;

[0032] The processing module is used to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and the configuration scheduling rules to obtain the processing result.

[0033] In a third aspect, the present application provides an electronic device comprising one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processor. When the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of the aforementioned embodiments.

[0034] In a fourth aspect, the present application provides a web data processing system based on Kettle, the system comprising a server and a plurality of user terminals communicating with the server, the server comprising an intermediate layer system and a Kettle application;

[0035] The user terminal is configured to obtain a processing request for accessing the intermediate layer system based on an operation on a web browser, and send the processing request to the server;

[0036] The server is configured to, upon receiving a processing request from any user terminal to access the intermediate layer system, start the intermediate layer system, obtain data processing tasks based on the processing request, and configure scheduling rules;

[0037] The server is further configured to simulate the communication protocol between the foreground and background of the kettle application through the middle layer system to call the interface of the kettle application;

[0038] The server is also used to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and configuration scheduling rules to obtain processing results.

[0039] The beneficial effects of the embodiments of the present application include, for example:

[0040] The present application provides a web data processing method, device, system and electronic device based on Kettle, which builds an intermediate layer system and Kettle application in the server, so that when the user terminal issues a processing request based on the operation of the web browser, the intermediate layer system is started, and the data processing task and the configuration scheduling rules are obtained based on the processing request. The server simulates the communication protocol between the foreground and background of the Kettle application through the intermediate layer system, thereby calling the interface of the Kettle application to access the function of the Kettle application, so as to perform corresponding data processing based on the data processing task and the configuration scheduling rules in the intermediate layer system to obtain the processing result. In this solution, it is only necessary to build the intermediate layer system and Kettle application in the server, and use the intermediate layer system to access the Kettle application function. The user terminal only needs to access the intermediate layer system based on the web browser operation to use all the functions of Kettle, saving the performance resources of the terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic diagram of an application scenario of a web data processing method based on Kettle provided in an embodiment of the present application;

[0043] Figure 2 A flowchart of a web data processing method based on Kettle provided in an embodiment of the present application;

[0044] Figure 3 for Figure 2 Flowchart of the sub-steps included in step S101;

[0045] Figure 4 for Figure 2 Flowchart of the sub-steps included in step S102;

[0046] Figure 5 for Figure 4 Flowchart of the sub-steps included in step S1021;

[0047] Figure 6 A flowchart of the analysis method in the web data processing method based on Kettle provided in an embodiment of the present application;

[0048] Figure 7 A structural block diagram of an electronic device provided in an embodiment of the present application;

[0049] Figure 8 This is a functional module block diagram of a Kettle-based web data processing device provided in an embodiment of the present application.

[0050] Icons: 110 - storage medium; 120 - processor; 130 - web data processing device based on Kettle; 131 - startup module; 132 - acquisition module; 133 - calling module; 134 - processing module; 140 - communication interface. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0054] It should be noted that, in the absence of conflict, the features in the embodiments of this application can be combined with each other.

[0055] See also Figure 1 , which is a schematic diagram of an application scenario of the Kettle-based web data processing method provided in an embodiment of the present application. The application scenario includes a Kettle-based web data processing system, which includes a server and multiple user terminals communicating with the server.

[0056] In the present embodiment, the server can be a separate server or a server cluster consisting of multiple servers. The server includes a middle-layer system and a kettel application. The user terminal can be a terminal device used by the user side, such as a personal computer, a tablet computer, or the like. The middle-layer system in the server can be accessed through a web browser on the user terminal, and then the kettel application is accessed to utilize the various functions of the kettel application.

[0057] See also Figure 2 , is a flowchart of a web data processing method based on Kettle provided in an embodiment of the present application. The method steps defined in the process related to the web data processing method based on Kettle can be implemented by the above server. Figure 2 The specific process is described in detail.

[0058] S101 , when a processing request for any user terminal to access the intermediate layer system is obtained, the intermediate layer system is started, wherein the processing request is issued based on an operation of a web browser of the user terminal.

[0059] S102: Obtain data processing tasks and configure scheduling rules based on the processing request.

[0060] S103, simulating the communication protocol between the foreground and backend of the kettle application through the middle layer system to call the interface of the kettle application.

[0061] S104: Access the function of the kettle application based on the called interface of the kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and the configuration scheduling rules to obtain a processing result.

[0062] As the times change, the importance of data has become increasingly prominent. However, data often comes from various business systems, is highly fragmented, and lacks unified data standards, making it difficult to unlock its value. To fully realize the value of data, it is necessary to unify data standards and aggregate data scattered across various business systems.

[0063] Kettle is a commonly used desktop application for data processing. In the prior art, users need to download and install Kettle on their terminals to utilize its related functions. Kettle's large installation package and high terminal configuration requirements place high demands on terminal performance and result in poor processing efficiency.

[0064] In this embodiment, the middle layer system and the Kettle application are configured in the server. When the user needs to use the Kettle application related functions, he can log in through the web browser on the user terminal, thereby sending a processing request to the server to access the middle layer system.

[0065] When the server receives a processing request from the user terminal, it will start the middle layer system so that the user terminal can access the middle layer system. After the user terminal accesses the middle layer system, it can synchronously implement the relevant processing process of the middle layer system on the web page of the web browser.

[0066] In this embodiment, on the server side, the middle layer system can simulate the communication protocol between the foreground and backstage of the Kettle application after startup. Wherein, the foreground of the Kettle application can be understood as the front-end page that presents the data processing process, and the backstage of the Kettle application can be understood as the backend that actually executes instructions and performs data processing.

[0067] The middle-tier system simulates the communication protocol between the front-end and back-end of the Kettle application, thereby being able to call the Kettle application's interface. Through the Kettle application's interface, the Kettle application's functions can be accessed. In other words, the middle-tier system can have the relevant functions of the Kettle application.

[0068] In addition, the processing request initiated by the user terminal includes a data processing task and a configuration scheduling rule, wherein the data processing task can be used to indicate what kind of data processing to perform, and the configuration scheduling rule can be used to indicate how to perform the data processing process.

[0069] Based on this, on the server side, corresponding data processing can be performed in the middle-layer system according to the data processing tasks and configuration scheduling rules to obtain processing results.

[0070] The web data processing method based on Kettle provided in this embodiment constructs an intermediate layer system and Kettle application in the server, and uses the intermediate layer system to access the Kettle application functions. The user terminal only needs to access the intermediate layer system based on the web browser operation to use all the functions of the Kettle application, thereby saving the performance resources of the terminal.

[0071] There is basically no concept of roles and permissions in existing kettel applications, which leads to information insecurity issues for some important data resources and is not conducive to the isolation of functions and data.

[0072] Based on this, see Figure 3 In this embodiment, the step of obtaining the processing request of the user terminal and starting the intermediate layer system in the above step S101 may include the following sub-steps:

[0073] S1011 , when a processing request for any user terminal to access the intermediate layer system is obtained, a user account corresponding to the user terminal is obtained, and user authority is determined according to the user account.

[0074] S1012: After starting the middle layer system, feeding back corresponding data resources to the user terminal based on the user authority, so that the user terminal displays the data resources in the web browser.

[0075] In this embodiment, on the user terminal side, the user can log in on the web browser using the user account and then access the middle-layer system. After accessing the middle-layer system, the web browser of the user terminal can display the relevant interface of data information, data configuration information, etc. This relevant interface can also be used by the user to select and configure data, and then the relevant configuration results are sent to the server for processing.

[0076] Different user accounts may correspond to different user roles, and different user roles may have different user permissions. Different user permissions may include but are not limited to viewing, modifying, configuring, deleting, and other processing permissions for different data resources.

[0077] Therefore, after receiving the processing request initiated by the user terminal, the user account can be verified to be legitimate, for example, whether it is a legally registered user, etc. If the user account is a legitimate account, the user rights corresponding to the user account can be determined, and then the data resources that the user account has processing rights for can be learned.

[0078] After the server starts the middle-tier system, it can provide the user terminal with data resources that it has processing permissions for based on the corresponding user permissions of the user account, so that the data resources can be displayed in the user terminal's web browser. This allows the user to perform operations such as configuration on the data resources through the relevant interface of the user terminal's web browser.

[0079] In this embodiment, different roles are assigned to different user accounts, and different user permissions are set up according to business needs to isolate functions and data permissions. Through permission grading, different roles can work together, improve work efficiency, and protect data access rights.

[0080] On the user terminal side, the user can select and configure the data resources with permissions, and then generate the above-mentioned processing request and send it to the server. After receiving the processing request, the server can obtain data processing tasks and configure scheduling rules based on the processing request. Figure 4 , this step may specifically include the following sub-steps:

[0081] S1021, obtaining a job and an executable task in the data processing task based on the processing request, wherein the job includes at least one executable task, and the executable task is obtained by operating a component based on the web browser of the user terminal.

[0082] S1022: Obtain trigger information of the job and trigger information of the at least one executable task in the configuration scheduling rule.

[0083] In this embodiment, data processing tasks are divided into jobs and tasks. A job is a collection of executable tasks, which can be divided into different business areas or business modules and have business significance. An executable task is the smallest execution unit.

[0084] You can create executable tasks under jobs. These tasks are based on components configured in a web browser. These components have the same functionality as components provided by the Kettle app. You can create an executable task by dragging and dropping, for example.

[0085] After configuring jobs and executable tasks, you can configure scheduling rules for the jobs and executable tasks, including job trigger information and each executable task's trigger information. This is used to subsequently trigger the jobs and executable tasks on a scheduled basis. The configured scheduling rules for the executable tasks have a higher priority than the job's. That is, during execution, the configured scheduling rules for the executable tasks take precedence.

[0086] In this embodiment, jobs are divided and configured based on business domains. One or more executable tasks are built under each job, and scheduling rules are configured for the jobs and executable tasks. This allows for support for job execution in different business domains and allows for the smallest execution unit to be broken down within the job.

[0087] In this embodiment, in the process of unified data management, it is usually necessary to migrate data to a specified database or data table. Therefore, the constructed jobs and executable tasks often involve processing data in the database or data table. For details, please refer to Figure 5 In this embodiment, the step of obtaining the jobs and executable tasks in the data processing task may include the following sub-steps:

[0088] S10211: Obtain a job created in the data processing task and an executable task created under the job based on the processing request.

[0089] S10212: Obtain the component corresponding to the executable task, and obtain the source database, source data table, target database, and target data table of the executable task.

[0090] S10213: Obtain query statements for the source data table and the target data table to determine matching fields between the source data table and the target data table.

[0091] In this embodiment, the data processing request initiated by the user terminal includes a newly created job and an executable task constructed under the newly created job. For example, a required component is selected in a related page of a web browser to form an executable task.

[0092] When building an executable task, for example, you can drag a table input component, select a source database and source data table, and generate a query statement, such as an SQL statement. Then, drag a table output component and select a target database and target data table. Based on the generated query statement, the fields of the source and target data tables are matched, and the executable task is generated and saved.

[0093] When the server receives the processing request, it can restore the job in the processing request and the specific information of the executable task in the above manner.

[0094] In this embodiment, through the above-mentioned management of the database and data tables, and the construction of corresponding jobs and executable tasks, data migration of corresponding fields can be smoothly achieved when data migration is subsequently achieved by executing tasks.

[0095] In this embodiment, the query statement generated above may be a query statement for a single field of the source data table and the target data table, or a joint query statement for multiple fields of the source data table and the target data table.

[0096] In this way, single-field data migration between data tables or multi-field linkage data migration between data tables can be achieved.

[0097] In this embodiment, if a server in the middle-tier system needs to perform corresponding data processing, it must be based not only on the aforementioned data processing tasks but also on the relevant configuration scheduling rules. As can be seen from the above, the configuration scheduling rules include the scheduling rules for jobs and the scheduling rules for each executable task. The scheduling rules for jobs and the scheduling rules for executable tasks may or may not be consistent. Specifically, data processing can be performed based on the configuration scheduling rules in the following manner.

[0098] In the middle-layer system, if the configured scheduling rules indicate that the trigger information of each executable task is consistent and consistent with the trigger information of the job, the data processing of each executable task under the job is synchronously triggered based on the trigger information of the job.

[0099] In this embodiment, when multiple executable tasks are constructed under a single job, the trigger information of each executable task is consistent and consistent with the trigger information of the job. To avoid triggering each task based on the trigger information of each executable task one by one, the executable tasks can be executed concurrently based on the trigger information of the job. This can reduce processing complexity while ensuring execution accuracy.

[0100] In addition, in the middle-layer system, if the configured scheduling rules indicate that the trigger information of each executable task is inconsistent, the data processing of each executable task under the job is triggered in sequence according to the trigger information of each executable task.

[0101] As can be seen above, the priority of executable tasks is higher than that of jobs. If the trigger information of each executable task is inconsistent, it is impossible for all of them to be consistent with the trigger information of the job. Therefore, in this case, it is necessary to trigger the execution data processing based on the trigger information of each executable task to ensure the accuracy of the task execution trigger.

[0102] In this embodiment, when the middle-tier system executes data processing based on the above information, if an error occurs during data processing, the scheduling status of the job and executable task will be automatically set to disabled. In this case, you can view the task execution log and choose to continue execution after the error is resolved.

[0103] In this embodiment, after executing the data processing task, the processing results can also be analyzed. Figure 6 Specifically, the following steps may also be included:

[0104] S105: Analyze the processing result to obtain an analysis result, which includes the number of successfully processed items, the number of failed processed items, and the data processing speed.

[0105] S106, accessing the SMS platform through the middle layer system to send the analysis result in the form of SMS via the SMS platform.

[0106] In this embodiment, an execution log may be generated during and after the data processing is executed. Based on the execution log, indicators such as the number of data successfully acquired, the number of data failed to be acquired, and the execution speed may be obtained during the data processing.

[0107] In addition, in this embodiment, the summary module can also be used to count the executed data from multiple dimensions, such as the number of data sources, the number of successfully executed data, the number of failed execution data, etc.

[0108] In addition to viewing the execution log, you can also download the execution log and access the SMS platform through the middle-layer system to send the above analysis results in the downloaded execution log to the user's terminal device via SMS.

[0109] The Kettle-based web data processing method provided in this embodiment configures a middle-tier system and a Kettle application on a server, enabling user terminals to access the middle-tier system. The middle-tier system then simulates the Kettle application's foreground and background communication protocols, invoking the Kettle application's interfaces and thereby accessing the Kettle application's functionality. This allows users to access the middle-tier system simply by logging into a web browser on their user terminal to configure data sources, jobs, and executable tasks within the Kettle application. Users can also view the execution status of executable tasks and error messages, enabling them to monitor process execution anytime, anywhere.

[0110] Furthermore, in this embodiment, the concept of authority management is introduced, and different roles can be assigned to different users to correspond to different user authorities, thereby achieving functional isolation and execution data isolation.

[0111] In addition, it has complete scheduling rules and can configure scheduling rules for jobs and executable tasks, realize the division of business areas, and the division of executable units under business areas.

[0112] In addition, the execution log can be analyzed from multiple dimensions and the execution status can be counted from multiple dimensions.

[0113] The processing method provided in this embodiment has a good experience in terms of interface usability, task scheduling, and statistical analysis, and can control each link of data processing more freely and conveniently.

[0114] From the above, it can be seen that an embodiment of the present application also provides a web data processing system based on kettel, which includes a server and multiple user terminals. Specifically, the user terminal is used to obtain a processing request for accessing the intermediate layer system based on the operation of the web browser, and send the processing request to the server.

[0115] The server is configured to start the intermediate layer system upon receiving a processing request from any user terminal to access the intermediate layer system, obtain a data processing task based on the processing request, and configure a scheduling rule.

[0116] The server is further used to simulate the communication protocol between the foreground and background of the kettle application through the middle layer system to call the interface of the kettle application.

[0117] The server is also used to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and configuration scheduling rules to obtain processing results.

[0118] The web data processing system based on Kettle provided in this embodiment builds an intermediate layer system and a Kettle application in a server, and uses the intermediate layer system to access the Kettle application function. The user terminal only needs to access the intermediate layer system based on a web browser to use all the functions of Kettle, thus saving the performance resources of the terminal.

[0119] For descriptions of the processing flow of each device in the system and the interaction flow between the devices, reference can be made to the relevant descriptions in the above method embodiments, which will not be described in detail here.

[0120] See also Figure 7 , is an exemplary component diagram of an electronic device provided in an embodiment of the present application, and the electronic device may be the above-mentioned server. The electronic device may include a storage medium 110, a processor 120, a kettel-based web data processing device 130 and a communication interface 140. In the present embodiment, the storage medium 110 and the processor 120 are both located in the electronic device and the two are separately arranged. However, it should be understood that the storage medium 110 may also be independent of the electronic device and may be accessed by the processor 120 through a bus interface. Alternatively, the storage medium 110 may also be integrated into the processor 120, for example, it may be a cache and / or a general register.

[0121] The Kettle-based web data processing device 130 can be understood as the above-mentioned electronic device, or the processor 120 of the electronic device, or can be understood as a software function module independent of the above-mentioned electronic device or processor 120 that implements the above-mentioned Kettle-based web data processing method under the control of the electronic device.

[0122] like Figure 8 As shown, the web data processing device 130 based on Kettle can include a startup module 131, an acquisition module 132, a call module 133 and a processing module 134. The functions of each functional module of the web data processing device 130 based on Kettle are described in detail below.

[0123] The starting module 131 is configured to start the middle-layer system upon receiving a processing request from any user terminal to access the middle-layer system, wherein the processing request is issued based on an operation of a web browser of the user terminal;

[0124] It can be understood that the starting module 131 can be used to execute the above step S101. For the detailed implementation of the starting module 131, reference can be made to the above content related to step S101.

[0125] An acquisition module 132, configured to acquire a data processing task based on the processing request and configure a scheduling rule;

[0126] It can be understood that the obtaining module 132 can be used to execute the above step S102. For the detailed implementation of the obtaining module 132, reference can be made to the above content related to step S102.

[0127] A calling module 133 is used to simulate the communication protocol between the foreground and backend of the kettle application through the middle layer system to call the interface of the kettle application;

[0128] It can be understood that the calling module 133 can be used to execute the above step S103. For the detailed implementation of the calling module 133, reference can be made to the above content related to step S103.

[0129] The processing module 134 is used to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and the configuration scheduling rules to obtain the processing result.

[0130] It can be understood that the processing module 134 can be used to execute the above step S104. For the detailed implementation of the processing module 134, reference can be made to the above content related to step S104.

[0131] In a possible implementation, the startup module 131 may be used to:

[0132] When receiving a processing request from any user terminal to access the intermediate layer system, obtaining a user account corresponding to the user terminal and determining user rights based on the user account;

[0133] After the middle layer system is started, the corresponding data resources are fed back to the user terminal based on the user authority, so that the user terminal displays the data resources in the web browser.

[0134] In a possible implementation, the obtaining module 132 may be used to:

[0135] Obtaining a job and an executable task in the data processing task based on the processing request, wherein the job includes at least one executable task, and the executable task is obtained by operating a component based on a web browser of the user terminal;

[0136] The trigger information of the job and the trigger information of the at least one executable task in the configuration scheduling rule are obtained.

[0137] In a possible implementation, the obtaining module 132 may be used to:

[0138] Obtaining a job created in the data processing task and an executable task created under the job based on the processing request;

[0139] Obtaining the component corresponding to the executable task, and obtaining the source database, source data table, target database, and target data table of the executable task;

[0140] A query statement for the source data table and the target data table is obtained to determine matching fields of the source data table and the target data table.

[0141] In a possible implementation, the query statement is a query statement for a single field of the source data table and the target data table, or the query statement is a joint query statement for multiple fields of the source data table and the target data table.

[0142] In a possible implementation, the processing module 134 may be used to:

[0143] In the middle-layer system, if the configuration scheduling rule indicates that the trigger information of each executable task is consistent and consistent with the trigger information of the job, then data processing of each executable task under the job is synchronously triggered based on the trigger information of the job;

[0144] If the configuration scheduling rule indicates that the trigger information of each executable task is inconsistent, then the data processing of each executable task under the job is triggered in sequence according to the trigger information of each executable task.

[0145] In a possible implementation, the kettel-based web data processing device 130 further includes an analysis module, which can be used to:

[0146] Analyzing the processing results to obtain analysis results, wherein the analysis results include the number of successfully processed items, the number of failed processed items, and the data processing speed;

[0147] The short message platform is accessed through the middle layer system to send the analysis result in the form of a short message through the short message platform.

[0148] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0149] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are executed, the web data processing method based on Kettle provided in the above embodiment is implemented.

[0150] Specifically, the computer-readable storage medium can be a general-purpose storage medium, such as a removable disk or hard disk. When the computer program on the computer-readable storage medium is executed, the above-mentioned Kettle-based web data processing method can be executed. Regarding the processes involved when the computer-readable storage medium and its executable instructions are executed, reference can be made to the relevant descriptions in the above-mentioned method embodiments and will not be described in detail here.

[0151] In summary, the kettle-based web data processing method, device, system and electronic device provided in the embodiments of the present application, by building an intermediate layer system and kettle application in the server, so that when the user terminal issues a processing request based on the operation of the web browser, the intermediate layer system is started, and the data processing task and the configuration scheduling rules are obtained based on the processing request. The server simulates the communication protocol between the foreground and background of the kettle application through the intermediate layer system, thereby calling the interface of the kettle application to access the function of the kettle application, so as to perform corresponding data processing based on the data processing task and the configuration scheduling rules in the intermediate layer system to obtain the processing result. In this solution, it is only necessary to build the intermediate layer system and kettle application in the server, and use the intermediate layer system to access the kettle application function. The user terminal only needs to access the intermediate layer system based on the web browser operation to use all the functions of Kettle, saving the performance resources of the terminal.

[0152] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A web data processing method based on kettle, characterized in that: Applied to a server that communicates with multiple user terminals, the server including an intermediate layer system and a kettle application, the method includes: Upon obtaining a processing request from any user terminal to access the intermediate layer system, starting the intermediate layer system, wherein the processing request is issued based on an operation of a web browser of the user terminal; Obtaining data processing tasks and configuring scheduling rules based on the processing request; The communication protocol between the foreground and background of the kettle application is simulated by the middle layer system to call the interface of the kettle application; Based on the interface of the called kettle application, the function of the kettle application is accessed to perform corresponding data processing in the middle layer system based on the data processing task and the configuration scheduling rules to obtain the processing result.

2. The kettle-based web data processing method according to claim 1, characterized in that: The step of starting the intermediate layer system when a processing request for accessing the intermediate layer system by any user terminal is obtained includes: When receiving a processing request from any user terminal to access the intermediate layer system, obtaining a user account corresponding to the user terminal and determining user rights based on the user account; After the middle layer system is started, the corresponding data resources are fed back to the user terminal based on the user authority, so that the user terminal displays the data resources in the web browser.

3. The kettle-based web data processing method according to claim 1, characterized in that: The step of obtaining a data processing task based on the processing request and configuring a scheduling rule includes: Obtaining a job and an executable task in the data processing task based on the processing request, wherein the job includes at least one executable task, and the executable task is obtained by operating a component based on a web browser of the user terminal; The trigger information of the job and the trigger information of the at least one executable task in the configuration scheduling rule are obtained.

4. The kettle-based web data processing method according to claim 3, characterized in that: The step of obtaining jobs and executable tasks in the data processing task based on the processing request includes: Obtaining a job created in the data processing task and an executable task created under the job based on the processing request; Obtaining the component corresponding to the executable task, and obtaining the source database, source data table, target database, and target data table of the executable task; A query statement for the source data table and the target data table is obtained to determine matching fields of the source data table and the target data table.

5. The kettle-based web data processing method according to claim 4, characterized in that: The query statement is a query statement for a single field of the source data table and the target data table, or the query statement is a query statement for multiple fields of the source data table and the target data table.

6. The kettle-based web data processing method according to claim 3, characterized in that: The step of performing corresponding data processing in the middle-layer system based on the data processing task and the configuration scheduling rule includes: In the middle-layer system, if the configuration scheduling rule indicates that the trigger information of each executable task is consistent and consistent with the trigger information of the job, then data processing of each executable task under the job is synchronously triggered based on the trigger information of the job; If the configuration scheduling rule indicates that the trigger information of each executable task is inconsistent, then the data processing of each executable task under the job is triggered in sequence according to the trigger information of each executable task.

7. The kettle-based web data processing method according to claim 1, characterized in that: The method further comprises: Analyzing the processing results to obtain analysis results, wherein the analysis results include the number of successfully processed items, the number of failed processed items, and the data processing speed; The short message platform is accessed through the middle layer system to send the analysis result in the form of a short message through the short message platform.

8. A web data processing device based on kettle, characterized in that: Applied to a server that communicates with multiple user terminals, the server including an intermediate layer system and a kettle application, the device includes: a startup module, configured to start the intermediate layer system upon obtaining a processing request from any user terminal to access the intermediate layer system, wherein the processing request is issued based on an operation of a web browser of the user terminal; An acquisition module, configured to obtain data processing tasks based on the processing request and configure scheduling rules; A calling module, configured to simulate the communication protocol between the foreground and backend of the kettle application through the middle-layer system to call the interface of the kettle application; The processing module is used to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and the configuration scheduling rules to obtain the processing result.

9. An electronic device, characterized in that: The electronic device comprises one or more storage media and one or more processors communicating with the storage media, wherein the one or more storage media store machine-executable instructions executable by the processors. When the electronic device is running, the processor executes the machine-executable instructions to perform the method steps described in any one of claims 1 to 7.

10. A web data processing system based on kettle, characterized in that: The system includes a server and a plurality of user terminals communicating with the server, wherein the server includes an intermediate layer system and a kettle application; The user terminal is configured to obtain a processing request for accessing the intermediate layer system based on an operation on a web browser, and send the processing request to the server; The server is configured to, upon receiving a processing request from any user terminal to access the intermediate layer system, start the intermediate layer system, obtain data processing tasks based on the processing request, and configure scheduling rules; The server is further configured to simulate the communication protocol between the foreground and background of the kettle application through the middle layer system to call the interface of the kettle application; The server is also used to access the function of the kettle application based on the interface of the called kettle application, so as to perform corresponding data processing in the middle layer system based on the data processing task and configuration scheduling rules to obtain processing results.

Images