ETL process arrangement method and device

Abstract

The invention discloses an ETL process arrangement method and device, and the method comprises the steps: obtaining a current ETL task; analyzing to obtain a data processing component and a data flow direction which need to be configured for executing the ETL task; performing parameter configuration of the ETL task according to the data processing component and the data flow direction, and converting the ETL task into a model description language; the parameter configuration comprises configuration of output content of an execution result; analyzing the model description language, and selecting a corresponding execution mode and a scheduling strategy according to an analysis result; the execution mode comprises stream processing and batch processing; executing the selected execution mode and the scheduling strategy to obtain a corresponding execution result; and according to the execution result, outputting a final result of the ETL task according to the configured output content, and storing a current ETL process arrangement strategy. According to the method, the ETL process arrangement efficiency can be improved, and the problem that a traditional ETL process arrangement method is difficult to meet real-time data analysis is solved.

Description

Technical Field

[0001] The present invention relates to an ETL process orchestration method and device, and belongs to the technical field of industrial Internet. Background Art

[0002] There are also some tools for ETL process orchestration on the market. The traditional ETL process includes three main steps: extraction, transformation, and loading. The details are as follows:

[0003] The Extraction step includes:

[0004] First, extract data from different source systems, which may include various databases, file systems, or other data stores;

[0005] You need to determine the method for data extraction, such as whether to actively extract data or wait for the source system to push data;

[0006] Decide whether to perform an incremental extraction (extract only the data that has changed since the last extraction) or a full extraction (extract all the data each time);

[0007] Determine the frequency of data extraction, such as daily or monthly.

[0008] The transformation step includes:

[0009] Convert the extracted data into the specific format required by the target system. This may include cleaning the data, normalizing the data structure, handling missing values, etc.

[0010] Enriching content with other data in the system or transforming data from a business model to an analytical model through a series of transformations;

[0011] ETL tools usually provide a visual operation interface, support the use of built-in code snippets, custom scripts, etc. to complete complex transformation logic, and can monitor the transformation status.

[0012] The Load step includes:

[0013] The transformed data will be loaded into the target database, data warehouse or other system;

[0014] Data can be loaded by directly connecting to the database to ensure efficiency;

[0015] In actual applications, the operation mode of data extraction and loading can be adjusted at any time according to needs, and can be flexibly integrated into other management systems;

[0016] Traditional ETL tools and processes are often designed for batch processing, which means that data processing between databases, files, and data warehouses is usually done in batches. This means that they are not suitable for real-time data processing. In modern business environments that require fast response and analysis, this delay can become a problem.

[0017] Moreover, the traditional process of performing ETL often requires a lot of computing and storage resources, especially when dealing with large data sets, which can lead to increased costs.

[0018] At the same time, traditional ETL architectures usually focus on existing technologies and tools. When new technologies are introduced, applications and tools may need to be rewritten, resulting in insufficient flexibility. With the rise of big data and real-time analysis, traditional ETL tools are difficult to adapt to new data processing requirements, such as stream data processing and advanced data analysis tasks.

[0019] Therefore, traditional ETL process orchestration methods are difficult to meet the needs of modern enterprises for real-time data analysis and have shortcomings in processing speed and resource consumption. Summary of the invention

[0020] The purpose of the present invention is to overcome the deficiencies in the prior art and provide an ETL process orchestration method and device to solve the problem that the traditional ETL process orchestration method is difficult to meet the needs of real-time data analysis.

[0021] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions:

[0022] In a first aspect, the present invention provides an ETL process orchestration method, comprising:

[0023] S1: Get the current ETL task;

[0024] S2: parsing the ETL task of step S1 to obtain the data processing components that need to be configured to execute the ETL task and the corresponding data flow of the data processing components;

[0025] S3: According to the data processing components and data flow of step S2, the parameters of the ETL task are configured, and the ETL task of step S1 is converted into a model description language; the parameter configuration includes configuring the output content of the execution result;

[0026] S4: parsing the model description language of step S3, and selecting a corresponding execution mode and scheduling strategy according to the parsing result; the execution mode includes stream processing and batch processing;

[0027] S5: Execute the execution mode and scheduling strategy of step S4 to obtain the corresponding execution result;

[0028] S6: According to the execution result of step S5, the final result of the ETL task is output according to the output content configured in step S3, and the current ETL process orchestration strategy is saved.

[0029] In the aforementioned ETL process orchestration method, step S1 further includes obtaining a saved ETL process orchestration strategy;

[0030] When there is an ETL process orchestration strategy adapted to the current ETL task in the saved ETL process orchestration strategy, the saved ETL process orchestration strategy is used to execute subsequent steps;

[0031] Alternatively, when there is an ETL process orchestration strategy adapted to the current ETL task in the saved ETL process orchestration strategy, the saved ETL process orchestration strategy is used to perform parameter configuration of step S3, and then subsequent steps are executed.

[0032] In the aforementioned ETL process orchestration method, the data processing component includes a source data processing component and a target data component;

[0033] In step S2, the data flow direction includes the data transmission direction and data transmission strategy between the source data processing component and the target data processing component;

[0034] The data transmission strategy includes switching to a successful transmission process when data transmission is successful and / or switching to a failed transmission process when data transmission fails. The failed transmission process includes transmission retry and / or system error reporting.

[0035] The aforementioned ETL process orchestration method uses a visualization engine to execute step S3, in which the ETL task of step S1 is converted into a model description language by editing parameter configuration, data processing components and data flow.

[0036] In the aforementioned ETL process arrangement method, in step S4, the model description language includes data flow,

[0037] The model description language of the analysis step S3 includes analyzing the data flow direction,

[0038] When there are multiple data flow directions, corresponding execution modes and scheduling strategies are selected according to the data flow directions.

[0039] In the aforementioned ETL process orchestration method, when there are multiple data flows, step S4 further includes analyzing the dependency relationship between target data processing components corresponding to each data flow;

[0040] When there is no dependency between the target data processing components corresponding to the various data flows, subsequent steps are executed in parallel.

[0041] In the aforementioned ETL process orchestration method, in step S6, the output strategy of the output content includes a result voter and a result combination; the result voter returns success or failure according to a voting rule, and the voting rule includes a set ratio and a veto; the result combination is used to output multiple results.

[0042] The aforementioned ETL process orchestration method, step S5 includes setting voting rules of result voters for target data processing components corresponding to each data flow, and outputting corresponding execution results;

[0043] Step S6 includes combining and outputting the final result of the ETL task according to the multiple execution results of step S5.

[0044] In the aforementioned ETL process orchestration method, the execution mode and scheduling strategy of executing step S4 in step S5 includes collecting execution logs; the execution logs are used to record and save system information during the execution process;

[0045] The execution mode and scheduling strategy of executing step S4 in step S5 also includes using a monitor to monitor the execution process.

[0046] In a second aspect, the present invention provides an ETL process arrangement device, including a task acquisition module, a task parsing module, a configuration module, a selection module, an execution module and a result output module;

[0047] The task acquisition module is used to: acquire the current ETL task;

[0048] The task parsing module is used to: parse the ETL task of the task acquisition module to obtain the data processing components that need to be configured to execute the ETL task and the corresponding data flow direction of the data processing components;

[0049] The configuration module is used to: perform parameter configuration of the ETL task according to the data processing components and data flow of the task parsing module, and convert the ETL task of step S1 into a model description language; the parameter configuration includes configuring the output content of the execution result;

[0050] The selection module is used to: parse the model description language of the configuration module, and select a corresponding execution mode and scheduling strategy according to the parsing result; the execution mode includes stream processing and batch processing;

[0051] The execution module is used to: execute the execution mode and scheduling strategy of the selection module to obtain corresponding execution results;

[0052] The result output module is used to: output the final result of the ETL task according to the execution result of the execution module and the output content configured in step S3, and save the current ETL process scheduling strategy.

[0053] Beneficial Effects

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention can select the execution mode of stream processing or batch processing according to the needs of ETL process orchestration. The user defines the output content of the execution result through parameter configuration, thereby improving the efficiency of ETL process orchestration and solving the problem that traditional ETL process orchestration methods are difficult to meet the needs of real-time data analysis.

[0056] The ETL process orchestration method provided by the present invention can call a saved ETL process orchestration strategy or edit it based on a saved ETL process orchestration strategy, thereby shortening the time for users to perform ETL process orchestration, improving the flexibility of ETL process orchestration, and further improving the efficiency of ETL process orchestration.

[0057] The ETL process scheduling method provided by the present invention enables users to select execution methods, scheduling strategies and output content of execution results according to specific ETL task classifications, and further select output strategies for output content according to the output content, thereby improving the accuracy of specific ETL process scheduling.

[0058] The ETL process orchestration method provided by the present invention can add log and monitoring functions, so that users can understand the ETL process orchestration process and execution status during the ETL process orchestration process and execution process, which is convenient for real-time viewing and historical viewing of ETL process orchestration, and improves the efficiency, flexibility and accuracy of ETL process orchestration. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is a flowchart of an ETL process arrangement method according to Embodiment 1 of the present invention;

[0060] Figure 2 This is a flow chart of a detection process in a process processing subtask of Example 1 of the present invention;

[0061] Figure 3 This is a flow chart of an ETL task outputting a final result in Example 1 of the present invention. DETAILED DESCRIPTION

[0062] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0063] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0064] Embodiment 1:

[0065] This embodiment 1 provides an ETL process arrangement method, such as Figure 1 As shown, including:

[0066] S1: Get the current ETL task;

[0067] S2: parsing the ETL task of step S1 to obtain the data processing components that need to be configured to execute the ETL task and the corresponding data flow of the data processing components;

[0068] S3: According to the data processing components and data flow of step S2, the parameters of the ETL task are configured, and the ETL task of step S1 is converted into a model description language; the parameter configuration includes configuring the output content of the execution result;

[0069] S4: parsing the model description language of step S3, and selecting a corresponding execution mode and scheduling strategy according to the parsing result; the execution mode includes stream processing and batch processing;

[0070] S5: Execute the execution mode and scheduling strategy of step S4 to obtain the corresponding execution result;

[0071] S6: According to the execution result of step S5, the final result of the ETL task is output according to the output content configured in step S3, and the current ETL process orchestration strategy is saved.

[0072] The components involved in the ETL process orchestration cover the entire data processing process from the data source to the target system, improving the efficiency and accuracy of data integration and utilization.

[0073] Step S1 also includes obtaining a saved ETL process orchestration strategy;

[0074] When there is an ETL process orchestration strategy adapted to the current ETL task in the saved ETL process orchestration strategy, the saved ETL process orchestration strategy is used to execute subsequent steps;

[0075] Alternatively, when there is an ETL process orchestration strategy adapted to the current ETL task in the saved ETL process orchestration strategy, the saved ETL process orchestration strategy is used to perform parameter configuration of step S3, and then subsequent steps are executed.

[0076] The data processing component includes a source data processing component and a target data component;

[0077] In step S2, the data flow direction includes the data transmission direction and data transmission strategy between the source data processing component and the target data processing component;

[0078] The data transmission strategy includes switching to a successful transmission process when data transmission is successful and / or switching to a failed transmission process when data transmission fails. The failed transmission process includes transmission retry and / or system error. The aforementioned transmission failure is divided into system error and non-compliance with transmission conditions. When a system error occurs, such as a program jump failure, the preset retry procedure is followed and / or the system error is reported to the system; when non-compliance with transmission conditions occurs, such as insufficient materials below to continue to execute the order production subtask, the preset retry procedure is followed to request materials and / or the execution failure is reported to the system.

[0079] The following example illustrates that the ETL process arrangement of the present invention is combined with a specific business, and the MES task distribution operation is completed through the cooperation of various ETL data processing components.

[0080] In industrial application scenarios, the MES system is a production information management system for the execution layer of the workshop of a manufacturing enterprise. The system uses the ETL process orchestration method of the present invention to issue ETL tasks. The specific extracted process parameters are as follows:

[0081]

[0082] As shown above, in step S1, the ETL task of the mes system is obtained;

[0083] In step S2, the ETL task of step S1 is parsed, and the string of request json data is parsed. Taking the above request data as an example, this ETL task includes three different types of subtasks: order production tasks, process processing and equipment shutdown. Therefore, the above json data contains three elements. By parsing the serviceType flag in each element data, it is found that they are located in three different data flows, that is, triggering different target processing components, which corresponds to different downstream devices in the production scenario. In this embodiment, the production machine tool A is responsible for issuing the order production task, the unit control system B is responsible for issuing the process processing, and the machine tool with the equipment code of funac-0930 is responsible for executing its own shutdown command.

[0084] In step S3, a visualization engine is used to execute step S3, and the ETL task of step S1 is converted into a model description language by editing parameter configuration, data processing components and data flow in the visualization engine. The ETL orchestration method allows users to configure parameters by dragging and dropping on the page of the visualization engine. The parameter configuration includes configuring the output content of the execution result; in this embodiment, the output content of the execution result is to output the execution result of the success or failure of each subtask. When the execution result is an execution failure, the execution failure and the reason for the execution failure are returned. The parameter configuration also includes other necessary configurations for the subsequent execution of the ETL task, which can be defined by the user according to the actual application scenario. Common configurations include one or more of the following: basic settings (names), timing configurations, attribute configurations, and description information formulated when the process is running.

[0085] In step S4, the model description language includes data flow, and the model description language of the parsing step S3 includes parsing the data flow. When there are multiple data flows, the corresponding execution mode and scheduling strategy are selected according to the data flow. The scheduling strategy includes a first-in-first-out (FIFO) scheduling strategy, a priority scheduling strategy, a weighted fair queue (WFQ) scheduling strategy, a time-slot polling scheduling strategy, a shortest packet first scheduling strategy, and a weighted shortest remaining queue scheduling strategy. In this embodiment, the three data flows all use the execution mode of stream processing to process data in real time, and the three data outflows all use a first-in-first-out (FIFO) scheduling strategy.

[0086] When there are multiple data flows, step S4 also includes analyzing the dependency relationship between the target data processing components corresponding to each data flow; when there is no dependency relationship between the target data processing components corresponding to each data flow, subsequent steps are executed in parallel.

[0087] The target data processing components corresponding to the three data flows in this embodiment have no dependency relationship and do not interfere with each other. They are independent tasks and the three subtasks execute subsequent steps in parallel.

[0088] In the above steps, the current ETL task is a complex business instruction sent by the user. The current ETL task is divided into three subtasks and distributed into three specific subtasks (order production tasks, process processing and equipment shutdown). Each subtask can also be used as the starting point of a complex business, while supporting parallel and serial execution of subsequent steps until the execution results of each subtask are obtained.

[0089] In step S6, the output strategy of the output content includes a result voter and a result combination;

[0090] The result voter returns execution success or execution failure according to voting rules, wherein the voting rules include setting ratios and one-vote veto; the result combination is used to output multiple results.

[0091] Step S5 includes setting voting rules of result voters for target data processing components corresponding to each data flow and outputting corresponding execution results;

[0092] For the order production task in the subtask of this embodiment, if there is insufficient material during the execution process, the system will send material requests to three different material units in parallel. Since all materials must be available for subsequent production, the present invention adopts a result voting device to output the material request result. The ratio is set to 100% in the voting rule, indicating that the execution result of successful material preparation will be output only when all materials are ready. Otherwise, the execution result of insufficient material and failed material preparation will be directly returned. When the material request fails, the order production task will also fail to execute, and the execution result of the order production task subtask execution failure is obtained. According to the output content of the execution result configured in step S3, the output content of the execution result of the order production task subtask is: execution failed, and the reason for the execution failure is insufficient materials; the order production task subtask process is terminated here.

[0093] The MES system order production task is executed according to the following procedures:

[0094]

[0095] The MES system issues n order production tasks at a time, and the final flow will be the interface of the target data processing component of the same order production task. The order production task adopts a first-in-first-out scheduling strategy, and will automatically assign the machine tool to execute the task according to the interface execution status. When all machine tools are working, the remaining unfinished order production tasks will be made into a queue and placed in the task scheduling system according to the scheduling strategy. The scheduling system will detect the operating status of the machine tool in real time. When a machine tool is idle, the corresponding order production task will be taken out from the queue according to the first-in-first-out scheduling strategy to let the corresponding idle machine tool execute production.

[0096] The process processing in the subtasks of this embodiment, such as Figure 2As shown: In the inspection process of a part processing subtask, the output execution result is qualified or unqualified, there are n inspection items, and the output strategy set by the present invention is a result voter. The ratio is set to 80% in the voting rule, and the weight detection is set as a veto item; the voting rule means that when the comprehensive qualified rate of all inspection items is above 80%, and the weight detection item must be qualified, the part will be defined as a qualified part. Therefore, even if the comprehensive qualified rate of the part is 95%, but the weight detection does not meet the standard, the part will still be deemed unqualified. In addition, the veto can be combined. For example, the gloss detection + weight detection can be defined as a veto item, which means that if any one of the gloss detection and weight detection does not meet the standard, it will be vetoed.

[0097] Step S6 includes combining the multiple execution results of step S5 to output the final result of the ETL task, such as Figure 3 As shown: result combination can obtain the final result by splicing, cutting and combining multiple execution results. After the three subtasks of this embodiment are executed, the three subtasks have three execution results. Since the three tasks are independent of each other and do not affect each other, the present invention uses the result combination method to output the final result, which is convenient for users to understand the execution status of each task.

[0098] Embodiment 2:

[0099] Based on the same inventive concept as the first embodiment, this embodiment introduces an ETL process orchestration method. On the basis of the method in the first embodiment,

[0100] In step S5, the execution mode and scheduling strategy of step S4 include collecting execution logs; the execution logs are used to record and save system information during the execution process;

[0101] And / or, executing the execution mode and scheduling strategy of step S4 in step S5 also includes using a monitor to monitor the execution process.

[0102] Logs include execution logs, process logs, error logs, etc. The following is a specific example of log content:

[0103] Taking a typical ETL task as an example, its log content includes but is not limited to the following aspects:

[0104] Execution log:

[0105] 2024-09-19 09:00:00 INFO ETL task started

[0106] 2024-09-19 09:00:05 INFO The data extraction phase begins, extracting data from the source database XXX

[0107] 2024-09-19 09:05:00 INFO Data extraction completed, a total of 10,000 rows of data extracted

[0108] 2024-09-19 09:05:05 INFO The data conversion phase begins, cleaning and converting data

[0109] ...(may contain logs of multiple transformation steps in between)

[0110] 2024-09-19 09:15:00 INFO Data conversion completed

[0111] 2024-09-19 09:15:05 INFO The data loading phase starts, loading data into the target database YYY

[0112] 2024-09-19 09:20:00 INFO Data loading completed

[0113] Error log:

[0114] 2024-09-19 09:10:15 ERROR An error occurred during data conversion. Error code: ERR001

[0115] Error message: Field type does not match, cannot convert string to date type

[0116] Error module: Conversion logic ZZZ

[0117] Overall log:

[0118] 2024-09-19 09:20:00 INFO ETL task completed

[0119] Execution result: Success

[0120] Start time: 2024-09-19 09:00:00

[0121] End time: 2024-09-19 09:20:00

[0122] Processing speed: 500 lines / minute

[0123] Every link, data and execution log in the above process have been recorded, which provides strong data support for later problem location, production data analysis, report preparation and other aspects.

[0124] Embodiment three:

[0125] Based on the same inventive concept as the first embodiment, this embodiment introduces an ETL process orchestration device, including a task acquisition module, a task parsing module, a configuration module, a selection module, an execution module and a result output module;

[0126] The task acquisition module is used to: obtain the current ETL task;

[0127] The task parsing module is used to: parse the ETL task of the task acquisition module to obtain the data processing components that need to be configured to execute the ETL task and the corresponding data flow of the data processing components;

[0128] The configuration module is used to: perform parameter configuration of the ETL task according to the data processing components and data flow of the task parsing module, and convert the ETL task of step S1 into a model description language; the parameter configuration includes configuring the output content of the execution result;

[0129] The selection module is used to: parse the model description language of the configuration module, and select a corresponding execution mode and scheduling strategy according to the parsing result; the execution mode includes stream processing and batch processing;

[0130] The execution module is used to: execute the execution mode and scheduling strategy of the selection module to obtain corresponding execution results;

[0131] The result output module is used to: output the final result of the ETL task according to the execution result of the execution module and the output content configured in step S3, and save the current ETL process orchestration strategy.

[0132] This embodiment and the first embodiment are based on the same inventive concept and have the same technical effects.

[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0137] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An ETL process orchestration method, characterized in that: include: S1: Get the current ETL task; S2: parsing the ETL task of step S1 to obtain the data processing components that need to be configured to execute the ETL task and the corresponding data flow of the data processing components; S3: According to the data processing components and data flow of step S2, the parameters of the ETL task are configured, and the ETL task of step S1 is converted into a model description language; the parameter configuration includes configuring the output content of the execution result; S4: parsing the model description language of step S3, and selecting a corresponding execution mode and scheduling strategy according to the parsing result; the execution mode includes stream processing and batch processing; S5: Execute the execution mode and scheduling strategy of step S4 to obtain the corresponding execution result; S6: According to the execution result of step S5, the final result of the ETL task is output according to the output content configured in step S3, and the current ETL process orchestration strategy is saved.

2. The ETL process arrangement method according to claim 1, characterized in that: Step S1 also includes obtaining a saved ETL process orchestration strategy; When there is an ETL process orchestration strategy adapted to the current ETL task in the saved ETL process orchestration strategy, the saved ETL process orchestration strategy is used to execute subsequent steps; Alternatively, when there is an ETL process orchestration strategy adapted to the current ETL task in the saved ETL process orchestration strategy, the saved ETL process orchestration strategy is used to perform parameter configuration of step S3, and then subsequent steps are executed.

3. The ETL process arrangement method according to claim 1, characterized in that: The data processing component includes a source data processing component and a target data component; In step S2, the data flow direction includes the data transmission direction and data transmission strategy between the source data processing component and the target data processing component; The data transmission strategy includes switching to a successful transmission process when data transmission is successful and / or switching to a failed transmission process when data transmission fails. The failed transmission process includes transmission retry and / or system error reporting.

4. The ETL process arrangement method according to claim 1, characterized in that: Step S3 is performed using a visualization engine, in which the ETL task of step S1 is converted into a model description language by editing parameter configuration, data processing components and data flow.

5. The ETL process arrangement method according to claim 3, characterized in that: In step S4, the model description language includes data flow, The model description language of the analysis step S3 includes analyzing the data flow direction, When there are multiple data flow directions, corresponding execution modes and scheduling strategies are selected according to the data flow directions.

6. The ETL process arrangement method according to claim 5, characterized in that: When there are multiple data flow directions, step S4 further includes analyzing the dependency relationship between the target data processing components corresponding to each data flow direction; When there is no dependency between the target data processing components corresponding to the various data flows, subsequent steps are executed in parallel.

7. The ETL process arrangement method according to claim 6, characterized in that: In step S6, the output strategy of the output content includes a result voter and a result combination; The result voter returns success or failure according to voting rules, wherein the voting rules include setting ratios and vetoes; The result combination is used to output a plurality of results.

8. The ETL process arrangement method according to claim 7, characterized in that: Step S5 includes setting voting rules of result voters for target data processing components corresponding to each data flow and outputting corresponding execution results; Step S6 includes combining the results according to the multiple execution results of step S5 to output the final result of the ETL task.

9. The ETL process arrangement method according to claim 1, characterized in that: In step S5, the execution mode and scheduling strategy of step S4 include collecting execution logs; the execution logs are used to record and save system information during the execution process; And / or, executing the execution mode and scheduling strategy of step S4 in step S5 also includes using a monitor to monitor the execution process.

10. An ETL process arrangement device, characterized in that: It includes task acquisition module, task parsing module, configuration module, selection module, execution module and result output module; The task acquisition module is used to: acquire the current ETL task; The task parsing module is used to: parse the ETL task of the task acquisition module to obtain the data processing components that need to be configured to execute the ETL task and the corresponding data flow of the data processing components; The configuration module is used to: perform parameter configuration of the ETL task according to the data processing components and data flow of the task parsing module, and convert the ETL task of step S1 into a model description language; the parameter configuration includes configuring the output content of the execution result; The selection module is used to: parse the model description language of the configuration module, and select a corresponding execution mode and scheduling strategy according to the parsing result; the execution mode includes stream processing and batch processing; The execution module is used to: execute the execution mode and scheduling strategy of the selection module to obtain corresponding execution results; The result output module is used to: output the final result of the ETL task according to the execution result of the execution module and the output content configured in step S3, and save the current ETL process scheduling strategy.

Images