ETL operator analytic conversion method and application thereof

Abstract

The application relates to an ETL operator parsing conversion algorithm and application thereof. The application obtains an operator graph, generates required translator parameters according to the operator graph, generates a corresponding ETL context according to the operator graph, and inputs the ETL context into a translator. The translator uses the translator parameters to translate the operator graph, generates a corresponding operator intermediate representation, converts the operator intermediate representation into a task engine general script, and sends the task engine general script to a task engine factory for execution. The operator graph can be topologically sorted and translated by using the translator, the operator intermediate representation obtained by using topology construction is converted to generate a corresponding task engine general script, the task engine factory can flexibly execute the task engine general script generated by the operator graph, and the task engine general script can be flexibly scheduled and queued for execution. Compared with a fixed logic model, the task execution logic is more flexible, and users can freely construct operators for data conversion and manage calculation time.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of data conversion, in particular to an ETL operator parsing conversion method, an application system thereof and an electronic device. BACKGROUND

[0002] ETL is the abbreviation of English Extract-Transform-Load, which is used to describe the process of extracting data from the source end, transforming and loading it to the destination end. The word ETL is more commonly used in data warehouse, but its object is not limited to data warehouse.

[0003] The data conversion process needs to perform data logic according to the operator graph constructed by the user, so as to complete the data conversion processing and make the data realize the data conversion from the input data set to the target output data set according to a certain operator logic program.

[0004] The execution of the existing technology on the operator graph, such as the execution logic of Spark-SQL, executes ETL operation on data of different formats. Although several data sources are supported, Spark-SQL will introduce Context and corresponding RDD every time ETL operation is executed, which will increase the operation difficulty and is not flexible to use. Moreover, with the development of business, the operator of ETL is becoming more and more complex, the main logic and the secondary logic of ETL are mixed, the several execution modes of ETL are not obvious, and there is a lack of business analysis (which may lead to insufficient testing). The existence of these problems also leads to the difficulty of expanding and modifying the operator, the function penetration of the data set leads to the frequent modification of the core logic of ETL, and the logic program of Spark-SQL means that the processing mode of the operator graph is limited, which can only analyze and process the operator graph of data according to the fixed logic model (read-only online analytical processing), and the degree of freedom is not high. Therefore, Spark-SQL is not flexible enough in executing ETL, and cannot freely construct and calculate the operator of data conversion. SUMMARY

[0005] In order to solve the above problems, the present application provides an ETL operator parsing conversion method, an application system thereof and an electronic device.

[0006] In one aspect of the present application, an ETL operator parsing conversion method is provided, comprising the following steps:

[0007] An operator graph is obtained, and a translator parameter required is generated according to the operator graph;

[0008] An ETL context corresponding to the operator graph is generated and input to the translator;

[0009] The translator translates the operator graph using the translator parameters to generate a corresponding operator intermediate representation;

[0010] The operator intermediate representation is converted into a task engine general script and sent to a task engine factory for execution.

[0011] As an optional embodiment of the present application, the operator graph is obtained, and the required translator parameters are generated according to the operator graph, including:

[0012] It is judged whether the operator graph to be processed already exists in the DB database:

[0013] If the operator graph exists, the corresponding operator graph is obtained from the DB database;

[0014] If the operator graph does not exist, a data set to be converted is selected through an interface provided by an ETL module, and the operator graph is constructed by dragging and dropping operators;

[0015] The required translator parameters are generated according to the content of the operator graph.

[0016] As an optional embodiment of the present application, the ETL context corresponding to the operator graph is generated and input to the translator, including:

[0017] The operator graph is input to an ETL context factory;

[0018] The ETL context factory generates the corresponding ETL context according to the content of the operator graph, and the ETL context includes data source, data target and data conversion information;

[0019] The ETL context is used to support the translator.

[0020] As an optional embodiment of the present application, the ETL context corresponding to the operator graph is generated and input to the translator, further including:

[0021] The ETL context is used to generate a data flow net, and the data flow net is topologically sorted to determine the execution order of each operator in the operator graph;

[0022] According to the result of the topological sorting, an ordered operator sequence is generated to obtain an ordered sequence of the operator graph;

[0023] The ordered sequence is input to the translator, and an execution environment of the ETL context is prepared according to the ordered sequence and the ETL context.

[0024] As an optional implementation of the present application, optionally, the translator translates the operator graph using the translator parameters to generate corresponding operator intermediate representations, including:

[0025] The operator graph is translated for each operator in the operator graph:

[0026] The translator uses the pre-generated translator parameters and translates each operator in the operator graph according to the ordered sequence of the operator graph to convert each operator into an intermediate representation;

[0027] The intermediate representation of each operator is combined to obtain the operator intermediate representation of the operator graph.

[0028] As an optional implementation of the present application, optionally, the operator intermediate representation is converted into a task engine general script and sent to a task engine factory for execution, including:

[0029] According to the operator intermediate representation of the operator graph, a corresponding task engine general script is generated and input into a task engine factory to prepare for execution of the corresponding task;

[0030] In the task engine factory, the task is queued, scheduled and executed to generate a corresponding task execution result;

[0031] The task execution result is obtained through a task management system and returned to the user;

[0032] The flow execution ends.

[0033] In another aspect of the present application, an application system for implementing the ETL operator parsing and conversion method is provided, including:

[0034] An operator graph acquisition module is configured to acquire an operator graph and generate the required translator parameters according to the operator graph;

[0035] An ETL context generation module is configured to generate a corresponding ETL context according to the operator graph and input it into a translator;

[0036] A translation module is configured to translate the operator graph using the translator parameters to generate corresponding operator intermediate representations;

[0037] A task execution module is configured to convert the operator intermediate representation into a task engine general script and send it to a task engine factory for execution.

[0038] In another aspect of the present application, an electronic device is also provided, including:

[0039] A processor;

[0040] a memory for storing processor-executable instructions;

[0041] wherein the processor is configured to implement the ETL operator parsing conversion method when executing the executable instructions.

[0042] Technical effects of the present application:

[0043] The present application obtains an operator graph, and generates a required translator parameter according to the operator graph; generates a corresponding ETL context according to the operator graph, and inputs the ETL context to a translator; the translator translates the operator graph by using the translator parameter, to generate a corresponding operator intermediate representation; converts the operator intermediate representation into a task engine general script, and sends the task engine general script to a task engine factory for execution. The operator graph can be topologically sorted and translated by using the translator. After the operator graph is translated, the corresponding operator intermediate representation is converted by using the topological structure to generate a corresponding task engine general script, so that the task engine factory can flexibly execute the task engine general script generated by the operator graph, and the task engine general script can be flexibly scheduled and queued for execution, thereby improving the execution flexibility of the operator graph. The present application uses the task engine general script as the intermediate code of operator execution, and the task execution logic is more flexible than the fixed logic model. Users can also freely construct the operator and manage the calculation timing for data conversion.

[0044] Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present disclosure and serve to explain the principles of the present disclosure.

[0046] Figure 1 An implementation flowchart diagram of the ETL operator parsing conversion method of the present application is shown;

[0047] Figure 2 An application system diagram of the present application is shown;

[0048] Figure 3 A component structure diagram of the present application is shown;

[0049] Figure 4 An application diagram of the electronic device of the present application is shown. DETAILED DESCRIPTION

[0050] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in different drawings represent the same or similar elements. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically noted.

[0051] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0052] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated in the disclosure will be omitted. It will be appreciated that those skilled in the art can practice the present disclosure without these specific details.

[0053] Embodiment 1

[0054] As Figure 1 shown, in one aspect of the present application, an ETL operator parsing conversion method is provided, including the following steps:

[0055] S1, an operator graph is obtained, and a required translator parameter is generated according to the operator graph;

[0056] S2, a corresponding ETL context is generated according to the operator graph, and is input to the translator;

[0057] S3, the translator uses the translator parameter to translate the operator graph, and generates a corresponding operator intermediate representation;

[0058] S4, the operator intermediate representation is converted into a task engine general script, and is sent to a task engine factory for execution.

[0059] As Figure 2 shown, the present scheme includes a DB database / front-end input, an ETL context factory, a translator, a task engine factory, and a task management system.

[0060] The present scheme is mainly used for data conversion. A low-code mode provided by an ETL module is used to construct a calculation process by dragging some visual operators, so as to realize data conversion from an input data set to a target output data set. An operator graph is composed of one or more input operators and at least one output operator. A user can export, import, delete, save, or modify the operator graph, and manage the calculation time of the operator graph, such as cascade triggering, timing triggering, and the like.

[0061] The user needs to perform logical processing, the data set to be executed, will be logically executed in the ETL module through each operator, and the operator execution process is accompanied by corresponding data processing.

[0062] The user can select the data set to be converted through the interface provided by the ETL module, and build an operator graph by dragging and dropping operators to realize the data conversion process. The operator graph can include multiple operators, each of which represents a data conversion operation, such as filtering, sorting, aggregation, etc. The user can select appropriate operators to build their own operator graph according to their own needs.

[0063] Once the user has built their own operator graph, the ETL module will convert this calculation process into an executable JobEngine general script, and submit the task to queue, schedule and execute in the task engine. The user can manage the calculation time of the operator graph, such as cascading trigger, timing trigger, etc. to meet their own needs.

[0064] Finally, the ETL module will generate one or more target data sets, which can be exported, imported, deleted, saved or modified by the user. The user can further process or analyze these data sets according to their own needs.

[0065] As shown in Figure 3 , it is a specific implementation logic diagram of the present scheme.

[0066] The purpose of operator analysis is to convert the operator into a script that can be executed by JobEngine. Ultimately, what is needed is an executable script to execute the script by submitting JobEngineTask to return the required results.

[0067] The way to convert the operator into a Javascript script is to define local variables, that is, each operator defines a local variable with its own Id as the name, and then combines them together through calculation relationship. As shown below, each local variable corresponds to each operator Id one by one.

[0068] JobEngine general script. The running logic of the script is described by a Js function, and the script compresses the running logic of all operators into a Js function. Each variable declared in the function represents the processing process of an operator. Finally, the script generates a result and produces a file, which is saved in the data set.

[0069] For example, in the following task engine general script (two data sets) example:

[0070] function (c) { return function () { / / id_1662111324978 is the ID of the operator / / what follows the equal sign is the calculation logic of the operator (Js notation) - DataFrame var id_1662111324978 =c.cu.loadParquetDF(c.sqc,'s3a: / / guandata-store / table_cache / demo / t6cf2cb9147b449abbb6a1df', true).selectExpr('`120` as `result4`', '`4`', '`setosa`', '`versicolor`', '`virginica`').limit(10); var id_1662343921066 =c.cu.loadParquetDF(c.sqc,'s3a: / / guandata-store / table_cache / demo / r5c1d8e1a59a04861bc23af5', true).selectExpr('`120`', '`4`', '`setosa`', '`versicolor`', '`virginica`').limit(10); var id_1663847429013 = id_1662111324978.alias('id_1662111324978')['join(Dataset, Column, String)'](id_1662343921066.alias('id_1662343921066'), c.f.expr('id_1662111324978.`virginica`').equalTo(c.f.expr('id_1662343921066.`virginica`')), 'INNER')['select(Column[])'](c.f.expr('id_1662111324978.`setosa`'), c.f.expr('id_1662343921066.`versicolor`')); return id_1663847429013;}()['select(Column[])'](c.f.col('setosa'), c.f.col('versicolor')).limit(200);}().

[0071] The running logic of this JS function is as follows:

[0072] The function function (c) receives a parameter c and returns a function.

[0073] In the returned function, first, the definition of two variables is executed, which are operator id_1662111324978 and id_1662343921066. These two variables represent two datasets loaded from S3 storage, and some column operations are performed on them, such as selecting specific columns, limiting the size of the dataset, etc.

[0074] Next, the operators id_1662111324978 and id_1662343921066 are performed an inner join operation, and the connection condition is that the virginica column in the two datasets is equal. The result of the connection is saved in the variable id_1663847429013.

[0075] Finally, two columns of data setosa and versicolor are selected from the operator id_1663847429013, the size of the result set is limited to 200, and the result is returned.

[0076] The function is to perform an inner join on two datasets and then select two columns of data for return. The implementation of this function uses the API of Spark SQL and performs some basic operations on the dataset, such as selection, limitation, inner join, etc.

[0077] All input operators implement Translatable, so they can call the translate method of Translatable to perform conversion. Translatable needs to rely on a context to process and convert BaseCommand to BaseRepresentationCommand (basic representation command). See the following description.

[0078] As an optional embodiment of the present application, the operator graph is obtained, and the required translator parameters are generated according to the operator graph, including:

[0079] Determine whether the DB database already exists the operator graph to be processed:

[0080] If it exists, the corresponding operator graph is obtained from the DB database;

[0081] If it does not exist, through the interface provided by the ETL module, select the dataset that needs to be converted, and construct the operator graph by dragging the operator;

[0082] The required translator parameters are generated according to the contents of the operator graph.

[0083] Determine whether the operator graph already exists in the DB database. If so, obtain the operator graph from the database. The operator graph file to be processed can be stored in the DB database in advance and can be directly retrieved for calculation.

[0084] If it does not exist, the user is asked to enter or build an operator graph on the front end. For details, see the description of building an operator graph in the ETL module above.

[0085] The operator graph will be translated later using a translator, so here we can generate the parameters required by the translator in advance based on the content of the subgraph.

[0086] As an optional implementation scheme of the present application, optionally, generating a corresponding ETL context according to the operator graph and inputting it into the translator includes:

[0087] Input the operator graph into the ETL context factory;

[0088] The ETL context factory generates a corresponding ETL context based on the contents of the operator graph. The ETL context includes data source, data target, and data conversion information.

[0089] The translator is supported using the ETL context.

[0090] The ETL context factory, i.e., the upper and lower level ETL factories, generates the ETL context based on the contents of the operator graph, including information such as data source, data target, and data transformation.

[0091] The ETL context is also used to prepare the context execution environment. Figure 3 As shown in the figure, the translator needs to translate operators within an execution environment. This requires preparing the execution environment based on the context. After generating the execution order for the corresponding operators, the translator then translates and executes each operator in the execution environment in an ordered sequence. The ETL context includes information such as the data source, data target, and data transformation.

[0092] As an optional embodiment of the present application, optionally, generating a corresponding ETL context according to the operator graph and inputting it into the translator further includes:

[0093] Generate a data flow network using the ETL context, and perform topological sorting on the data flow network to determine the execution order of each operator in the operator graph;

[0094] Generate an ordered sequence of operators according to the result of the topological sorting, and obtain an ordered sequence of the operator graph;

[0095] According to inputting the ordered sequence into a translator, and according to the ordered sequence and the ETL context, an execution environment of the ETL context is prepared.

[0096] An ETL context factory converts the operators in the operator graph into a data flow net (corresponding to the topological data net, but the network is sorted according to the requirements of topological sorting) according to the generated ETL context (topological data net of the operator graph), and performs sorting processing on the data flow net.

[0097] The data flow net is topologically sorted by using a topological sorting algorithm to determine the execution order of each operator. According to the result of topological sorting, an ordered operator sequence is generated. According to the ETL context and the ordered operator sequence, a context execution environment is prepared, including a data source, a conversion process, and the like.

[0098] The purpose of operator analysis is to convert the operator into a script that can be executed by the JobEngine. Ultimately, what is needed is an executable script that can be executed by submitting the JobEngineTask to return the required result.

[0099] Input operator (BaseCommand): This is a mesh structure, and each operator references its predecessor.

[0100] Operator conversion (Translatable.translate): After topological sorting of the mesh structure of the input operator (using the edge deletion method), the operator is converted into a certain representation of the operator (BaseRepresentationCommand) by sequentially traversing each operator.

[0101] In this embodiment, the implementation of topological sorting can refer to the topological sorting algorithm in the prior art, which will not be described here.

[0102] The algorithm is re-implemented using Scala in this module, and is optimized based on the characteristics of the Scala language.

[0103] As an optional embodiment of the present application, the translator translates the operator graph using the translator parameters to generate the corresponding operator intermediate representation, including:

[0104] The translator translates each operator in the operator graph:

[0105] The translator uses the translator parameters generated in advance and translates each operator in the operator graph according to the ordered sequence of the operator graph, and converts each operator into an intermediate representation;

[0106] The intermediate representations of each operator are combined to obtain the operator intermediate representation of the operator graph.

[0107] As an optional embodiment of the present application, the operator intermediate representation is optionally converted into a task engine general script and sent to a task engine factory for execution, including:

[0108] According to the operator intermediate representation of the operator graph, a corresponding task engine general script is generated, and the task engine general script is input into a task engine factory, ready for execution of the corresponding task;

[0109] In the task engine factory, the task is queued, scheduled and executed to generate a corresponding task execution result;

[0110] The task execution result is obtained through a task management system, and the task execution result is returned to the user;

[0111] The flow execution ends.

[0112] Task engine general script: convert the operator representation into a Javascript script.

[0113] The translator uses the generated translator parameters to convert each operator in the operator graph into an intermediate representation.

[0114] Operator intermediate representation: combine the intermediate representation of each operator into the intermediate representation of the entire operator graph.

[0115] Convert to task engine general script: convert the intermediate representation of the operator graph into a task engine general script for execution in the task engine. The script compresses the running logic of all operators into a Js function.

[0116] Submit the task to the task engine factory: submit the task engine general script to the task engine factory for queuing, scheduling and execution.

[0117] Get the task result through the task management: get the task execution result through the task management system and return the result to the user.

[0118] Stop: the flow execution ends. The script generates a result and produces a file, which is saved in the data set.

[0119] The task engine factory and the task management system can be combined with the system platform of the ETL module for specific application, which will not be described in detail in the present embodiment.

[0120] Therefore, the present application converts the operator graph input by the user into an executable task engine general script, and queues, schedules and executes through the task engine factory. In this process, multiple processing and conversion are required, including generation of translator parameters, generation of ETL context, generation of data flow network, topological sorting, generation of ordered sequence, preparation of execution environment, etc. Finally, the task execution result is obtained through the task management system, and the result is returned to the user.

[0121] In the data conversion process, the operator graph parsing is converted into the corresponding task engine general script, and the user can participate in the ordering, scheduling and online reconstruction of the task engine general script, thereby providing the user with greater flexibility in operator parsing and task execution convenience, and increasing the execution flexibility of the operator graph.

[0122] Obviously, those skilled in the art should understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The program can include the processes of the above-mentioned embodiments when executed. Those skilled in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware, and the program can be stored in a computer readable storage medium. The program can include the processes of the above-mentioned embodiments when executed. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage medium can also include a combination of the above types of memories.

[0123] Embodiment 2

[0124] Based on the implementation principle of embodiment 1, in another aspect, the present application proposes an application system for implementing the ETL operator parsing conversion method, comprising:

[0125] An operator graph acquisition module is configured to acquire an operator graph and generate required translator parameters according to the operator graph;

[0126] An ETL context generation module is configured to generate a corresponding ETL context according to the operator graph and input the ETL context to a translator;

[0127] A translation module is configured to translate the operator graph using the translator parameters to generate a corresponding operator intermediate representation;

[0128] A task execution module is configured to convert the operator intermediate representation into a task engine universal script and send the task engine universal script to a task engine factory for execution.

[0129] The modules can be understood from the application description in Embodiment 1, and thus will not be described herein.

[0130] The modules or steps of the present application can be implemented by using a general computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Alternatively, they can be implemented by using program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively manufactured into integrated circuit modules, or multiple modules or steps among them can be manufactured into a single integrated circuit module. Thus, the present application is not limited to any specific combination of hardware and software.

[0131] Embodiment 3

[0132] As shown in Figure 4 Further, another aspect of the present application also provides an electronic device, comprising:

[0133] a processor;

[0134] a memory for storing processor-executable instructions;

[0135] The processor is configured to implement the ETL operator parsing and conversion method when executing the executable instructions.

[0136] The electronic device according to the present application includes a processor and a memory for storing processor-executable instructions. The processor is configured to implement any of the ETL operator parsing and conversion methods described above when executing the executable instructions.

[0137] It should be noted that the number of processors can be one or more. Meanwhile, the electronic device according to the present application can also include an input device and an output device. The processor, the memory, the input device and the output device can be connected by a bus or by other means, which is not limited herein.

[0138] The memory as a computer readable storage medium can be used to store software programs, computer executable programs and various modules, such as the programs or modules corresponding to the ETL operator parsing and conversion method according to the present application. The processor executes the software programs or modules stored in the memory, thereby performing various functions and data processing of the electronic device.

[0139] The input device can be used to receive inputted numbers or signals. Among them, the signal can be a key signal generated in relation to user settings and function control of the device / terminal / server. The output device can include a display device such as a display screen.

[0140] Embodiments of the disclosure have been described above, the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles, practical applications, or technical improvements of the technology in the market, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. An ETL operator parsing conversion method, characterized in that: The steps include: Obtaining an operator graph and generating required translator parameters according to the operator graph; Generating a corresponding ETL context according to the operator graph and inputting the context into the translator includes: inputting the operator graph into an ETL context factory; the ETL context factory generating a corresponding ETL context according to the contents of the operator graph, the ETL context including data source, data target, and data conversion information; and supporting the translator with the ETL context; The translator translates the operator graph using the translator parameters to generate a corresponding operator intermediate representation; The operator intermediate representation is converted into a task engine general script and sent to the task engine factory for execution.

2. The ETL operator parsing conversion method according to claim 1, characterized in that: Obtain an operator graph and generate the required translator parameters based on the operator graph, including: Determine whether the operator graph to be processed already exists in the DB database: If it exists, the corresponding operator graph is retrieved from the DB database; If it does not exist, select the data set to be converted through the interface provided by the ETL module, and construct the operator graph by dragging and dropping operators; The required translator parameters are generated according to the contents of the operator graph.

3. The ETL operator parsing conversion method according to claim 1, characterized in that: Generate the corresponding ETL context according to the operator graph and input it into the translator, which also includes: Generate a data flow network using the ETL context, and perform topological sorting on the data flow network to determine the execution order of each operator in the operator graph; Generate an ordered sequence of operators according to the result of the topological sorting, and obtain an ordered sequence of the operator graph; The ordered sequence is input into a translator, and an execution environment of the ETL context is prepared according to the ordered sequence and the ETL context.

4. The ETL operator parsing conversion method according to claim 1, characterized in that: The translator uses the translator parameters to translate the operator graph and generate a corresponding operator intermediate representation, including: Translate each operator in the operator graph: The translator uses the translator parameters generated in the previous step and translates each operator in the operator graph according to the ordered sequence of the operator graph, converting each operator into an intermediate representation; The intermediate representation of each operator is combined to obtain the operator intermediate representation of the operator graph.

5. The ETL operator parsing conversion method according to claim 4, characterized in that: Convert the operator intermediate representation into a task engine general script and send it to the task engine factory for execution, including: Generate a corresponding task engine general script according to the operator intermediate representation of the operator graph, and input the task engine general script into the task engine factory to prepare to execute the corresponding task; In the task engine factory, the tasks are queued, scheduled and executed, and corresponding task execution results are generated; Obtain the task execution result through the task management system and return the task execution result to the user; The process execution ends.

6. An application system for implementing the ETL operator parsing and conversion method according to any one of claims 1 to 5, characterized in that: include: An operator graph acquisition module, used to acquire an operator graph and generate required translator parameters according to the operator graph; An ETL context generation module is used to generate a corresponding ETL context according to the operator graph and input it into the translator; A translation module, configured to translate the operator graph using the translator parameters to generate a corresponding operator intermediate representation; The task execution module is used to convert the operator intermediate representation into a task engine general script and send it to the task engine factory for execution.

7. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to implement the ETL operator parsing and conversion method according to any one of claims 1 to 5 when executing the executable instructions.

Images