Data acquisition method, device and electronic equipment

Abstract

This invention discloses a data acquisition method, apparatus, and electronic device applied in the field of data processing, comprising: acquiring a target ETL task and target configuration items for specifying a target execution engine; converting the target ETL task into a standard job configuration; generating an application package suitable for execution by the target execution engine based on the target configuration items; and submitting the application package to a target service cluster based on the target configuration items, so that the target service cluster uses the target execution engine to execute the application package to run an ETL data acquisition job. This invention improves data acquisition efficiency and reduces user operational complexity.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more particularly to a data acquisition method, apparatus, and electronic device. Background Technology

[0002] Users extract the required data from the data source, perform data cleaning, and finally load the data into the data warehouse according to a predefined data warehouse model. ETL (Extract-Transform-Load) is a key tool in this process. ETL data acquisition tools extract, clean, and transform data from business systems before loading it into the data warehouse. The goal is to integrate scattered, disorganized, and inconsistent data within an enterprise, providing analytical support for decision-making. ETL is a crucial component of BI (Business Intelligence) projects.

[0003] ETL data acquisition tools all have internally integrated execution engines. Therefore, their functionality is tied to the execution engine, and switching execution engines is not possible. If the current execution engine is unsuitable for a certain data source, the entire ETL data acquisition tool needs to be replaced to acquire data from that data source. This results in scenarios such as ETL jobs with multi-source heterogeneous data requiring users to use multiple ETL data acquisition tools to perform ETL jobs on different data sources separately. This leads to low data acquisition efficiency, and the operation of switching between different ETL tools is difficult and costly for users. Summary of the Invention

[0004] The embodiments of the present invention provide a data acquisition method, device, and electronic device, which solves the technical problems of low data acquisition efficiency and high user operation difficulty in the prior art.

[0005] In a first aspect, the present invention provides a data acquisition method through an embodiment of the present invention, comprising:

[0006] Retrieve the target ETL task and the target configuration items used to specify the target execution engine;

[0007] Convert the target ETL task into a standard job configuration;

[0008] Based on the target configuration items, generate an application package suitable for execution by the target execution engine;

[0009] According to the target configuration items, the application package is submitted to the target service cluster so that the target service cluster can use the target execution engine to execute the application package to run an ETL data acquisition job.

[0010] Optionally, converting the target ETL task into a standard job configuration includes: invoking a pre-created configuration conversion plugin to convert the target ETL task into a standard job configuration; generating an application package suitable for execution by the target execution engine based on the target configuration items includes: invoking a pre-created program generation plugin to generate the application package based on the target configuration items.

[0011] Optionally, submitting the application package to the target service cluster according to the target configuration item includes:

[0012] The application package is submitted to the target service cluster by invoking a pre-created submission plugin.

[0013] Optionally, submitting the application package to the target service cluster according to the target configuration item includes:

[0014] Based on the preset mapping relationship between each configuration item and the startup command, the target startup command that matches the target configuration item is determined.

[0015] The application package is submitted to the target service cluster using the target startup command, so that the target service cluster can start the target execution engine using the target startup command and execute the application package using the started target execution engine.

[0016] Optionally, there may be multiple candidate execution engines distributed across different service clusters or within the same service cluster, and the target execution engine belongs to one of the multiple candidate execution engines.

[0017] Optionally, an engine configuration interface containing multiple configuration items is pre-set, and obtaining the target configuration item for specifying the target execution engine includes:

[0018] Using the engine configuration interface, the configuration item selected by the user is obtained as the target configuration item, wherein the target configuration item is used to specify an execution engine that is compatible with the target ETL task from the plurality of candidate execution engines as the target execution engine.

[0019] Optionally, the application package includes any one of the following: the Spark application package, the Flink application package, and the DataX application package.

[0020] Secondly, embodiments of the present invention provide a data acquisition device, comprising:

[0021] The data acquisition unit is used to acquire the target ETL task and the target configuration items used to specify the target execution engine.

[0022] A configuration conversion unit is used to convert the target ETL task into a standard job configuration;

[0023] The program generation unit is used to generate an application package suitable for execution by the target execution engine based on the target configuration items.

[0024] The submission unit is used to submit the application package to the target service cluster according to the target configuration item, so that the target service cluster can use the target execution engine to execute the application package to run an ETL data acquisition job.

[0025] Thirdly, embodiments of the present invention provide an electronic device, including: a memory, a processor, and code stored in the memory and executable on the processor, characterized in that the processor, when executing the code, implements the steps of the method described in any embodiment of the first aspect.

[0026] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any embodiment of the first aspect.

[0027] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0028] The process involves converting the acquired target ETL task into a standard job configuration; then generating an application package suitable for the target execution engine based on the target configuration items; and finally submitting the application package to the target service cluster according to the target configuration items. This allows the target service cluster to start the target execution engine to execute the application package, thus running the ETL data acquisition job. In this way, users only need to change the configuration items to select the actual target execution engine to generate an application package suitable for that engine. This expands the range of ETL execution engines and allows for flexible switching between different engines. Furthermore, based on the ETL task and the framework advantages of different execution engines, a suitable execution engine can be selected to complete the ETL job. Users do not need to learn various ETL tools, making the process seamless and improving the efficiency of ETL jobs for multi-source data. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart of a data acquisition method provided in an embodiment of the present invention;

[0031] Figure 2 A functional block diagram of the data acquisition device provided in the embodiments of the present invention;

[0032] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] To address the technical problems of low data acquisition efficiency and high user operation difficulty in existing technologies, this invention provides a data acquisition method, the overall concept of which is as follows:

[0034] The process involves converting the acquired target ETL task into a standard job configuration; generating an application package suitable for execution by the target execution engine based on the target configuration items; submitting the application package to the target service cluster based on the target configuration items; and enabling the target service cluster to start the target execution engine to execute the application package in order to run the ETL data acquisition job.

[0035] The above technical solution achieves low coupling between the upper-level functions of ETL jobs and the underlying execution engine. Thus, users only need to change the configuration items to select the target execution engine to generate an application package suitable for that target execution engine. This expands the ETL execution engines and allows for flexible switching between different execution engines. Furthermore, based on the ETL task and the framework advantages of different execution engines, a suitable execution engine can be selected to complete the ETL job. Users do not need to learn various ETL tools, making the process seamless and improving the efficiency of ETL jobs for multi-source data.

[0036] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0037] First, it should be clarified that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0038] This invention provides a data acquisition method that utilizes a single ETL data acquisition tool. Specifically, this ETL data acquisition tool includes a business layer and an execution engine, and extends to multiple execution engines. This invention achieves the following data acquisition method using a single ETL data acquisition tool, without employing multiple different ETL data acquisition tools.

[0039] refer to Figure 1As shown, the data acquisition method provided in this embodiment of the invention includes the following steps:

[0040] S101: Obtain the target ETL task and the target configuration items used to specify the target execution engine.

[0041] The target ETL task can specifically collect data from data sources such as log files and streaming data. Specifically, if the target ETL task is created by the user, then the user-created target ETL task is retrieved. This task is configured with read operations (reader), transformation operations (transformer), and write operations (writer), etc. The read operation configuration specifies the data source from which the data is extracted; the transformation operation configuration specifies the transformation operation content, such as filtering, settering, sorting, field mapping, deduplication, etc.; the write operation configuration specifies the target source of the data to be written, i.e., which data warehouse the transformed data will be written to.

[0042] The ETL data acquisition tool used in this embodiment of the invention provides a visual interface for users to create ETL tasks. This allows users to perform operations related to creating a target ETL task within the visual interface. Furthermore, the visual interface includes an engine configuration interface. Using this interface, the configuration items selected by the user are obtained as target configuration items. These target configuration items are used to specify an execution engine suitable for the target ETL task from multiple candidate execution engines as the target execution engine.

[0043] Specifically, a pre-configured engine configuration interface is provided, which includes multiple configuration items that correspond one-to-one with the execution engine of the ETL data acquisition tool. These configuration items can include various options such as DataX, Flink, and Spark, allowing users to select one of these engines as the target execution engine.

[0044] It should be noted that in this embodiment of the invention: Flink is a framework and distributed processing engine for stateful computation on unbounded and bounded data streams. Flink is designed to run in all common cluster environments, performing computations at memory speed and any scale; Spark is a distributed open-source processing system for big data workloads. Spark uses in-memory caching and optimized query execution to perform fast analytical queries on data of any size; DataX is an open-source ETL acquisition framework that supports data synchronization from multiple data sources and target sources.

[0045] S102: Convert the target ETL task into a standard job configuration.

[0046] In the ETL data acquisition tool used in this embodiment of the invention, a pre-created configuration conversion plugin is invoked to convert the target ETL task into a standard job configuration. Specifically, this involves converting each operation configured in the target ETL task, such as converting the configurations for read, transform, and write operations, thereby obtaining the standard job configuration. The standard job configuration contains the standard format configuration content for read, transform, and write operations. This standard job configuration is the basis for switching execution engines. Since ETL data acquisition generally uses a universal architecture that includes both a data source and a target source, any ETL task can be converted into the aforementioned standard job configuration.

[0047] S103: Generate an application package suitable for execution by the target execution engine based on the target configuration items.

[0048] The ETL data acquisition tool used in this embodiment of the invention has a pre-created program generation plugin. By calling the pre-created program generation plugin, the standard job configuration is converted into an application package suitable for execution by the target execution engine.

[0049] In practice, different target configuration options will result in different generated application packages. The generated application package can be any of the following: Spark application package, Flink application package, or DataX application package.

[0050] Specifically, if the user selects "spark" as the target configuration option, the standard job configuration will be transformed to generate a Spark application package; if the user selects "flink" as the target configuration option, the standard job configuration will be transformed to generate a Flink application package; and if the user selects "datax" as the target configuration option, the standard job configuration will be transformed to generate a Datax application package.

[0051] Specifically, if the goal is to transform and generate a Spark application package, the process involves converting the standard job configuration into RDD (Resilient Distributed Datasets) operators that can be executed by the Spark execution engine. Then, the resulting RDD operators are packaged to obtain the Spark application package. An RDD operator consists of multiple partitions, and the RDD operator operates on these partitions; it is a function that computes the partitions.

[0052] Specifically, if the goal is to transform and generate a Flink application package, the process is as follows: convert the standard job configuration into an execution graph structure that can be executed by the Flink execution engine, and then package the transformed execution graph structure to obtain the Spark application package.

[0053] If the conversion generates a datax application package, the specific method is as follows: convert the standard job configuration into a configuration file in the standard data source configuration format suitable for the datax execution engine, specifically in JSON format, and pass the converted configuration file as a parameter to the datax.py script to obtain the datax application package.

[0054] Step S102 generates an application package suitable for execution by the target execution engine.

[0055] S103. Based on the target configuration items, submit the application package to the target service cluster so that the target service cluster can use the target execution engine to execute the application package to run the ETL data acquisition job.

[0056] In step S103, the application package is submitted to the target service cluster by invoking a pre-created submission plugin, and then executed on the target service cluster. The target service cluster refers to the service cluster where the target execution engine resides.

[0057] Different candidate execution engines require different startup commands; therefore, by using different defined startup commands, the corresponding execution engine can be started. The startup command for each execution engine defines the relevant information for the execution engine to be started.

[0058] In an optional implementation, a mapping relationship between each configuration item and the startup command can be pre-created. This mapping relationship is used to ensure that the generated application package is submitted to the appropriate location and executed by the appropriate execution engine.

[0059] For example, the "Spark configuration" item maps to the first startup command; the "Flink configuration" item maps to the second startup command; and the "DataX configuration" item maps to the third startup command. Based on the target configuration item selected by the user, the startup command used to start the target execution engine can be matched.

[0060] Specifically, step S103 includes: determining the target startup command that matches the target configuration item based on the preset mapping relationship between each configuration item and the startup command; using the target startup command to submit the application package to the target service cluster, so that the target service cluster can use the target startup command to start the target execution engine and use the started target execution engine to execute the application package.

[0061] Of course, in practice, the application package and target configuration items can also be submitted directly to the target service cluster. Based on the target configuration items, the target service cluster can match the target startup command, and then use the target startup command to start the target execution engine for executing the application package.

[0062] In practice, multiple candidate execution engines can be distributed across different service clusters, or they can be distributed across the same service cluster. For example, the Spark execution engine is located in service cluster A (service nodes 1, 2, 3, 4), the Flink execution engine is located in service cluster B (service nodes 3, 4, 6, 7), and the DataX execution engine is located in service cluster B (service nodes 5, 6, 7, 8).

[0063] In an alternative implementation, if multiple candidate execution engines are distributed in the same service cluster, the application package is submitted to the service cluster corresponding to the fixed IP address.

[0064] In an optional implementation, if multiple candidate execution engines are distributed across different service clusters, a mapping relationship is pre-created between each configuration item and the IP address of the corresponding execution engine's service node to determine the target IP address of the service cluster where the target execution engine resides. The application package and target startup command are then submitted to the determined target IP address, ensuring they are delivered to the service cluster where the target execution engine resides. Upon receiving the target startup command and application package, the service cluster uses the target startup command to start the target execution engine.

[0065] Based on the same inventive concept, another embodiment of the present invention provides a data acquisition device, with reference to... Figure 2 Shown, including:

[0066] Data acquisition unit 201 is used to acquire the target ETL task and the target configuration items for specifying the target execution engine;

[0067] The configuration conversion unit 202 is used to convert the target ETL task into a standard job configuration;

[0068] The program generation unit 203 is used to generate an application package suitable for execution by the target execution engine based on the target configuration items;

[0069] Submission unit 204 is used to submit the application package to the target service cluster according to the target configuration item, so that the target service cluster can use the target execution engine to execute the application package to run ETL data acquisition jobs.

[0070] In an optional implementation, the configuration conversion unit 202 is specifically used to: call a pre-created configuration conversion plugin to convert the target ETL task into a standard job configuration; the program generation unit 203 is specifically used to: call a pre-created program generation plugin to generate the application package according to the target configuration items.

[0071] In an optional implementation, the submission unit 204 is specifically used to: invoke a pre-created submission plugin to submit the application package to the target service cluster.

[0072] In an optional implementation, the submission unit 204 is specifically used for:

[0073] Based on the preset mapping relationship between each configuration item and the startup command, the target startup command that matches the target configuration item is determined.

[0074] The application package is submitted to the target service cluster using the target startup command, so that the target service cluster can start the target execution engine using the target startup command and execute the application package using the started target execution engine.

[0075] In one optional implementation, there are multiple candidate execution engines distributed across different service clusters or the same service cluster, and the target execution engine is one of the multiple candidate execution engines.

[0076] In an optional implementation, an engine configuration interface containing multiple configuration items is pre-configured, and the data acquisition unit 201 is specifically used for:

[0077] Using the engine configuration interface, the configuration item selected by the user is obtained as the target configuration item, wherein the target configuration item is used to specify an execution engine that is compatible with the target ETL task from the plurality of candidate execution engines as the target execution engine.

[0078] In an alternative implementation, the application package includes any one of the following: the Spark application package, the Flink application package, and the DataX application package.

[0079] Based on the same inventive concept, embodiments of the present invention provide an electronic device. (See reference...) Figure 3 As shown, the electronic device provided in this embodiment of the invention includes: a memory 301, a processor 302, and code 303 stored in the memory and executable on the processor. When the controller executes the code, it implements any of the embodiments of the data acquisition method described above.

[0080] Among them, Figure 3In this document, a bus architecture (represented by bus 300) is used. Bus 300 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 302 and memory represented by memory 301. Bus 300 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 306 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 302 is responsible for managing bus 300 and general processing, while memory 304 can be used to store data used by processor 302 during operation.

[0081] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0082] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0083] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0085] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0086] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A data acquisition method, characterized in that, include: Obtain the target ETL task and the target configuration items for specifying the target execution engine, wherein the target ETL task is created by user operation; Converting the target ETL task into a standard job configuration includes: calling a pre-created configuration conversion plugin to convert the configuration of each operation configured in the target ETL task to convert the target ETL task into a standard job configuration. The operations include: read operations, conversion operations, and write operations. Generate an application package suitable for execution by the target execution engine based on the target configuration items, including: calling a pre-created program generation plugin, and converting the standard job configuration into an application package suitable for execution by the target execution engine based on the target configuration items; According to the target configuration item, the application package is submitted to the target service cluster so that the target service cluster can use the target execution engine to execute the application package to run an ETL data acquisition job. There are multiple candidate execution engines distributed in different service clusters or in the same service cluster. The target execution engine is one of the multiple candidate execution engines. Each service cluster includes multiple service nodes. The target configuration item is used to specify an execution engine that is compatible with the target ETL task from the multiple candidate execution engines as the target execution engine. The target service cluster refers to the service cluster where the target execution engine is located.

2. The method as described in claim 1, characterized in that, The step of submitting the application package to the target service cluster according to the target configuration item includes: The application package is submitted to the target service cluster by invoking a pre-created submission plugin.

3. The method as described in claim 1 or 2, characterized in that, The step of submitting the application package to the target service cluster according to the target configuration item includes: Based on the preset mapping relationship between each configuration item and the startup command, the target startup command that matches the target configuration item is determined. The application package is submitted to the target service cluster using the target startup command, so that the target service cluster can start the target execution engine using the target startup command and execute the application package using the started target execution engine.

4. The method as described in claim 1, characterized in that, There are multiple candidate execution engines distributed across different service clusters or within the same service cluster, and the target execution engine is one of the multiple candidate execution engines.

5. The method as described in claim 4, characterized in that, An engine configuration interface with multiple configuration items is pre-set. The step of obtaining the target configuration item for specifying the target execution engine includes: Using the engine configuration interface, the configuration item selected by the user is obtained as the target configuration item.

6. The method as described in claim 1, characterized in that, The application package includes any one of the following: Spark application package, Flink application package, and DataX application package.

7. A data acquisition device, characterized in that, include: The data acquisition unit is used to acquire the target ETL task and the target configuration item for specifying the target execution engine, wherein the target ETL task is created by user operation; A configuration conversion unit is used to convert the target ETL task into a standard job configuration, including: calling a pre-created configuration conversion plugin to convert the configuration of each operation configured in the target ETL task to convert the target ETL task into a standard job configuration, wherein the operations include: read operations, conversion operations, and write operations; The program generation unit is used to generate an application package suitable for execution by the target execution engine based on the target configuration item, including: calling a pre-created program generation plugin to convert the standard job configuration into an application package suitable for execution by the target execution engine based on the target configuration item; The submission unit is configured to submit the application package to the target service cluster according to the target configuration item, so that the target service cluster can use the target execution engine to execute the application package to run an ETL data acquisition job. There are multiple candidate execution engines distributed in different service clusters or in the same service cluster. The target execution engine is one of the multiple candidate execution engines. Each service cluster includes multiple service nodes. The target configuration item is configured to specify an execution engine adapted to the target ETL task from the multiple candidate execution engines as the target execution engine. The target service cluster refers to the service cluster where the target execution engine is located.

8. An electronic device, comprising: A memory, a processor, and code stored in the memory and executable on the processor, characterized in that the processor, when executing the code, implements the steps of any of the methods described in claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method of any one of claims 1-6.

Images