ETL data processing method and device, electronic equipment and computer storage medium

Abstract

The application provides an ETL data processing method and device, electronic equipment and computer storage medium, the method is applied to the technical field of data warehouse, the method comprises the following steps: receiving the ETL job sent by the front end; obtaining the related information of the ETL job, splitting the related information of the ETL job to obtain split information; the split information comprises data source detail information and job detail information; the data source detail information is stored to the first database, and the job detail information is stored to the second database; wherein the first database and the second database are different databases; when receiving the execution request for the ETL job, the execution of the ETL job is completed based on the data source detail information obtained from the first database and the job detail information obtained from the second database. In this way, the pressure of the database can be reduced, the stability of the system can be improved, and the application requirements can be better met.

Description

Technical Field

[0001] This application relates to the field of data warehouse technology, and in particular to an ETL data processing method, apparatus, electronic device and computer storage medium. Background Technology

[0002] ETL (Extract-Transform-Load) is the process of extracting, cleaning, and transforming data from business systems and then loading it into a data warehouse. Its purpose is to integrate scattered, disorganized, and inconsistent data within an enterprise to provide analytical support for decision-making.

[0003] In related technologies, when storing ETL jobs, the content related to the ETL jobs is usually stored in the same database. However, when there are a large number of ETL jobs to be executed, the above storage method will not only increase the storage pressure on the database, but also affect the stability of the system. Summary of the Invention

[0004] This application provides an ETL data processing method, apparatus, electronic device, and computer storage medium that can improve system stability.

[0005] The technical solution of this application is implemented as follows:

[0006] This application provides an ETL data processing method applied to a backend server, the method comprising:

[0007] Receive ETL jobs sent from the front end;

[0008] Obtain relevant information about the ETL job, and then break down the relevant information of the ETL job to obtain the broken down information; the broken down information includes data source details and job details.

[0009] The data source details are stored in a first database, and the job details are stored in a second database; wherein the first database and the second database are different databases.

[0010] Upon receiving an execution request for the ETL job, the execution of the ETL job is completed based on the data source details obtained from the first database and the job details obtained from the second database.

[0011] In some embodiments, receiving the ETL job sent by the front end includes:

[0012] The system receives the converted ETL job sent by the front end; the converted ETL job is generated by the front end from the initial ETL job format generated according to the user operation instructions, converted into YAML format, and then sent to the backend server.

[0013] In some embodiments, the split information further includes job attribute information, and the method further includes:

[0014] When storing the job details information in the second database, obtain the current timestamp;

[0015] The current timestamp is determined as the version information of the ETL job, and the version information is added to the job attribute information.

[0016] In some embodiments, the job attribute information further includes the ID and status information of the ETL job, and the method further includes:

[0017] Obtain the ID, status information, and version information of the ETL job;

[0018] A key value is generated based on the ID, status information, and version information of the ETL job, and the key value is stored in the first database;

[0019] The step of storing the job details information in the second database includes:

[0020] The job details are stored in the second database as the value corresponding to the key.

[0021] In some embodiments, the execution of the ETL job based on the data source details obtained from the first database and the job details obtained from the second database includes:

[0022] The scheduling engine retrieves the data source details and the key value from the first database, and retrieves the job details from the second database based on the key value.

[0023] The scheduling engine submits the data source details and job details to the execution engine, so that the execution engine can complete the execution of the ETL job.

[0024] In some embodiments, the method further includes:

[0025] When storing the job details information in the second database, the job details information is backed up to a distributed cache.

[0026] In some embodiments, the method further includes:

[0027] When an anomaly occurs in the second database, the job details information is retrieved from the distributed cache based on the key value.

[0028] This application provides an ETL data processing apparatus, which includes a receiving module, a splitting module, a storage module, and an execution module, wherein...

[0029] The receiving module is used to receive ETL jobs sent from the front end;

[0030] The splitting module is used to obtain relevant information of the ETL job, split the relevant information of the ETL job, and obtain split information; the split information includes data source details and job details.

[0031] A storage module is used to store the data source details in a first database and the job details in a second database; wherein the first database and the second database are different databases.

[0032] The execution module is configured to, upon receiving an execution request for the ETL job, execute the ETL job based on the data source details obtained from the first database and the job details obtained from the second database.

[0033] This application provides an electronic device, the device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the ETL data processing method provided by one or more of the foregoing technical solutions.

[0034] This application provides a computer storage medium storing a computer program; when the computer program is executed, it can implement the ETL data processing method provided by one or more of the aforementioned technical solutions.

[0035] This application provides an ETL data processing method, apparatus, electronic device, and computer storage medium. The method is applied to a backend server and includes: receiving an ETL job sent by a frontend; obtaining relevant information of the ETL job; splitting the relevant information of the ETL job to obtain split information; the split information includes data source details and job details; storing the data source details in a first database and the job details in a second database; wherein the first database and the second database are different databases; and upon receiving an execution request for the ETL job, executing the ETL job based on the data source details obtained from the first database and the job details obtained from the second database.

[0036] As can be seen from the embodiments of this application, after receiving the ETL job sent by the front end, the relevant information of the ETL job is split, and the data source details and job details obtained after splitting are stored in different databases respectively. In this way, not only can the possibility of ETL job execution failure caused by modifying the data source details be effectively reduced, but the reading speed can also be improved. Especially when a large number of ETL jobs are scheduled concurrently, the pressure on the database can be reduced, the stability of the system can be improved, and the application needs can be better met. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating an ETL data processing method according to an embodiment of this application.

[0038] Figure 2 This application provides a schematic diagram of the structure for storing information after ETL job splitting;

[0039] Figure 3 A flowchart illustrating another ETL data processing method provided in this application embodiment;

[0040] Figure 4 This is a schematic diagram illustrating a process for backing up job details information, provided as an embodiment of this application.

[0041] Figure 5 This is a schematic diagram of the composition structure of the ETL data processing device according to an embodiment of this application;

[0042] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0043] The technical solutions in this application will now be clearly and completely described with reference to the accompanying drawings.

[0044] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments provided herein are merely illustrative of the present application and are not intended to limit the present application. Furthermore, the embodiments provided below are some embodiments for implementing the present application, and not all embodiments for implementing the present application. Unless otherwise specified, the technical solutions described in the present application can be implemented in any combination.

[0045] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly stated, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising a..." does not exclude the presence of other related elements (e.g., steps in the method or units in the apparatus, such as portions of a processor, program, or software, etc.) in the method or apparatus that includes that element.

[0046] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, C and / or D can represent three cases: C exists alone, C and D exist simultaneously, and D exists alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of C, D, and E can mean including any one or more elements selected from the set consisting of C, D, and E.

[0047] For example, the ETL data processing method provided in this application includes a series of steps, but the ETL data processing method provided in this application is not limited to the steps described herein. Similarly, the ETL data processing apparatus provided in this application includes a series of modules, but the ETL data processing apparatus provided in this application is not limited to the modules explicitly described herein, but may also include modules that need to be set up for obtaining relevant information or processing based on information.

[0048] This application can be implemented based on electronic devices, which may be thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, networked personal computers, minicomputer systems, etc.

[0049] Terminal devices, servers, and other electronic devices can achieve corresponding functions through the execution of program modules. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc. They perform specific tasks or implement specific abstract data types. Computer systems can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0050] Currently, common ETL tools typically store ETL jobs in relational databases such as MySQL and Oracle in JavaScript Object Notation (JSON) format. Both the transmission and display of ETL jobs use JSON format. Each time, the scheduling engine reads the JSON file from the relational database and submits it to the job execution engine, which then executes the job. However, relational databases are not suitable for storing JSON data because ETL job files can sometimes be very large, exceeding the length limits of the MySQL text type, and impacting the system's queries per second (QPS). Furthermore, these ETL tools lack version control for workflow files. During transmission, JSON files contain many punctuation marks, and the keys in objects are often identifiers; these are redundant symbols that consume excessive bandwidth and reduce data transmission speed.

[0051] In addition, in related technologies, when storing ETL jobs, the content related to the ETL jobs is usually stored in the same database. However, when there are a large number of ETL jobs to be executed, the above storage method will not only increase the storage pressure on the database, but also affect the stability of the system.

[0052] To address the above problems, the following embodiments are proposed.

[0053] In some embodiments of this application, the ETL data processing method can be implemented using a processor in an ETL data processing device. The processor can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor.

[0054] Figure 1 This is a flowchart illustrating an ETL data processing method according to an embodiment of this application, applied to a backend server, such as... Figure 1As shown, the process may include:

[0055] Step 100: Receive the ETL job sent by the front end.

[0056] Here, the front end, or client, can be a page running on a browser such as a computer or mobile device, used to display content to the user or perform some user interaction operations. In this embodiment of the application, the ETL data processing method can be applied to the back end server, also known as the back end or backend, which refers to a computer system that can provide certain services to the aforementioned front end. The content of the services can be providing resources to the front end, storing front end data, etc.

[0057] In some embodiments, receiving an ETL job sent by the front end may include: receiving a converted ETL job sent by the front end; wherein the converted ETL job is a YAML format that the front end converts into the format of the initial ETL job generated according to the user's operation instructions and then sends to the backend server.

[0058] For example, user operation instructions can be instructions generated after the user performs interactive operations such as dragging, pulling, and dragging on the front-end page; after receiving the user operation instructions, the front-end can generate the corresponding initial ETL job according to the user operation instructions; then, the front-end converts the format of the initial ETL job, for example, it can convert the format of the initial ETL job into yaml (yaml aint markup language) format, and then obtain the yaml format ETL job, that is, the converted ETL job.

[0059] For example, after the front end receives the ETL job in YAML format, it will transmit the ETL job to the back end server. At this time, the back end server receives the ETL job sent by the front end.

[0060] Understandably, compared with JSON format ETL jobs in related technologies, the embodiments of this application convert ETL jobs into YAML format on the front end, which can reduce unnecessary identifiers and save job transmission bandwidth and storage resources.

[0061] Step 101: Obtain relevant information about the ETL job, break down the relevant information of the ETL job, and obtain the broken down information.

[0062] For example, after receiving the ETL job sent by the frontend, the backend server will obtain the relevant information of the ETL job. Here, the relevant information of the ETL job may include data source information related to the ETL job and information related to the ETL job itself. Further, after obtaining the relevant information of the ETL job, the relevant information of the ETL job will be split to obtain the split information.

[0063] For example, the split information may include data source details and job details. Since the main function of an ETL job is to extract, clean, and transform data from one data source and load it into another data source, data source details are an important component of the ETL job. The data source details may include the data source's primary key identity document (ID), Internet Protocol (IP), port, username, and password. The job details may include relevant information required for the subsequent execution of the ETL job, such as job priority, source data source information, and destination data source information.

[0064] In some embodiments, the split information may further include job attribute information, which is information describing the attributes of the ETL job itself, such as the job ID, name, path, type, status information, project group to which it belongs, and primary key of scheduling information.

[0065] In this embodiment of the application, the business scenario for ETL jobs is not limited; it can be understood that for each ETL job obtained under different business scenarios, the corresponding data source details, job details and job attribute information may be different.

[0066] Step 102: Store the data source details in the first database and the job details in the second database; the first database and the second database are different databases.

[0067] For example, the first database can be a relational database, such as MySQL or Oracle. Taking MySQL as an example, the data source details are usually structured data, and MySQL stores data in a structured way. Each table defines each field and stores the data in rows and columns. Therefore, using MySQL to store the data source details will facilitate subsequent reading and querying.

[0068] Understandably, to ensure that the latest data source details are obtained in real time, when storing data source details in a relational database, the primary key ID of the data source and other detailed information (such as the IP address, port, username, and password of the data source) can be stored separately. For example, the primary key ID of the data source can be stored in the relational database, while other detailed information can be stored in a separate metadata management module, which can be accessed through the representational state transfer (REST) ​​interface.

[0069] In some embodiments, since the split information may also include job attribute information, which is usually structured data, this part of the information may also be stored in the first database to facilitate subsequent reading and querying.

[0070] For example, the second database can be an in-memory database, such as Redis or Memcached. Taking Redis as an example, because job details are unstructured information, not only is the data volume large, but it is also frequently accessed. If the job details are stored in a relational database, there will be input / output (I / O) operations every time the database is accessed. If the database is accessed repeatedly and frequently, it will reduce the data retrieval efficiency. Redis, on the other hand, is a cache database used to store frequently used data, thus reducing the number of database accesses and improving operating efficiency. In addition, Redis supports a wider range of data storage types than relational databases, including strings, lists, sets, and sorted sets, which is convenient for storing unstructured data such as job details.

[0071] Understandably, storing ETL job details in a Redis database can improve data retrieval speed, especially when a large number of jobs are scheduled and run concurrently, greatly reducing the pressure on relational databases and improving system stability.

[0072] In some embodiments, the above method may further include: obtaining the current timestamp when storing job details information in the second database; determining the current timestamp as the version information of the ETL job; and adding the version information to the job attribute information.

[0073] For example, the current timestamp is a timestamp generated by the backend server to describe the ETL job. After obtaining the current timestamp, the backend server can use the current timestamp as the version information of the ETL job and add the version information to the job attribute information. In this way, version management of the ETL job can be realized.

[0074] In some embodiments, the above method may further include: obtaining the ID, status information and version information of the ETL job; generating a key value based on the ID, status information and version information of the ETL job, and storing the key value in a first database;

[0075] For example, during the data storage process, the ID, status information, and version information of the ETL job can be obtained from the job attribute information. Then, a key value is generated based on the obtained ETL job ID, status information, and version information. This key value is used to perform read and write operations on the job details information stored in the second database. After obtaining the key value, the key value is stored in the first database.

[0076] In some embodiments, storing job details in a second database may include storing the job details as a value corresponding to a key value in the second database.

[0077] For example, after determining the above key value, the job details information can be stored in the second database as the value corresponding to the key value.

[0078] Understandably, since the information of the data source may be modified, in order to reduce the failure of subsequent ETL jobs due to the modification of the data source information, this application embodiment splits the relevant information of the ETL job and stores the data source details and job details obtained after splitting into different databases. This not only improves the data reading speed, but also effectively increases the probability of successful execution of the ETL job.

[0079] Step 103: Upon receiving an execution request for an ETL job, the execution of the ETL job is completed based on the data source details obtained from the first database and the job details obtained from the second database.

[0080] For example, the user first initiates an execution request for the ETL job on the front end. After the front end receives the execution request, it sends it to the back end server, so that the back end server can receive the execution request.

[0081] For example, the backend server may also include a scheduling engine and an execution engine. After the backend server receives the execution request, it can obtain data source details and job details from the first database and the second database respectively based on the scheduling engine. Then, the execution engine completes the execution of the ETL job based on these two parts of information, which will be explained in detail below.

[0082] In some embodiments, the execution of an ETL job is completed based on data source details obtained from a first database and job details obtained from a second database. This may include: obtaining data source details and a key value from the first database through a scheduling engine, and obtaining job details from the second database based on the key value; and submitting the data source details and job details to the execution engine through the scheduling engine so that the execution engine can complete the execution of the ETL job.

[0083] For example, the scheduling engine first retrieves job details from the second database based on the key value obtained from the first database; then, it retrieves data source details from the first database based on the primary key ID of the data source; after injecting the obtained data source IP, port, username, and password into the job details information in YAML format, it submits it to the execution engine, and finally the execution engine completes the execution of the ETL job.

[0084] In some embodiments, the above method may further include: when storing job details information in the second database, backing up the job details information to a distributed cache.

[0085] For example, to reduce the possibility of ETL job execution failure due to data loss in the Redis database, a timed thread can be started when an execution request for an ETL job is received to read the job details information stored in the second database and save it to a distributed cache. Here, there is no limitation on the distributed cache, for example, it can be a distributed file system (Hadoop Distributed File System, HDFS), etc.

[0086] For example, when saving job details to the distributed cache, to save storage space, the job details can be packaged and compressed, and the previously stored compressed package can be deleted. This way, when the second database service encounters an anomaly, the job details can be retrieved from the distributed cache based on the key value. This effectively ensures the integrity of ETL job information, avoids the loss of ETL job information due to second database failures, and improves the success rate of ETL job execution. Furthermore, when the second database service encounters an anomaly, an alarm can be triggered to notify operations and maintenance personnel for timely handling of the problem.

[0087] This application provides an ETL data processing method, apparatus, electronic device, and computer storage medium. The method is applied to a backend server and includes: receiving an ETL job sent from a frontend; obtaining relevant information about the ETL job; splitting the relevant information to obtain split information; the split information includes data source details and job details; storing the data source details in a first database and the job details in a second database; wherein the first database and the second database are different databases; upon receiving an execution request for the ETL job, executing the ETL job based on the data source details obtained from the first database and the job details obtained from the second database. It can be seen that in this application embodiment, after receiving the ETL job sent from the frontend, the relevant information of the ETL job is split, and the resulting data source details and job details are stored in different databases. This not only effectively reduces the possibility of ETL job execution failure due to modification of data source details, but also improves read speed. Especially when a large number of ETL jobs are scheduled concurrently, it reduces database pressure, improves system stability, and better meets application requirements.

[0088] To better illustrate the purpose of this application, further examples are provided based on the above embodiments.

[0089] Figure 2 This application provides a schematic diagram of the structure for storing information after ETL job splitting, as shown in the embodiment of the present application. Figure 2 As shown, the split information includes data source details, job attribute information, and job details; among them, data source details and job attribute information are stored in a MySQL database, job details are stored in a Redis database, and data source details are backed up to HDFS.

[0090] exist Figure 2 Based on the storage method shown, Figure 3 This application provides a schematic flowchart of another ETL data processing method according to an embodiment. Figure 3 As shown, the process may include:

[0091] Step 200: The front end transmits the ETL job in YAML format to the backend server.

[0092] For example, after the front end receives the generated initial ETL job, it converts the format of the initial ETL job into YAML format and transmits it to the backend server. At the same time, it saves only one data source primary key ID for the data source details in the MySQL database. Other details of the data source are stored in a separate metadata management module, which can be accessed through the REST interface.

[0093] Step 201: Generate version information using timestamps, and generate key values ​​based on the ETL job's ID, status information, and version information.

[0094] For example, when the backend server stores job details, the current timestamp generated by the backend server is used as the version information of the ETL job, and a key value is generated based on the ETL job's ID, status information, and version information.

[0095] Step 202: Store the key value and job attribute information in the MySQL database.

[0096] Step 203: Store the job details in the Redis database.

[0097] Step 204: Before each ETL job is executed, retrieve job details from the Redis database. If a Redis failure is detected, read from HDFS, then inject the latest data source details from the metadata management module before submitting them to the execution engine.

[0098] Figure 4 This application provides a schematic diagram of a process for backing up job details information, as shown in the embodiments of this application. Figure 4 As shown, the process may include:

[0099] Step 300: When the program starts, a timed thread is started to read the job details information from the Redis database.

[0100] For example, to reduce the possibility of ETL job execution failure due to data loss in the Redis database, a timed thread is started when the background server program starts to read the job details information from the Redis database.

[0101] Step 301: Store the job details information in HDFS.

[0102] Step 302: Package and compress the job details information stored in HDFS.

[0103] For example, to save storage space, job details stored in HDFS can be packaged and compressed, that is, stored in the form of a compressed package.

[0104] Step 303: Delete the previously stored compressed file.

[0105] For example, after storing a new compressed file in HDFS, the previously stored compressed file is deleted. In this way, when the in-memory database Redis service is abnormal, job details can be read from HDFS to ensure the successful execution of the ETL job. In addition, an alarm can be triggered to notify the operation and maintenance personnel.

[0106] As can be seen, this embodiment uses YAML to represent ETL jobs, which reduces the size of job files and bandwidth usage for job transmission to a certain extent compared to the many punctuation marks included in JSON format. Furthermore, storing job details in Redis improves read speed, especially when a large number of jobs are scheduled and run concurrently, significantly reducing the pressure on relational databases and improving system stability. Separating data source details from job details ensures that data source details are injected from the metadata management module each time the job is executed, preventing job execution failure due to modifications to data source information. Moreover, using HDFS as a backup storage medium significantly improves job file integrity, preventing job information loss due to Redis database issues, while also addressing the high availability requirements of microservices.

[0107] Figure 5 This is a schematic diagram of the composition structure of the ETL data processing apparatus according to an embodiment of this application, as shown below. Figure 5 As shown, the device includes: a receiving module 400, a splitting module 401, a storage module 402, and an execution module 403, wherein:

[0108] The receiving module 400 is used to receive ETL jobs sent by the front end;

[0109] The splitting module 401 is used to obtain relevant information of the ETL job, split the relevant information of the ETL job, and obtain split information; the split information includes data source details and job details.

[0110] Storage module 402 is used to store the data source details in a first database and the job details in a second database; wherein the first database and the second database are different databases;

[0111] The execution module 403 is used to execute the ETL job based on the data source details obtained from the first database and the job details obtained from the second database when it receives an execution request for the ETL job.

[0112] In some embodiments, the receiving module 400 is configured to receive an ETL job sent by the front end, including:

[0113] The system receives the converted ETL job sent by the front end; the converted ETL job is generated by the front end from the initial ETL job format generated according to the user operation instructions, converted into YAML format, and then sent to the backend server.

[0114] In some embodiments, the split information further includes job attribute information, and the storage module 402 is further configured to:

[0115] When storing the job details information in the second database, obtain the current timestamp;

[0116] The current timestamp is determined as the version information of the ETL job, and the version information is added to the job attribute information.

[0117] In some embodiments, the job attribute information further includes the ETL job's identification number (Id) and status information, and the storage module 402 is further configured to:

[0118] Obtain the ID, status information, and version information of the ETL job;

[0119] A key value is generated based on the ID, status information, and version information of the ETL job, and the key value is stored in the first database;

[0120] The storage module 402 is used to store the job details information to the second database, including:

[0121] The job details are stored in the second database as the value corresponding to the key.

[0122] In some embodiments, the execution module 403 is configured to execute the ETL job based on data source details obtained from the first database and job details obtained from the second database, including:

[0123] The scheduling engine retrieves the data source details and the key value from the first database, and retrieves the job details from the second database based on the key value.

[0124] The scheduling engine submits the data source details and job details to the execution engine, so that the execution engine can complete the execution of the ETL job.

[0125] In some embodiments, the storage module 402 is further configured to:

[0126] When storing the job details information in the second database, the job details information is backed up to a distributed cache.

[0127] In some embodiments, the execution module 403 is further configured to:

[0128] When an anomaly occurs in the second database, the job details information is retrieved from the distributed cache based on the key value.

[0129] In practical applications, the receiving module 400, splitting module 401, storage module 402 and execution module 403 can all be implemented by a processor located in an electronic device. The processor can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller and microprocessor.

[0130] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0131] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] Specifically, the computer program instructions corresponding to an ETL data processing method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the computer program instructions corresponding to an ETL data processing method in the storage media are read or executed by an electronic device, any of the ETL data processing methods in the aforementioned embodiments are implemented.

[0133] Based on the same technical concept as the foregoing embodiments, see Figure 6It illustrates an electronic device 500 provided in an embodiment of this application, which may include: a memory 501 and a processor 502; wherein,

[0134] Memory 501 is used to store computer programs and data;

[0135] The processor 502 is configured to execute a computer program stored in the memory to implement any of the ETL data processing methods described in the foregoing embodiments.

[0136] In practical applications, the aforementioned memory 501 can be volatile memory, such as RAM; or non-volatile memory, such as ROM, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 502.

[0137] The processor 502 described above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It is understood that for different data processing systems, the electronic device used to implement the above processor function can also be other types, and this application embodiment does not specifically limit the specific types.

[0138] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0139] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0140] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0141] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0142] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0143] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0144] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0146] The above are merely preferred embodiments of this application and are not intended to limit the scope of protection of this application.

Claims

1. An ETL data processing method, characterized in that, Applied to a backend server, the method includes: Receive ETL jobs sent from the front end; Obtain relevant information about the ETL job, and then break down the relevant information of the ETL job to obtain the broken down information; the broken down information includes data source details, job details, and job attribute information; The data source details are stored in a first database, and the job details are stored in a second database; wherein the first database and the second database are different databases. Upon receiving an execution request for the ETL job, the execution of the ETL job is completed based on the data source details obtained from the first database and the job details obtained from the second database. The method further includes: when storing the job details information in the second database, obtaining the current timestamp; determining the current timestamp as the version information of the ETL job, and adding the version information to the job attribute information; The job attribute information also includes the ID and status information of the ETL job; the method further includes: generating a key value based on the ID, status information and version information of the ETL job, and storing the key value in the first database; storing the job details information in the second database includes: storing the job details information as the value corresponding to the key value in the second database.

2. The method according to claim 1, characterized in that, The ETL job sent by the receiving front-end includes: The system receives the converted ETL job sent by the front end; the converted ETL job is generated by the front end from the initial ETL job format generated according to the user operation instructions, converted into YAML format, and then sent to the backend server.

3. The method according to claim 1, characterized in that, The step of executing the ETL job based on the data source details obtained from the first database and the job details obtained from the second database includes: The scheduling engine retrieves the data source details and the key value from the first database, and retrieves the job details from the second database based on the key value. The scheduling engine submits the data source details and job details to the execution engine, so that the execution engine can complete the execution of the ETL job.

4. The method according to claim 1 or 3, characterized in that, The method further includes: When storing the job details information in the second database, the job details information is backed up to a distributed cache.

5. The method according to claim 4, characterized in that, The method further includes: When an anomaly occurs in the second database, the job details information is retrieved from the distributed cache based on the key value.

6. An ETL data processing apparatus, characterized in that, The device, applied to a backend server, includes: The receiving module is used to receive ETL jobs sent from the front end; The splitting module is used to obtain relevant information of the ETL job, split the relevant information of the ETL job, and obtain split information; the split information includes data source details, job details, and job attribute information. A storage module is used to store the data source details in a first database and the job details in a second database; wherein the first database and the second database are different databases. An execution module is configured to, upon receiving an execution request for the ETL job, execute the ETL job based on the data source details obtained from the first database and the job details obtained from the second database; The storage module is further configured to obtain the current timestamp when storing the job details information in the second database; determine the current timestamp as the version information of the ETL job; and add the version information to the job attribute information. The job attribute information also includes the ID and status information of the ETL job; the storage module is further configured to generate a key value based on the ID, status information and version information of the ETL job, and store the key value in the first database; and store the job details information as the value corresponding to the key value in the second database.

7. An electronic device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method according to any one of claims 1 to 5.

8. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the method described in any one of claims 1 to 5.

Images