ETL engine switching method and device, computer equipment, storage medium and product
By selecting and translating the initial job configuration to adapt to the target ETL engine during the ETL engine switching process, the problem of low engine switching efficiency is solved, and efficient and accurate engine switching is achieved.
Patent Information
- Application Number
- CN202511760276.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-02-27
AI Technical Summary
Switching between different ETL engines is inefficient, resulting in high time costs.
By selecting the job configuration of the initial ETL engine, translating the job configuration based on the pre-selected target ETL engine, obtaining the target job configuration that the target ETL engine can recognize, and transmitting it to the target ETL engine, the engine switching is realized.
It improves the efficiency and accuracy of ETL engine switching, ensuring that the target job configuration can be adapted to the target ETL engine, thus achieving effective switching.
Smart Images
Figure CN121579577A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of big data collection and analysis technology, and in particular to an ETL engine switching method, apparatus, computer equipment, storage medium and product. Background Technology
[0002] The ETL (Extract, Transform, Load) engine is one of the key components used in the data transformation phase of the ETL process. It is responsible for processing data extracted from different data sources and performing operations such as data cleaning, integration, calculation, and formatting according to predefined transformation rules, and finally loading the processed data into the target system (such as a database, data warehouse, etc.).
[0003] However, when switching between different ETL engines is required, the differences between them result in low switching efficiency, often requiring a significant amount of time. Summary of the Invention
[0004] Therefore, it is necessary to provide an ETL engine switching method, apparatus, computer equipment, storage medium, and product that can efficiently complete ETL engine switching in response to the above-mentioned technical problems.
[0005] Firstly, this application provides an ETL engine switching method. The method includes:
[0006] Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine;
[0007] Based on the pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain a target job configuration that can be recognized by the target ETL engine.
[0008] The target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0009] In one embodiment, the step of translating the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine includes:
[0010] Based on a pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain a candidate job configuration.
[0011] The validity of the candidate job configuration is verified;
[0012] If the validity verification result is passed, the candidate job configuration will be used as the target job configuration that the target ETL engine can recognize.
[0013] In one embodiment, the step of translating the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine includes:
[0014] Based on a pre-selected target ETL engine, the initial job configuration is translated multiple times to obtain at least two candidate job configurations.
[0015] Select the target job configuration that has the highest compatibility with the target ETL engine from all the candidate job configurations.
[0016] In one embodiment, based on a pre-selected target ETL engine, the initial job configuration is translated multiple times to obtain at least two candidate job configurations, including:
[0017] At least two pre-trained artificial intelligence translation models are used to translate the initial job configuration based on a pre-selected target ETL engine, resulting in candidate job configurations output by each artificial intelligence translation model.
[0018] In one embodiment, the step of translating the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine includes:
[0019] If a translation template corresponding to the target ETL engine exists, obtain the translation template corresponding to the target ETL engine;
[0020] Based on the translation template, the initial job configuration is translated to obtain a target job configuration that can be recognized by the target ETL engine.
[0021] In one embodiment, the step of transferring the target job configuration to the target ETL engine to switch the initial ETL engine to the target ETL engine includes:
[0022] If a user requests an adjustment to the target job configuration, the target job configuration is adjusted based on the adjustment content in the request.
[0023] The adjusted target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0024] Secondly, this application also provides an ETL engine switching device. The device includes:
[0025] The selection module is used to select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine.
[0026] The translation module is used to translate the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration that can be recognized by the target ETL engine.
[0027] The transmission module is used to transmit the target job configuration to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0028] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0029] Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine;
[0030] Based on the pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain a target job configuration that can be recognized by the target ETL engine.
[0031] The target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0032] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:
[0033] Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine;
[0034] Based on the pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain a target job configuration that can be recognized by the target ETL engine.
[0035] The target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0036] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:
[0037] Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine;
[0038] Based on the pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain a target job configuration that can be recognized by the target ETL engine.
[0039] The target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0040] The aforementioned ETL engine switching method, apparatus, computer equipment, storage medium, and product, by selecting an initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine, achieves job configuration translation of the initial job configuration based on a pre-selected target ETL engine, obtaining a target job configuration recognizable by the target ETL engine, and then transmitting the target job configuration to the target ETL engine, thereby achieving the switching from the initial ETL engine to the target ETL engine. As can be seen from the above, during the ETL engine switching process, this application pre-selects an initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine, achieving the acquisition of relevant information about the initial ETL engine, ensuring the smooth progress of subsequent job configuration translation of the initial job configuration. Furthermore, by translating the initial job configuration based on the pre-selected target ETL engine, a configuration conversion is achieved for the initial job configuration, ensuring that the target job configuration can adapt to the target ETL engine, realizing an effective switch from the initial ETL engine to the target ETL engine, improving the efficiency of ETL engine switching, and ensuring the accuracy of ETL engine switching. Attached Figure Description
[0041] Figure 1 An application environment diagram for an ETL engine switching method provided in this application embodiment;
[0042] Figure 2 A flowchart illustrating the first ETL engine switching method provided in this application embodiment;
[0043] Figure 3 A flowchart illustrating the second ETL engine switching method provided in this application embodiment;
[0044] Figure 4 A flowchart illustrating the third ETL engine switching method provided in this application embodiment;
[0045] Figure 5 A flowchart illustrating the fourth ETL engine switching method provided in this application embodiment;
[0046] Figure 6 A flowchart illustrating the fifth ETL engine switching method provided in this application embodiment;
[0047] Figure 7A structural block diagram of an ETL engine switching device provided in an embodiment of this application;
[0048] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0050] The ETL engine switching method provided in this application embodiment can be applied to, for example, Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. By selecting the initial job configuration corresponding to the initial extraction and transformation loading ETL engine, the initial job configuration is translated based on the pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine. Then, the target job configuration is transmitted to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster composed of multiple servers.
[0051] In one embodiment, such as Figure 2 As shown, an ETL engine switching method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:
[0052] S201, Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine.
[0053] It should be noted that when it is necessary to select the initial job configuration corresponding to the initial extraction, transformation and loading ETL engine, the following can be included: after determining the initial ETL engine, query the job list under the initial ETL engine, select the ETL job from the job list, and obtain the configuration of the ETL job to obtain the initial job configuration, thereby realizing the steps of selecting the initial job configuration corresponding to the initial extraction, transformation and loading ETL engine.
[0054] Furthermore, the Target ETL engine can be selected to enable a detailed preview of the initial job configuration.
[0055] In one embodiment of this application, when it is necessary to select an ETL job from the job list, the job type of each job can be obtained in advance; then, according to the actual demand type of the target ETL engine, the job with a high degree of similarity to the actual demand type is selected as the ETL job corresponding to the initial job configuration.
[0056] In another embodiment of this application, when it is necessary to select an ETL job from the job list, a random selection method can be used to select any ETL job as the ETL job corresponding to the initial job configuration.
[0057] S202, based on the pre-selected target ETL engine, translates the initial job configuration into a job configuration that can be recognized by the target ETL engine.
[0058] It should be noted that when it is necessary to translate the initial job configuration based on a pre-selected target ETL engine, there are several possible methods. Specifically: The initial job configuration can be translated to obtain candidate job configurations, and then the target job configuration can be determined by validating the candidate configurations; alternatively, the initial job configuration can be translated multiple times to determine the target job configuration; or the target job configuration can be determined based on the translation template corresponding to the target ETL engine. In summary, there are many methods for translating the initial job configuration. The following will explain the three methods mentioned above:
[0059] In one embodiment of this application, based on a pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain a candidate job configuration; the candidate job configuration is validated for validity; if the validity validation result is passed, the candidate job configuration is used as the target job configuration that can be recognized by the target ETL engine.
[0060] In another embodiment of this application, based on a pre-selected target ETL engine, the initial job configuration is translated multiple times to obtain at least two candidate job configurations; the target job configuration with the highest compatibility with the target ETL engine is selected from each candidate job configuration.
[0061] In another embodiment of this application, if a translation template corresponding to the target ETL engine exists, the translation template corresponding to the target ETL engine is obtained; based on the translation template, the initial job configuration is translated to obtain a target job configuration that can be recognized by the target ETL engine.
[0062] S203, the target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0063] In one embodiment of this application, when it is necessary to transfer the target job configuration to the target ETL engine, the target job configuration can be transferred to the target ETL engine by submitting an API to the target ETL engine.
[0064] It should be noted that when it is necessary to transfer the target job configuration to the target ETL engine, the following may also be included: if a user's adjustment request for the target job configuration is received, the target job configuration is adjusted based on the adjustment content in the adjustment request; the adjusted target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0065] The aforementioned ETL engine switching method selects the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine; it then translates the initial job configuration based on the pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine, and finally transmits the target job configuration to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine. As can be seen from the above, this application, during the ETL engine switching process, pre-selects the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine, achieving the acquisition of relevant information about the initial ETL engine, ensuring the smooth progress of subsequent job configuration translation of the initial job configuration. Furthermore, based on the pre-selected target ETL engine, it performs job configuration translation of the initial job configuration, achieving configuration transformation for the initial job configuration, thus ensuring that the target job configuration can adapt to the target ETL engine. This achieves an effective switch from the initial ETL engine to the target ETL engine, improving the efficiency and accuracy of ETL engine switching.
[0066] In one embodiment, such as Figure 3 As shown, when it is necessary to translate the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration that can be recognized by the target ETL engine, the following can be included:
[0067] S301, based on the pre-selected target ETL engine, translates the initial job configuration into a job configuration to obtain candidate job configurations.
[0068] It should be noted that when it is necessary to translate the initial job configuration, a pre-trained translation model can be invoked. Then, by inputting the initial job configuration into the translation model and obtaining the output result of the translation model, the output result is the candidate job configuration.
[0069] The training process of the translation model includes: obtaining at least one first sample job configuration corresponding to the initial ETL engine, and determining the second sample job configuration corresponding to each first sample job configuration through manual annotation and other means; wherein each second job configuration is a job configuration that can be recognized by the target ETL engine. Therefore, the initial model is trained by each first job configuration and the second job configuration corresponding to each first job configuration to obtain the trained translation model.
[0070] To further explain, when it is necessary to translate the initial job configuration, a pre-registered artificial intelligence translation model can be invoked. Then, by inputting the initial job configuration into the artificial intelligence translation model and obtaining the output result of the artificial intelligence translation model, the output result is the candidate job configuration.
[0071] The registration process for AI translation models includes obtaining registration information for at least one AI translation model, which includes model identity information, and registering each AI translation model separately using the registration information of each AI translation model.
[0072] S302, validate the validity of candidate job configurations.
[0073] It should be noted that when it is necessary to verify the validity of candidate job configurations, the candidate job configurations can be displayed so that users can perform configuration checks on them. If the check is correct, a validity verification pass command is obtained from the user through the terminal, and the candidate job configuration validity verification result is determined to be passed. If the check is incorrect, a validity verification fail command is obtained from the user through the terminal, and the candidate job configuration validity verification result is determined to be failed.
[0074] S303: If the validity verification result is passed, the candidate job configuration will be used as the target job configuration that the target ETL engine can recognize.
[0075] It should be noted that if the validity verification result is passed, it indicates that the candidate job configuration is a job configuration that can be effectively recognized by the target ETL engine. Therefore, the candidate job configuration is regarded as the target job configuration that the target ETL engine can recognize.
[0076] The above-described ETL engine switching method translates the initial job configuration, thereby achieving configuration conversion for the initial job configuration. This ensures that the target job configuration can adapt to the target ETL engine, realizing an effective switch from the initial ETL engine to the target ETL engine, improving the efficiency of ETL engine switching, and ensuring the accuracy of ETL engine switching.
[0077] In one embodiment, such as Figure 4 As shown, when it is necessary to translate the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration that can be recognized by the target ETL engine, the following can be included:
[0078] S401, based on a pre-selected target ETL engine, performs multiple job configuration translations on the initial job configuration to obtain at least two candidate job configurations.
[0079] In one embodiment of this application, when it is necessary to perform multiple job configuration translations on the initial job configuration to obtain at least two candidate job configurations, the following may be included: using at least two pre-trained artificial intelligence translation models, based on a pre-selected target ETL engine, to perform job configuration translations on the initial job configuration respectively, to obtain the candidate job configuration output by each artificial intelligence translation model.
[0080] Specifically, the ETL type of the target ETL engine can be obtained in advance. Then, based on the ETL type, at least two pre-trained artificial intelligence translation models are used to translate the initial job configuration based on the pre-selected target ETL engine, so as to obtain the candidate job configuration that conforms to the ETL type output by each artificial intelligence translation model.
[0081] S402: Select the target job configuration that is most compatible with the target ETL engine from among the candidate job configurations.
[0082] In one embodiment of this application, when it is necessary to select the target job configuration with the highest compatibility with the target ETL engine from among the candidate job configurations, an analysis model can be pre-trained, and each candidate job configuration can be input into the analysis model. The output result of the analysis model can be obtained, and the output result is the target job configuration with the highest compatibility with the target ETL engine.
[0083] In another embodiment of this application, when it is necessary to select the target job configuration with the highest compatibility with the target ETL engine from among the candidate job configurations, each candidate job configuration can also be displayed to the user through a display screen, so that the user can evaluate their own experience and actual needs to select the target job configuration with the highest compatibility with the target ETL engine from among the candidate job configurations.
[0084] As an example, when switching initial ETL engines, the following steps can be taken: Launch the job management interface of the ETL tool to enter the initial scenario of job configuration and execution; In the tool's engine list, click the initial ETL engine; In the types supported by the ETL engine, select the target type (such as "Batch Job," "Streaming Job," "Real-time Job," etc.). In the tool's "Job Management" module, browse the job list under the TLI engine (assuming it's a specific engine, such as "Target LoadIntegration"). In the job list, select the target ETL job (such as "Data Cleaning Job," "Report Generation Job," etc.), and determine the initial job configuration corresponding to the target ETL job; In the job configuration interface, click the "Job Translation" button (or menu item). In the tool's "AI Model Library," select the target large model. In the tool interface, click the "New Window" button and select the "Processing Window" type. In the processing window, click the "Output Process" button (or trigger automatically). In the tool interface, click the "Multi-Window Display" button and select multiple large AI models. In the tool's "AI Model Selection" module, switch to other large AI models to obtain multiple candidate job configurations. After comparing the results of multiple models, and considering business needs and model performance (such as accuracy and efficiency), select the target job configuration that best matches the target ETL engine from among the candidate configurations. In the job configuration interface, modify parameters such as "Input Source, Transformation Rules, and Output Target" (e.g., changing the data source or adjusting cleaning rules). After modifying the configuration, click the "Confirm" or "Cancel" button. Finally, transfer the target job configuration to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0085] The above-described ETL engine switching method translates the initial job configuration, thereby achieving configuration conversion for the initial job configuration. This ensures that the target job configuration can adapt to the target ETL engine, realizing an effective switch from the initial ETL engine to the target ETL engine, improving the efficiency of ETL engine switching, and ensuring the accuracy of ETL engine switching.
[0086] In one embodiment, such as Figure 5 As shown, when it is necessary to translate the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration that can be recognized by the target ETL engine, the following can be included:
[0087] S501, if a translation template corresponding to the target ETL engine exists, obtain the translation template corresponding to the target ETL engine.
[0088] In one embodiment of this application, the existence of a translation template can be pre-queried. Specifically, a template query for the target ETL engine can be performed in the translation template library to determine the translation template corresponding to the target ETL engine.
[0089] S502, based on the translation template, translates the initial job configuration into a job configuration that can be recognized by the target ETL engine.
[0090] In one embodiment of this application, the initial job configuration is translated into a job configuration by filling the corresponding position in the translation template, thereby translating the initial job configuration into a target job configuration that can be recognized by the target ETL engine.
[0091] In one embodiment, when an initial ETL engine switch is required, the job management interface of the ETL tool is launched, entering the initial scenario of job configuration and execution. In the tool's engine list, the SourceETL engine (i.e., the initial ETL engine) is selected. The tool's "Engine Information Query" function is used to obtain information such as the Source ETL engine's connection parameters, supported job types, and component capabilities. In the tool's "Job Management" module, the job list under the Source ETL engine is browsed. In the job list, the target ETL job is clicked (e.g., "MySQL CDC To Doris," used to synchronize changed MySQL data to Doris). In the tool's "Target Engine Selection" module, the target ETL engine is clicked (e.g., Flink CDC, used for streaming changed data). In the job configuration interface, the "AI Translation Conversion" button (or menu item) is clicked. The tool's "Translation Template Library" is used to check if a translation template for "Initial ETL → Target ETL" (e.g., the MySQL CDC → Flink CDC template) exists. Retrieve the corresponding translation template from the template library to complete the mapping from "Source ETL configuration to Target ETL configuration." Use the tool's "API query" function to call the Source ETL engine's configuration query API to obtain the job's detailed configuration parameters. Transfer the target job configuration to the target ETL engine to switch the initial ETL engine to the target ETL engine. Package the translated job configuration, translation rules, version information, etc., and store them in the tool's "Translation Template Library."
[0092] The above-described ETL engine switching method translates the initial job configuration, thereby achieving configuration conversion for the initial job configuration. This ensures that the target job configuration can adapt to the target ETL engine, realizing an effective switch from the initial ETL engine to the target ETL engine, improving the efficiency of ETL engine switching, and ensuring the accuracy of ETL engine switching.
[0093] In one embodiment, such as Figure 6 As shown, when it is necessary to switch from the initial ETL engine to the target ETL engine, the following may be included;
[0094] S601, Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine.
[0095] S602 employs at least two pre-trained artificial intelligence translation models, based on a pre-selected target ETL engine, to perform job configuration translation on the initial job configuration, thereby obtaining the candidate job configuration output by each artificial intelligence translation model.
[0096] S603 selects the target job configuration with the highest compatibility with the target ETL engine from among the candidate job configurations.
[0097] S604 If a user requests an adjustment to the target job configuration, the target job configuration is adjusted based on the adjustment content in the request.
[0098] S605 transmits the adjusted target job configuration to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0099] The aforementioned ETL engine switching method selects the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine; it then translates the initial job configuration based on the pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine, and finally transmits the target job configuration to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine. As can be seen from the above, this application, during the ETL engine switching process, pre-selects the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine, achieving the acquisition of relevant information about the initial ETL engine, ensuring the smooth progress of subsequent job configuration translation of the initial job configuration. Furthermore, based on the pre-selected target ETL engine, it performs job configuration translation of the initial job configuration, achieving configuration transformation for the initial job configuration, thus ensuring that the target job configuration can adapt to the target ETL engine. This achieves an effective switch from the initial ETL engine to the target ETL engine, improving the efficiency and accuracy of ETL engine switching.
[0100] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0101] Based on the same inventive concept, this application also provides an ETL engine switching device for implementing the ETL engine switching method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more ETL engine switching device embodiments provided below can be found in the limitations of the ETL engine switching method described above, and will not be repeated here.
[0102] In one embodiment, such as Figure 7 As shown, an ETL engine switching device is provided, including: a selection module 10, a translation module 20, and a transmission module 30, wherein:
[0103] Select module 10, which is used to select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine.
[0104] Translation module 20 is used to translate the initial job configuration based on the pre-selected target ETL engine to obtain a target job configuration that can be recognized by the target ETL engine.
[0105] The transmission module 30 is used to transmit the target job configuration to the target ETL engine, thereby enabling the initial ETL engine to be switched to the target ETL engine.
[0106] In one embodiment, based on a pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain a candidate job configuration.
[0107] Verify the validity of candidate job configurations;
[0108] If the validity verification result is passed, the candidate job configuration will be used as the target job configuration that the target ETL engine can recognize.
[0109] In one embodiment, based on a pre-selected target ETL engine, the initial job configuration is translated multiple times to obtain at least two candidate job configurations.
[0110] Select the target job configuration that best matches the target ETL engine from among the candidate job configurations.
[0111] In one embodiment, at least two pre-trained AI translation models are used to translate the initial job configuration based on a pre-selected target ETL engine, resulting in candidate job configurations output by each AI translation model. In another embodiment, if a translation template corresponding to the target ETL engine exists, the translation template corresponding to the target ETL engine is obtained.
[0112] Based on the translation template, the initial job configuration is translated to obtain the target job configuration that can be recognized by the target ETL engine.
[0113] In one embodiment, if a user requests an adjustment to the configuration of a target job, the configuration of the target job is adjusted based on the adjustment content in the adjustment request.
[0114] The adjusted target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0115] The aforementioned ETL engine switching device selects an initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine; it then translates the initial job configuration based on the pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine, and finally transmits the target job configuration to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine. As can be seen from the above, this application, during the ETL engine switching process, pre-selects the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine, achieving the acquisition of relevant information about the initial ETL engine, ensuring the smooth progress of subsequent job configuration translation of the initial job configuration. Furthermore, based on the pre-selected target ETL engine, it performs job configuration translation of the initial job configuration, achieving configuration transformation for the initial job configuration, thereby ensuring that the target job configuration can adapt to the target ETL engine, realizing an effective switch from the initial ETL engine to the target ETL engine, improving the efficiency of ETL engine switching, and ensuring the accuracy of ETL engine switching.
[0116] The modules in the aforementioned ETL engine switching device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.
[0117] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an ETL engine switching method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0118] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0119] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0120] Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine;
[0121] Based on the pre-selected target ETL engine, the initial job configuration is translated into a target job configuration that can be recognized by the target ETL engine.
[0122] The target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0123] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0124] Based on a pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain candidate job configurations.
[0125] Verify the validity of candidate job configurations;
[0126] If the validity verification result is passed, the candidate job configuration will be used as the target job configuration that the target ETL engine can recognize.
[0127] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0128] Based on the pre-selected target ETL engine, the initial job configuration is translated multiple times to obtain at least two candidate job configurations;
[0129] Select the target job configuration that best matches the target ETL engine from among the candidate job configurations.
[0130] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0131] At least two pre-trained AI translation models are used to translate the initial job configuration based on a pre-selected target ETL engine, resulting in candidate job configurations output by each AI translation model. In one embodiment, the processor further implements the following steps when executing the computer program:
[0132] If a translation template for the target ETL engine exists, obtain the translation template for the target ETL engine.
[0133] Based on the translation template, the initial job configuration is translated to obtain the target job configuration that can be recognized by the target ETL engine.
[0134] In one embodiment, the processor, when executing a computer program, also performs the following steps:
[0135] If a user requests an adjustment to the target job configuration, the target job configuration will be adjusted based on the adjustment content in the request.
[0136] The adjusted target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0137] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:
[0138] Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine;
[0139] Based on the pre-selected target ETL engine, the initial job configuration is translated into a target job configuration that can be recognized by the target ETL engine.
[0140] The target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0141] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0142] Based on a pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain candidate job configurations.
[0143] Verify the validity of candidate job configurations;
[0144] If the validity verification result is passed, the candidate job configuration will be used as the target job configuration that the target ETL engine can recognize.
[0145] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0146] Based on the pre-selected target ETL engine, the initial job configuration is translated multiple times to obtain at least two candidate job configurations;
[0147] Select the target job configuration that best matches the target ETL engine from among the candidate job configurations.
[0148] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0149] At least two pre-trained AI translation models are used to translate the initial job configuration based on a pre-selected target ETL engine, resulting in candidate job configurations output by each AI translation model. In one embodiment, the computer program, when executed by a processor, further implements the following steps:
[0150] If a translation template for the target ETL engine exists, obtain the translation template for the target ETL engine.
[0151] Based on the translation template, the initial job configuration is translated to obtain the target job configuration that can be recognized by the target ETL engine.
[0152] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0153] If a user requests an adjustment to the target job configuration, the target job configuration will be adjusted based on the adjustment content in the request.
[0154] The adjusted target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0155] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:
[0156] Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine;
[0157] Based on the pre-selected target ETL engine, the initial job configuration is translated into a target job configuration that can be recognized by the target ETL engine.
[0158] The target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0159] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0160] Based on a pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain candidate job configurations.
[0161] Verify the validity of candidate job configurations;
[0162] If the validity verification result is passed, the candidate job configuration will be used as the target job configuration that the target ETL engine can recognize.
[0163] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0164] Based on the pre-selected target ETL engine, the initial job configuration is translated multiple times to obtain at least two candidate job configurations;
[0165] Select the target job configuration that best matches the target ETL engine from among the candidate job configurations.
[0166] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0167] At least two pre-trained AI translation models are used to translate the initial job configuration based on a pre-selected target ETL engine, resulting in candidate job configurations output by each AI translation model. In one embodiment, the computer program, when executed by a processor, further implements the following steps:
[0168] If a translation template for the target ETL engine exists, obtain the translation template for the target ETL engine.
[0169] Based on the translation template, the initial job configuration is translated to obtain the target job configuration that can be recognized by the target ETL engine.
[0170] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:
[0171] If a user requests an adjustment to the target job configuration, the target job configuration will be adjusted based on the adjustment content in the request.
[0172] The adjusted target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0174] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0175] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0176] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for switching ETL engines, characterized in that, The method includes: Select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine; Based on the pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain a target job configuration that can be recognized by the target ETL engine. The target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
2. The method according to claim 1, characterized in that, The step of translating the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine includes: Based on a pre-selected target ETL engine, the initial job configuration is translated into a job configuration to obtain a candidate job configuration. The validity of the candidate job configuration is verified; If the validity verification result is passed, the candidate job configuration will be used as the target job configuration that the target ETL engine can recognize.
3. The method according to claim 1, characterized in that, The step of translating the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine includes: Based on a pre-selected target ETL engine, the initial job configuration is translated multiple times to obtain at least two candidate job configurations. Select the target job configuration that has the highest compatibility with the target ETL engine from all the candidate job configurations.
4. The method according to claim 3, characterized in that, Based on a pre-selected target ETL engine, the initial job configuration is translated multiple times to obtain at least two candidate job configurations, including: At least two pre-trained artificial intelligence translation models are used to translate the initial job configuration based on a pre-selected target ETL engine, resulting in candidate job configurations output by each artificial intelligence translation model.
5. The method according to claim 1, characterized in that, The step of translating the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration recognizable by the target ETL engine includes: If a translation template corresponding to the target ETL engine exists, obtain the translation template corresponding to the target ETL engine; Based on the translation template, the initial job configuration is translated to obtain a target job configuration that can be recognized by the target ETL engine.
6. The method according to any one of claims 1-5, characterized in that, The step of transferring the target job configuration to the target ETL engine to switch the initial ETL engine to the target ETL engine includes: If a user requests an adjustment to the target job configuration, the target job configuration is adjusted based on the adjustment content in the request. The adjusted target job configuration is transferred to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
7. An ETL engine switching device, characterized in that, The device includes: The selection module is used to select the initial job configuration corresponding to the initial extraction, transformation, and loading ETL engine. The translation module is used to translate the initial job configuration based on a pre-selected target ETL engine to obtain a target job configuration that can be recognized by the target ETL engine. The transmission module is used to transmit the target job configuration to the target ETL engine, thereby switching the initial ETL engine to the target ETL engine.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.