Method and apparatus for splitting ETL files

Abstract

The application discloses an ETL file splitting method and device, and relates to the technical field of automatic programming, wherein the method comprises the following steps: extracting system setting information of a plurality of ETL jobs in an ETL file according to the content of the ETL file; extracting processing configuration information of the plurality of ETL jobs in the ETL file according to the content of the ETL file; and splicing the system setting information and the processing configuration information of each ETL job into a separate ETL file. The application relates to the technical field of automatic programming, and the embodiment of the application extracts the system setting information and the processing configuration information of the plurality of ETL jobs in the ETL file, splices the system setting information and the processing configuration information of each ETL job into a separate ETL file, thereby effectively realizing the splitting of the ETL file and facilitating the operation and maintenance of users.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic programming, and particularly relates to an ETL file splitting method and device. BACKGROUND

[0002] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein does not constitute admission of prior art.

[0003] ETL is the abbreviation of English Extract-Transform-Load, which is used to describe the process of extracting data from the source end, transforming and loading to the destination end. The purpose is to integrate the scattered, disordered and non-uniform standard data in the enterprise together to provide analysis basis for the decision of the enterprise, and ETL is an important link of BI (business intelligence) project.

[0004] The process of ETL can be developed by using any programming language, since ETL is a very complex process, and it is not easy to manage the handwritten program, so more and more enterprises use tools to assist the development of ETL, and use the built-in metadata function to store the mapping and conversion rules of the source and the destination. For example, DataStage is a kind of data integration software platform of IBM company, which helps to carry out ETL processing, and DSJOB is the ETL job generated by DataStage software. In practical application, different scenes and applications will carry out multiple different ETL processing. However, when these ETL processing is exported, all the ETL jobs will be generated under the same ETL file, which is not convenient for the user to operate and maintain.

[0005] The prior art provides a merging and splitting processing method of a configuration file, which comprises the following steps: marking configuration items corresponding to each host role in a software system according to the host role identification of each host role in advance; when it is required to write the configuration file into a production environment, performing merging processing on the configuration items contained in the configuration file corresponding to each host role in the software system, to generate a configuration file corresponding to the host; and when it is required to collect the configuration file from the production environment, performing splitting processing on the configuration items contained in the configuration file of the host according to a role list of the host, to generate a configuration file corresponding to each host role. In the case where different host roles in the software system are deployed on the same host, the method can merge the configuration files of multiple host roles in real time according to the host role division of the production environment, or can split the configuration items of different host roles in real time according to the host role division of the production environment. However, the method is only applicable to simple configuration file splitting, and is only applicable to splitting the configuration files of different hosts, and is not applicable to splitting complex ETL files. SUMMARY

[0006] The embodiment of the present application provides an ETL file splitting method, which is used for effectively splitting an ETL file, and the method comprises the following steps:

[0007] extracting system setting information of multiple ETL jobs in the ETL file according to the content of the ETL file;

[0008] extracting processing configuration information of the multiple ETL jobs in the ETL file according to the content of the ETL file;

[0009] splicing the system setting information and the processing configuration information of each ETL job into a separate ETL file.

[0010] In the embodiment, the system setting information of the multiple ETL jobs in the ETL file is extracted according to the content of the ETL file, which comprises the following steps: in the form of a Python script, the system setting information of the multiple ETL jobs in the ETL file is recognized according to the content of the ETL file.

[0011] In addition, the processing configuration information of the multiple ETL jobs in the ETL file is extracted according to the content of the ETL file, which comprises the following steps: in the form of a Python script, the processing configuration information of the multiple ETL jobs in the ETL file is recognized according to the content of the ETL file.

[0012] In the embodiment, before the system setting information and the processing configuration information of the multiple ETL jobs in the ETL file are extracted, the following steps are further included:

[0013] judging whether the format of the ETL file is correct;

[0014] When the format of the ETL file is incorrect, the ETL file is re-read.

[0015] In an embodiment, the system setting information includes one or any combination of the following information:

[0016] The export tool information, tool version, host name, tool project name, MDI version, system time, and server version are exported.

[0017] In an embodiment, the system setting information of the plurality of ETL jobs in the ETL file is extracted according to the content of the ETL file, including:

[0018] The system setting information of the plurality of ETL jobs in the ETL file is found by keyword.

[0019] In an embodiment, the processing configuration information of the plurality of ETL jobs in the ETL file is extracted according to the content of the ETL file, including:

[0020] The processing configuration information of the plurality of ETL jobs in the ETL file is found by keyword.

[0021] In an embodiment, after the system setting information of the plurality of ETL jobs in the ETL file is extracted according to the content of the ETL file, the method further includes:

[0022] The extracted system setting information of the plurality of ETL jobs is stored in a first temporary text file;

[0023] After the processing configuration information of the plurality of ETL jobs in the ETL file is extracted according to the content of the ETL file, the method further includes:

[0024] The extracted processing configuration information of the plurality of ETL jobs is stored in a second temporary text file;

[0025] The system setting information and the processing configuration information of each ETL job are spliced into a separate ETL file, including:

[0026] The system setting information of each ETL job is extracted from the first temporary text file, and the processing configuration information of each ETL job is extracted from the second temporary text file.

[0027] In an embodiment, after the processing configuration information of the ETL job is extracted, the method further includes:

[0028] The name of the ETL job is extracted from the processing configuration information of the ETL job;

[0029] The ETL file name corresponding to the ETL job is determined according to the name of the ETL job.

[0030] The embodiment of the present application also provides an ETL file splitting device for effectively splitting an ETL file.

[0031] A system setting information extraction module is configured to extract system setting information of a plurality of ETL jobs in the ETL file according to the content of the ETL file.

[0032] A processing configuration information extraction module is configured to extract processing configuration information of the plurality of ETL jobs in the ETL file according to the content of the ETL file.

[0033] An ETL file splicing module is configured to splice the system setting information and the processing configuration information of each ETL job into a separate ETL file.

[0034] In the embodiment, the system setting information extraction module is specifically configured to recognize the system setting information of the plurality of ETL jobs in the ETL file according to the content of the ETL file in the form of a Python script.

[0035] And / or, the processing configuration information extraction module is specifically configured to recognize the processing configuration information of the plurality of ETL jobs in the ETL file according to the content of the ETL file in the form of a Python script.

[0036] In the embodiment, the device further comprises a file reading module configured to read the ETL file, judge whether the format of the ETL file is correct, and re-read the ETL file when the format of the ETL file is incorrect.

[0037] In the embodiment, the system setting information in the system setting information extraction module comprises one or any combination of the following information:

[0038] Export tool information, tool version, host name, tool project name, MDI version, system time and server version.

[0039] In the embodiment, the system setting information extraction module is specifically configured to:

[0040] Find the system setting information of the plurality of ETL jobs in the ETL file through a keyword.

[0041] In the embodiment, the processing configuration information extraction module is specifically configured to:

[0042] Find the processing configuration information of the plurality of ETL jobs in the ETL file through a keyword.

[0043] In the embodiment, the device further comprises a storage module configured to store the extracted system setting information of the plurality of ETL jobs in a first temporary text file and store the extracted processing configuration information of the plurality of ETL jobs in a second temporary text file.

[0044] The ETL file splicing module is specifically used for:

[0045] The system setting information of each ETL job is extracted from the first temporary text file respectively, and the processing configuration information of each ETL job is extracted from the second temporary text file respectively.

[0046] In the embodiment, the processing configuration information extraction module is further specifically used for:

[0047] The name of the ETL job is extracted from the processing configuration information of the ETL job.

[0048] The ETL file name corresponding to the ETL job is determined according to the name of the ETL job.

[0049] The embodiment of the present application further provides a computer device for effectively implementing the splitting of the ETL file, comprising a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the above-mentioned ETL file splitting method when executing the computer program.

[0050] The embodiment of the present application further provides a computer readable storage medium for effectively implementing the splitting of the ETL file, and the computer readable storage medium stores the computer program for executing the above-mentioned ETL file splitting method.

[0051] In the embodiment of the present application, the system setting information of multiple ETL jobs in the ETL file is extracted respectively according to the content of the ETL file, the processing configuration information of multiple ETL jobs in the ETL file is extracted respectively according to the content of the ETL file, and the system setting information and the processing configuration information of each ETL job are spliced into a separate ETL file. Compared with the prior art in which all ETL jobs are generated in the same ETL file when ETL processing is exported, the system setting information and the processing configuration information of multiple ETL jobs in the ETL file are extracted respectively, the system setting information and the processing configuration information of each ETL job are spliced into a separate ETL file, thereby effectively implementing the splitting of the ETL file and facilitating the operation and maintenance of the user. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0053] Figure 1A processing flowchart of the ETL file splitting method in the embodiment of the present application is shown in the figure.

[0054] Figure 2 A general flowchart of the ETL file splitting method in the embodiment of the present application is shown in the figure.

[0055] Figure 3 A specific flowchart of the system setting information extraction of the multiple ETL jobs in the embodiment of the present application is shown in the figure.

[0056] Figure 4 A specific flowchart of the processing configuration information extraction of the multiple ETL jobs in the embodiment of the present application is shown in the figure.

[0057] Figure 5 A structural schematic diagram of the ETL file splitting device in the embodiment of the present application is shown in the figure.

[0058] Figure 6 A structural schematic diagram of a specific example of the ETL file splitting device in the embodiment of the present application is shown in the figure.

[0059] Figure 7 A structural schematic diagram of another specific example of the ETL file splitting device in the embodiment of the present application is shown in the figure.

[0060] Figure 8 A structural schematic diagram of the computer device of an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0061] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer and more apparent, the embodiments of the present application are further described in detail below with reference to the drawings. Herein, the illustrative embodiments of the present application and the descriptions thereof are used to explain the present application but not as a limitation of the present application.

[0062] To solve the problem that all the ETL jobs are generated under the same ETL file when the existing technology concentrates on exporting ETL processing, which is inconvenient for users to operate and maintain, the embodiments of the present application respectively extract the system setting information and the processing configuration information of the multiple ETL jobs in the ETL file, splice the system setting information and the processing configuration information of each ETL job into a separate ETL file, thereby effectively splitting the ETL file and facilitating users to operate and maintain.

[0063] Figure 1 A processing flowchart of the ETL file splitting method in the embodiment of the present application is shown in the figure. Figure 1 As shown in the figure, the ETL file splitting method in the embodiment of the present application can include:

[0064] Step 101, according to the content of the ETL file, respectively extract the system setting information of the multiple ETL jobs in the ETL file;

[0065] Step 102: Based on the contents of the ETL file, extract the processing configuration information of multiple ETL jobs from the ETL file;

[0066] Step 103: Combine the system settings information and processing configuration information of each ETL job into a separate ETL file.

[0067] Depend on Figure 1 As can be seen from the process shown, the ETL file splitting method in this embodiment of the invention is different from the method in the prior art that only applies to simple configuration file splitting and only splits configuration files of different hosts. In this embodiment of the invention, the system setting information and processing configuration information of multiple ETL jobs in the ETL file are extracted respectively, and then the system setting information and processing configuration information of the ETL job are spliced ​​together to form a separate ETL file for each ETL job.

[0068] In practice, the system settings information of multiple ETL jobs in the ETL file is extracted based on its content. In this example, a Python script can be used to identify the system settings information of multiple ETL jobs in the ETL file based on its content.

[0069] And / or, based on the content of the ETL file, extract the processing configuration information of multiple ETL jobs in the ETL file respectively. In the embodiment, a Python script can be used to identify the processing configuration information of multiple ETL jobs in the ETL file based on the content of the ETL file.

[0070] Python is an object-oriented, interpreted, dynamically typed high-level programming language with concise syntax rules that make learning programming easier. It also boasts powerful functionality, meeting the development needs of most application areas. Besides Python, ETL processes can be developed using many other programming languages. Those skilled in the art will understand that identifying system settings and processing configuration information for multiple ETL jobs within an ETL file can also be done using other programming languages; however, the scope of protection of this invention is not limited to the examples given.

[0071] In practice, before extracting the system settings and processing configuration information of multiple ETL jobs from the ETL file, it is also possible to determine whether the ETL file format is correct; if the ETL file format is incorrect, the ETL file can be read again.

[0072] ETL processing is a very complex process, and hand-written programs are not easy to manage, so more and more enterprises use data integration software platforms to help with ETL processing and use the built-in Metadata function to store the mapping and conversion rules of the source and destination. Metadata, also known as intermediate data or relay data, is data that describes data, mainly information that describes data attributes, and is used to support functions such as indicating storage location, historical data, resource lookup, file recording, etc. For example, DataStage is a data integration software platform from IBM that helps with ETL processing, but the scope of protection of the embodiments of the present invention is not limited to the example. The multiple job information derived from DataStage is stored in the ETL file in the form of multiple independent jobs in the dsx format.

[0073] In one embodiment, before extracting the system setting information and processing configuration information of the multiple ETL jobs in the ETL file respectively, it can be determined whether the format of the ETL file is dsx format; if the format of the ETL file is not dsx format, the ETL file is re-read until the format of the ETL file is dsx format.

[0074] In specific implementation, the system setting information of the ETL job includes one or any combination of the following information: export tool information, tool version, host name, tool project name, MDI version, system time, and server version, etc. In the embodiment, the key of the system setting information of the ETL job can be determined by the name of the system setting information of the ETL job; the key of the system setting information of the ETL job can also be determined by the name of the processing configuration information of the ETL job.

[0075] In specific implementation, according to the content of the ETL file, the system setting information of the multiple ETL jobs in the ETL file can be extracted by keyword search. According to the content of the ETL file, the processing configuration information of the multiple ETL jobs in the ETL file can also be extracted by keyword search.

[0076] In one embodiment, for example, by keyword searching the system setting information of the multiple ETL jobs in the ETL file, one or any combination of the names of the export tool information, tool version, host name, tool project name, MDI version, system time, and server version in the ETL processing process can be used as a keyword to search for the processing configuration information of the ETL job in the ETL file containing multiple ETL jobs and extract it.

[0077] In the implementation, according to the content of the ETL file, after extracting the system setting information of the plurality of ETL jobs in the ETL file respectively, the method further comprises storing the extracted system setting information of the plurality of ETL jobs into a first temporary text file.

[0078] In one embodiment, when storing the extracted system setting information of the plurality of ETL jobs into the first temporary text file, the system setting information of the plurality of ETL jobs can be stored into the same first temporary text file, or the system setting information of each ETL job can be stored into different first temporary text files. For example, when storing the system setting information of each ETL job into different first temporary text files, each first temporary text file can be numbered as: first temporary text file A, first temporary text file B, first temporary text file C, first temporary text file D, and so on.

[0079] In the implementation, according to the content of the ETL file, after extracting the processing configuration information of the plurality of ETL jobs in the ETL file respectively, the method further comprises storing the extracted processing configuration information of the plurality of ETL jobs into a second temporary text file.

[0080] In one embodiment, when storing the extracted processing configuration information of the plurality of ETL jobs into the second temporary text file, the processing configuration information of the plurality of ETL jobs can be stored into the same second temporary text file, or the processing configuration information of each ETL job can be stored into different second temporary text files. For example, when storing the processing configuration information of each ETL job into different second temporary text files, each second temporary text file can be numbered according to the number corresponding to the ETL job record, for example, can be numbered as: second temporary text file a, second temporary text file b, second temporary text file c, second temporary text file d, and so on (corresponding to first temporary text file A, first temporary text file B, first temporary text file C, first temporary text file D, and so on).

[0081] In the implementation, when splicing the system setting information and the processing configuration information of each ETL job into a separate ETL file, the method comprises extracting the system setting information of each ETL job from the first temporary text file respectively, and extracting the processing configuration information of each ETL job from the second temporary text file respectively.

[0082] In one embodiment, when the system setting information and the processing configuration information of each ETL job are spliced into a single ETL file, if the system setting information and the processing configuration information of multiple ETL jobs are stored in temporary text files, the system setting information and the processing configuration information of multiple ETL jobs are stored in two temporary text files respectively, the system setting information of each ETL job is extracted from the first temporary text file respectively, the processing configuration information of each ETL job is extracted from the second temporary text file respectively, then the one-to-one correspondence between the system setting information of each ETL job and the processing configuration information of each ETL job needs to be determined, and the system setting information of each ETL job and the processing configuration information of the ETL job corresponding thereto are spliced into a single ETL file.

[0083] If the system setting information and the processing configuration information of multiple ETL jobs are stored in temporary text files by storing the system setting information and the processing configuration information of multiple ETL jobs in temporary text files numbered corresponding to each other respectively, the system setting information of each ETL job is extracted from the first temporary text file respectively, the processing configuration information of each ETL job is extracted from the second temporary text file respectively, then the system setting information of each ETL job numbered corresponding to each other and the processing configuration information of the ETL job corresponding thereto need to be spliced into a single ETL file.

[0084] In specific implementation, after the processing configuration information of the ETL job is extracted, the name of the ETL job is further extracted from the processing configuration information of the ETL job, and the ETL file name corresponding to the ETL job is determined according to the name of the ETL job.

[0085] In the embodiment, the processing configuration content of the ETL job is the core content of each ETL job, therefore when the single ETL file is named, the name of the ETL job can be extracted from the processing configuration information of each ETL job as the name of each single ETL file, that is, after the processing configuration information of each ETL job is filtered, the ETL file name corresponding to each ETL job is determined. The format of each single ETL file is the same as that of the original ETL file, and is a fixed format exported by the data integration software platform for assisting ETL processing.

[0086] For example, if the data integration software platform for assisting ETL processing is DataStage, which is a data integration software platform of IBM, the multiple job information exported by the data integration software platform in the processing process is sequentially stored in the ETL file in the form of multiple independent jobs in dsx format, therefore the format of each single ETL file obtained by splitting the ETL file is also dsx format.

[0087] The following will be described in combination with Figure 2The example illustrates the overall process of ETL file splitting. As shown in Figure 2 , the overall process of ETL file splitting in this example can include:

[0088] Step 201, input the ETL file;

[0089] Step 202, determine whether the ETL file format is correct, if yes, execute step 203, otherwise return to step 201 to re-input the ETL file;

[0090] Step 203, read the ETL file;

[0091] Step 204, extract the system setting information of multiple ETL jobs in the ETL file;

[0092] Step 205, extract the processing configuration information of multiple ETL jobs in the ETL file;

[0093] Step 206, splice the system setting information and the processing configuration information of each ETL job into a separate ETL file;

[0094] Step 207, determine whether there is still information of ETL jobs in the ETL file, if not, end the process, if yes, return to step 204 to continue extracting the information of ETL jobs.

[0095] The following will illustrate the specific process of extracting the system setting information of multiple ETL jobs in the ETL file in conjunction with the example of Figure 3 . As shown in Figure 3 , the specific process of extracting the system setting information of multiple ETL jobs in this example can include:

[0096] Step 301, input the ETL file;

[0097] Step 302, determine whether the ETL file format is correct, if yes, execute step 303, otherwise return to step 301 to re-input the ETL file;

[0098] Step 303, read the ETL file;

[0099] Step 304, locate the system setting information of the ETL file according to the keyword;

[0100] Step 305, extract the system setting information of multiple ETL jobs and store it in a first temporary text file, and end the process.

[0101] The following will illustrate the specific process of extracting the processing configuration information of multiple ETL jobs in the ETL file in conjunction with the example of Figure 4 . As shown in Figure 4 , the specific process of extracting the processing configuration information of multiple ETL jobs in this example can include:

[0102] Step 401, input the ETL file;

[0103] Step 402, judge whether the ETL file format is correct, if yes, execute step 403, otherwise return to step 401 to input the ETL file again;

[0104] Step 403, read the ETL file;

[0105] Step 404, locate the ETL file processing configuration information according to the key word;

[0106] Step 405, extract the processing configuration information of multiple ETL jobs and store it in the second temporary text file;

[0107] Step 406, extract the name of the ETL job from the processing configuration information of the ETL job, determine the ETL file name corresponding to the ETL job, and end the process.

[0108] After Figure 3 With Figure 4 After the extraction process of the ETL file system setting information and the ETL file processing configuration information, the system setting information and the processing configuration information of each ETL job are spliced into a separate ETL file.

[0109] The embodiment of the application also provides an ETL file splitting device, as described in the following embodiment. Since the principle of solving the problem of the device is similar to that of the ETL file splitting method, the implementation of the device can be referred to the implementation of the ETL file splitting method, and the repeated parts will not be described again.

[0110] Figure 5 The structure diagram of the ETL file splitting device in the embodiment of the application is shown in the figure. Figure 5 As shown in the figure, the ETL file splitting device in the embodiment of the application can include:

[0111] The system setting information extraction module 501 is used to extract the system setting information of multiple ETL jobs in the ETL file according to the content of the ETL file;

[0112] The processing configuration information extraction module 502 is used to extract the processing configuration information of multiple ETL jobs in the ETL file according to the content of the ETL file;

[0113] The ETL file splicing module 503 is used to splice the system setting information and the processing configuration information of each ETL job into a separate ETL file.

[0114] In the embodiment, the system setting information extraction module 501 is specifically used to: in the form of a Python script, according to the content of the ETL file, identify the system setting information of multiple ETL jobs in the ETL file;

[0115] And / or, the processing configuration information extraction module 502 is specifically used to: in the form of a Python script, according to the content of the ETL file, identify the processing configuration information of the plurality of ETL jobs in the ETL file.

[0116] Figure 6 A structural schematic diagram of a specific example of the ETL file splitting device in the embodiment of the application. As shown in the figure, in the example Figure 6 As shown in the figure, the ETL file splitting device Figure 5 may further include:

[0117] The file reading module 601 is used to read the ETL file and judge whether the format of the ETL file is correct; when the format of the ETL file is incorrect, the ETL file is re-read.

[0118] In the embodiment, the system setting information in the system setting information extraction module 501 includes one or any combination of the following information:

[0119] Export tool information, tool version, host name, tool project name, MDI version, system time and server version.

[0120] In the embodiment, the system setting information extraction module 501 is specifically used to:

[0121] Find the system setting information of the plurality of ETL jobs in the ETL file through a keyword.

[0122] In the embodiment, the processing configuration information extraction module 502 is specifically used to:

[0123] Find the processing configuration information of the plurality of ETL jobs in the ETL file through a keyword.

[0124] Figure 7 A structural schematic diagram of another specific example of the ETL file splitting device in the embodiment of the application. As shown in the figure, in the example Figure 7 As shown in the figure, in the example Figure 5 As shown in the figure, the ETL file splitting device may further include:

[0125] The storage module 701 is used to store the extracted system setting information of the plurality of ETL jobs into a first temporary text file; and store the extracted processing configuration information of the plurality of ETL jobs into a second temporary text file.

[0126] The ETL file splicing module 503 is specifically used to:

[0127] Extract the system setting information of each ETL job from the first temporary text file respectively, and extract the processing configuration information of each ETL job from the second temporary text file respectively.

[0128] The storage module 701 in the embodiment can also include Figure 6 in the ETL file splitting device shown.

[0129] In the embodiment, the processing configuration information extraction module 502 is further specifically configured to:

[0130] extract the name of the ETL job from the processing configuration information of the ETL job;

[0131] determine the ETL file name corresponding to the ETL job according to the name of the ETL job.

[0132] Based on the foregoing inventive concept, as Figure 8 shown, the present application also proposes a computer device 800, which comprises a memory 810, a processor 820, and a computer program 830 stored on the memory 810 and capable of running on the processor 820, and the processor 820 implements the foregoing ETL file splitting method when executing the computer program 830.

[0133] The embodiment of the present application also provides a computer readable storage medium to solve the problem that all ETL jobs will be generated under the same ETL file when the existing technology concentrates on exporting ETL processing, which is inconvenient for users to operate and maintain, and the computer readable storage medium stores a computer program for executing the foregoing ETL file splitting method.

[0134] To sum up, in the embodiment of the present application, the system setting information of multiple ETL jobs in an ETL file is extracted according to the content of the ETL file; the processing configuration information of multiple ETL jobs in the ETL file is extracted according to the content of the ETL file; and the system setting information and the processing configuration information of each ETL job are spliced into a separate ETL file. Compared with the technical solution that all ETL jobs will be generated under the same ETL file when the existing technology concentrates on exporting ETL processing, which is inconvenient for users to operate and maintain, the embodiment of the present application extracts the system setting information and the processing configuration information of multiple ETL jobs in an ETL file respectively, splices the system setting information and the processing configuration information of each ETL job into a separate ETL file, thereby effectively splitting the ETL file and facilitating users to operate and maintain.

[0135] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program codes.

[0136] The present application is described in reference to the flowchart and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for carrying out the function specified by the block or blocks.

[0137] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for carrying out the function specified by the block or blocks.

[0138] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for carrying out the function specified by the block or blocks.

[0139] The specific embodiments were described above to enable implementation of the present application. It should be understood, however, that the above-described embodiments are merely examples of the implementation of the present application and are not intended to limit the scope of the present application. It is intended that any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. An ETL file splitting method, characterized by, The method comprises the following steps: According to the content of the ETL file, the system setting information of the multiple ETL jobs in the ETL file is extracted respectively; According to the content of the ETL file, the processing configuration information of the multiple ETL jobs in the ETL file is extracted respectively; wherein, the system setting information of the ETL jobs is found in the ETL file by keyword search, and the name of one or any combination of the export tool information, tool version, host name, tool project name, MDI version, system time and server version in the ETL processing process is taken as the keyword, and the processing configuration information of the ETL jobs is found and extracted in the ETL file containing multiple ETL jobs according to the keyword; The system setting information and the processing configuration information of each ETL job are spliced into a separate ETL file; After the system setting information of the multiple ETL jobs is extracted according to the content of the ETL file, the system setting information of the multiple ETL jobs is stored in a first temporary text file; wherein, the system setting information of the multiple ETL jobs is stored in the same first temporary text file, or the system setting information of each ETL job is stored in different first temporary text files respectively; After the processing configuration information of the multiple ETL jobs is extracted according to the content of the ETL file, the processing configuration information of the multiple ETL jobs is stored in a second temporary text file; wherein, the processing configuration information of the multiple ETL jobs is stored in the same second temporary text file, or the processing configuration information of each ETL job is stored in different second temporary text files respectively; The system setting information and the processing configuration information of each ETL job are spliced into a separate ETL file, which comprises the following steps: the system setting information of each ETL job is extracted from the first temporary text file respectively, and the processing configuration information of each ETL job is extracted from the second temporary text file respectively; if the system setting information and the processing configuration information of the multiple ETL jobs are stored in the temporary text file when the system setting information and the processing configuration information of the multiple ETL jobs are stored in the temporary text file, the system setting information and the processing configuration information of each ETL job are spliced into a separate ETL file according to the corresponding number; Before the system setting information and the processing configuration information of the multiple ETL jobs in the ETL file are extracted respectively, it is further included that whether the format of the ETL file is correct is judged; when the format of the ETL file is incorrect, the ETL file is read again; wherein, whether the format of the ETL file is dsx format is judged; if the format of the ETL file is not dsx format, the ETL file is read again until the format of the ETL file is dsx format.

2. The method of claim 1, wherein, According to the content of the ETL file, system setting information of multiple ETL jobs in the ETL file is extracted respectively, including: in the form of a Python script, according to the content of the ETL file, the system setting information of multiple ETL jobs in the ETL file is identified; And / or, according to the content of the ETL file, processing configuration information of multiple ETL jobs in the ETL file is extracted respectively, including: in the form of a Python script, according to the content of the ETL file, the processing configuration information of multiple ETL jobs in the ETL file is identified.

3. The method of claim 1, wherein, The system setting information includes one or any combination of the following information: Export tool information, tool version, host name, tool project name, MDI version, system time and server version.

4. The method of claim 1, wherein, According to the content of the ETL file, processing configuration information of multiple ETL jobs in the ETL file is extracted respectively, including: The processing configuration information of multiple ETL jobs in the ETL file is found by keyword.

5. The method according to any one of claims 1 to 4, characterized in that, After extracting the processing configuration information of the ETL job, it further includes: The name of the ETL job is extracted from the processing configuration information of the ETL job; The ETL file name corresponding to the ETL job is determined according to the name of the ETL job.

6. An ETL file splitting apparatus characterized by comprising: Including: The system setting information extraction module is used for extracting the system setting information of multiple ETL jobs in the ETL file according to the content of the ETL file; The processing configuration information extraction module is used for extracting the processing configuration information of multiple ETL jobs in the ETL file according to the content of the ETL file; wherein the system setting information of multiple ETL jobs in the ETL file is found by keyword, one or any combination of the names of the export tool information, tool version, host name, tool project name, MDI version, system time and server version in the ETL processing process is used as the keyword, and the processing configuration information of the ETL job is found and extracted in the ETL file containing multiple ETL jobs according to the keyword; The ETL file splicing module is used for splicing the system setting information and the processing configuration information of each ETL job into a separate ETL file; Further including: The storage module is used for storing the extracted system setting information of multiple ETL jobs into a first temporary text file; and storing the extracted processing configuration information of multiple ETL jobs into a second temporary text file; wherein the system setting information of multiple ETL jobs is stored into the same first temporary text file or the system setting information of each ETL job is stored into different first temporary text files respectively; and the processing configuration information of multiple ETL jobs is stored into the same second temporary text file or the processing configuration information of each ETL job is stored into different second temporary text files respectively; The ETL file splicing module is specifically used for: extract system setting information of each ETL job from the first temporary text file respectively, and extract processing configuration information of each ETL job from the second temporary text file respectively; if the system setting information and the processing configuration information of the multiple ETL jobs are stored in the temporary text files in a manner that the system setting information and the processing configuration information of the multiple ETL jobs are stored in the temporary text files numbered corresponding to each other respectively, then after the system setting information of each ETL job numbered corresponding to each other and the processing configuration information of the ETL job are spliced into a single ETL file; Further comprising: a file reading module, configured to read the ETL file, and judge whether the format of the ETL file is correct; when the format of the ETL file is incorrect, re-read the ETL file; wherein, whether the format of the ETL file is dsx format is judged; if the format of the ETL file is not dsx format, then re-read the ETL file until the format of the ETL file is dsx format.

7. The apparatus of claim 6, wherein, The system setting information extraction module is specifically configured to: in the form of a Python script, according to the content of the ETL file, identify the system setting information of the multiple ETL jobs in the ETL file. And / or, the processing configuration information extraction module is specifically configured to: in the form of a Python script, according to the content of the ETL file, identify the processing configuration information of the multiple ETL jobs in the ETL file.

8. The apparatus of claim 6, wherein, The system setting information in the system setting information extraction module includes one or any combination of the following information: export tool information, tool version, host name, tool project name, MDI version, system time and server version.

9. The apparatus of claim 6, wherein, The processing configuration information extraction module is specifically configured to: find the processing configuration information of the multiple ETL jobs in the ETL file through a keyword.

10. The device of any one of claims 6 to 9, wherein, The processing configuration information extraction module is further specifically configured to: extract the name of the ETL job from the processing configuration information of the ETL job; determine the ETL file name corresponding to the ETL job according to the name of the ETL job.

11. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program for executing the method of any one of claims 1 to 5.

Images