Parallel decision-making method and device for etl process
A decision-making method and decision-making technology, applied in the direction of instruments, calculations, electrical digital data processing, etc., can solve problems such as the complexity of ETL process development, and achieve the effects of saving time, cost, and good results in developing ETL processes
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0071] As described above, the ETL process parallel decision-making method, the difference of this embodiment is that, as Figure 5 Shown in the flow chart of ETL process parallel decision-making method step a of the present invention, it is; Wherein, step a comprises:
[0072] In step a1, the ETL process script file is divided into operations, and the modification of the same field or the same group of fields is regarded as an operation;
[0073] The behavior of modifying the field includes: deleting the field, adding a field (adding a constant field separately, and calculating a new field from other fields), modifying the format of the field (changing the floating-point type to an integer type), changing the field ( times 100) and so on. Dividing the ETL process script file into operations, and treating the modification of the same field or the same group of fields as an operation means: first divide the script file into operations, and then number these operations accordin...
Embodiment 2
[0080] As described above, the ETL process parallel decision-making method, the difference of this embodiment is that, as Image 6 Shown in the flow chart of ETL process parallel decision-making method step b of the present invention, it is; Wherein, step b comprises:
[0081] Step b1, executing the ETL process script file with the operation information to process the source data file;
[0082] Execute the ETL process script file with operation information to process the source data file, that is, modify the source data file to execute the script in the script file.
[0083] In this step, the operation information includes: whether to make statistics, and information about statistics fields.
[0084] In this step, pipeline parallel technology is used when processing source data files. Source data files are transferred between operations in units of data rows, and data rows are transferred between operations in the form of data streams, unless there is a special case. The sp...
Embodiment 3
[0088] As the ETL process parallel decision-making method described above, the present embodiment is different from it in that, as Figure 7 Shown in the flowchart of step c of ETL process parallel decision-making method of the present invention, it is; Wherein, step c comprises:
[0089] Step c1, analyzing the input ETL process script file with the operation information, judging whether each operation can be parallelized, and in what way the parallel operations will be parallelized, and outputting the information obtained by judging to the Parallel parameter table;
[0090] The operation information described in this step includes: whether to count, the information of the statistical field, the execution time, the number of field rows;
[0091] In this step, it is judged whether there is a statistical operation, if not, then this step can be performed in parallel with a single operation; if so, it is judged whether there are continuous statistical operations on different fie...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More