A visual ETL process data processing method, device and equipment and storage medium

Abstract

The application discloses a kind of visual ETL process data processing method, device, computing device and storage medium.It is according to the technical scheme provided in the application, define process component, the function operator of the process component and process control expression;In the icon corresponding to the process component is combined in visual platform, constructs ETL process;According to the function operator of the process component and the process control expression, determine ETL process function expression;According to ETL process function expression, online ETL process.Through the application, ETL process can be constructed, and different dimensions in ETL process function expression are pre-checked, the ETL process is optimized from function operator, cyclomatic complexity and the like, simplifies ETL process function expression, greatly reduces the complexity of the ETL process and the probability of error, improves information expression efficiency, while monitoring ETL process by slicing preset mode, improves the maintainability of ETL process.

Description

Technical Field

[0001] This invention relates to the field of mobile communications, and specifically to a method, apparatus, computing device, and computer storage medium for visualizing ETL process data processing. Background Technology

[0002] With the development of big data technology, enterprises typically use ETL (Extract-Transform-Load) to aggregate and integrate data. However, increased business volume leads to the ETL process incorporating numerous components, resulting in a large volume of process data. Simultaneously, the complexity of business operations increases the number of control conditions such as branches, loops, and waits within the ETL process, making it difficult to find, locate, and monitor problems, thus reducing the reliability and usability of the entire system. Therefore, it is necessary to reduce the processing complexity of the ETL process and improve its development and application efficiency.

[0003] In existing technologies, one approach is to establish an ETL process and then describe it using XML or JSON format to reduce development complexity. However, the data structure of XML or JSON tag configuration languages ​​is inherently redundant, and the description process is also quite complex, involving generation, parsing, loading, and storage. This results in an excessive amount of information, especially with large datasets and complex ETL scenarios. Furthermore, optimization and verification of the ETL process are not addressed. Another approach is to decompose the ETL process into subtasks based on hierarchical relationships, with each subtask corresponding to a visual canvas interface. This breaks down the complex problem into several independent sub-problems, addressing complex ETL scenarios. However, this method may encounter data transmission issues with large datasets, prevents further optimization of the process before deployment, and makes it difficult to locate and debug errors. Summary of the Invention

[0004] In view of the above problems, the present invention is proposed to provide a visual ETL process data processing method and corresponding visual ETL process data processing apparatus, computing device and computer storage medium to overcome or at least partially solve the above problems.

[0005] According to one aspect of the present invention, a method for visualizing ETL process data processing is provided, the method comprising:

[0006] Define process components, function operators of the process components, and process control expressions;

[0007] In the visualization platform, the icons corresponding to the process components are combined to construct the ETL process;

[0008] Based on the function operators of the process components and the process control expression, determine the ETL process function expression;

[0009] Based on the ETL process function expression, deploy the ETL process.

[0010] In the above scheme, the process components include: an extraction component, a loading component, and a conversion component;

[0011] The defined process components, the function operators of the process components, and the process control expressions further include:

[0012] Define the input and output conditions of the function operators of the process components;

[0013] The process components are connected in series according to a preset process control method to obtain a process control expression.

[0014] In the above scheme, determining the ETL process function expression based on the function operator of the process component and the process control expression further includes:

[0015] Based on the ETL process, the function operators of the process components are combined with the process control expression to construct the ETL process function expression.

[0016] In the above scheme, the step of deploying the ETL process based on the ETL process function expression further includes:

[0017] The ETL process function expression is examined from multiple dimensions to obtain an optimization scheme;

[0018] Based on the optimization scheme, the optimized ETL process function expression is obtained, and a log probe is added to the function operator using the aspect pre-setting method to set the sampling parameters;

[0019] Deploy the ETL process using the optimized ETL process function expression.

[0020] In the above scheme, the multiple dimensions include: circular reference dimension, cyclomatic complexity dimension, and function optimization dimension.

[0021] In the above scheme, the step of adding log probes to the function operator using the aspect pre-setting method and setting sampling parameters further includes:

[0022] Traverse the optimized ETL process function expressions, extract logical points, and store the logical points in the aspect configuration file;

[0023] The process logic is configured by modifying the aspect configuration file; wherein the logic configuration includes at least: adding log probes, setting sampling frequency, setting sampling time, and setting log level;

[0024] Based on the logical points in the aspect configuration file, a new optimized ETL process function expression is formed.

[0025] In the above scheme, the step of using the optimized ETL process function expression to deploy the ETL process further includes:

[0026] The optimized ETL process function expression is submitted to the server, which then parses the optimized ETL process function expression and puts the ETL process online.

[0027] According to another aspect of the present invention, a visual ETL process data processing apparatus is provided, comprising: a definition module, a construction module, a determination module, and an online module; wherein,

[0028] The definition module is used to define process components, function operators of the process components, and process control expressions;

[0029] The construction module is used to combine the icons corresponding to the process components in the visualization platform to construct an ETL process.

[0030] The determining module is used to determine the ETL process function expression based on the function operators of the process components and the process control expression;

[0031] The online module is used to launch the ETL process based on the ETL process function expression.

[0032] According to another aspect of the present invention, a computing device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus;

[0033] The memory is used to store at least one executable instruction that causes the processor to perform operations corresponding to the visual ETL process data processing method described above.

[0034] According to another aspect of the present invention, a computer storage medium is provided, wherein at least one executable instruction is stored in the storage medium, the executable instruction causing a processor to perform operations corresponding to the above-described visual ETL process data processing method.

[0035] According to the technical solution provided by this invention, process components, function operators of the process components, and process control expressions are defined; icons corresponding to the process components are combined in a visualization platform to construct an ETL process; the ETL process function expression is determined based on the function operators of the process components and the process control expression; and the ETL process is deployed based on the ETL process function expression. This solves the problems in the prior art, such as large amounts of intermediate data, high network load, easy transmission timeouts, and difficulty in process optimization in ultra-large and complex ETL processes due to redundant data formats; and the difficulty in testing and locating problems when they occur during process processing. The technical solution provided by this invention effectively improves information expression efficiency by utilizing ETL process function expressions, a pre-check scheme for the process, and a slice pre-setting method, thereby improving data transmission and storage efficiency, reducing the probability of errors in the ETL process, improving the reliability of the ETL process, and facilitating monitoring of the ETL process, thus improving its maintainability.

[0036] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0037] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0038] Figure 1 A flowchart illustrating a method for visualizing ETL process data processing according to an embodiment of the present invention is shown.

[0039] Figure 2 A schematic diagram illustrating the construction of an ETL process using a visualization platform according to an embodiment of the present invention is shown;

[0040] Figure 3 A schematic diagram of a circular reference optimization process according to another embodiment of the present invention is shown;

[0041] Figure 4 A schematic diagram of a cyclomatic complexity optimization process according to an embodiment of the present invention is shown;

[0042] Figure 5 A schematic diagram of a higher-order function combination method according to an embodiment of the present invention is shown;

[0043] Figure 6 A schematic diagram of a parameter optimization method according to an embodiment of the present invention is shown;

[0044] Figure 7 A schematic diagram of an expression compression method according to an embodiment of the present invention is shown;

[0045] Figure 8 A schematic diagram of the slice presetting process according to an embodiment of the present invention is shown;

[0046] Figure 9 A structural block diagram of a visual ETL process data processing apparatus according to an embodiment of the present invention is shown;

[0047] Figure 10 A schematic diagram of the structure of a computing device according to an embodiment of the present invention is shown. Detailed Implementation

[0048] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0049] Figure 1 A flowchart illustrating a visual ETL process data processing method according to an embodiment of the present invention is shown, as follows: Figure 1 As shown, the method includes the following steps:

[0050] Step S101: Define the process component, the function operator of the process component, and the process control expression.

[0051] Specifically, by defining the function operators of the process components, each process component becomes a single task processing unit.

[0052] Preferably, the inputs and outputs of the process components conform to the following definitions:

[0053] x∈{...params}→fn(...params):result→y∈{result}

[0054] Specifically, the process components include: an extraction component, a loading component, and a transformation component;

[0055] The defined process components, the function operators of the process components, and the process control expressions further include:

[0056] Define the input and output conditions of the function operators of the process components;

[0057] The process components are connected in series according to a preset process control method to obtain a process control expression.

[0058] Specifically, the extraction component is used to extract data from a data source, such as HDFS data. The input conditions for the function operator of the extraction component can be data source information and data format, and the output condition is the data format.

[0059] Preferably, the definition of the extraction component can be:

[0060] input(format, source):format. Here, format represents the data format, and source represents the data source information.

[0061] Specifically, the loading component is used to write data to a data source, such as writing data to HDFS. The input conditions for the function operators of the loading component can be the data format, and the output conditions are the data source information.

[0062] Preferably, the definition of the loading component can be:

[0063] output(format):source.

[0064] Specifically, the conversion component typically includes operations such as deduplication, summation, filtering, JOIN, and replacement. The input conditions for the function operators of the conversion component can be data format and processing logic, and the output condition is data format.

[0065] Preferably, the definition of the conversion component can be:

[0066] transfer(format,function):format;

[0067] The transfer component can be: unique (removing duplicates), sum (summing), filter (filtering), join (merging), replace (replacing), etc., format represents the data format, and function represents the processing logic.

[0068] Specifically, the preset process control method may include timers, scheduling cycles, asynchronous control, serial condition control, parallel condition control, loop control, etc., used to connect the extraction component, loading component and conversion component in series.

[0069] Preferably, for higher-order functions, functional programming is performed using nested ETL processes;

[0070] Pure functions are used in the ETL process. A pure function, in the context of function integration during data cleaning, produces the same output for the same input and has no observable side effects. For example, for the function `(Function(array A)) {processing logic; return B;}`, the caller inputs array A, processes it, and then inputs array B. After the input parameters are processed by the function, the value of array A remains unchanged. Because pure functions produce the same output for the same input, they do not have side effects on the input parameters. Including pure functions in the ETL process avoids the influence of function side effects. Therefore, the ETL process does not need to include logic for observing and processing ETL component inputs or observing function side effects, simplifying the ETL process and reducing its overall complexity. Step S102: Combine the icons corresponding to the process components in the visualization platform to construct the ETL process.

[0071] Specifically, the process components are combined by dragging and dropping the corresponding icons in the visualization canvas. Figure 2 A schematic diagram illustrating the construction of an ETL process using a visualization platform according to an embodiment of the present invention is shown, such as... Figure 2 As shown. In the ETL process, the icons corresponding to each process component are dragged and combined using the visualization platform to form the final ETL process. Specifically, the ETL process obtained by combining the icons corresponding to the process components may include: HDFS extraction, conditional filtering, deduplication, sorting, timer, aggregation, function execution, looping, SQL execution, etc., that is, extracting data from the HDFS data source, filtering and deduplicating the extracted data, sorting the steps in the process and using a timer to determine the running time, aggregating the various parts of the process, and finally executing through function execution or looping followed by SQL execution.

[0072] Step S103: Determine the ETL process function expression based on the function operators of the process component and the process control expression.

[0073] Specifically, based on the ETL process, the function operators of the process components are combined with the process control expression to construct the ETL process function expression.

[0074] Preferably, the flow control expression can be defined as shown in Table 1:

[0075]

[0076]

[0077] Table 1

[0078] Constructing an ETL process is a combination of function operators and flow control. The method of constructing the ETL process function expression based on the flow control expression in functional terms offers concise syntax, effectively reducing the size of the data in ultra-large-scale processes, thereby lowering transmission and storage costs. Furthermore, functional expressions facilitate parameter optimization of process components; they are also function-friendly, allowing external interpolation of input parameters without affecting the function itself; and they are easily parsed by ETL execution engines. In short, the flow control expression in functional terms allows for a more concise syntax to describe ETL processes and supports syntax compression, enabling the expression of complex ETL processes with a smaller data volume, saving data transmission bandwidth and data storage space.

[0079] After obtaining the ETL process function expression, the ETL process function expression can be optimized.

[0080] Specifically, the ETL process function expression is examined from multiple dimensions to obtain an optimization scheme; based on the optimization scheme, an optimized ETL process function expression is obtained, and a log probe is added to the function operator using aspect pre-setting, and sampling parameters are set.

[0081] The multiple dimensions include: circular reference dimension, cyclomatic complexity dimension, and function optimization dimension.

[0082] The step of adding log probes to the function operator using a pre-defined aspect method and setting sampling parameters further includes:

[0083] The optimized ETL process function expressions are traversed, logical points are extracted, and stored in the aspect configuration file. The process logic is configured by modifying the aspect configuration file. The logical configuration includes at least: adding log probes, setting sampling frequency, setting sampling time, and setting log levels. Based on the logical points in the aspect configuration file, a new optimized ETL process function expression is formed. The logical points are the entry parameter functions of the process components, and the actual operation and processing logic of each process component is executed by these entry parameter functions.

[0084] Optimization of circular references refers to reducing the number of dependencies between two or more process components when building an ETL process. Circular references easily generate a large amount of redundant data and processes, causing the overall process execution to fall into an infinite loop, resulting in reduced ETL process execution efficiency or even execution failure.

[0085] Preferably, the directed acyclic graph data structure based on the ETL process uses the Kahn algorithm to determine whether circular references exist. Figure 3 A schematic diagram of a circular reference optimization process according to an embodiment of the present invention is shown:

[0086] Step S301: Set up a directed acyclic graph data structure for the set S corresponding to the ETL process, traverse the nodes, and put the nodes with an in-degree of 0 into S.

[0087] Step S302: Input set S and execute Kahn's algorithm to perform topological sorting on the flowchart.

[0088] Step S303: Determine whether the topological sorting has finished. If yes, proceed to step S304; otherwise, proceed to step S305.

[0089] Step S304: Submit the ETL process.

[0090] Specifically, if the sorting is successful, meaning all nodes can be topologically sorted, it indicates that there are no circular references, and the process can be submitted normally.

[0091] Step S305: Prohibit the submission of the ETL process.

[0092] Specifically, if topological sorting fails, it indicates the existence of a circular reference, and process submission is prohibited.

[0093] The optimization for the cyclomatic complexity dimension refers to judging the cyclomatic complexity of the ETL process, and prompting the user to make modifications when the cyclomatic complexity is too high. Figure 4 A schematic diagram of a cyclomatic complexity optimization process according to an embodiment of the present invention is shown:

[0094] Cyclomatic complexity is a metric for the structural complexity of an ETL process. Higher cyclomatic complexity indicates a more complex ETL process logic, a higher probability of potential errors, and more hidden defects. The ETL process is a directed acyclic flowchart, and its cyclomatic complexity can be calculated using a point-edge computation method. Furthermore, different processes within the ETL process have varying impacts on ETL scheduling and execution; therefore, weighting is necessary to assign weights to the edges. Preferably, the weight coefficients for parallel condition control and loop control permissions are set to 2, while the weight coefficients for other control conditions are set to 1.

[0095] Step S401: Define weight rules and write all process components and weight rules into the dictionary.

[0096] Step S402: Extract the control flow from the ETL process function expression.

[0097] Step S403: Calculate the edges and vertices of the directed acyclic graph.

[0098] Preferably, the edges of a directed acyclic graph are defined as E, and the vertices of a directed acyclic graph are defined as N.

[0099] Step S404: Calculate the cyclomatic complexity of the input parameter function of the point-corresponding operator and count the number of all functions in the ETL process.

[0100] Preferably, the cyclomatic complexity of the operator's input parameters is defined as G, and the total number of functions in the ETL process is defined as P.

[0101] Step S405: Determine whether the cyclomatic complexity of a single function is greater than the first preset threshold.

[0102] Preferably, the first preset threshold is 5; if the cyclomatic complexity of a single function is greater than 5, then step S406 is executed; if the cyclomatic complexity of a single function is not greater than 5, then step S407 is executed.

[0103] Step S406: Prompt the user that the cyclomatic complexity of the specific function operator is too high.

[0104] Step S407: Calculate the process cyclomatic complexity using the weighted and sub-function point-edge calculation formula, and determine whether the process cyclomatic complexity is greater than the second preset threshold.

[0105] Preferably, the cyclomatic complexity of the process is defined as V(G), and the second preset threshold is 10; if the cyclomatic complexity is greater than 10, then step S408 is executed; if the cyclomatic complexity is not greater than 10, then step S409 is executed.

[0106] Step S408: Prompt the user that the ETL process circle is too complex and suggest that the user modify it.

[0107] Step S409: End cyclomatic complexity optimization.

[0108] The function optimization includes three aspects: repetition of function operator expressions, splitting of complex function input conditions, and combination of higher-order functions.

[0109] The optimization method for the repetition of the function operator expression is to issue a repetition prompt and merging suggestion to the user when the same function operator is detected to appear repeatedly.

[0110] The optimization method for splitting the input conditions of the complex function is as follows: in the above cyclomatic complexity optimization process, if the cyclomatic complexity of the function operator input conditions is too large, the user is prompted to split the function.

[0111] The optimization method for the aforementioned higher-order function composition is as follows: for multiple functions scattered under the same execution conditions, the definition of higher-order functions can be used to perform function composition and nesting, thereby achieving function decomposition and reuse as well as single responsibility, such as... Figure 5 As shown. Figure 5A schematic diagram of a higher-order function combination method according to an embodiment of the present invention is shown.

[0112] Preferably, the function optimization may also include two methods: parameter optimization and expression compression.

[0113] The parameter optimization mentioned refers to the optimization of function expressions in serial and parallel process control, taking into account the characteristic that the output of the upper-level component is the input of the lower-level component. This optimization can further refine the input and output formats, reducing invalid information that needs to be transmitted and improving transmission and storage efficiency. Figure 6 As shown. Figure 6 A schematic diagram of a parameter optimization method according to an embodiment of the present invention is shown.

[0114] The expression compression refers to compressing redundant characters, line breaks, and comments in the process during compression, further simplifying the data, such as... Figure 7 As shown. Figure 7 A schematic diagram of an expression compression method according to an embodiment of the present invention is shown.

[0115] Step S104: Launch the ETL process based on the ETL process function expression.

[0116] Specifically, the optimized ETL process function expression is submitted to the server, which then parses the optimized ETL process function expression and puts the ETL process online.

[0117] According to the visualized ETL process data processing method provided in this embodiment, process components, function operators of the process components, and process control expressions are defined; the icons corresponding to the process components are combined in the visualization platform to construct the ETL process; the ETL process function expression is determined according to the function operators of the process components and the process control expression; and the ETL process is launched based on the ETL process function expression. Using the technical solution provided by this invention, the ETL process for processing data can be represented in the form of function expressions, the process components within the ETL process can be optimized and reasonably set, and the ETL process can be monitored using aspect pre-setting, effectively reducing the complexity of individual components and the overall process within the ETL process, improving information expression efficiency, and thus improving data transmission and storage efficiency; at the same time, optimization is performed before the ETL process is launched, and sampling monitoring is performed after launch, improving the maintainability of the ETL process.

[0118] Figure 8 A schematic diagram of the slice presetting process according to an embodiment of the present invention is shown, as follows: Figure 8 As shown, where:

[0119] The pre-defined slice method is used for process configuration to quickly obtain process health data or error logs during routine maintenance monitoring of complex ETL processes or when errors occur. A complex ETL process refers to one built from numerous process components, where the process function expression contains multiple branches, loops, wait statements, etc. In such cases, the amount of information describing the complex ETL process is enormous, and process designers may find it difficult to detect errors in the constructed process. Therefore, the pre-defined slice method is needed for monitoring during operation.

[0120] Preferably, a complex ETL process can be built from no fewer than 100 process components.

[0121] Step S801: Traverse the optimized ETL process function expressions, extract logical points, and store the logical points in the aspect configuration file.

[0122] Specifically, the logical point refers to the entry parameter passing function of the process component in the ETL process function expression; the logical point is the actual executor of the actual operation and processing logic of each component.

[0123] Step S802: Configure the logic of the process by modifying the aspect configuration file.

[0124] Specifically, the logical configuration includes at least: adding log probes, setting sampling frequency, setting sampling time, and setting log level.

[0125] Step S803: Based on the logical points of the aspect configuration file, form a new optimized ETL process function expression.

[0126] Specifically, after forming a new optimized ETL process function expression, the process is submitted.

[0127] Figure 9 A structural block diagram of a visual ETL process data processing apparatus according to an embodiment of the present invention is shown, such as... Figure 9 As shown, the device includes: a definition module 901, a construction module 902, a determination module 903, and an online module 904; wherein,

[0128] The definition module 901 is used to define process components, function operators of the process components, and process control expressions.

[0129] Specifically, the process components include: an extraction component, a loading component, and a transformation component;

[0130] The definition of the process components, the function operators of the process components, and the process control expression further includes: defining the input conditions and output conditions of the function operators of the process components; and concatenating the process components according to a preset process control method to obtain the process control expression.

[0131] The construction module 902 is used to combine the icons corresponding to the process components in the visualization platform to construct an ETL process.

[0132] The determining module 903 is used to determine the ETL process function expression based on the function operator of the process component and the process control expression.

[0133] Specifically, based on the ETL process, the function operators of the process components are combined with the process control expression to construct the ETL process function expression.

[0134] The ETL process function expression is examined from multiple dimensions to obtain an optimization scheme. Based on the optimization scheme, an optimized ETL process function expression is obtained, and a log probe is added to the function operator using aspect pre-setting, with sampling parameters set. The multiple dimensions include: circular reference dimension, cyclomatic complexity dimension, and function optimization dimension.

[0135] Optionally, the optimized ETL process function expression is traversed, logical points are extracted, and the logical points are stored in the aspect configuration file; the process logic is configured by modifying the aspect configuration file; wherein, the logical configuration includes at least: adding log probes, setting sampling frequency, setting sampling time, and setting log level; and a new optimized ETL process function expression is formed based on the logical points in the aspect configuration file.

[0136] The online module 904 is used to launch the ETL process based on the ETL process function expression.

[0137] The optimized ETL process function expression is submitted to the server, which then parses the optimized ETL process function expression and puts the ETL process online.

[0138] According to the visual ETL process data processing device provided in this embodiment, process components, function operators of the process components, and process control expressions are defined; the icons corresponding to the process components are combined in the visualization platform to construct the ETL process; the ETL process function expression is determined according to the function operators of the process components and the process control expression; and the ETL process is launched based on the ETL process function expression. Using the technical solution provided by this invention, the ETL process for processing data can be represented in the form of function expressions, the process components within the ETL process can be optimized and reasonably set, and the ETL process can be monitored using aspect pre-setting, effectively reducing the complexity of individual components and the overall process within the ETL process, improving information expression efficiency, and thus improving data transmission and storage efficiency; at the same time, optimization is performed before the ETL process is launched, and sampling monitoring is performed after launch, improving the maintainability of the ETL process.

[0139] The present invention also provides a non-volatile computer storage medium storing at least one executable instruction that can execute the visual ETL process data processing method in any of the above method embodiments.

[0140] Figure 10 The diagram illustrates the structure of a computing device according to an embodiment of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.

[0141] like Figure 10 As shown, the computing device may include: a processor 1002, a communications interface 1004, a memory 1006, and a communications bus 1008.

[0142] in:

[0143] The processor 1002, communication interface 1004, and memory 1006 communicate with each other via communication bus 1008.

[0144] Communication interface 1004 is used to communicate with other network elements such as clients or other servers.

[0145] The processor 1002 is used to execute program 1010, which can specifically execute the relevant steps in the above-described visualization ETL process data processing embodiment.

[0146] Specifically, program 1010 may include program code that includes computer operation instructions.

[0147] The processor 1002 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The computing device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0148] Memory 1006 is used to store program 1010. Memory 1006 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0149] Specifically, program 1010 can be used to cause processor 1002 to execute the visualized ETL process data processing method in any of the above method embodiments. The specific implementation of each step in program 1010 can be found in the corresponding descriptions of the steps and units in the above-described visualized ETL process data processing method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0150] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this invention is not directed to any particular programming language. It should be understood that the contents of the invention described herein can be implemented using various programming languages, and the above description of specific languages ​​is for the purpose of disclosing the best mode of implementation of the invention.

[0151] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0152] Similarly, it should be understood that, in order to streamline this disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0153] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0154] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0155] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as a device or apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0156] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

Claims

1. A method for visualizing ETL process data processing, comprising: Define process components, function operators of the process components, and process control expressions; In the visualization platform, the icons corresponding to the process components are combined to construct the ETL process; Based on the function operators of the process components and the process control expression, determine the ETL process function expression; Based on the ETL process function expression, deploy the ETL process; The ETL process deployment process, based on the ETL process function expression, further includes: The ETL process function expression is examined from multiple dimensions to obtain an optimization scheme; Based on the optimization scheme, the optimized ETL process function expression is obtained, and a log probe is added to the function operator using the aspect pre-setting method to set the sampling parameters; Deploy the ETL process using the optimized ETL process function expression.

2. The method according to claim 1, wherein, The process components include: an extraction component, a loading component, and a conversion component; The defined process components, the function operators of the process components, and the process control expressions further include: Define the input and output conditions of the function operators of the process components; The process components are connected in series according to a preset process control method to obtain a process control expression.

3. The method according to claim 1, wherein, The step of determining the ETL process function expression based on the function operators of the process components and the process control expression further includes: Based on the ETL process, the function operators of the process components are combined with the process control expression to construct the ETL process function expression.

4. The method according to claim 1, wherein, The multiple dimensions include: circular reference dimension, cyclomatic complexity dimension, and function optimization dimension.

5. The method according to claim 1, wherein, The method of adding log probes to the function operator using aspect pre-setting and setting sampling parameters further includes: Traverse the optimized ETL process function expressions, extract logical points, and store the logical points in the aspect configuration file; The process logic is configured by modifying the aspect configuration file; wherein the logic configuration includes at least: adding log probes, setting sampling frequency, setting sampling time, and setting log level; Based on the logical points in the aspect configuration file, a new optimized ETL process function expression is formed.

6. The method according to claim 1, wherein, The process of deploying the ETL process using the optimized ETL process function expression further includes: The optimized ETL process function expression is submitted to the server, which then parses the optimized ETL process function expression and puts the ETL process online.

7. A visual ETL process data processing device, comprising: The module includes a definition module, a construction module, a determination module, and a deployment module; among them, The definition module is used to define process components, function operators of the process components, and process control expressions; The construction module is used to combine the icons corresponding to the process components in the visualization platform to construct an ETL process. The determining module is used to determine the ETL process function expression based on the function operators of the process components and the process control expression; The online module is used to launch the ETL process based on the ETL process function expression; The online module is further used for: The ETL process function expression is examined from multiple dimensions to obtain an optimization scheme; Based on the optimization scheme, the optimized ETL process function expression is obtained, and a log probe is added to the function operator using the aspect pre-setting method to set the sampling parameters; Deploy the ETL process using the optimized ETL process function expression.

8. A computing device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the visual ETL process data processing method as described in any one of claims 1-6.

9. A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the visual ETL process data processing method as described in any one of claims 1-6.

Images