ETL task processing method and device

Abstract

The present invention discloses a method and device for processing ETL tasks, and relates to the field of data processing technology. The main purpose of the present invention is to discover and clean up similar ETL tasks to save the computing and storage resources of the system. The main technical scheme of the present invention is: obtaining at least two ETL tasks; judging whether the complexity of the processing logic of the ETL tasks is greater than a first threshold; if it is less than the first threshold, determining the similarity value of the ETL tasks according to the metadata in the data warehouse; if it is not less than the first threshold, determining the similarity value of the ETL tasks by using a graph matching method; outputting prompt information, wherein the prompt information contains the ETL task pairs whose similarity values ​​are greater than a second threshold.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and device for processing ETL tasks. Background Art

[0002] ETL is the abbreviation of Extract-Transform-Load, which is used to describe the process of extracting, transposing and loading data from the source to the destination. The term ETL is more commonly used in data warehouses, but its objects are not limited to data warehouses. ETL is responsible for extracting data from distributed and heterogeneous data sources such as relational data and flat data files to a temporary intermediate layer for cleaning, transformation, and integration, and finally loading it into a data warehouse or data mart, becoming the basis for online analytical processing and data mining. ETL is a very important part of the data warehouse. It is a necessary step that connects the past and the future. Compared with relational databases, data warehouse technology does not have a strict mathematical theoretical basis, and it is more oriented towards practical engineering applications.

[0003] However, for various business departments within the same company, due to the different levels of data demand among their departments, the ETL tasks of different data positions are not visible to external departments. This leads to a large number of ETL tasks with similar calculations on the company's overall cluster. These ETL tasks cause a huge waste of computing resources and storage resources. Summary of the invention

[0004] In view of the above problems, the present invention proposes a method and device for processing ETL tasks, the main purpose of which is to discover and clean up similar ETL tasks to save the computing and storage resources of the system.

[0005] In order to achieve the above object, the present invention mainly provides the following technical solutions:

[0006] On the one hand, the present invention provides a method for processing an ETL task, which specifically includes:

[0007] Get at least two ETL tasks;

[0008] Determine whether the complexity of the processing logic of the ETL task is greater than a first threshold;

[0009] If it is less than the first threshold, determining the similarity value of the ETL task according to the metadata in the data warehouse;

[0010] If it is not less than the first threshold, determining the similarity value of the ETL task by using a graph matching method;

[0011] Outputting prompt information, wherein the prompt information contains the ETL task pairs whose similarity values ​​are greater than the second threshold.

[0012] On the other hand, the present invention provides an ETL task processing device, specifically comprising:

[0013] An acquisition unit, used for acquiring at least two ETL tasks;

[0014] A judging unit, configured to judge whether the complexity of the processing logic of the ETL task obtained by the acquiring unit is greater than a first threshold;

[0015] A first determining unit, configured to determine a similarity value of the ETL task according to metadata in the data warehouse when the determining unit determines that the complexity is less than a first threshold;

[0016] A second determining unit, configured to determine the similarity value of the ETL task by using a graph matching method when the determination unit determines that the complexity is not less than the first threshold;

[0017] The output unit is used to output prompt information, wherein the prompt information contains the ETL task pairs whose similarity values ​​determined by the first determining unit and the second determining unit are greater than a second threshold.

[0018] On the other hand, the present invention provides a processor, wherein the processor is used to run a program, wherein the program executes the above-mentioned ETL task processing method when running.

[0019] By means of the above technical scheme, the present invention provides a method and device for processing ETL tasks. By judging the processing complexity of ETL tasks, ETL tasks are divided into two categories, simple tasks and complex tasks, based on a first threshold value. For simple tasks, the metadata in the data warehouse can be used as a standard to directly match the similarities between simple ETL tasks. For complex tasks, the ETL tasks can be converted into graphs, and then the similarities between complex ETL tasks can be determined by graph matching. Finally, ETL task pairs with similarities greater than the second threshold value are output to prompt users to clean up the ETL task pairs, thereby reducing the occupancy of system computing and storage resources.

[0020] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0022] Figure 1 A flowchart of a method for processing an ETL task proposed in an embodiment of the present invention is shown;

[0023] Figure 2 A flowchart of building a graph structure based on an ETL task in an embodiment of the present invention is shown;

[0024] Figure 3 A block diagram showing a composition of an ETL task processing device proposed in an embodiment of the present invention;

[0025] Figure 4 A block diagram showing the composition of another ETL task processing device proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0027] In the ETL task processing method proposed in the embodiment of the present invention, the main purpose is to discover the existing and newly added similar ETL tasks on the cluster, and inform the user to clean up these ETL tasks through prompts to ensure the effective use of system resources. The specific steps of the embodiment of the present invention are as follows Figure 1 As shown, including:

[0028] Step 101: Obtain at least two ETL tasks.

[0029] Among them, ETL tasks are proposed by various business departments according to their own data needs, and there are a large number of ETL tasks proposed by these business departments in the system cluster. The ETL tasks obtained in this step are these existing or newly added ETL tasks in the system cluster. Therefore, the ETL tasks obtained in this step have nothing to do with how each business department proposes ETL tasks.

[0030] It should be noted that the embodiment of the present invention extracts ETL tasks for similarity judgment and cleans up some similar ETL tasks, which also consumes certain system resources. Therefore, the execution of the embodiment of the present invention may have certain triggering conditions, such as when the number of ETL tasks in the cluster reaches a certain threshold, the calculation and cleanup of similar ETL tasks is triggered, or when the utilization rate of system resources reaches a certain threshold, the calculation and cleanup of similar ETL tasks is triggered. A combination of multiple composite conditions can also be used to trigger the calculation and cleanup of similar ETL tasks. This embodiment does not make specific limitations on this.

[0031] In addition, for each ETL task obtained, it is necessary to obtain the complete task information of the ETL task, including relevant information of data extraction, conversion and loading, so that the similarity of the ETL tasks can be judged based on the task information later.

[0032] Step 102: Determine whether the complexity of the processing logic of the ETL task is greater than a first threshold.

[0033] Since the ETL task belongs to a series of data processing processes, the ETL task can be regarded as an operation combination composed of multiple data processing operations with logical relationships. The complexity of the ETL task can be considered from two dimensions: on the one hand, the data involved in the processing. The more types of data and the larger the amount of data, the higher the complexity of the corresponding ETL task group; on the other hand, the operations involved. The more operations there are, the higher the complexity of the corresponding ETL task group. Therefore, the complexity of the ETL task in this step can be obtained from different dimensions through weighted calculation, and the weight values ​​corresponding to different dimensions can be customized according to the specific application scenario requirements.

[0034] The first threshold in this step is a custom setting, which is used to distinguish the difficulty of the complexity of the ETL task. Since the purpose of the embodiment of the present invention is to calculate the similarity of different ETL tasks and find similar ETL tasks, and the complexity of the ETL task can reflect its processing logic, for the similarity judgment of the ETL task, the similarity of ETL tasks of different complexity is extremely low. Therefore, through this step, the ETL tasks can be preliminarily screened and divided into two categories: complex and simple. In addition, for ETL tasks of different complexity, the embodiment of the present invention adopts different similarity identification methods. Specifically, for the case less than the first threshold, the ETL task is considered to be a simple task. At this time, the similarity between the ETL tasks is determined by step 103; and for the case greater than or equal to the first threshold, the ETL task is considered to be a complex task. At this time, the similarity between the ETL tasks is determined by step 104.

[0035] Step 103: Determine the similarity value of the ETL task according to the metadata in the data warehouse.

[0036] This step is to calculate the similarity of simple ETL tasks. Generally, after screening in step 102, a batch of simple ETL tasks can be obtained. For these ETL tasks, the data content or the number of operations they process are relatively simple. To determine the similarity between different ETL tasks, the similarity can be directly measured by comparing the data content they process and the corresponding operations.

[0037] In actual applications, based on the data content and the corresponding operations, the matched objects can also be preset with weights. For example, different weight values ​​can be set when comparing different types of data or different operations in an ETL task.

[0038] In addition, since the data processed in the ETL task will eventually be loaded into the data warehouse, the data processed by the ETL task and the metadata in the data warehouse have a certain correspondence. In other words, the data content of the same metadata in the data warehouse may come from multiple ETL tasks, and from the perspective of the metadata, these ETL tasks can be considered to be similar. Therefore, the metadata of the data warehouse can be used to judge the similarity of ETL tasks. Similarly, the operations in the ETL task can also be used to judge the similarity of ETL tasks. The metadata can also be matched with the corresponding operations to comprehensively determine the similarity value between ETL tasks.

[0039] Step 104: Determine the similarity value of the ETL task by using a graph matching method.

[0040] This step is to perform similarity calculation for ETL tasks with high complexity. High complexity means that the processing logic of the ETL task is relatively complex and there are many data contents to be processed. To match the similarity of such ETL tasks, the above-mentioned matching method cannot be used, that is, the method of matching the input and output of the ETL task is not enough to represent the internal processing logic of the ETL task. For this reason, the embodiment of the present invention adopts a graph matching method for similarity matching of such ETL tasks. The graph is a natural representation of the coding relationship structure and has been applied in many fields. It can be seen that converting the ETL task into a graph can effectively represent its internal processing logic. Therefore, the similarity calculation of the ETL task in this step can be converted into a graph similarity calculation. For this reason, this step needs to convert the ETL task into a graph first. The specific implementation method is not limited in this embodiment. After that, the similarity value between the ETL tasks is calculated using the graph matching method. Among them, the specific method of graph matching is widely used in the field of artificial intelligence. Therefore, it is not described in detail in this embodiment.

[0041] Step 105: output prompt information, wherein the prompt information contains the ETL task pairs whose similarity values ​​are greater than the second threshold.

[0042] The purpose of this step is to provide prompts and reports. For ETL tasks that are considered similar, that is, ETL task pairs with a similarity value greater than the second threshold, the system administrator can be notified by outputting prompt information, and they can decide whether to clean up the similar ETL tasks. For dissimilar ETL tasks, no processing is required, and the system can perform subsequent processing according to the normal process.

[0043] Through the description of the above embodiments, the processing method of an ETL task provided by the embodiment of the present invention is mainly intended to identify a large number of similar ETL tasks in the cluster to prompt the management personnel to clean up. In the identification process, the embodiment of the present invention first classifies the ETL tasks according to the complexity of processing them, uses the metadata in the data warehouse as the standard to directly match the similarity between simple ETL tasks, converts the complex ETL tasks into a graph, and then uses the graph matching method to determine the similarity between the complex ETL tasks. Finally, the ETL task pairs with similarity greater than the second threshold are output so that the management personnel can quickly clean up, thereby reducing the occupation and waste of system computing and storage resources. It can be seen that the method provided by the present invention can be applied to various application scenarios that have a demand for data processing and data development, especially data warehouses in various data processing and development platforms. Through the present invention, the ETL tasks in the cluster where the platform is located can be analyzed and cleaned more effectively, saving relevant computing and storage resources for platform users, and also saving computing and storage costs for platform users. And the application of the present invention in the development platform can assist users in cleaning up similar ETL tasks, reduce the waste of resources caused by repeated development, and improve the development efficiency of users.

[0044] Furthermore, the embodiments of the present invention will focus on the above Figure 1 The specific implementation of the important steps in the method is described one by one, wherein the important steps specifically refer to the above Figure 1 Step 102, step 103 and step 104 in .

[0045] With respect to step 102, ie, determining the complexity of the ETL task processing logic, in a specific embodiment of the present invention, determining the complexity of an ETL task requires parsing the task, and using the parsed data to determine its complexity.

[0046] Specifically, the ETL task is first parsed according to the processing level and operation to obtain the processing information of the ETL task. Among them, the processing level refers to the logical level determined according to the specified protocol. For example, taking the ETL automation protocol of Ant as an example, an ETL task can be parsed into three layers: Job, Target, and Task, and the automatic scheduling of tasks is realized based on these three layers. These three layers have a top-down disassembly relationship, that is, the Job is the top layer, which contains multiple middle-level Target targets, and the Target contains multiple Task tasks. The data content in different levels also has corresponding operation information, such as counting and accumulating operations on the data in the data table. The hierarchical data content and the operations corresponding to the data obtained after parsing are the processing information obtained by the ETL task parsing. The data content in the processing information specifically includes the data table and the fields contained in the table, and the operation specifically includes the actions performed on the data content.

[0047] Afterwards, the complexity of the processing logic of the ETL task is determined by comparing the amount of processed information with the first threshold. The amount of processed information can be specified according to the needs, which can be the amount of data content, the amount of operations, or the total amount of the two. Of course, according to the needs, it can also be specified as the amount of data and / or operations of a certain level of content in the processed information. It can also be calculated according to the preset weights. The weighted sum of the data table, field, and the operations performed is used to highlight the importance of certain data content or operations. The first threshold is an empirical value that is custom set according to the needs. When it is lower than the first threshold, the ETL task is determined to be a simple task, otherwise it is determined to be a complex task.

[0048] For step 103, i.e., similarity calculation for simple ETL tasks, it uses metadata in the data warehouse for simple matching. It should be noted here that the reason why the graph matching method of complex tasks is not used for simple ETL tasks is that: the processing logic of simple ETL tasks is relatively small, and it is easy to generate sparse graphs by converting them into graphs, and sparse graphs do not use the graph matching method to calculate similarity, that is, using the graph matching method for simple ETL tasks will reduce the accuracy of similarity calculation. Therefore, for simple ETL tasks, only the input and output data of the task can be used for similarity judgment, that is, the metadata in the data warehouse is used to determine the similarity. Specifically: first, the metadata in the data warehouse is obtained, and the metadata at least includes data tables and fields in the tables; then, the corresponding relationship between the processing information in the ETL task and the metadata is determined, where the processing information is the data content obtained after parsing, at least including data tables, fields in the tables, and the operations performed; finally, the similarity value between the ETL tasks is determined according to the corresponding relationship. The similarity can be measured according to the number of identical metadata corresponding to different ETL tasks. Generally, the similarity value can be calculated by normalizing the total amount of metadata possessed by all ETL tasks.

[0049] Regarding step 104, i.e., similarity calculation for complex ETL tasks, the specific implementation of the graph matching method used is mainly divided into two steps. The first step is to convert the ETL task into a graph, and the second step is to match the similarity of the graph.

[0050] For the first step of graph construction, a specific implementation method of the embodiment of the present invention is to construct a directed acyclic graph based on the processing information obtained by parsing the ETL task. The information corresponding to the metadata in the data warehouse contained in the processing information is the node of the directed acyclic graph, and the operation information in the processing information is the edge of the directed acyclic graph. The specific steps of constructing the directed acyclic graph are as follows: Figure 2 As shown, including:

[0051] Step 201: parse the data information corresponding to the metadata in the ETL task according to the processing level, and use the data information as a node in a directed acyclic graph.

[0052] Step 202: parse the operation information corresponding to the data information in the ETL task, and use the operation information as an edge in a directed acyclic graph.

[0053] Step 203: construct a directed acyclic graph using the data information and the operation information.

[0054] Based on the above example of parsing ETL tasks with Ant's ETL automation protocol, the process of building a directed acyclic graph is explained: Since ETL tasks can be parsed into three layers: Job, Target, and Task, where the data in the Task layer is the smallest execution unit, a Task specifically refers to an executable SQL code, and this code is a minimum ETL unit that cannot be further split. In addition, this code also includes objects and operations, where an object can specifically refer to a data table or a field in a table, and an operation refers to a specific operation corresponding to the object, such as counting, summing, and other operations. Through top-down parsing, it can be seen that a Job can be parsed into a series of Targets, and a Target can be further parsed into a series of Tasks. In addition, the data in the same layer also has a certain execution order. Therefore, based on the parsed data content, operations, and related execution order, a directed acyclic graph can be constructed. The nodes of the directed acyclic graph are the objects in the data content of each layer parsed, and the edges between the nodes are the operations and execution order corresponding to the objects. It should be noted that the data objects, operations, and execution sequences in each layer in the above examples can each constitute an independent directed acyclic graph.

[0055] Furthermore, when constructing a directed acyclic graph, based on the parsed hierarchy, an embodiment of the present invention can also mark the corresponding hierarchical information for each node in the graph to distinguish nodes of different hierarchies. The main purpose of this marking operation is to facilitate the configuration of weight values ​​for each node in the graph according to the weights of different hierarchies during subsequent similarity matching.

[0056] For the graph matching in the second step, a specific matching method of the embodiment of the present invention is as follows:

[0057] First, obtain the processing level corresponding to each node in the directed acyclic graph and the weight value corresponding to each processing level; specifically, it can be obtained through the level information marked on the node mentioned above, and the weight value is customized according to the needs of the specific application.

[0058] Afterwards, the weight values ​​are used to perform fuzzy matching on the nodes in the directed acyclic graph corresponding to different ETL tasks to determine the similarity values ​​between different ETL tasks.

[0059] For fuzzy matching of graph similarity, the embodiment of the present invention can adopt the Graph Kernal method, because this method is directly oriented to graph structure data, which not only retains the advantage of efficient kernel function calculation, but also contains the structured information of graph data in Hilbert high-dimensional space. The specific process of this graph matching method is: first encode the node labels in the graph, and construct a dictionary to store the mapping relationship between node labels and codes; then, based on the decomposition of the nodes in the graph, take each node as the root node to decompose the tree structure containing all its adjacent nodes, and obtain the subtree structure after the graph decomposition; then, define a hash operation, and use the label encoding of the root node and its adjacent nodes to generate a new encoding to update the node encoding. In this way, each updated code actually contains the information of its subtree structure, and such an update can be iterated multiple times, and each iteration will contain more information. Finally, the similarity of graphs is determined by measuring the similarity of node encodings in different graphs. In the process of calculating the similarity of matching graphs, by adding weights to nodes at different levels, the important operations in the ETL task can be more highlighted, making the graph matching have a certain tendency. In this way, the obtained similarity value can be more inclined to the artificially set processing logic, thereby improving the accuracy of similarity matching.

[0060] In addition, since the above-mentioned Graph Kernal method is a widely used approximate measurement method of graph structure similarity, the specific implementation principle of the method is not described in detail in the embodiment of the present invention.

[0061] Based on the above Figure 1 From the detailed description of each step, it can be seen that the embodiment of the present invention is mainly aimed at a large number of ETL tasks received in the cluster, classifies these ETL tasks according to their complexity, simply matches simple ETL tasks according to their input and output data to find similar tasks, converts complex ETL tasks into a directed acyclic graph, and finds similar tasks through graph matching. Finally, similar ETL tasks are output in the form of prompt information so that system administrators can perform further cleanup operations.

[0062] Furthermore, as a response to the above Figure 1 In order to realize the method shown in the figure, the embodiment of the present invention provides a processing device for ETL tasks, the main purpose of which is to discover and clean up similar ETL tasks to save the computing and storage resources of the system. For the sake of reading, the present device embodiment will not repeat the details of the above method embodiments one by one, but it should be clear that the device in this embodiment can realize all the contents of the above method embodiments. The device is as follows Figure 3 As shown, specifically including:

[0063] An acquisition unit 31, used for acquiring at least two ETL tasks;

[0064] A judging unit 32, configured to judge whether the complexity of the processing logic of the ETL task obtained by the obtaining unit 31 is greater than a first threshold;

[0065] A first determining unit 33, configured to determine a similarity value of the ETL task according to metadata in the data warehouse when the determining unit 31 determines that the complexity is less than a first threshold;

[0066] A second determining unit 34 is configured to determine the similarity value of the ETL task by using a graph matching method when the determining unit 32 determines that the complexity is not less than the first threshold;

[0067] The output unit 35 is used to output prompt information, wherein the prompt information contains the ETL task pairs whose similarity values ​​determined by the first determining unit 33 and the second determining unit 34 are greater than the second threshold.

[0068] Further, such as Figure 4 As shown, the judging unit 32 includes:

[0069] The parsing module 321 is used to parse the ETL task according to the processing level and operation to obtain the processing information of the ETL task;

[0070] The comparison module 322 is used to compare the amount of processing information obtained by the analysis module 321 with the first threshold value to determine the complexity of the processing logic of the ETL task.

[0071] Furthermore, the processing information of the ETL task includes: data tables, fields and operations performed; the quantity of the processing information is a weighted sum quantity value of the data tables, fields and operations performed calculated according to preset weights.

[0072] Further, such as Figure 4 As shown, the second determining unit 34 includes:

[0073] A graph construction module 341 is used to construct a directed acyclic graph according to the processing information obtained by parsing the ETL task, wherein the information corresponding to the metadata in the data warehouse contained in the processing information is the node of the directed acyclic graph, and the operation information in the processing information is the edge of the directed acyclic graph;

[0074] The graph matching module 342 is used to match the similarity values ​​between ETL tasks using the directed acyclic graph obtained by the graph construction module.

[0075] Further, such as Figure 4 As shown, the graph construction module 341 is specifically used for:

[0076] Parsing the data information in the ETL task corresponding to the metadata in the data warehouse according to the processing level, and using the data information as nodes in a directed acyclic graph;

[0077] Parsing operation information corresponding to the data information in the ETL task, and using the operation information as an edge in a directed acyclic graph;

[0078] A directed acyclic graph is constructed using the data information and the operation information.

[0079] Further, such as Figure 4 As shown, the second determining unit 34 includes:

[0080] The marking module 343 is used to mark the corresponding hierarchical information in the nodes of the directed acyclic graph constructed by the graph construction module 341.

[0081] Furthermore, the graph matching module 342 is specifically used for:

[0082] Obtaining a processing level corresponding to each node in the directed acyclic graph and a weight value corresponding to each processing level;

[0083] The weight values ​​are used to perform fuzzy matching on the nodes in the directed acyclic graph corresponding to different ETL tasks to determine the similarity values ​​between the different ETL tasks.

[0084] Further, such as Figure 4 As shown, the first determining unit 33 includes:

[0085] The acquisition module 331 is used to acquire metadata in the data warehouse, where the metadata at least includes data tables and fields;

[0086] A determination module 332, used to determine the correspondence between the processing information in the ETL task and the metadata obtained by the acquisition module 331, wherein the processing information at least includes a data table, a field, and an executed operation;

[0087] The matching module 333 is used to determine the similarity values ​​between the ETL tasks according to the corresponding relationship obtained by the determining module 332 .

[0088] In addition, an embodiment of the present invention further provides a processor, which is used to run a program, wherein the program executes the ETL task processing method provided by any one of the above embodiments when running.

[0089] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0090] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, but do not represent the advantages and disadvantages of the embodiments.

[0091] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0092] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the best mode of the present invention.

[0093] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0094] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0095] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0096] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0097] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0098] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0099] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0100] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0101] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0102] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0103] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for processing an ETL task, the method comprising: Get at least two ETL tasks; Determine whether the complexity of the processing logic of the ETL task is greater than a first threshold; If it is less than the first threshold, determining the similarity value of the ETL task according to the metadata in the data warehouse; Determining the similarity value of the ETL task according to the metadata in the data warehouse includes: Obtaining metadata in a data warehouse, the metadata at least including data tables and fields; determining a correspondence between processing information in the ETL task and the metadata, the processing information at least including data tables, fields and performed operations; determining a similarity value between ETL tasks based on the correspondence; If it is not less than the first threshold, determining the similarity value of the ETL task by using a graph matching method; The determining the similarity value of the ETL task by using a graph matching method includes: Constructing a directed acyclic graph according to the processing information obtained by parsing the ETL task, wherein the information corresponding to the metadata in the data warehouse contained in the processing information is the node of the directed acyclic graph, and the operation information in the processing information is the edge of the directed acyclic graph; using the directed acyclic graph to match the similarity values ​​between the ETL tasks; Outputting prompt information, wherein the prompt information contains the ETL task pairs whose similarity values ​​are greater than the second threshold.

2. The method according to claim 1, characterized in that: Determining whether the complexity of the processing logic of the ETL task is greater than a first threshold includes: Parsing the ETL task according to the processing level and operation to obtain processing information of the ETL task; The complexity of the processing logic of the ETL task is determined according to the comparison between the amount of the processed information and the first threshold.

3. The method according to claim 2, characterized in that The processing information of the ETL task includes: Data tables, fields, and operations performed; The amount of processed information is a weighted sum of the data tables, fields, and operations performed according to preset weights.

4. The method according to claim 1, characterized in that A directed acyclic graph is constructed according to the processing information obtained by parsing the ETL task, including: Parsing the data information in the ETL task corresponding to the metadata in the data warehouse according to the processing level, and using the data information as nodes in a directed acyclic graph; Parsing operation information corresponding to the data information in the ETL task, and using the operation information as an edge in a directed acyclic graph; A directed acyclic graph is constructed using the data information and the operation information.

5. The method according to claim 4, characterized in that The method further comprises: The corresponding hierarchical information is marked in the nodes of the directed acyclic graph.

6. The method according to claim 1, characterized in that Using the directed acyclic graph to match similarity values ​​between ETL tasks includes: Obtaining a processing level corresponding to each node in the directed acyclic graph and a weight value corresponding to each processing level; The weight values ​​are used to perform fuzzy matching on the nodes in the directed acyclic graph corresponding to different ETL tasks to determine the similarity values ​​between the different ETL tasks.

7. A device for processing an ETL task, the device comprising: An acquisition unit, used for acquiring at least two ETL tasks; A judging unit, configured to judge whether the complexity of the processing logic of the ETL task obtained by the acquiring unit is greater than a first threshold; A first determining unit, configured to determine a similarity value of the ETL task according to metadata in the data warehouse when the determining unit determines that the complexity is less than a first threshold; Determining the similarity value of the ETL task according to the metadata in the data warehouse includes: acquiring the metadata in the data warehouse, the metadata at least including data tables and fields; determining the corresponding relationship between the processing information in the ETL task and the metadata, the processing information at least including data tables, fields and performed operations; determining the similarity value between the ETL tasks according to the corresponding relationship; A second determination unit is used to determine the similarity value of the ETL task by using a graph matching method when the judgment unit determines that the complexity is not less than the first threshold; the determination of the similarity value of the ETL task by using a graph matching method includes: constructing a directed acyclic graph according to the processing information obtained by parsing the ETL task, wherein the information corresponding to the metadata in the data warehouse contained in the processing information is the node of the directed acyclic graph, and the operation information in the processing information is the edge of the directed acyclic graph; and using the directed acyclic graph to match the similarity values ​​between the ETL tasks; The output unit is used to output prompt information, wherein the prompt information contains the ETL task pairs whose similarity values ​​determined by the first determining unit and the second determining unit are greater than a second threshold.

8. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the ETL task processing method described in any one of claims 1 to 6.

Images