Data Lake Long-Link Processing Methods, Devices, Equipment and Storage Media

Abstract

This invention discloses a method, apparatus, device, and storage medium for processing long data links in a data lake. The method includes: traversing the end jobs in the data lake through existing table dependencies to obtain the corresponding long links and their lengths; extracting jobs from the data sets within the long links based on the link lengths to obtain the occurrence frequency and average depth of the long links; locating key points in the long links based on the occurrence frequency and average depth to obtain the key upstream nodes in the long links; and visualizing the long links based on the key upstream nodes and the data sets to obtain a link visualization diagram. Because this invention locates the key upstream nodes of long data links by calculating the occurrence frequency and average depth of the long links within the data lake, and provides a visual representation of each data cluster within the long links, it facilitates database developers in analyzing the data flow in the long links, thereby improving the efficiency of long link analysis.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and storage medium for long-link data lake processing. Background Technology

[0002] As the financial industry continues to develop and business types rapidly increase, the scale of data lakes in the fintech field is expanding accordingly, providing technical support for the development of the financial industry. Within a data lake, newly emerging data tables typically use data from existing data tables for processing. Data tables with usage relationships are usually referred to as forming a link, where the used data table is the upstream of the link, and the data table that uses the data is the downstream of the link. Each data table is a node in the link.

[0003] As a fundamental building block of a data lake, the number of tables and jobs grows rapidly during its expansion. For example, if the end table of a particular data link is repeatedly accessed, the total length of that link may far exceed the average length of links within the data lake. Such links are called long links within the data lake. The emergence of long links can cause many problems, such as poor timeliness of end tables, difficulty in transmitting change notifications from upstream tables, and difficulty in tracing and maintaining the data sources of end tables.

[0004] Traditional long-link governance strategies mainly involve reading each script in the long link, finding the reference relationships of fields in the link, and thus discovering ways to move downstream nodes forward. This method is complex, inefficient, and requires analysis of each link separately, and the results of each analysis are not universally applicable.

[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main objective of this invention is to provide a method, apparatus, device, and storage medium for processing long data lake links, aiming to solve the technical problem of low analysis efficiency caused by the traditional long link governance method which involves manually analyzing the links layer by layer to find the reference relationships of fields in the links.

[0007] To achieve the above objectives, the present invention provides a method for processing long data lake links, the method comprising the following steps:

[0008] By traversing the end jobs in the data lake through the existing table dependencies, the long links and their lengths corresponding to the end jobs can be obtained.

[0009] Based on the link length, the data set in the long link is extracted to obtain the occurrence frequency and average depth of the long link;

[0010] Based on the occurrence frequency and the average depth, key points are located in the long link to obtain the key upstream nodes in the long link.

[0011] The long link is visualized based on the key upstream node and the data set to obtain a link view corresponding to the long link.

[0012] Optionally, the step of extracting data from the dataset in the long link based on the link length to obtain the occurrence frequency and average depth corresponding to the long link includes:

[0013] Extract the corresponding upstream jobs in the long link based on the terminal job and the link length, and construct a data set based on the upstream jobs and the terminal jobs;

[0014] The frequency of occurrence of each job in the dataset is obtained by extracting the frequency using a frequency calculation formula.

[0015] The average depth of the dataset is obtained by performing depth analysis on the dataset and the frequency of occurrence of each job using a depth calculation formula.

[0016] Optionally, the frequency calculation formula is:

[0017] ;

[0018] Where F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long chain.

[0019] Optionally, the step of performing depth analysis on the dataset and the frequency of occurrence of each job using a depth calculation formula to obtain the average depth of the dataset includes:

[0020] The depth calculation formula is as follows:

[0021] ;

[0022] Where D(job_name) represents the average depth, Len(job_name) is the job depth of each job in the link, F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long link;

[0023] Extract the job depth of each job in the dataset;

[0024] The average depth of the dataset is obtained by performing depth analysis on the task depth and occurrence frequency of each task using the depth calculation formula.

[0025] Optionally, the step of traversing the end-point jobs in the data lake through existing table dependencies to obtain the corresponding long links and their lengths in the end-point jobs includes:

[0026] Graph databases corresponding to the data lake are constructed by building graph databases based on existing table dependencies within the data lake.

[0027] The graph database is traversed to obtain the end jobs in the graph database;

[0028] Link extraction is performed based on the terminal operation to obtain long links in the graph database, and the link length of the long links is recorded.

[0029] Optionally, the step of extracting links based on the end-point operation to obtain long links in the graph database and recording the link length of the long links includes:

[0030] The link lengths of each link are extracted from the graph database based on the terminal operation.

[0031] Based on the chain length, determine the long links in the graph database;

[0032] When the chain length reaches a preset threshold, the chain corresponding to the chain length reaching the preset threshold is taken as a long chain in the graph database, and the link length of the long chain is recorded.

[0033] Optionally, the step of visualizing the long link based on the key upstream node and the data set to obtain a link view corresponding to the long link includes:

[0034] Data information of each operation in the long link and key data information of the key upstream nodes are obtained through the data lake;

[0035] The data information and the key data information are standardized to obtain JSON string format data;

[0036] By visualizing the JSON string data through the set relationships of the data set, a link view corresponding to each job in the data lake can be obtained.

[0037] Furthermore, to achieve the above objectives, the present invention also proposes a data lake long-link processing device, the device comprising:

[0038] The long link acquisition module is used to traverse the end jobs in the data lake through the existing table dependencies of the data lake to obtain the long links and their lengths in the end jobs.

[0039] The indicator acquisition module is used to extract the data set in the long link according to the link length, and obtain the occurrence frequency and average depth of the long link.

[0040] The key point localization module is used to locate key points in the long link based on the occurrence frequency and the average depth, and to obtain the key upstream nodes in the long link.

[0041] The link visualization module is used to visualize the long link based on the key upstream node and the data set to obtain a link view corresponding to the long link.

[0042] Furthermore, to achieve the above objectives, the present invention also proposes a long-link data lake processing device, the device comprising: a memory, a processor, and a long-link data lake processing program stored in the memory and executable on the processor, the long-link data lake processing program being configured to implement the steps of the long-link data lake processing method described above.

[0043] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing a data lake long-link processing program, which, when executed by a processor, implements the steps of the data lake long-link processing method as described above.

[0044] This invention traverses the end-point jobs in the data lake by leveraging existing table dependencies to obtain the corresponding long links and their lengths. Then, based on the link length, it extracts jobs from the data sets within the long links to obtain the frequency and average depth of occurrence of each long link. Next, it locates key points in the long links based on the frequency and average depth to identify the critical upstream nodes. Finally, it visualizes the long links using the critical upstream nodes and the data sets to obtain a link view. Because this invention locates the critical upstream nodes of long links within the data lake by calculating their frequency and average depth, and provides visualization of each data cluster within the long links, it facilitates database developers' analysis of data flow within long links, thereby improving the efficiency of long link analysis. Attached Figure Description

[0045] Figure 1 This is a schematic diagram of the structure of the data lake long-link processing device in the hardware operating environment involved in the embodiments of the present invention;

[0046] Figure 2 This is a flowchart illustrating the first embodiment of the data lake long-link processing method of the present invention;

[0047] Figure 3 This is a flowchart illustrating the second embodiment of the data lake long-link processing method of the present invention;

[0048] Figure 4 This is a flowchart illustrating the third embodiment of the data lake long-link processing method of the present invention;

[0049] Figure 5 This is a schematic diagram illustrating the implementation process of visualizing the link in the third embodiment of the data lake long-link processing method of the present invention;

[0050] Figure 6 This is a structural block diagram of the first embodiment of the data lake long-link processing device of the present invention.

[0051] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0053] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of the data lake long-link processing device in the hardware operating environment involved in the embodiments of the present invention.

[0054] like Figure 1 As shown, the data lake long-link processing device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk storage device. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0055] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the long-link processing equipment for the data lake, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0056] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and a data lake long-link processing program.

[0057] exist Figure 1 In the data lake long-link processing device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the data lake long-link processing device of the present invention can be set in the data lake long-link processing device, and the data lake long-link processing device calls the data lake long-link processing program stored in the memory 1005 through the processor 1001 and executes the data lake long-link processing method provided in the embodiment of the present invention.

[0058] This invention provides a method for processing long data lake links, referring to... Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the data lake long-link processing method of the present invention.

[0059] In this embodiment, the data lake long-link processing method includes the following steps:

[0060] Step S10: Traverse the end jobs in the data lake through the existing table dependencies of the data lake to obtain the long links and their lengths in the end jobs.

[0061] It should be noted that the execution subject of the method in this embodiment can be a computing service device with long link extraction, key point localization, and data visualization capabilities, such as a personal computer or server. It can also be other electronic devices capable of performing the same or similar functions, such as the aforementioned data lake long link processing device. This embodiment does not limit this. Here, the aforementioned data lake long link processing device (hereinafter referred to as the long link processing device) will be used to specifically describe this embodiment and the following embodiments.

[0062] Understandably, a data lake refers to a storage system or architecture that stores large amounts of raw, unprocessed data.

[0063] It should be understood that existing table dependencies refer to the relationships between data within a data lake. New data appearing in the data lake typically uses data from existing tables for processing. Tables with usage relationships are often referred to as a "link," where the used table is the upstream and the downstream table is the downstream, with each table being a node in the link. Data tables within the data lake require data to be written through jobs. Correspondingly, jobs within the data lake can also form links, with the nodes being jobs within the data lake. The upstream and downstream relationships between link nodes are defined based on the upstream and downstream relationships of the data tables.

[0064] It should be noted that an end-point job is a task or operation located at the end of a data link layer, used to receive data and perform corresponding processing or responses. By calculating the length of the link where the end-point job resides, the longest link among all upstream links of the end-point job can be determined, and this link is taken as the long link corresponding to the end-point job.

[0065] As a fundamental building block of a data lake, the number of data tables and jobs grows rapidly during the data lake's expansion. This growth manifests in two directions: one is the increase in downstream nodes for a single data table, meaning the same node appears in more links; the other is the increase in link length, as new data tables draw data from the original end table in the link during expansion, increasing the total link length. If, during data lake expansion, the end table of a link is continuously accessed, the total link length will far exceed the average link length within the data lake; such links are called long links within the data lake. The emergence of long links causes many problems, such as poor timeliness of end tables, difficulty in transmitting upstream table change notifications, and difficulty in tracing and maintaining the data source of end tables. Therefore, this embodiment visualizes the links by identifying the long links corresponding to end jobs, facilitating database developers' analysis of data flow within long links and improving the efficiency of long link analysis.

[0066] Step S20: Extract the data set in the long link according to the link length to obtain the occurrence frequency and average depth of the long link.

[0067] It should be noted that the dataset is a collection of data from the jobs at each node in a long link. Frequency of occurrence refers to the frequency with which each job appears in the long link within the data lake. Average depth refers to the depth of each job within each link in the data lake.

[0068] In a specific implementation, the long link processing device can query the number of each job and the node depth of the data set in the long link according to the link length, and determine the occurrence frequency and average depth of the long link according to the number of each job and the node depth.

[0069] Step S30: Based on the occurrence frequency and the average depth, perform key point localization on the long link to obtain the key upstream nodes in the long link.

[0070] It should be noted that the key upstream nodes are those that appear frequently and deeply in long links.

[0071] Step S40: Visualize the long link based on the key upstream node and the data set to obtain a link view corresponding to the long link.

[0072] In practical implementation, long-link processing devices can perform format conversion based on key upstream nodes and data sets, such as processing into JSON string format, to facilitate graphical parsing and ultimately achieve visualization of long-link data, obtaining a link view. This makes it easier for database developers to analyze the data flow in long links, thereby improving the analysis efficiency of long links.

[0073] Furthermore, in this embodiment, step S40 includes: obtaining data information of each job in the long link and key data information of the key upstream node through the data lake; standardizing the data information and the key data information to obtain JSON string format data; and visualizing the JSON string format data through the set relationship of the data set to obtain a link view corresponding to each job in the data lake.

[0074] It's worth noting that JSON (JavaScript Object Notation) is a lightweight data exchange format that can be used for data transfer between the front-end and back-end. Because JSON represents structured data in a concise text format, it is more readable and easier to parse when used for graphical visualization, thus improving the efficiency of graphical visualization.

[0075] In its implementation, the long-link processing device graphically displays all jobs in the long link according to their cluster location. First, it retrieves the data areas and key data information of each job in the long link from the data lake, and standardizes this information into JSON string format. Then, it parses the JSON string data into a visual graph, obtaining a link view corresponding to the long link. Because JSON string format data represents structured data in a concise textual form, it is more readable and easier to parse during graphical visualization, thus improving the efficiency of graphical visualization.

[0076] In this embodiment, the long-link processing device traverses the end-point jobs in the data lake by querying existing table dependencies and calculates the length of the links containing the end-point jobs. The longest link among all upstream links of the end-point job is identified as the long link corresponding to that job. Then, based on the link length, the device queries the number of jobs and node depths of the data set within the long link, determining the frequency and average depth of the long link based on these parameters. Next, key points are located within the long link based on the frequency and average depth, identifying nodes with high frequency and depth as key upstream nodes. Finally, the format of the key upstream nodes and data set is converted, for example, to JSON format for easier graphical parsing, ultimately achieving visualization of the long link data and obtaining a link visualization diagram. Because this embodiment locates key upstream nodes in the data lake's long links by calculating the frequency and average depth of occurrence, it provides a visual representation of each data cluster within the long link, facilitating database developers' analysis of data flow within the long link and thus improving the efficiency of long link analysis.

[0077] refer to Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the data lake long-link processing method of the present invention.

[0078] Based on the first embodiment described above, in this embodiment, considering the accuracy of extracting occurrence frequency and average depth, step S20 includes:

[0079] Step S21: Extract the corresponding upstream job in the long link according to the terminal job and the link length, and construct a data set based on the upstream job and the terminal job.

[0080] It should be noted that upstream operations are tasks or operations located at the beginning or sending end of the data link in the data link layer, used to send data from the source node to the target node and ensure reliable transmission.

[0081] Step S22: Extract the frequency of each job in the dataset using the frequency calculation formula.

[0082] Step S23: Perform depth analysis on the data set and the occurrence frequency of each job using the depth calculation formula to obtain the average depth of the data set.

[0083] It should be noted that the frequency calculation formula is used to calculate the frequency of each job in the dataset. The depth calculation formula is used to calculate the depth of each job in the dataset.

[0084] In practical implementation, upstream jobs corresponding to long links can be extracted based on end-user jobs and link length, and a dataset can be constructed based on upstream and end-user jobs. Then, frequency extraction is performed on the dataset using a frequency calculation formula to obtain the occurrence frequency of each job in the dataset. Finally, depth analysis is performed on the dataset and the occurrence frequency of each job using a depth calculation formula to obtain the average depth of the dataset. This yields a more accurate occurrence frequency and average depth corresponding to long links.

[0085] Furthermore, the frequency calculation formula in this embodiment is as follows:

[0086] ;

[0087] Where F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long chain.

[0088] Further, in this embodiment, step S23 includes: wherein the depth calculation formula is:

[0089] ;

[0090] Where D(job_name) represents the average depth, Len(job_name) is the job depth of each job in the link, F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long link;

[0091] In a specific implementation, by extracting the job depth of each job in the dataset, and then performing depth analysis on the job depth and occurrence frequency of each job using the depth calculation formula, the average depth of the dataset can be obtained, thus obtaining a more accurate average depth corresponding to long links.

[0092] In this embodiment, the long-link processing device predefines a value N in the data lake, defining links with a length greater than or equal to N as long links in the data lake. Simultaneously, for each terminal job of a long link, all its upstream job names are extracted, and the upstream jobs of all terminal jobs of long links are summarized to calculate the occurrence frequency and average depth of each job. By obtaining the occurrence frequency and average depth indicators through the aforementioned frequency calculation formula and depth calculation formula, the key upstream nodes of long links in the data lake can be efficiently located, providing a universally applicable modification scheme for link optimization of various long-link jobs in the data lake.

[0093] refer to Figure 4 , Figure 4 This is a flowchart illustrating the third embodiment of the data lake long-link processing method of the present invention.

[0094] Based on the above embodiments, in this embodiment, considering the determination of long links, step S10 includes:

[0095] Step S11: Construct the graph database corresponding to the data lake by leveraging the existing table dependencies within the data lake.

[0096] Step S12: Traverse the graph database to obtain the end jobs in the graph database.

[0097] It's important to note that graph databases are built upon existing table dependencies. Since a data lake is a storage system or architecture that stores large amounts of raw, unprocessed data, it's necessary to first integrate the data lake's relationships to obtain a graph database, facilitating data relationship lookup.

[0098] Step S13: Extract links based on the end-point operation, obtain long links in the graph database, and record the link length of the long links.

[0099] In its implementation, the long-link processing device constructs a graph database based on existing table dependencies within the data lake, obtaining a graph database corresponding to the data lake. Then, it traverses the graph database based on data relationships to obtain end-point jobs. Finally, it extracts links based on the end-point jobs, obtaining long links in the graph database and recording their lengths. This process integrates the unprocessed data lake into a graph database, facilitating data relationship lookup and improving the efficiency of long-link identification.

[0100] Furthermore, considering the length of long links, in this embodiment, step S13 includes: extracting the length of links in the graph database according to the end-user operation to obtain the chain length corresponding to each link; judging the links in the graph database as long links according to the chain length; when the chain length reaches a preset threshold, taking the link corresponding to the chain length reaching the preset threshold as a long link in the graph database, and recording the link length of the long link.

[0101] It should be noted that the preset threshold is a pre-set threshold used to distinguish long links. When the link length is greater than the preset threshold, the corresponding link is considered a long link. The preset threshold can be set based on the proportion of links in the graph database. For example, if 10% of the links in the graph database are considered long links, then the length of the 10% link can be used as the preset threshold. Alternatively, other common link lengths can be used as the preset threshold. This embodiment does not limit this.

[0102] In a specific implementation, the long link processing device extracts the length of the links in the graph database according to the end-user operation to obtain the chain length corresponding to each link; then, it judges the links in the graph database as long links based on the chain length; when the chain length reaches a preset threshold, the link corresponding to the chain length reaching the preset threshold is taken as the long link in the graph database, thereby accurately determining the long links in the graph database.

[0103] For ease of understanding, the process of visualizing the link is used as an example, but this does not limit the scope of this solution. References Figure 5 , Figure 5 This is a schematic diagram illustrating the implementation process of visualizing the data lake long-link processing method in the third embodiment of the present invention. Based on the above embodiments, the data lake long-link processing method can be divided into three modules, such as... Figure 5 The module shown comprises a link extraction module, a key node localization module, and a link visualization module. The specific process is as follows:

[0104] 1. Link Extraction Module: The link extraction module can build a graph database based on the existing table dependencies within the data lake, and then perform hierarchical extraction, traversing the end jobs in the graph database to obtain a list of long-link end jobs. Then, it performs upstream extraction of end jobs, finds the longest link among all upstream links of the end job, and records the link length of each end job.

[0105] Because upstream and downstream dependencies exist within the data lake, and to ensure the daily normal operation of jobs, multiple job dependencies cannot form a cycle. Therefore, the generated graph database is a directed acyclic graph, which satisfies the condition for extracting the upstream links of the end nodes.

[0106] 2. Critical Node Location Module: In a data lake, a value N is defined, and links with a length greater than or equal to N are called long links in that data lake. The critical node location module can extract the link job details of the terminal job of each long link, obtain all its upstream job names, and summarize the upstream jobs of each terminal job of long links to calculate the occurrence frequency and average depth of each job.

[0107] Based on the two indicators of frequency of occurrence and average depth, key upstream nodes of long links in the data lake can be efficiently located, providing a universally applicable transformation solution for link optimization of various long link operations in the data lake.

[0108] 3. Link Visualization Module: This module graphically displays all jobs in a long link by their respective clusters. First, it retrieves data such as the data area, job name, and cluster information for each job in the long link from the graph database. This information is then standardized into a JSON string format, which is subsequently parsed into a visual graph to obtain the link visualization. Ultimately, this completes the partitioned display of the cluster and data area of ​​the long link, facilitating database developers' analysis of data flow within the long link and thus improving the efficiency of long link analysis.

[0109] In this embodiment, the long-link processing device constructs a graph database based on existing table dependencies within the data lake, obtaining a graph database corresponding to the data lake. Then, it traverses the data relationships within the graph database to obtain end-user jobs. Finally, it extracts links based on the end-user jobs to obtain long links in the graph database and records their lengths. This integrates the unprocessed data lake into a graph database, facilitating data relationship lookup and improving the efficiency of long-link identification. Furthermore, the long-link processing device extracts the lengths of links in the graph database based on end-user jobs, obtaining the chain lengths corresponding to each link. Then, it determines whether a link in the graph database is a long link based on its chain length. When a chain length reaches a preset threshold, the link corresponding to that threshold is considered a long link in the graph database, thus accurately identifying long links in the graph database.

[0110] Furthermore, this embodiment of the invention also proposes a storage medium storing a data lake long-link processing program, which, when executed by a processor, implements the steps of the data lake long-link processing method described above.

[0111] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the data lake long-link processing device of the present invention.

[0112] like Figure 6 As shown, the data lake long-link processing device proposed in this embodiment of the invention includes:

[0113] The long link acquisition module 601 is used to traverse the end jobs in the data lake through the existing table dependencies of the data lake to obtain the long links and their lengths in the end jobs.

[0114] The indicator acquisition module 602 is used to perform job extraction on the data set in the long link according to the link length, and obtain the occurrence frequency and average depth of the long link.

[0115] The key point localization module 603 is used to locate key points in the long link based on the occurrence frequency and the average depth, and to obtain the key upstream nodes in the long link.

[0116] The link visualization module 604 is used to perform visualization processing on the long link based on the key upstream node and the data set to obtain the link visualization view corresponding to the long link.

[0117] In this embodiment, the long-link processing device traverses the end-point jobs in the data lake by querying existing table dependencies and calculates the length of the links containing the end-point jobs. The longest link among all upstream links of the end-point job is identified as the long link corresponding to that job. Then, based on the link length, the device queries the number of jobs and node depths of the data set within the long link, determining the frequency and average depth of the long link based on these parameters. Next, key points are located within the long link based on the frequency and average depth, identifying nodes with high frequency and depth as key upstream nodes. Finally, the format of the key upstream nodes and data set is converted, for example, to JSON format for easier graphical parsing, ultimately achieving visualization of the long link data and obtaining a link visualization diagram. Because this embodiment locates key upstream nodes in the data lake's long links by calculating the frequency and average depth of occurrence, it provides a visual representation of each data cluster within the long link, facilitating database developers' analysis of data flow within the long link and thus improving the efficiency of long link analysis.

[0118] Based on the first embodiment of the data lake long-link processing device of the present invention, a second embodiment of the data lake long-link processing device of the present invention is proposed.

[0119] In this embodiment, the indicator acquisition module 602 is further configured to extract the corresponding upstream jobs in the long link based on the terminal job and the link length, and construct a data set based on the upstream jobs and the terminal jobs; extract the frequency of occurrence of each job in the data set by using a frequency calculation formula; and perform depth analysis on the data set and the frequency of occurrence of each job by using a depth calculation formula to obtain the average depth of the data set.

[0120] Furthermore, the indicator acquisition module 602 also includes: the frequency calculation formula is:

[0121] ;

[0122] Where F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long chain.

[0123] Furthermore, the indicator acquisition module 602 also includes: the depth calculation formula is:

[0124] ;

[0125] Where D(job_name) represents the average depth, Len(job_name) is the job depth of each job in the link, F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long link;

[0126] The indicator acquisition module 602 is also used to extract the job depth of each job in the data set; and to perform depth analysis on the job depth and the frequency of occurrence of each job through the depth calculation formula to obtain the average depth of the data set.

[0127] Furthermore, the long link acquisition module 601 is also used to construct a graph database through the existing table dependencies in the data lake to obtain the graph database corresponding to the data lake; traverse the graph database to obtain the end jobs in the graph database; extract links based on the end jobs to obtain long links in the graph database, and record the link length of the long links.

[0128] Furthermore, the long link acquisition module 601 is also used to extract the length of the links in the graph database according to the end job to obtain the chain length corresponding to each link; to determine the long links in the graph database according to the chain length; and when the chain length reaches a preset threshold, to take the link corresponding to the chain length reaching the preset threshold as the long link in the graph database and record the link length of the long link.

[0129] Furthermore, the link visualization module 604 is also used to obtain data information of each job in the long link and key data information of the key upstream node through the data lake; to standardize the data information and the key data information to obtain JSON string format data; and to visualize the JSON string format data through the set relationship of the data set to obtain the link visualization view corresponding to each job in the data lake.

[0130] Other embodiments or specific implementations of the data lake long-link processing device of the present invention can be referred to the above-described method embodiments, and will not be repeated here.

[0131] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0132] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0134] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for processing long data lake links, characterized in that, The data lake long-link processing method includes: By traversing the end jobs in the data lake through the existing table dependencies, the long links and their lengths corresponding to the end jobs are obtained; wherein, a value N is defined in the data lake, and links with a length greater than or equal to the value N are the long links of the data lake. Based on the link length, the data set in the long link is extracted to obtain the occurrence frequency and average depth of the long link; Based on the occurrence frequency and the average depth, key points are located in the long link to obtain the key upstream nodes in the long link. Based on the key upstream nodes and the data set, the long link is visualized to obtain a link view corresponding to the long link; The step of extracting data from the data set in the long link based on the link length to obtain the occurrence frequency and average depth corresponding to the long link includes: Extract the corresponding upstream jobs in the long link based on the terminal job and the link length, and construct a data set based on the upstream jobs and the terminal jobs; Frequency extraction is performed on the dataset to obtain the occurrence frequency of each job in the dataset. The formula for calculating the occurrence frequency is as follows: ； Where F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long chain; Extract the job depth of each job in the dataset; A depth analysis is performed on the task depth and frequency of occurrence of each task to obtain the average depth of the dataset. The formula for calculating the average depth is as follows: ； Where D(job_name) represents the average depth, Len(job_name) is the job depth of each job in the link, F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long link.

2. The data lake long-link processing method as described in claim 1, characterized in that, The step of traversing the end-point jobs in the data lake based on existing table dependencies to obtain the corresponding long links and their lengths in the end-point jobs includes: Graph databases are constructed by leveraging existing table dependencies within the data lake to obtain the graph database corresponding to the data lake. The graph database is traversed to obtain the end jobs in the graph database; Link extraction is performed based on the terminal operation to obtain long links in the graph database, and the link length of the long links is recorded.

3. The data lake long-link processing method as described in claim 2, characterized in that, The step of extracting links based on the end-point operation to obtain long links in the graph database and recording the link length of the long links includes: The link lengths of each link are extracted from the graph database based on the terminal operation; Based on the chain length, determine the long links in the graph database; When the chain length reaches a preset threshold, the chain corresponding to the chain length reaching the preset threshold is taken as a long chain in the graph database, and the link length of the long chain is recorded.

4. The data lake long-link processing method as described in claim 1, characterized in that, The step of visualizing the long link based on the key upstream node and the data set to obtain a link view corresponding to the long link includes: Data information of each operation in the long link and key data information of the key upstream nodes are obtained through the data lake; The data information and the key data information are standardized to obtain JSON string format data; By visualizing the JSON string data through the set relationships of the data set, a link view corresponding to each job in the data lake can be obtained.

5. A data lake long-link processing device, characterized in that, The device includes: The long link acquisition module is used to traverse the end jobs in the data lake through the existing table dependencies of the data lake to obtain the long links and their lengths in the end jobs; wherein, a value N is defined in the data lake, and links with a length greater than or equal to the value N are the long links of the data lake. The indicator acquisition module is used to extract the data set in the long link according to the link length, and obtain the occurrence frequency and average depth of the long link. The key point localization module is used to locate key points in the long link based on the occurrence frequency and the average depth, and to obtain the key upstream nodes in the long link. The link visualization module is used to visualize the long link based on the key upstream node and the data set to obtain a link visualization view corresponding to the long link. The indicator acquisition module is further configured to extract the corresponding upstream jobs in the long link based on the terminal job and the link length, and construct a data set based on the upstream jobs and the terminal jobs; perform frequency extraction on the data set to obtain the occurrence frequency of each job in the data set, wherein the calculation formula for the occurrence frequency is: ； Where F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long chain; The indicator acquisition module is further configured to extract the job depth of each job in the dataset; perform depth analysis on the job depth and the frequency of occurrence of each job to obtain the average depth of the dataset, wherein the formula for calculating the average depth is: ； Where D(job_name) represents the average depth, Len(job_name) is the job depth of each job in the link, F(job_name) represents the frequency of occurrence of each job, and up_job_name_list represents the data set of the end job and its upstream jobs in the long link.

6. A data lake long-link processing device, characterized in that, The device includes: a memory, a processor, and a data lake long-link processing program stored in the memory and executable on the processor, the data lake long-link processing program being configured to implement the steps of the data lake long-link processing method as described in any one of claims 1 to 4.

7. A storage medium, characterized in that, The storage medium stores a data lake long-link processing program, which, when executed by a processor, implements the steps of the data lake long-link processing method as described in any one of claims 1 to 4.

Images