Data processing link determination method and device and computer readable storage medium

Abstract

The invention relates to the technical field of cloud computing, and provides a data processing link determination method and device and a computer readable storage medium. The method comprises the following steps: extracting data tables from a database and an operation dependency relationship between the data tables, constructing a directed weighted graph according to the data tables and the operation dependency relationship, obtaining a target table pair, searching a shortest processing path of the target table pair by adopting a first graph search algorithm according to the directed weighted graph, and obtaining a second processing path of the target table pair according to the directed weighted graph. Determining a first preset number of shortest and lengthy processing links of the target table pair by adopting a second graph search algorithm, and merging redundant steps in the first preset number of shortest and lengthy processing links according to a preset reconstruction processing logic to obtain a plurality of extended processing paths, and sending the shortest processing path and the plurality of extended processing paths corresponding to the target table pair to the user side. The problem of computing resource waste caused by low link optimization efficiency in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and more specifically, to a method, apparatus, computer-readable storage medium, and computer program product for determining a data processing link. Background Technology

[0002] Data lineage graphs contain massive amounts of data, reaching hundreds of millions of entities, and are crucial for the maintenance, management, and protection of data assets. In practice, numerous data processing chains exist, and developers often randomly invoke these chains based on business needs and scenarios without considering optimization. This results in a large number of lengthy and redundant data processing chains, leading to significant waste of computing resources and a substantial increase in subsequent data maintenance and governance costs. Summary of the Invention

[0003] The main objective of this application is to provide a method, apparatus, computer-readable storage medium, and computer program product for determining data processing links, so as to at least solve the problem of low link optimization efficiency leading to wasted computing resources in the prior art.

[0004] To achieve the above objectives, according to one aspect of this application, a method for determining data processing links is provided, comprising: a construction step: extracting data tables and job dependencies between the data tables from a database, and constructing a directed weighted graph based on the data tables and job dependencies, wherein the directed weighted graph includes nodes, edges, and edge weights, the nodes representing data tables, the edges representing the job dependencies between the data tables, and the edge weights representing the computational cost of the edges; a determination step: obtaining target table pairs, searching for the shortest processing path of the target table pairs using a first graph search algorithm based on the directed weighted graph, and determining the first preset number of shortest redundant processing links of the target table pairs using a second graph search algorithm based on the directed weighted graph, wherein the target table pairs include a starting data table and an ending data table of the data processing links; and a processing step: merging redundant steps in the first preset number of shortest redundant processing links according to a preset reconstruction processing logic to obtain multiple extended processing paths, and sending the shortest processing path and the multiple extended processing paths corresponding to the target table pairs to a user terminal.

[0005] Optionally, the method further includes: obtaining a set of endpoint data tables, traversing each endpoint data table in the set of endpoint data tables, determining the starting data table corresponding to each endpoint data table, and, based on each endpoint data table in the set of endpoint data tables and the starting data table corresponding to each endpoint data table, executing the determination step and the processing step multiple times to obtain a set of processing paths, wherein the set of processing paths includes the shortest processing path and multiple extended processing paths corresponding to each endpoint data table in the set of endpoint data tables and the starting data table corresponding to each endpoint data table.

[0006] Optionally, after obtaining the set of processing paths, the method further includes: determining the degree of each of the data tables in each of the shortest processing paths, wherein the degree represents the number of processing relationships associated with the data table; and determining the high-voltage table in the directed weighted graph based on the degree of each of the data tables.

[0007] Optionally, after determining the high-voltage table in the directed weighted graph based on the degree of each of the data tables, the method further includes: performing backup processing on the high-voltage table in each of the shortest processing paths.

[0008] Optionally, after determining the high-voltage table in the directed weighted graph based on the degree of each of the data tables, the method further includes: creating multiple high-voltage sub-tables corresponding to each of the high-voltage tables based on the degree of each of the high-voltage tables in the directed weighted graph; and evenly distributing the operation processing relationship of each of the high-voltage tables to the multiple high-voltage sub-tables corresponding to each of the high-voltage tables.

[0009] Optionally, determining the high-voltage table in the directed weighted graph based on the degree of each of the data tables includes: identifying the data tables whose degree exceeds a degree threshold as the high-voltage table in the directed weighted graph.

[0010] Optionally, the method further includes: determining the edge weights in the directed weighted graph based on the number of processor cores used, processor utilization, memory usage, and computation time used to calculate the job dependencies between the data tables.

[0011] According to another aspect of this application, a data processing link determination apparatus is provided, comprising: a construction unit, configured to perform the following construction steps: extracting data tables from a database and job dependencies between the data tables, and constructing a directed weighted graph based on the data tables and job dependencies, wherein the directed weighted graph includes nodes, edges and edge weights, the nodes representing data tables, the edges representing the job dependencies between the data tables, and the edge weights representing the computational cost of the edges; a first determination unit, configured to perform the following determination steps: obtaining target table pairs, searching for the shortest processing path of the target table pairs using a first graph search algorithm based on the directed weighted graph, and determining the first preset number of shortest redundant processing links of the target table pairs using a second graph search algorithm based on the directed weighted graph, wherein the target table pairs include a starting data table and an ending data table of the data processing link; and a processing unit, configured to perform the following processing steps: merging redundant steps in the first preset number of shortest redundant processing links according to a preset reconstruction processing logic to obtain multiple extended processing paths, and sending the shortest processing path and the multiple extended processing paths corresponding to the target table pairs to a user terminal.

[0012] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform any of the data processing link determination methods described above.

[0013] According to another aspect of this application, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods for determining the data processing link.

[0014] By applying the technical solution of this application, and through constructing a directed weighted graph and employing a graph search algorithm, the problems of redundant data processing links leading to wasted computing resources and the lack of efficient disaster recovery and load balancing strategies in existing technologies are solved. Specifically, in the construction step, the method extracts data tables and job dependencies from the database and constructs a directed weighted graph including nodes, edges, and edge weights, where the edge weights reflect the computational cost of data processing. In the determination step, a first graph search algorithm, such as Dijkstra's algorithm, is used to search for the shortest processing path for the target table pair. Simultaneously, a second graph search algorithm, such as Yen's K shortest path algorithm, is used to determine the first preset number of shortest redundant processing links for the target table pair, achieving automated identification of redundant processing links. In the processing step, according to the preset reconstruction processing logic, redundant steps in the determined redundant processing links are merged to generate multiple extended processing paths, significantly reducing redundant calculations and thus saving computing resources. Simultaneously, the optimized shortest processing path and multiple extended processing paths are sent to the user terminal, facilitating user decision-making and execution, and improving the efficiency and flexibility of the data processing link. Attached Figure Description

[0015] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0016] Figure 1 A hardware structure block diagram of a mobile terminal for determining a data processing link according to an embodiment of this application is shown.

[0017] Figure 2 A flowchart illustrating a method for determining a data processing link according to an embodiment of this application is shown.

[0018] Figure 3 A structural block diagram of a data processing link determination apparatus provided according to an embodiment of this application is shown. Detailed Implementation

[0019] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] As described in the background section, the low efficiency of link optimization in the prior art leads to a waste of computing resources. To address this problem, embodiments of this application provide a method, apparatus, computer-readable storage medium, and computer program product for determining data processing links.

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0024] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a data processing link determination method according to an embodiment of the present invention. For example... Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1The different configurations shown.

[0025] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data processing link determination method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0026] This embodiment provides a method for determining a data processing link running on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0027] Figure 2 This is a flowchart of a method for determining a data processing link according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0028] Step S201, Construction Step: Extract data tables from the database and the job dependencies between the data tables, and construct a directed weighted graph based on the data tables and the job dependencies. The directed weighted graph includes nodes, edges and edge weights. The nodes represent data tables, the edges represent the job dependencies between the data tables, and the edge weights represent the computational cost of the edges.

[0029] Job dependencies include job extraction, job transformation, and job loading.

[0030] Step S202, Determine the steps: Obtain the target table pair, and according to the above directed weighted graph, use the first graph search algorithm to search for the shortest processing path of the target table pair, and according to the above directed weighted graph, use the second graph search algorithm to determine the first preset number of shortest redundant processing links of the target table pair, wherein the target table pair includes the starting data table and the ending data table of the data processing link;

[0031] Specifically, the first graph search algorithm can employ Dijkstra's algorithm or A. The second graph search algorithm can use Yen's K shortest path algorithm. The preset number can be set according to specific optimization needs, such as 5 or 10.

[0032] Step S203, Processing steps: According to the preset reconstruction processing logic, the redundant steps in the preset number of the above-mentioned shortest redundant processing links are merged and processed to obtain multiple extended processing paths, and the above-mentioned target table pairs corresponding to the above-mentioned shortest processing paths and multiple above-mentioned extended processing paths are sent to the user terminal.

[0033] The data processing chain is the data processing path from the starting data table to the ending data table, consisting of a series of ETL (Extract-Transform-Load) jobs.

[0034] This embodiment, by applying steps S201, S202, and S203, constructs a directed weighted graph and employs a graph search algorithm, thus solving the problems of redundant data processing links leading to wasted computing resources and the lack of efficient disaster recovery and load balancing strategies in existing technologies. Specifically, in the construction step, the method extracts data tables and job dependencies from the database to construct a directed weighted graph including nodes, edges, and edge weights, where the edge weights reflect the computational cost of data processing. In the determination step, a first graph search algorithm, such as Dijkstra's algorithm, is used to search for the shortest processing path for the target table pair. Simultaneously, a second graph search algorithm, such as Yen's K shortest path algorithm, is used to determine the first preset number of shortest redundant processing links for the target table pair, achieving automated identification of redundant processing links. In the processing step, according to a preset reconstruction processing logic, redundant steps in the determined redundant processing links are merged to generate multiple extended processing paths, significantly reducing redundant calculations and thus saving computing resources. At the same time, the optimized shortest processing path and multiple extended processing paths are sent to the user terminal to facilitate user decision-making and execution, thereby improving the efficiency and flexibility of the data processing chain.

[0035] In the specific implementation process, the above method further includes: obtaining a set of endpoint data tables, traversing each endpoint data table in the set of endpoint data tables, determining the starting data table corresponding to each endpoint data table, and, based on each endpoint data table in the set of endpoint data tables and the starting data table corresponding to each endpoint data table, executing the determination step and the processing step multiple times to obtain a set of processing paths. The set of processing paths includes the shortest processing path and multiple extended processing paths corresponding to each endpoint data table in the set of endpoint data tables and the starting data table corresponding to each endpoint data table.

[0036] In this embodiment, after obtaining the set of endpoint data tables, each endpoint data table within the set is traversed to accurately determine its corresponding starting data table. Based on this determination result, the above determination and processing steps are executed multiple times to generate a comprehensive set of processing paths for each endpoint data table, including the shortest processing path and multiple extended processing paths. This extended process ensures that the processing links of all key data tables are meticulously optimized and analyzed, not only improving resource utilization efficiency but also enhancing the reliability and response speed of data processing. In this way, the method can effectively cope with the complexity of data lineage graphs and provide a more comprehensive solution for link optimization in big data environments.

[0037] Specifically, after obtaining the set of processing paths, the above method further includes: determining the degree of each of the above data tables in each of the above shortest processing paths, wherein the degree represents the number of processing relationships associated with the above data tables; and determining the high-voltage table in the above directed weighted graph based on the degree of each of the above data tables.

[0038] This embodiment further includes a process for identifying high-pressure tables, which is achieved by analyzing the degree of each data table in the shortest processing path. The degree, i.e., the number of job processing relationships associated with a data table, is used to identify table entities that play a key role in the data lineage graph. These entities are defined as high-pressure tables due to their high frequency of use. The strategy for identifying high-pressure tables helps us to specifically monitor and protect tables that may become bottlenecks or face data quality risks during data processing link optimization. The shortest path calculated by the graph algorithm can accurately count the degree of each table, thereby effectively identifying high-pressure tables. This step not only enhances the robustness of the data processing link but also provides data support for subsequent disaster recovery backup and load balancing strategies, ensuring the efficiency and security of the data link, reducing the risk of data loss or contamination of high-pressure tables, and simultaneously reducing the read / write pressure on high-pressure tables through data migration and job scheduling optimization, thereby improving overall data processing efficiency and system stability.

[0039] Furthermore, after determining the high-voltage table in the directed weighted graph based on the degrees of each of the aforementioned data tables, the method further includes: performing backup processing on the high-voltage table in each of the aforementioned shortest processing paths.

[0040] In this embodiment, in addition to identifying and optimizing the lengthy data processing chain based on the data lineage graph, disaster recovery processing for high-voltage tables is further included. Specifically, once the high-voltage tables in the directed weighted graph are determined by calculating the degree of each data table, backup measures are implemented for them. This strategy aims to prevent data risks caused by frequent access to high-voltage tables, such as data loss or data corruption. By backing up the high-voltage tables, even if the original table experiences data quality problems, the backup table can still serve as a source for data recovery, thereby ensuring the continuity of the data chain and the integrity of the data, and reducing the possibility of business interruption.

[0041] Furthermore, after determining the high-voltage tables in the directed weighted graph based on the degrees of each of the aforementioned data tables, the method further includes: creating multiple high-voltage sub-tables corresponding to each of the aforementioned high-voltage tables based on the degrees of each of the aforementioned high-voltage tables in the directed weighted graph; and evenly distributing the aforementioned operational processing relationships of each of the aforementioned high-voltage tables to the multiple high-voltage sub-tables corresponding to each of the aforementioned high-voltage tables.

[0042] In this embodiment, after determining the high-voltage tables in the directed weighted graph, it further involves creating corresponding high-voltage sub-tables based on the degrees of the high-voltage tables. Specifically, for each high-voltage table, multiple sub-tables are created according to its degree, and then the job processing relationships involving the high-voltage tables are evenly distributed across these sub-tables. This strategy aims to reduce the load pressure on individual high-voltage tables through distributed data storage, avoiding performance bottlenecks caused by frequent reads and writes. By splitting the processing links and directing them to different high-voltage sub-tables, the use of cluster resources can be effectively balanced, data link latency can be reduced, and the overall data processing efficiency and reliability can be improved.

[0043] Specifically, determining the high-voltage table in the directed weighted graph based on the degrees of each of the aforementioned data tables includes: identifying the aforementioned data tables whose degrees exceed the degree threshold as the aforementioned high-voltage tables in the directed weighted graph.

[0044] In this embodiment, high-pressure tables in the directed weighted graph are determined based on the degree of each data table. Specifically, data tables with a degree exceeding a set degree threshold P are identified as high-pressure tables. This strategy is based on fundamental graph theory concepts, where degree refers to the number of edges connecting a node to it; in the context of a data lineage graph, it is the number of times a data table is referenced or depended upon by other tables. By setting a threshold P, data tables that play a crucial role in the network and are frequently accessed can be filtered out. These high-pressure tables may become bottlenecks in the data processing process, affecting the performance and stability of the entire system.

[0045] Furthermore, the above method also includes: determining the edge weights in the directed weighted graph based on the number of processor cores used, processor utilization, memory usage, and computation time used to calculate the job dependencies between the above data tables.

[0046] This embodiment also includes the precise quantification of edge weights in the data lineage graph. Specifically, it involves determining the cost metric for inter-table dependencies by analyzing key parameters such as the number of processor cores used, processor utilization, memory usage, and computation time of ETL jobs. This process aims to comprehensively consider the actual resource consumption and time overhead of data processing jobs, thereby constructing a weighted graph model that more accurately reflects the true state of the data processing chain. In the graph model, the weight of each edge represents the actual cost of processing data from one data table entity to another, making the decision-making basis for exploring the shortest processing chain and rectifying redundant chains more reliable. By converting resource consumption and computation time into graph weights, not only can truly efficient data processing paths be revealed, but also resource-intensive or time-consuming chains can be identified, providing a clear direction for subsequent optimization.

[0047] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the method for determining the data processing link of this application will be described in detail below with reference to specific embodiments.

[0048] This embodiment relates to a specific method for determining a data processing link, including the following steps:

[0049] Step S1: Construct a data lineage map;

[0050] Metadata extraction: Pull entity information of all relevant data tables and their dependent ETL jobs from the enterprise's metadata center to ensure data comprehensiveness and accuracy.

[0051] Graph Construction: Using the acquired data and ETL job dependencies, a directed weighted graph G=(V,E) is constructed, where node V represents a data table and edge E represents a data processing link. The weight w of each edge is determined by factors such as historical job execution time and resource consumption.

[0052] Step S2: Shortest processing link detection;

[0053] For a given target table pair (A, B), set the distance of the starting table A to 0, and the distance of other tables to infinity, and establish a priority queue to store the nodes in the graph.

[0054] An improved Dijkstra algorithm is applied to search for the shortest path from A to B, and Yen's K shortest path algorithm is used to determine the multi-path expansion of the target table pair, obtaining the Top-K paths for subsequent redundant link determination.

[0055] Step S3: Rectification of redundant links;

[0056] Redundant link identifier: Mark processing links that are not the shortest path as redundant links.

[0057] Refactoring Processing Logic: Multiple ETL steps in a lengthy process chain are merged into a single job, processed directly according to the shortest path logic. An example of this refactored processing logic is as follows:

[0058] If there are two processing links from table A to table D, namely table A->table B->table C->table D and table A->table C->table D, then the former is removed and the latter is retained. If some processing logic of table A->table B->table C is not in the processing logic of table A->table C, it can be merged into the processing logic of table A->table C (single job).

[0059] Data migration: The data in the original redundant intermediate tables of the link is migrated to the output table of the new link after reconstruction and processing to ensure data consistency and integrity.

[0060] Job scheduling update: In the enterprise's job scheduling system, replace the original lengthy chain jobs and deploy new jobs with optimized processing logic.

[0061] Step S4: Batch optimization of the entire supply chain;

[0062] Selection of important tables: Determine the set of key data tables T={T1,T2,...,Tn} that need to be optimized.

[0063] Upstream dependency analysis: For each table Ti in set T, trace back all its upstream source tables Sj and evaluate their dependencies.

[0064] Pairwise optimization: For each pair (Sj, Ti), the shortest path exploration and redundant link rectification steps S2~S3 are executed sequentially to generate an optimization instruction set.

[0065] Batch operation execution: Through transaction control mechanisms, optimization operations are committed all at once, ensuring atomic updates of data lineage graphs and processing logic, and avoiding data inconsistency.

[0066] Step S5: High-voltage meter disaster recovery;

[0067] High-voltage meter identification: Calculate the degree of all meters in the data lineage map and find the high-voltage meters whose degree exceeds the preset threshold P.

[0068] Disaster recovery backup: Set up regular backup strategies for all high-voltage meters to ensure rapid recovery even in the event of data loss or corruption, reducing business downtime.

[0069] Step S6: High-voltage meter load balancing;

[0070] High-voltage meter multi-replica creation: Create multiple replicas (e.g., N replicas) for the high-voltage meter, distributing storage to reduce the load on a single meter. The value of N is determined by the meter's readings, and N = (high-voltage meter reading / 100). Rounding up means that 3.1 becomes 4 after rounding up; that is, any non-integer digit is rounded up to the nearest integer.

[0071] Link splitting and balancing: Based on the usage frequency and time distribution of the processing links, the links passing through the high-voltage meters are split, and traffic is directed to different high-voltage meter replicas to achieve load balancing.

[0072] N loads by backing up tables and distributing related processing relationships by associating one high-voltage table copy with every 100 processing relationships. This reduces the access pressure on each backup table and achieves the splitting of the high-voltage table processing chain in this way.

[0073] This embodiment specifically achieves the following technical effects:

[0074] 1) Save computing resources: Eliminate redundant ETL steps and reduce cluster CPU / memory consumption.

[0075] 2) Reduced maintenance costs: Automated link analysis replaces manual investigation, reducing the complexity of operation and maintenance.

[0076] 3) Improved timeliness: Shorten the processing time of critical data.

[0077] 4) Global optimality: Graph algorithms ensure the global optimal solution across multiple levels of dependency, breaking through the locality limitation of the rule base.

[0078] 5) Reduces the load pressure on high-voltage meters and improves the disaster recovery capability of the data processing link.

[0079] This application also provides a data processing link determination device. It should be noted that the data processing link determination device of this application can be used to execute the data processing link determination method provided in this application. This device is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0080] The following describes the data processing link determination device provided in the embodiments of this application.

[0081] Figure 3 This is a schematic diagram of a data processing link determination device according to an embodiment of this application. Figure 3 As shown, the device includes:

[0082] Construction unit 31 is used to perform the construction steps: extracting data tables from the database and the job dependencies between the data tables, and constructing a directed weighted graph based on the data tables and the job dependencies. The directed weighted graph includes nodes, edges and edge weights. The nodes represent data tables, the edges represent the job dependencies between the data tables, and the edge weights represent the computational cost of the edges.

[0083] The first determining unit 32 is used to perform the determining steps: obtaining the target table pair, searching for the shortest processing path of the target table pair using the first graph search algorithm according to the above directed weighted graph, and determining the first preset number of shortest redundant processing links of the target table pair using the second graph search algorithm according to the above directed weighted graph, wherein the target table pair includes the starting data table and the ending data table of the data processing link;

[0084] Processing unit 33 is used to perform the following processing steps: according to the preset reconstruction processing logic, the redundant steps in the first preset number of the above-mentioned shortest redundant processing links are merged and processed to obtain multiple extended processing paths, and the above-mentioned target table pairs are sent to the user terminal along with the above-mentioned shortest processing paths and multiple above-mentioned extended processing paths.

[0085] In this embodiment, the construction unit is used to perform the construction steps: extracting data tables and job dependencies between data tables from the database, and constructing a directed weighted graph based on the data tables and job dependencies, wherein the directed weighted graph includes nodes, edges and edge weights, nodes represent data tables, edges represent job dependencies between data tables, and edge weights represent the computational cost of edges; the first determination unit is used to perform the determination steps: obtaining target table pairs, searching for the shortest processing path of the target table pairs using a first graph search algorithm based on the directed weighted graph, and determining the first preset number of shortest redundant processing links of the target table pairs using a second graph search algorithm based on the directed weighted graph, wherein the target table pairs include the starting data table and the ending data table of the data processing links; the processing unit is used to perform the processing steps: merging redundant steps in the first preset number of shortest redundant processing links according to a preset reconstruction processing logic to obtain multiple extended processing paths, and sending the shortest processing path and multiple extended processing paths corresponding to the target table pairs to the user terminal. By constructing a directed weighted graph and applying a graph search algorithm, this method addresses the problems of redundant data processing links leading to wasted computing resources and the lack of efficient disaster recovery and load balancing strategies in existing technologies. Specifically, in the construction step, the method extracts data tables and job dependencies from the database to construct a directed weighted graph including nodes, edges, and edge weights, where the edge weights reflect the computational cost of data processing. In the determination step, a first graph search algorithm, such as Dijkstra's algorithm, is used to search for the shortest processing path for the target table pair. Simultaneously, a second graph search algorithm, such as Yen's K shortest path algorithm, is employed to determine the first preset number of shortest redundant processing links for the target table pair, achieving automated identification of redundant processing links. In the processing step, based on a preset reconstruction processing logic, redundant steps in the determined redundant processing links are merged to generate multiple extended processing paths, significantly reducing redundant calculations and thus saving computing resources. Simultaneously, the optimized shortest processing path and multiple extended processing paths are sent to the user terminal, facilitating user decision-making and execution, and improving the efficiency and flexibility of the data processing link.

[0086] As an optional solution, the apparatus further includes an acquisition unit, configured to acquire a set of endpoint data tables, traverse each endpoint data table in the set of endpoint data tables, determine the starting data table corresponding to each endpoint data table, and, based on each endpoint data table in the set of endpoint data tables and the starting data table corresponding to each endpoint data table, execute the determination step and the processing step multiple times to obtain a processing path set. The processing path set includes the shortest processing path and multiple extended processing paths corresponding to each endpoint data table in the set of endpoint data tables and the starting data table corresponding to each endpoint data table.

[0087] In one optional embodiment, the apparatus further includes a second determining unit and a third determining unit; the second determining unit is used to determine the degree of each of the aforementioned data tables in each of the aforementioned shortest processing paths after obtaining the processing path set, wherein the degree represents the number of processing relationships associated with the aforementioned data tables; the third determining unit is used to determine the high voltage table in the aforementioned directed weighted graph based on the degree of each of the aforementioned data tables.

[0088] In an alternative embodiment, the apparatus further includes a backup unit for backing up the high-voltage meters in each of the shortest processing paths after determining the high-voltage meters in the directed weighted graph based on the degrees of each of the aforementioned data tables.

[0089] In one optional embodiment, the apparatus further includes a creation unit and an allocation unit; the creation unit is used to create multiple high-voltage sub-tables corresponding to each high-voltage table after determining the high-voltage table in the directed weighted graph based on the degree of each of the aforementioned data tables; the allocation unit is used to evenly distribute the aforementioned operation processing relationship of each of the aforementioned high-voltage tables to the multiple high-voltage sub-tables corresponding to each of the aforementioned high-voltage tables.

[0090] In one alternative, the third determining unit includes a determining module for determining the data table whose degree exceeds the degree threshold as the high voltage table in the directed weighted graph.

[0091] In an alternative embodiment, the apparatus further includes a fourth determining unit, configured to determine the edge weights in the directed weighted graph based on the number of processor cores used, processor utilization, memory usage, and computation time used to calculate the job dependencies between the data tables.

[0092] The aforementioned data processing link determination device includes a processor and a memory. The aforementioned construction unit, first determination unit, processing unit, etc., are all stored as program units in the memory, and the processor executes these program units stored in the memory to achieve the corresponding functions. All of the aforementioned modules are located in the same processor; alternatively, the aforementioned modules may be located in different processors in any combination.

[0093] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can address the problem of low link optimization efficiency leading to wasted computing resources in existing technologies.

[0094] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0095] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the data processing link determination method.

[0096] Specifically, the methods for determining the data processing chain include:

[0097] Step S201, Construction Step: Extract data tables from the database and the job dependencies between the data tables, and construct a directed weighted graph based on the data tables and the job dependencies. The directed weighted graph includes nodes, edges and edge weights. The nodes represent data tables, the edges represent the job dependencies between the data tables, and the edge weights represent the computational cost of the edges.

[0098] Step S202, Determine the steps: Obtain the target table pair, and according to the above directed weighted graph, use the first graph search algorithm to search for the shortest processing path of the target table pair, and according to the above directed weighted graph, use the second graph search algorithm to determine the first preset number of shortest redundant processing links of the target table pair, wherein the target table pair includes the starting data table and the ending data table of the data processing link;

[0099] Step S203, Processing steps: According to the preset reconstruction processing logic, the redundant steps in the preset number of the above-mentioned shortest redundant processing links are merged and processed to obtain multiple extended processing paths, and the above-mentioned target table pairs corresponding to the above-mentioned shortest processing paths and multiple above-mentioned extended processing paths are sent to the user terminal.

[0100] This invention provides a processor for running a program, wherein the program executes the method for determining the data processing link.

[0101] Specifically, the methods for determining the data processing chain include:

[0102] Step S201, Construction Step: Extract data tables from the database and the job dependencies between the data tables, and construct a directed weighted graph based on the data tables and the job dependencies. The directed weighted graph includes nodes, edges and edge weights. The nodes represent data tables, the edges represent the job dependencies between the data tables, and the edge weights represent the computational cost of the edges.

[0103] Step S202, Determine the steps: Obtain the target table pair, and according to the above directed weighted graph, use the first graph search algorithm to search for the shortest processing path of the target table pair, and according to the above directed weighted graph, use the second graph search algorithm to determine the first preset number of shortest redundant processing links of the target table pair, wherein the target table pair includes the starting data table and the ending data table of the data processing link;

[0104] Step S203, Processing steps: According to the preset reconstruction processing logic, the redundant steps in the preset number of the above-mentioned shortest redundant processing links are merged and processed to obtain multiple extended processing paths, and the above-mentioned target table pairs corresponding to the above-mentioned shortest processing paths and multiple above-mentioned extended processing paths are sent to the user terminal.

[0105] This invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0106] Step S201, Construction Step: Extract data tables from the database and the job dependencies between the data tables, and construct a directed weighted graph based on the data tables and the job dependencies. The directed weighted graph includes nodes, edges and edge weights. The nodes represent data tables, the edges represent the job dependencies between the data tables, and the edge weights represent the computational cost of the edges.

[0107] Step S202, Determine the steps: Obtain the target table pair, and according to the above directed weighted graph, use the first graph search algorithm to search for the shortest processing path of the target table pair, and according to the above directed weighted graph, use the second graph search algorithm to determine the first preset number of shortest redundant processing links of the target table pair, wherein the target table pair includes the starting data table and the ending data table of the data processing link;

[0108] Step S203, Processing steps: According to the preset reconstruction processing logic, the redundant steps in the preset number of the above-mentioned shortest redundant processing links are merged and processed to obtain multiple extended processing paths, and the above-mentioned target table pairs corresponding to the above-mentioned shortest processing paths and multiple above-mentioned extended processing paths are sent to the user terminal.

[0109] The devices mentioned in this article can be servers, PCs, tablets, mobile phones, etc.

[0110] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0111] Step S201, Construction Step: Extract data tables from the database and the job dependencies between the data tables, and construct a directed weighted graph based on the data tables and the job dependencies. The directed weighted graph includes nodes, edges and edge weights. The nodes represent data tables, the edges represent the job dependencies between the data tables, and the edge weights represent the computational cost of the edges.

[0112] Step S202, Determine the steps: Obtain the target table pair, and according to the above directed weighted graph, use the first graph search algorithm to search for the shortest processing path of the target table pair, and according to the above directed weighted graph, use the second graph search algorithm to determine the first preset number of shortest redundant processing links of the target table pair, wherein the target table pair includes the starting data table and the ending data table of the data processing link;

[0113] Step S203, Processing steps: According to the preset reconstruction processing logic, the redundant steps in the preset number of the above-mentioned shortest redundant processing links are merged and processed to obtain multiple extended processing paths, and the above-mentioned target table pairs corresponding to the above-mentioned shortest processing paths and multiple above-mentioned extended processing paths are sent to the user terminal.

[0114] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0115] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0117] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0118] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0119] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0120] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0121] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0124] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining a data processing link, characterized in that, include: Construction steps: Extract data tables and job dependencies between the data tables from the database, and construct a directed weighted graph based on the data tables and job dependencies. The directed weighted graph includes nodes, edges and edge weights. The nodes represent data tables, the edges represent the job dependencies between the data tables, and the edge weights represent the computational cost of the edges. Determining steps: Obtain the target table pair; according to the directed weighted graph, use the first graph search algorithm to search for the shortest processing path of the target table pair; and according to the directed weighted graph, use the second graph search algorithm to determine the first preset number of shortest redundant processing links of the target table pair. The target table pair includes the starting data table and the ending data table of the data processing link. Processing steps: Based on the preset reconstruction processing logic, the redundant steps in the first preset number of the shortest redundant processing links are merged to obtain multiple extended processing paths, and the target table corresponding to the shortest processing path and the multiple extended processing paths are sent to the user terminal.

2. The method according to claim 1, characterized in that, The method further includes: Obtain a set of endpoint data tables, traverse each endpoint data table in the set of endpoint data tables, determine the starting data table corresponding to each endpoint data table, and based on each endpoint data table in the set of endpoint data tables and the starting data table corresponding to each endpoint data table, execute the determination step and the processing step multiple times to obtain a set of processing paths. The set of processing paths includes the shortest processing path and multiple extended processing paths corresponding to each endpoint data table in the set of endpoint data tables and the starting data table corresponding to each endpoint data table.

3. The method according to claim 2, characterized in that, After obtaining the processing path set, the method further includes: Determine the degree of each data table in each of the shortest processing paths, wherein the degree represents the number of processing relationships associated with the data table; The high-voltage table in the directed weighted graph is determined based on the degree of each of the data tables.

4. The method according to claim 3, characterized in that, After determining the high-voltage table in the directed weighted graph based on the degrees of each of the data tables, the method further includes: The high-voltage meters in each of the shortest processing paths are backed up.

5. The method according to claim 3, characterized in that, After determining the high-voltage table in the directed weighted graph based on the degrees of each of the data tables, the method further includes: Based on the degree of each high-voltage meter in the directed weighted graph, multiple high-voltage sub-meters corresponding to each high-voltage meter are created; The operation processing relationship of each high voltage meter is evenly distributed to the multiple high voltage sub-meters corresponding to each high voltage meter.

6. The method according to claim 3, characterized in that, Determining the high-voltage table in the directed weighted graph based on the degree values ​​of each of the aforementioned data tables includes: The data table whose degree exceeds the degree threshold is identified as the high-voltage table in the directed weighted graph.

7. The method according to claim 1, characterized in that, The method further includes: The edge weights in the directed weighted graph are determined based on the number of processor cores used, processor utilization, memory usage, and computation time used to calculate the job dependencies between the data tables.

8. A device for determining a data processing link, characterized in that, include: A construction unit is used to perform the construction steps: extracting data tables and job dependencies between the data tables from the database, and constructing a directed weighted graph based on the data tables and job dependencies, wherein the directed weighted graph includes nodes, edges and edge weights, the nodes represent data tables, the edges represent the job dependencies between the data tables, and the edge weights represent the computational cost of the edges; The first determining unit is used to perform the determining steps: obtaining the target table pair, searching for the shortest processing path of the target table pair using a first graph search algorithm according to the directed weighted graph, and determining the first preset number of shortest redundant processing links of the target table pair using a second graph search algorithm according to the directed weighted graph, wherein the target table pair includes the starting data table and the ending data table of the data processing link; The processing unit is used to perform the following processing steps: merging redundant steps in the first preset number of shortest redundant processing links according to the preset reconstruction processing logic to obtain multiple extended processing paths, and sending the target table corresponding to the shortest processing path and the multiple extended processing paths to the user terminal.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the data processing link determination method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method for determining the data processing link as described in any one of claims 1 to 7.

Images