Graph-based data optimization method and device, computer device and medium

Abstract

The application belongs to the technical field of data processing, is applied to the field of insurance claim settlement of financial technology, and relates to a graphical data optimization method, which comprises the following steps: obtaining a table blood relationship graph by including and analyzing a data query script corresponding to a large database table; identifying leaf table nodes in the table blood relationship graph, and obtaining all calling paths between the leaf table nodes and root table nodes according to the table blood relationship graph; identifying a ring calling loop in the calling paths, disassembling the ring calling loop, and obtaining a one-way table relationship graph; determining the longest calling path between the leaf table nodes and the root table nodes based on the one-way table relationship graph; and optimizing the longest calling path according to a target business to obtain a table optimized blood relationship graph. The application also provides a graphical data optimization device, a computer device and a medium. In addition, the application also relates to blockchain technology, and the table blood relationship graph can be stored in the blockchain. The application can simplify a calling link of a data table and reduce the complexity of a data processing link.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a graphical data optimization method, apparatus, computer equipment, and medium. Background Technology

[0002] Big data architecture typically includes an Operational Data Store (ODS) layer, a Data Warehouse (DW) layer, and an Application Data Store (ADS) layer. Each layer contains different sub-layers, and each sub-layer actually contains multiple data processing logics. As the complexity of data requirements continues to increase, the processing process often exhibits three "more" phenomena: more processing layers, more upstream tables that depend on the same layer, and more circular calls between tables. These situations lead to increasingly later data output times for downstream applications and a deteriorating user experience.

[0003] Especially in large-scale big data clusters, there are numerous business database tables. A single end table may depend on thousands or even tens of thousands of upstream tables, which will reduce the performance of data querying and reading, resulting in reduced efficiency in processing business data. At the same time, the lack of clear table and table call relationships is not conducive to business development and modification. Summary of the Invention

[0004] The purpose of this application is to propose a graphical data optimization method, apparatus, computer device, and medium to solve the technical problems in the related art, such as complex data table calls, low data processing efficiency, and difficulties in business development and modification when there are a large number of business database tables.

[0005] To address the aforementioned technical problems, this application provides a graphical data optimization method, employing the following technical solution:

[0006] Obtain the data query script corresponding to the large database table, and parse the data query script to obtain the table lineage diagram;

[0007] Identify the leaf table nodes in the table lineage graph, and obtain all call paths between the leaf table nodes and the root table nodes based on the table lineage graph;

[0008] Identify the circular call loop in the call path, disassemble the circular call loop, and obtain a one-way table relationship graph;

[0009] Based on the one-way table relationship graph, determine the longest call path between the leaf table node and the root table node;

[0010] Based on the target business, optimize the longest call path to obtain the optimized table lineage diagram.

[0011] Furthermore, the step of parsing the data query script to obtain the table lineage graph includes:

[0012] The data query script is parsed to obtain an abstract syntax tree;

[0013] Extract data table information based on the abstract syntax tree;

[0014] The table nodes are determined based on the information in the data table.

[0015] The hierarchical relationship between the table nodes is determined based on the hierarchical relationship of the statements corresponding to the data table information in the abstract syntax tree;

[0016] A table lineage graph is obtained based on the hierarchical relationship between the table nodes.

[0017] Furthermore, the step of obtaining all call paths from the leaf table node to the root table node based on the table lineage graph includes:

[0018] The directed edges of the leaf table nodes are obtained based on the bloodline relationship graph.

[0019] The call path is obtained by traversing upstream of the leaf table node based on the directed edge until the root table node is reached.

[0020] Furthermore, the step of identifying the circular call loop in the call path includes:

[0021] Extract all table nodes along the call path;

[0022] Remove duplicate nodes from the table to obtain the number of path nodes;

[0023] The call path length is obtained based on the call path.

[0024] Based on the number of path nodes and the length of the call path, a circular call loop is determined.

[0025] Furthermore, the step of determining the circular call loop based on the number of path nodes and the call path length includes:

[0026] Compare the number of path nodes with the length of the call path;

[0027] When the length of the call path is greater than or equal to the number of path nodes, the call path is a circular call path.

[0028] Furthermore, the step of dismantling the circular call loop includes:

[0029] Determine the smallest loop in the circular call loop;

[0030] The minimum loop is disassembled according to the directed edges in the minimum loop and a preset disassembly rule.

[0031] Furthermore, the step of determining the minimum loop in the circular call loop includes:

[0032] Obtain the ring table nodes on the ring call path and form a node set;

[0033] Traverse the circular table nodes in the node set according to the directed edges to obtain a circular path with the same starting and ending nodes;

[0034] The number of deduplicated ring nodes and the length of the ring path are obtained based on the ring path.

[0035] Compare the length of the ring path with the number of ring nodes;

[0036] When the length of the circular path is equal to the number of circular nodes, the circular path is a minimum loop.

[0037] To address the aforementioned technical problems, this application also provides a graphical data optimization device, employing the following technical solution:

[0038] The parsing module is used to obtain the data query scripts corresponding to the large database tables, and parse the data query scripts to obtain the table lineage diagram;

[0039] The identification module is used to identify leaf table nodes in the table lineage graph and obtain all call paths between the leaf table nodes and the root table nodes according to the table lineage graph.

[0040] The disassembly module is used to identify the circular call loop in the call path, disassemble the circular call loop, and obtain a one-way table relationship diagram.

[0041] The module is used to determine the longest call path between the leaf table node and the root table node based on the one-way table relationship graph;

[0042] The optimization module is used to optimize the longest call path according to the target business and obtain the optimized lineage graph of the table.

[0043] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0044] The computer device includes a memory and a processor, the memory storing computer-readable instructions, and the processor executing the computer-readable instructions to implement the steps of the graphical data optimization method described above.

[0045] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0046] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the graphical data optimization method described above.

[0047] Compared with the prior art, the embodiments of this application have the following main advantages:

[0048] This application obtains and parses the data query scripts corresponding to large database tables to obtain a table lineage diagram; identifies leaf table nodes in the table lineage diagram and obtains all call paths from leaf table nodes to root table nodes based on the table lineage diagram; identifies circular call loops in the call paths, disassembles the circular call loops to obtain a unidirectional table relationship diagram; determines the longest call path from leaf table nodes to root table nodes based on the unidirectional table relationship diagram; optimizes the longest call path according to the target business to obtain an optimized table lineage diagram. This application obtains the call paths between data tables through the table lineage diagram of a large data database, identifies and disassembles circular call loops in the call paths, simplifies the call chain of data tables, reduces the complexity of the data processing chain, and improves the efficiency of data acquisition and processing. At the same time, a clear table call chain is conducive to locating and analyzing problems and to business development and modification. Attached Figure Description

[0049] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0051] Figure 2 This is a flowchart of an embodiment of the graphical data optimization method according to this application;

[0052] Figure 3 This is a table of lineage relationships according to a specific embodiment of the graphical data optimization method of this application;

[0053] Figure 4This is a one-way table relationship diagram according to a specific embodiment of the graphical data optimization method of this application;

[0054] Figure 5 This is a schematic diagram of the structure of an embodiment of the graphical data optimization device according to this application;

[0055] Figure 6 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0057] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0058] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0059] This application provides a graphical data optimization method that can be applied to, for example... Figure 1 In the system architecture 100 shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 is used as a medium to provide a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0060] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0061] Terminal devices 101, 102, and 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers, etc.

[0062] Server 105 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal devices 101, 102, and 103.

[0063] It should be noted that the graphical data optimization method provided in this application is generally executed by a server / terminal device, and correspondingly, the graphical data optimization device is generally located in the server / terminal device.

[0064] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0065] Continue to refer to Figure 2 The flowchart illustrates an embodiment of a graphical data optimization method according to this application, including the following steps:

[0066] Step S201: Obtain the data query script corresponding to the large database table, and parse the data query script to obtain the table lineage diagram.

[0067] Big data architecture typically includes a Data Operations Layer (ODS), a Data Warehouse Layer (DW), and an Application Data Layer (ADS). Each layer contains further sub-layers, and each sub-layer actually contains multiple data processing logics. Data in a big data database is stored in the form of tables, and there are calling relationships between the data in these tables, i.e., data lineage relationships. Data lineage is a concept in data governance, referring to finding the connections between related data during the data tracing process.

[0068] Big data has its corresponding data query scripts. By parsing the data query scripts, we can obtain the lineage diagram between tables in the big data database, i.e., the table lineage diagram.

[0069] Data query scripts include SQL (Structured Query Language) scripts, which can include program instructions for data access, querying, updating and management operations.

[0070] In this embodiment, the data query script can be statically or dynamically parsed to obtain a table lineage diagram.

[0071] The table lineage graph includes table nodes and directed edges. Table nodes are nodes corresponding to source or target tables in the large database. Directed edges are directed connections from the target table's corresponding table node to the source table's corresponding table node, representing the calling relationship. For example, target table A and source table B, where A and B represent table nodes, A→B means that table A calls table B.

[0072] It is important to note that source table and target table are relative concepts; a data table can be both a source table and a target table at the same time.

[0073] It should be emphasized that, to further ensure the privacy and security of the table kinship diagram, the aforementioned table kinship diagram can also be stored in a node of a blockchain.

[0074] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0075] Step S202: Identify the leaf table nodes in the table lineage graph, and obtain all call paths between the leaf table nodes and the root table nodes based on the table lineage graph.

[0076] In this embodiment, the table closest to the source layer data in the big data database is called the root node. The root node has no upstream and is the first layer of the big data processing chain. The table at the end of the big data data processing is called the leaf node. The leaf node has no downstream and is closest to the application end.

[0077] Determine the leaf table nodes in the table lineage graph, and obtain the directed edges of the leaf table nodes based on the table lineage graph; traverse upstream of the leaf table nodes based on the directed edges until the root table node is reached to obtain the call path.

[0078] Based on the directed edges corresponding to the leaf table nodes, i.e., the calling relationships of the leaf table nodes, the search for the parent table node is traversed along the arrows on the directed edges in the table lineage graph until the root table node is encountered, at which point the search stops. It is important to note that edges traversed during the search process cannot be repeated.

[0079] There are multiple paths between the root table node and the leaf table node, and each path is a call path.

[0080] This embodiment uses directed edges in the lineage graph to traverse from leaf table nodes to root table nodes, ensuring both efficiency and accuracy in obtaining the call path.

[0081] Step S203: Identify the circular call loop in the call path, disassemble the circular call loop, and obtain the one-way table relationship diagram.

[0082] Because large database table processing lacks primary key / foreign key / index restrictions, data can be read from within the table itself during write operations. Circular calls between different tables are also possible, creating loops in the table lineage graph. In actual processing, numerous circular calls exist. These loops arise from various reasons: some involve numerous joins in the processing logic of wide table fields; others result from cross-calls between different data partitions within the table; still others stem from data model design flaws or specific data processing requirements. Circular calls increase the complexity of the data model and the risk of inefficient scheduling. Furthermore, the upstream processing complexity of leaf table nodes directly impacts data output timeliness and results, reducing the user experience on the application side.

[0083] In this embodiment, identifying and breaking down the circular call loop in the call path can reduce the complexity of the call chain and simplify the data model.

[0084] In some alternative implementations, the steps for identifying circular call loops in the call path include:

[0085] Extract all table nodes along the call path;

[0086] Remove duplicate nodes from the table to obtain the number of path nodes;

[0087] The length of the call path is obtained from the call path;

[0088] Based on the number of path nodes and the length of the call path, a circular call loop is determined.

[0089] There are multiple call paths between the root table node and the leaf table node. Suppose the path length of a certain call path is M (i.e. the number of directed edges passed through, which also represents the processing level), and the number of path nodes after deduplication of all table nodes (including the root table node and leaf table nodes) passed through this path is N (i.e. the number of physical tables).

[0090] The relationship between M and N can be categorized into the following two cases:

[0091] 1) M = N-1 is a directed acyclic graph, meaning that traversing through N table nodes requires only N-1 directed edges;

[0092] 2) M≥N indicates that there is a circular call loop between the table nodes on the call path.

[0093] In this embodiment, the number of path nodes N and the call path length M are compared; when the call path length M is greater than or equal to the number of path nodes N, the call path is a circular call path.

[0094] In some embodiments, obtaining the call path from the leaf node to the root node and identifying the circular call loop in the call path is equivalent to solving for the table node z traversed by the directed edge between the leaf node x and the root node y, where z = f(x, y), z ∈ {z1, z2, ..., z}. n Then, solve for the table node u through which the directed edge of table node z, which is both the start and end point, passes, u = f(z). i ,z i ), u∈{u1,u2,…,u m If u is empty, the upstream link of leaf table node x is unidirectional and non-repeating; if u is not empty, u is the set of simplest table nodes that can form a circular call loop.

[0095] The identified circular call loops are broken down to obtain a unidirectional table relationship graph without loops, making the call chain unidirectional and clear.

[0096] Example, reference Figure 3 Leaf nodes only call one upstream table A, and root nodes only call one downstream table D. Analyzing the calling relationship between root nodes and leaf nodes can be simplified to analyzing the calling path between table A and table D. The parsing of the calling path between A and D is shown in Table 1.

[0097] Table 1

[0098] M N Passing through points (in sequence) Passing through the edges (in sequence) contrast in conclusion 2 3 ABD 27 M = N-1 Acyclic 2 3 AED 16 M = N-1 Acyclic 3 3 AEED 156 M = N Single-point loop 5 5 ABCAED 24316 M = N Single-point loop 6 5 AEDCABD 168327 M>N Multi-point ring 6 5 ABCAEED 243156 M>N Multi-point ring 6 5 ABDCAED 278316 M>N Multi-point ring 7 5 AEEDCABD 1568327 M>N Multi-point ring 7 5 ABDCAEED 2783156 M>N Multi-point ring

[0099] As shown in the table above, there are 7 call paths that form a circular call loop.

[0100] This embodiment determines the circular call loop by the number of path nodes and the length of the call path, which is simple and quick.

[0101] In some alternative embodiments, the steps for dismantling the circular call loop described above include:

[0102] Determine the smallest loop in the circular call loop;

[0103] Based on the directed edges in the minimum cycle, the minimum cycle is disassembled according to the preset disassembly rules.

[0104] A minimum cycle is defined as a path that can be returned to any node in the cycle by starting from any node in the table, and that follows the same directed edges. For example, a single-point cycle is always a minimum cycle.

[0105] As can be seen from the above example, there are 7 circular call loops in the call path. Although the directed edges are not repeated, it will cause the leaf table node to take a long detour to the root table node, which will reduce the efficiency of data query and retrieval.

[0106] The number of minimum loops can determine the complexity of data calls. This embodiment decomposes the circular call loop by breaking down the minimum loops. The decomposition is based on the directed edges in the minimum loops, i.e., the call relationships, and is carried out according to preset decomposition rules.

[0107] The preset splitting rules are set according to the actual business logic to ensure data accuracy and business availability. For example, tables can be split into multiple tables based on table fields according to business logic, and the split tables can be placed in the corresponding call locations in the call path.

[0108] Example, reference Figure 4 , Figure 4 This demonstrates a method for dismantling circular call loops. To ensure that the processing content of the upstream and downstream tables of tables C and E remains unchanged, table C is split into tables C1 and C2, and table E is split into tables E1 and E2, placed at different processing levels to obtain a unidirectional table relationship graph. Although the number of tables in the unidirectional table relationship graph increases, the call path is still unidirectional and clear.

[0109] In a circular call loop, the longest path M of table node AD is 9, and there are two "leaf AEEDCABD root" and "leaf ABDCAEED root" paths. In a unidirectional table graph without loops, the longest path M of AD is 7, and there is only one "leaf C1AE1E2DC2 root". Obviously, the latter has better data hierarchy clarity and fluency.

[0110] It should be understood that the above decomposition method is only one possibility. In reality, there can be multiple ways to decompose loops. There is no optimal or worst decomposition method. As long as the loops are eliminated, the traversal complexity of the entire path can be simplified.

[0111] This embodiment breaks down the circular call loop by disassembling the smallest loop, which can achieve disassembly more efficiently and accurately, ensuring that the disassembled table can be used in business operations.

[0112] Step S204: Determine the longest call path between the leaf table node and the root table node based on the one-way table relationship graph.

[0113] Based on a unidirectional list graph without loops, the longest call path between leaf nodes and root nodes is obtained. The longest call path metric measures the complexity of the data model; a high metric indicates multiple and complex data levels. Further link reduction of the longest call path is necessary to completely simplify data calls and achieve a simplification of the data processing model.

[0114] Step S205: Optimize the longest call path according to the target business to obtain the table optimization lineage diagram.

[0115] Optimizing the longest call path requires a comprehensive assessment based on the data query script logic, the characteristics of the related tables, the timeliness of data updates, and business requirements. Different requirements will result in different table optimization lineage diagrams.

[0116] In some alternative implementations, after the step of obtaining all call paths from leaf table nodes to root table nodes based on the table lineage graph, the following is also included:

[0117] If no circular call loop is found in the call path, the longest call path between the leaf table node and the root table node is determined based on the table lineage graph.

[0118] Based on the target business, optimize the longest call path to obtain the optimized table lineage diagram.

[0119] If there is no circular call loop in the table lineage graph, it means that the call paths in the table lineage graph are unidirectional and clear, and can be directly optimized according to the business requirements.

[0120] After optimization, the number of minimum loops between leaf table nodes and root table nodes can be used to determine whether the model complexity has been reduced.

[0121] This application uses a table lineage diagram of a big data database to obtain the call paths between data tables, identify and decompose circular call loops in the call paths, simplify the call chain of data tables, and improve the efficiency of data acquisition and processing. At the same time, a clear table call chain is conducive to locating and analyzing problems, and to business development and modification. Secondly, it uses a graphical representation of the data table lineage. By identifying the number of minimum loops in the upstream path of leaf table nodes and calculating the longest upstream call path corresponding to the leaf table node, it can be used to measure the complexity of the data model and provide a reference for optimizing the model.

[0122] In some alternative implementations, the steps of the above data parsing query script to obtain the table lineage diagram include:

[0123] The data query script is parsed to obtain an abstract syntax tree;

[0124] Extract data table information based on the abstract syntax tree;

[0125] Determine table nodes based on data table information;

[0126] The hierarchical relationship between table nodes is determined based on the hierarchical relationship of the statements corresponding to the data table information in the abstract syntax tree;

[0127] A table lineage graph is obtained based on the hierarchical relationship between table nodes.

[0128] Abstract syntax trees (ASTs) can be used to abstractly represent the syntactic and lexical information of data query scripts. The data query script is broken down into multiple syntactic units, and syntactic analysis is performed on each unit to determine the syntactic nodes in the AST and the hierarchical relationships between them. Data table information is extracted from these nodes, including table information and field information. Table information can identify the table (e.g., the table name), and field information can identify the field (e.g., the field name).

[0129] Each piece of information in a table and each field can correspond to a table node in the table lineage diagram. The hierarchical relationship between table nodes in the table lineage diagram is determined based on the hierarchical relationship of the table information and the field information.

[0130] In this embodiment, the data query script can be statically or dynamically parsed.

[0131] For example, static scripts can be obtained from git / svn code repositories. You can obtain SQL script files, write Python (sqlparse) or Java code to format and split the SQL code in the script, and identify the tables following the keywords insert / withas / create as / from / join to extract the source and / or target tables.

[0132] Alternatively, the table lineage graph can be obtained by parsing dynamically executed scripts. This involves retrieving and parsing the SQL script from a query that is in execution or has finished executing. Before the Hive script is executed, a hook (LineageLogger) is set, which writes the lineage information of the query to a specified log file. By parsing the log file, the accurate tables and their call relationships can be obtained.

[0133] It should be understood that there are many ways to parse data query scripts, not limited to the two mentioned above. Sometimes, in order to obtain a more accurate blood relationship, the above methods can be combined.

[0134] This application obtains the table lineage diagram by parsing the data query script, which can guarantee the accuracy of the obtained table lineage.

[0135] In some alternative implementations, the steps for determining the minimum loop in a circular call loop include:

[0136] Retrieve the circular table nodes along the circular call path and form a node set;

[0137] By traversing the circular table nodes in the node set according to the directed edges, a circular path with the same starting and ending nodes can be obtained.

[0138] The number of duplicate ring nodes and the length of the ring path are obtained from the ring path after deduplication.

[0139] Compare the length of the loop path and the number of loop nodes;

[0140] When the length of the circular path and the number of circular nodes are equal, the circular path is a minimum loop.

[0141] The minimum number of loops can be used to measure the complexity of the data processing link. In this embodiment, it can be determined using algorithms and scripts provided by a large database.

[0142] The above example will be used as a starting point for a detailed explanation. The steps are as follows:

[0143] 1) Extract all table nodes in the AD path and form a node set;

[0144] 2) Iterate through the table nodes in the node set in sequence. Let the current node n be the starting node. If a path p can be found that returns from node n and the number of duplicate nodes N on the path is equal to the number of directed edges M (i.e. the length of the circular path), then p is the minimum cycle. If no such path can be found, it means that none of the edges passing through node n belong to the circular path.

[0145] 3) Obtain all the table nodes on the loop p, which are the points that constitute this loop. Return the set of these nodes, which is the minimum loop.

[0146] The final minimum loop is shown in Table 2.

[0147] Table 2

[0148]

[0149]

[0150] By determining the minimum number of loops in a circular call loop, we can measure the complexity of the data model and provide a reference for optimizing the model.

[0151] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0152] This application can be applied to the fintech field by retrieving data query scripts corresponding to financial data tables from large databases, parsing these scripts to obtain a table lineage diagram of the financial data, identifying and breaking down circular call paths within the lineage diagram to obtain a unidirectional table relationship diagram. This approach simplifies the call chain of financial data tables, improves the efficiency of financial data acquisition and processing, and the clear table call chain facilitates problem localization and analysis, as well as the development and modification of financial business applications.

[0153] Specifically, this application can be applied to insurance claims in the fintech field. Specifically, it involves obtaining a claim request, extracting claim parameters from the claim request, retrieving the corresponding data query script from the insurance database based on the claim parameters, parsing the data query script, and obtaining the table lineage diagram corresponding to the claim request. For example, the table lineage diagram includes table nodes such as the policy table, protection plan table, insurance type table, liability table, payment liability table, and billing item details table. The calling relationships include: Policy table → Protection plan table A → Insurance type table B → Liability table C → Billing item details table, Policy table → Protection plan table A → Insurance type table B → Liability table C → Payment liability table D → Protection plan table A → Protection plan table E → Billing item details table; and determining the claim parameters based on the data query script. The leaf table nodes in the root table relationship diagram are the policy tables, thus obtaining all call paths from the policy tables to the root table nodes. Circular call loops within these paths are identified and broken down to obtain a one-way table relationship diagram. For example, the payment liability table D is split into payment liability tables D1 and D2, and the protection plan table A is split into protection plan tables A1 and A2. These are then placed at different processing levels, resulting in the following call relationships: Policy table → Protection plan table A1 → Insurance type table B → Liability table C → Payment liability table D1 → Billing item details table; Policy table → Protection plan table A1 → Insurance type table B → Liability table C → Protection plan table A2 → Protection plan table E → Billing item details table. The one-way table relationship diagram can then be optimized.

[0154] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware with computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0155] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0156] Further reference Figure 5 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a graphical data optimization device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0157] like Figure 5 As shown, the graphical data optimization device 500 described in this embodiment includes: a parsing module 501, an identification module 502, a disassembly module 503, an acquisition module 504, and an optimization module 505. Wherein:

[0158] The parsing module 501 is used to obtain the data query scripts corresponding to the large database tables, and parse the data query scripts to obtain the table lineage diagram;

[0159] The identification module 502 is used to identify leaf table nodes in the table lineage graph and obtain all call paths between leaf table nodes and root table nodes based on the table lineage graph.

[0160] The disassembly module 503 is used to identify the circular call loop in the call path, disassemble the circular call loop, and obtain the one-way list relationship diagram;

[0161] Module 504 is used to determine the longest call path between leaf table nodes and root table nodes based on the one-way table relationship graph;

[0162] The optimization module 505 is used to optimize the longest call path based on the target business and obtain the optimized table lineage diagram.

[0163] It should be emphasized that, to further ensure the privacy and security of the table kinship diagram, the aforementioned table kinship diagram can also be stored in a node of a blockchain.

[0164] Based on the aforementioned graphical data optimization device, the call paths between data tables are obtained through the table lineage diagram of the big data database. The circular call loops in the call paths are identified and broken down, simplifying the call chain of data tables, reducing the complexity of the data processing chain, and improving the efficiency of data acquisition and processing. At the same time, the clear table call chain is conducive to locating and analyzing problems, and is beneficial to business development and modification.

[0165] In some alternative implementations, the parsing module 501 is further used for:

[0166] The data query script is parsed to obtain an abstract syntax tree;

[0167] Extract data table information based on the abstract syntax tree;

[0168] Determine table nodes based on data table information;

[0169] The hierarchical relationship between table nodes is determined based on the hierarchical relationship of the statements corresponding to the data table information in the abstract syntax tree;

[0170] A table lineage graph is obtained based on the hierarchical relationship between table nodes.

[0171] This embodiment obtains the table lineage graph by parsing the data query script, which ensures the accuracy of the obtained table lineage.

[0172] In this embodiment, the identification module 502 includes an acquisition submodule and a traversal submodule. The acquisition submodule is used to obtain the directed edges of the leaf table nodes according to the table lineage graph. The traversal submodule is used to traverse upstream of the leaf table nodes based on the directed edges until the root table node stops, thereby obtaining the call path.

[0173] This embodiment uses directed edges in the lineage graph to traverse from leaf table nodes to root table nodes, ensuring both efficiency and accuracy in obtaining the call path.

[0174] In this embodiment, the disassembly module 503 includes an identification submodule, which includes an extraction unit, a deduplication unit, an acquisition unit, and a determination unit, wherein:

[0175] The extraction unit is used to extract all table nodes along the call path;

[0176] The deduplication unit is used to remove duplicate nodes from the table to obtain the number of path nodes;

[0177] The obtain unit is used to get the call path length based on the call path;

[0178] The determination unit is used to determine the circular call loop based on the number of path nodes and the length of the call path.

[0179] This embodiment determines the circular call loop by the number of path nodes and the length of the call path, which is simple and quick.

[0180] In this embodiment, the determining unit is further configured to: compare the number of path nodes and the call path length; when the call path length is greater than or equal to the number of path nodes, the call path is a circular call path.

[0181] In some optional implementations, the disassembly module 503 also includes a disassembly submodule for: determining the minimum loop in the circular call loop; and disassembling the minimum loop according to a preset disassembly rule based on the directed edges in the minimum loop.

[0182] This embodiment breaks down the circular call loop by disassembling the smallest loop, which can achieve disassembly more efficiently and accurately, ensuring that the disassembled table can be used in business operations.

[0183] In this embodiment, the disassembly submodule includes an acquisition unit, a traversal unit, an obtaining unit, and a comparison unit, wherein:

[0184] The acquisition unit is used to acquire the circular table nodes on the circular call path and form a node set;

[0185] The traversal unit is used to traverse the circular table nodes in the node set according to the directed edges to obtain the circular path with the same starting and ending nodes;

[0186] The obtaining unit is used to obtain the number of deduplicated ring nodes and the length of the ring path based on the ring path;

[0187] The comparison unit is used to compare the length of the loop path and the number of loop nodes; when the length of the loop path and the number of loop nodes are equal, the loop path is the minimum loop.

[0188] By determining the minimum number of loops in a circular call loop, we can measure the complexity of the data model and provide a reference for optimizing the model.

[0189] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0190] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only the computer device 6 with components 61-63 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0191] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0192] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may include both the internal storage unit and its external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions based on graphical data optimization methods. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or will be output.

[0193] In some embodiments, the processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to execute computer-readable instructions stored in the memory 61 or to process data, for example, to execute computer-readable instructions based on the graphical data optimization method.

[0194] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 6 and other electronic devices.

[0195] This embodiment implements the steps of the graphical data optimization method described above by executing computer-readable instructions stored in memory through the processor. By using the table lineage diagram of the big data database, the call paths between data tables are obtained, and circular call loops in the call paths are identified and broken down, simplifying the call chain of data tables, reducing the complexity of the data processing chain, and improving the efficiency of data acquisition and processing. At the same time, a clear table call chain is conducive to locating and analyzing problems, and is beneficial to business development and modification.

[0196] This application also provides another implementation, namely, a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to perform the steps of the graphical data optimization method described above. This method obtains the call paths between data tables through a table lineage diagram of a large data database, identifies and breaks down circular call loops in the call paths, simplifies the call chain of data tables, reduces the complexity of the data processing chain, and improves the efficiency of data acquisition and processing. Furthermore, a clear table call chain facilitates problem location and analysis, and is beneficial for business development and redesign.

[0197] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0198] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A graphical data optimization method, characterized in that, Includes the following steps: Obtain the data query script corresponding to the large database table, and parse the data query script to obtain the table lineage diagram; Identify the leaf table nodes in the table lineage graph, and obtain all call paths between the leaf table nodes and the root table nodes based on the table lineage graph; Identify the circular call loop in the call path, disassemble the circular call loop, and obtain a one-way table relationship graph; Based on the one-way table relationship graph, determine the longest call path between the leaf table node and the root table node; Based on the target business, optimize the longest call path to obtain the optimized table lineage diagram; The step of disassembling the ring call loop includes: Determine the smallest loop in the circular call loop; Based on the directed edges in the minimum loop, the minimum loop is disassembled according to a preset disassembly rule; The step of determining the minimum loop in the circular call loop includes: Obtain the ring table nodes on the ring call loop and form a node set; Traverse the circular table nodes in the node set according to the directed edges to obtain a circular path with the same starting and ending nodes; The number of deduplicated ring nodes and the length of the ring path are obtained based on the ring path. Compare the length of the ring path with the number of ring nodes; When the length of the circular path is equal to the number of circular nodes, the circular path is a minimum loop. The preset disassembly rules are set according to the actual business logic, and the step of disassembling the minimum loop according to the preset disassembly rules includes: Based on the business logic, the table consisting of all call paths from the leaf table node to the root table node is split into multiple tables according to the table fields, and the split tables are placed at the corresponding call positions in the call paths.

2. The graphical data optimization method according to claim 1, characterized in that, The step of parsing the data query script to obtain the table lineage diagram includes: The data query script is parsed to obtain an abstract syntax tree; Extract data table information based on the abstract syntax tree; The table nodes are determined based on the information in the data table. The hierarchical relationship between the table nodes is determined based on the hierarchical relationship of the statements corresponding to the data table information in the abstract syntax tree; A table lineage graph is obtained based on the hierarchical relationship between the table nodes.

3. The graphical data optimization method according to claim 1, characterized in that, The step of obtaining all call paths from the leaf table node to the root table node based on the table lineage graph includes: The directed edges of the leaf table nodes are obtained based on the bloodline relationship graph. The call path is obtained by traversing upstream of the leaf table node based on the directed edge until the root table node is reached.

4. The graphical data optimization method according to claim 3, characterized in that, The step of identifying the circular call loop in the call path includes: Extract all table nodes along the call path; Remove duplicate nodes from the table to obtain the number of path nodes; The call path length is obtained based on the call path. Based on the number of path nodes and the length of the call path, a circular call loop is determined.

5. The graphical data optimization method according to claim 4, characterized in that, The step of determining the circular call loop based on the number of path nodes and the call path length includes: Compare the number of path nodes with the length of the call path; When the length of the call path is greater than or equal to the number of path nodes, the call path is a circular call path.

6. A graphical data optimization device, characterized in that, include: The parsing module is used to obtain the data query scripts corresponding to the large database tables, and parse the data query scripts to obtain the table lineage diagram; The identification module is used to identify leaf table nodes in the table lineage graph and obtain all call paths between the leaf table nodes and the root table nodes according to the table lineage graph. The disassembly module is used to identify the circular call loop in the call path, disassemble the circular call loop, and obtain a one-way table relationship diagram. The module is used to determine the longest call path between the leaf table node and the root table node based on the one-way table relationship graph; The optimization module is used to optimize the longest call path according to the target business and obtain the optimized table lineage graph; The disassembly module also includes a disassembly sub-module; The disassembly submodule is used to determine the smallest loop in the circular call loop; and disassemble the smallest loop according to the directed edges in the smallest loop and a preset disassembly rule. The disassembly submodule includes an acquisition unit, a traversal unit, an acquisition unit, and a comparison unit; The acquisition unit is used to acquire the ring table nodes on the ring call loop and form a node set; The traversal unit is used to traverse the circular table nodes in the node set according to the directed edges to obtain a circular path with the same starting node and ending node. The obtaining unit is used to obtain the number of deduplicated ring nodes and the length of the ring path based on the ring path. The comparison unit is used to compare the length of the loop path and the number of loop nodes; when the length of the loop path and the number of loop nodes are equal, the loop path is the minimum loop. The preset disassembly rules are set according to the actual business logic, and the disassembly submodule also includes a splitting unit; The splitting unit is used to split the table consisting of all call paths from the leaf table node to the root table node according to the table fields, based on the business logic, into multiple tables, and place the split tables at the corresponding call positions in the call paths.

7. A computer device comprising a memory and a processor, the memory storing computer-readable instructions, wherein the processor, when executing the computer-readable instructions, implements the steps of the graphical data optimization method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the graphical data optimization method as described in any one of claims 1 to 5.

Images