Business data relation analysis method and device, equipment and storage medium

Abstract

The invention discloses a business data relation analysis method and device, equipment and a storage medium. Comprising the following steps: acquiring system data traffic, and preprocessing the system data traffic to determine interface request data; determining a locator in the interface request data, constructing a dictionary tree based on the locator, and determining each transaction related system and a corresponding interface according to the dictionary tree; performing intra-system blood relationship analysis on each transaction related system to generate an intra-system data table; and performing inter-system blood relationship analysis on each transaction related system according to a preset regular expression and the intra-system data table to generate a business data relationship analysis result. By constructing a dictionary tree and a blood relationship analysis technology, system architecture can be sorted efficiently, the association relationship between systems can be presented clearly, and the cost and error of manual registration of interfaces and data relationships can be reduced. An intra-system data table is generated through intra-system blood relationship analysis, an internal data flow path of the system can be mined, cross-system association is established, and mastering of the full-link flow direction of the data is assisted.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a business data relationship analysis method, device, equipment and storage medium. Background Art

[0002] In today's digital business environment, data interactions between systems are becoming increasingly frequent and complex. The smooth operation of the business depends on accurate and efficient data transmission and processing between systems. Clearly understanding the data flow and interfaces between systems is crucial for architectural management and control. For example, in e-commerce transactions, multiple systems, such as order systems, payment systems, and logistics systems, collaborate with each other, and data constantly flows between them. Failure to effectively organize the relationships and interface information between systems will seriously impact business stability and scalability.

[0003] Currently, manual registration is commonly used for interface monitoring and inter-system relationship management. This allows managers to access the system's external interface management, interface information, and data information. Furthermore, registering system data consumption relationships allows managers to understand the flow of transaction-related data across the entire system, assisting with architecture management.

[0004] However, existing manual maintenance methods require technicians to constantly and accurately register system interfaces and inter-system consumption relationships. However, in practice, required registrations are often not made, leading to insufficient interface management and inaccurate data flow relationships between systems. This hinders long-term dynamic monitoring of the system, makes it difficult to adapt to rapid business changes and frequent system iterations, and makes it difficult to ensure accurate and timely analysis of business data relationships. Summary of the Invention

[0005] The present invention provides a business data relationship analysis method, device, equipment and storage medium, establishes a set of methods for automatically identifying system-related business interfaces through system data traffic, and at the same time, through data lineage technology, under the premise of the existence of consumption relationships between data, assists technical personnel to more accurately and completely sort out the data flow relationships between the entire system, and establishes an overall view of the overall background technology system link corresponding to the identified business services.

[0006] According to one aspect of the present invention, a method for analyzing business data relationships is provided, the method comprising:

[0007] Obtain system data traffic and pre-process the system data traffic to determine interface request data;

[0008] Determine the locator in the interface request data, and build a dictionary tree based on the locator. According to the dictionary tree, determine each transaction-related system and its corresponding interface;

[0009] Conduct intra-system lineage analysis on each transaction-related system to generate intra-system data tables;

[0010] Based on preset regular expressions and system data tables, inter-system lineage analysis is performed on each transaction-related system to generate business data relationship analysis results.

[0011] Optionally, the system data traffic is preprocessed to determine the interface request data, including: determining the non-interface request data in the system data traffic according to the specified field and preset filtering rules; filtering the non-interface request data in the system data traffic and retaining the interface request data.

[0012] Optionally, each transaction-related system and its corresponding interface are determined based on the dictionary tree, including: taking the child nodes with the same parent node in the dictionary tree as the target child nodes; clustering each target child node using the cosine similarity algorithm to generate each cluster node; and determining each transaction-related system and its corresponding interface by identifying the root path of each cluster node.

[0013] Optionally, each target sub-node is clustered using a cosine similarity algorithm to generate each cluster node, including: determining a locator string corresponding to each target sub-node; converting each locator string into a vector, and calculating the similarity between each vector according to the cosine similarity algorithm; and storing the target sub-nodes whose similarity is greater than a preset similarity threshold close to each other to generate each cluster node.

[0014] Optionally, perform an intra-system lineage analysis on each transaction-related system to generate an intra-system data table, including: taking each transaction-related system as a target system and determining the target interface corresponding to the target system; extracting processing metadata from the SQL statement of the target interface based on static analysis technology, and determining each data table related to data processing based on the processing metadata; analyzing the data flow relationship between each data table to generate the intra-system data table of the target system.

[0015] Optionally, determining various data tables related to data processing based on processing metadata includes: using a preset parsing tool to convert the processing metadata into a tree structure; using the data after the first specified field in the tree structure as the target table for processing, and the table involved after the second specified field as the source table; traversing the tree structure based on the target table and the source table to obtain various data tables related to data processing.

[0016] Optionally, an inter-system lineage analysis is performed on each transaction-related system based on a preset regular expression and data tables within the system to generate a business data relationship analysis result, including: matching the data tables within the system through a preset regular expression to determine the downstream table and the downstream table name; removing the prefix from the downstream table name to obtain the upstream table name; establishing a mapping relationship between the upstream table name and the downstream table name to generate a lineage view, and using the lineage view as the business data relationship analysis result.

[0017] According to another aspect of the present invention, there is provided a device for analyzing business data relationships, the device comprising:

[0018] An interface request data determination module is used to obtain system data traffic and pre-process the system data traffic to determine the interface request data;

[0019] The system and interface determination module is used to determine the locator in the interface request data, and build a dictionary tree based on the locator, and determine each transaction-related system and its corresponding interface according to the dictionary tree;

[0020] The intra-system lineage analysis module is used to perform intra-system lineage analysis on each transaction-related system to generate intra-system data tables;

[0021] The inter-system lineage analysis module is used to perform inter-system lineage analysis on each transaction-related system based on preset regular expressions and system data tables to generate business data relationship analysis results.

[0022] According to another aspect of the present invention, an electronic device is provided, comprising:

[0023] at least one processor;

[0024] and a memory communicatively coupled to the at least one processor;

[0025] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute a business data relationship analysis method described in any embodiment of the present invention.

[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement a business data relationship analysis method according to any embodiment of the present invention when executed.

[0027] The technical solution of the present invention, through the construction of a dictionary tree and lineage analysis technology, can efficiently organize system architecture, intuitively present inter-system relationships, and reduce the cost and error of manually registering interfaces and data relationships. By performing intra-system lineage analysis on each transaction-related system to generate internal data tables, it is possible to deeply explore the internal data flow paths of the system, accurately locate the data processing process and the relationships between tables, and provide comprehensive and accurate data support for business decision-making.

[0028] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 This is a flowchart of a business data relationship analysis method provided according to the first embodiment of the present invention;

[0031] Figure 2 This is a flowchart of another method for analyzing business data relationships provided according to the second embodiment of the present invention;

[0032] Figure 3 is a schematic diagram of a blood relationship view provided according to the second embodiment of the present invention;

[0033] Figure 4 This is a schematic diagram of the structure of a business data relationship analysis device provided according to the third embodiment of the present invention;

[0034] Figure 5 It is a structural diagram of an electronic device for implementing a business data relationship analysis method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Example 1

[0038] Figure 1 A flowchart of a business data relationship analysis method is provided for the first embodiment of the present invention. This embodiment is applicable to sorting out business relationships between systems. The method can be executed by a business data relationship analysis device. The business data relationship analysis device can be implemented in the form of hardware and / or software. The business data relationship analysis device can be configured in a computer controller. Figure 1 As shown, the method includes:

[0039] S110 , obtaining system data traffic, and preprocessing the system data traffic to determine interface request data.

[0040] System data traffic refers to the flow of all data and information transmitted over the network during system operation, including but not limited to API requests, data transmission, file exchange, and other business-related data interactions. Interface request data refers to request data in system data traffic that complies with API interface call specifications.

[0041] Specifically, the controller collects all network traffic data generated during system operation, covering the business entry system (i.e., the first-hop system) and downstream related systems. Data preprocessing involves filtering out valid API interface request data from the full traffic and excluding non-API requests, such as page views and static resource loading.

[0042] Optionally, the system data traffic is preprocessed to determine the interface request data, including: determining the non-interface request data in the system data traffic according to the specified field and preset filtering rules; filtering the non-interface request data in the system data traffic and retaining the interface request data.

[0043] Specifically, by parsing the request header field Content-Type in the system data traffic, you can determine whether the request is an API interface call, thereby distinguishing interface request data from non-interface request data. The Content-Type field is located in the HTTP request header and is used to identify the media type of the request body. If the request header field value matches an exclusion rule, such as text / html, it can be determined as non-interface request data and directly filtered and discarded. If the field value matches a matching rule, such as application / json, it is determined to be interface request data and retained for the next step of processing.

[0044] S120: Determine the locator in the interface request data, and construct a dictionary tree based on the locator. Determine each transaction-related system and its corresponding interface according to the dictionary tree.

[0045] The locator is a Uniform Resource Locator (URL), which identifies the specific location of a resource in a system, such as the access address of an API. A trie is a tree-like data structure that can be constructed using the common prefix of a URL as the parent node and the differences as the child nodes.

[0046] Specifically, during construction, all URLs can be traversed and split by path level, such as splitting by " / ", and then URLs with the same prefix are attributed to the same parent node to form a tree-like hierarchical relationship.

[0047] Optionally, each transaction-related system and its corresponding interface are determined based on the dictionary tree, including: taking the child nodes with the same parent node in the dictionary tree as the target child nodes; clustering each target child node using the cosine similarity algorithm to generate each cluster node; and determining each transaction-related system and its corresponding interface by identifying the root path of each cluster node.

[0048] The target child node refers to all child nodes under the same parent node in the dictionary tree. Cosine similarity is used to measure the similarity between two strings. It converts the strings into vectors in vector space and calculates the cosine of the angle between the vectors. The closer the value is to 1, the more similar they are.

[0049] Specifically, the controller will calculate the cosine similarity of any two child node strings for the target child nodes under the same parent node, that is, the interface path fragments under the same system, and perform node aggregation operations based on the cosine similarity. The root path is the complete public prefix path from the root node to the clustering node in the dictionary tree. For example, " / v1 / app / " represents the first-level subdirectory of a business system. Assuming that the first-level subdirectory of the domain name corresponding to the business transaction is the system identifier, if the child node with a similarity of 1 under the first-level subdirectory is selected for aggregation, the first-level subdirectory in the root path can be directly mapped to the transaction-related system. The aggregated clustering node represents the interface template, and the variables in the template, such as {id} and {string}, are parameterized identifiers.

[0050] In summary, the common prefix feature of the dictionary tree aggregates similar URLs into the same template, avoiding duplicate interface records and improving data simplicity. By calculating numerical similarity, thresholds can be flexibly set to differentiate interfaces, adapting to the classification needs of parameterized and instantiated interfaces.

[0051] Optionally, each target sub-node is clustered using a cosine similarity algorithm to generate each cluster node, including: determining a locator string corresponding to each target sub-node; converting each locator string into a vector, and calculating the similarity between each vector according to the cosine similarity algorithm; and storing the target sub-nodes whose similarity is greater than a preset similarity threshold close to each other to generate each cluster node.

[0052] Specifically, the string is the differentiated part after the parent node in the URL path. For example, if the parent node is " / v1 / app / ", its child nodes may include "1009 / create", "{string} / show", etc. The string is used as the input data to be processed, and only the path part after the parent node is retained, excluding irrelevant prefixes. When converting a string into a vector, the character set is first defined, and the characters that appear in all target child nodes are selected to build a global list. Then, for each string, on the corresponding dimension of the character set, if the character exists, the vector value is 1, otherwise it is 0. Then, the cosine similarity formula is used to calculate the similarity between vectors. The formula is the dot product of the vectors divided by the product of the vector moduli. The dot product reflects the number of shared characters, and the modulus product is used for normalization. The closer the result value is to 1, the higher the similarity.

[0053] Finally, nodes are clustered and stored based on preset thresholds. These thresholds typically include aggregation and proximity thresholds. For example, subnodes with a similarity of 1 are clustered together to form a parameterized interface template, such as "{id} / create." Subnodes with a similarity ≥ 0.5 are stored closely under the corresponding template node as similar interface instances, such as "18sj-01ka-38dh / show" corresponding to the template "{string} / show." Paths with similarities below the threshold are considered independent paths. Each cluster node in the clustering results contains the interface template, a list of similar instances, and the corresponding similarity value, enabling subsequent analysis of interface standardization and usage.

[0054] S130: Perform intra-system lineage analysis on each transaction-related system to generate an intra-system data table.

[0055] Intra-system lineage analysis analyzes the flow of data within a single system during data processing, specifically the flow of data from source tables to target tables, to establish data lineage relationships within the system. Intra-system data tables refer to the collection of source and target tables involved in the system's data processing, as determined through intra-system lineage analysis. These tables include table names, field names, and data flow relationships.

[0056] Optionally, perform an intra-system lineage analysis on each transaction-related system to generate an intra-system data table, including: taking each transaction-related system as a target system and determining the target interface corresponding to the target system; extracting processing metadata from the SQL statement of the target interface based on static analysis technology, and determining each data table related to data processing based on the processing metadata; analyzing the data flow relationship between each data table to generate the intra-system data table of the target system.

[0057] Specifically, the controller identifies each transaction-related system as a target system and extracts all interfaces involved in data processing within the target system. It then parses the SQL statements corresponding to the target interfaces and constructs an abstract syntax tree to break down the statement structure, identifying the data tables relevant to data processing. These tables include both target and source tables. Furthermore, the controller removes duplicate table names extracted from all target interface SQL statements, generating a complete list of the target system's data tables, distinguishing between source and target tables and annotating the corresponding interfaces for each table.

[0058] Furthermore, when analyzing the data flow relationship, the controller will use the association conditions and field processing logic in the SQL statement, such as aggregation and filtering, to sort out the flow path of data from the source table to the target table. For example, the "Order Table" and the "Payment Table" are associated through the order ID. After screening and aggregation, the data flows into the "Order Summary Table", forming a clear blood relationship chain, and these relationships are recorded in the form of lists or charts, marking the data processing method. Finally, the system data table of the target system is generated, which includes two parts: one is the table list, which lists the table name, type and associated interface; the other is the blood relationship record, which explains the source table source and field mapping logic of each target table. By automatically parsing SQL statements, the accurate sorting of data flow within the system is achieved, avoiding the tediousness and errors of manual registration, and providing structured underlying data support for the full-link analysis of business data.

[0059] Optionally, determining various data tables related to data processing based on processing metadata includes: using a preset parsing tool to convert the processing metadata into a tree structure; using the data after the first specified field in the tree structure as the target table for processing, and the table involved after the second specified field as the source table; traversing the tree structure based on the target table and the source table to obtain various data tables related to data processing.

[0060] Specifically, the controller can use a pre-defined parsing tool, such as an SQL parser, to process the processing metadata (i.e., SQL statements) and convert it into a tree structure, which can be an abstract syntax tree. This tree structure can display the grammatical components of SQL statements in a hierarchical manner, for example, with keywords such as INSERT, SELECT, FROM, and WHERE as different nodes, and sub-nodes containing specific table names, field names, and operation logic.

[0061] Furthermore, the controller will locate two types of key nodes in the tree structure, namely the first specified field and the second specified field. The first specified field is a keyword indicating data output such as "INSERT INTO" or "CREATE TABLE", and the data table corresponding to its subsequent nodes is the target table for processing. For example, in the "INSERT INTO order_summary..." statement, "order_summary" is the table name after the first specified field and is determined to be the target table. The second specified field is a keyword indicating data input such as "FROM" or "JOIN", and the data table involved in its subsequent nodes is the source table. For example, in the "SELECT...FROM orders JOIN payments..." statement, "orders" and "payments" are the table names after the second specified field and are determined to be the source table.

[0062] Finally, based on the identified target and source tables, the tree structure is deeply traversed to extract all tables related to data processing. During the traversal process, not only are the target and source tables directly extracted, but the associated child node tables, such as tables in subqueries, are also recursively searched to ensure that no tables involved in data processing are missed. For example, if the source table references "temp_table" through a subquery, the traversal process will also include "temp_table" in the data table list. By traversing the tree structure, a complete set of data tables related to data processing in the target system can be obtained, including the target table, source table, and all intermediate related tables, and the role of each table in data processing can be clarified.

[0063] S140. Perform inter-system lineage analysis on each transaction-related system based on a preset regular expression and a data table within the system to generate a business data relationship analysis result.

[0064] Among them, the preset regular expression refers to a predefined string matching pattern, which is used to identify the mapping relationship between the downstream system table name and the upstream system table name. Inter-system lineage analysis refers to the analysis of the data consumption relationship between different systems, that is, how the downstream system consumes the data of the upstream system through the interface or data transmission to form a cross-system data lineage relationship. The business data relationship analysis result refers to the overall view of the flow of business data between systems, including the system list and API interface list corresponding to the business transaction and the complete flow path of data from the upstream system to the downstream system. By generating the business data relationship analysis results, it can help technical personnel intuitively grasp the full-link relationship of business data, avoid omissions and erroneous registrations caused by manual registration, and support dynamic monitoring and architecture management.

[0065] Specifically, the final output of the business data relationship analysis includes: a system link list, which lists all systems involved in the business and their order, such as the first-hop system and the second-hop system; interfaces and data flows, which lists the API interfaces of each system and the complete path of data flow from the source table to the target table and across systems; and a visualization view, which provides a graphical display based on lineage relationships and supports dynamic monitoring of data flows.

[0066] The technical solution of the present invention, through the construction of a dictionary tree and lineage analysis technology, can efficiently organize system architecture, intuitively present inter-system relationships, and reduce the cost and error of manually registering interfaces and data relationships. By performing intra-system lineage analysis on each transaction-related system to generate internal data tables, it is possible to deeply explore the internal data flow paths of the system, accurately locate the data processing process and the relationships between tables, and provide comprehensive and accurate data support for business decision-making.

[0067] Example 2

[0068] Figure 2This is a flowchart of a business data relationship analysis method provided by Example 2 of the present invention. This example adds a specific process of performing inter-system lineage analysis on each transaction-related system based on a preset regular expression and a data table within the system to generate a business data relationship analysis result. The specific contents of steps S210-S230 are roughly the same as those of steps S110-S130 in Example 1, so they will not be repeated in this embodiment. Figure 2 As shown, the method includes:

[0069] S210: Obtain system data traffic, and pre-process the system data traffic to determine interface request data.

[0070] Optionally, the system data traffic is preprocessed to determine the interface request data, including: determining the non-interface request data in the system data traffic according to the specified field and preset filtering rules; filtering the non-interface request data in the system data traffic and retaining the interface request data.

[0071] S220: Determine the locator in the interface request data, and construct a dictionary tree based on the locator. Determine each transaction-related system and its corresponding interface according to the dictionary tree.

[0072] Optionally, each transaction-related system and its corresponding interface are determined based on the dictionary tree, including: taking the child nodes with the same parent node in the dictionary tree as the target child nodes; clustering each target child node using the cosine similarity algorithm to generate each cluster node; and determining each transaction-related system and its corresponding interface by identifying the root path of each cluster node.

[0073] Optionally, each target sub-node is clustered using a cosine similarity algorithm to generate each cluster node, including: determining a locator string corresponding to each target sub-node; converting each locator string into a vector, and calculating the similarity between each vector according to the cosine similarity algorithm; and storing the target sub-nodes whose similarity is greater than a preset similarity threshold close to each other to generate each cluster node.

[0074] S230: Perform intra-system lineage analysis on each transaction-related system to generate an intra-system data table.

[0075] Optionally, perform an intra-system lineage analysis on each transaction-related system to generate an intra-system data table, including: taking each transaction-related system as a target system and determining the target interface corresponding to the target system; extracting processing metadata from the SQL statement of the target interface based on static analysis technology, and determining each data table related to data processing based on the processing metadata; analyzing the data flow relationship between each data table to generate the intra-system data table of the target system.

[0076] Optionally, determining various data tables related to data processing based on processing metadata includes: using a preset parsing tool to convert the processing metadata into a tree structure; using the data after the first specified field in the tree structure as the target table for processing, and the table involved after the second specified field as the source table; traversing the tree structure based on the target table and the source table to obtain various data tables related to data processing.

[0077] S240: Match the data tables in the system using a preset regular expression to determine the downstream table and the downstream table name.

[0078] Specifically, the preset regular expression can be based on the table name naming conventions agreed upon by upstream and downstream systems. For example, downstream system table names use the "prefix + upstream table name" format, such as "P_00orders," where "P_00" is the prefix and "orders" is the upstream table name. During analysis, the controller uses the regular expression to match all table names in the system. Table names that meet this rule are identified as downstream tables. The capture group portion of the regular expression, such as "(\w+)", extracts the original table name fragment corresponding to the downstream table. For example, after the table name "P_00orders" matches the regular expression, "orders" is captured as the original table name.

[0079] S250. Remove the prefix from the downstream table name to obtain the upstream table name.

[0080] Specifically, according to naming rules, downstream table names typically have a fixed prefix, such as "P_00" or "S_." Using a regular expression to remove the prefix yields the upstream table name. For example, removing the "P_00" prefix from "P_00orders" yields the upstream table name "orders." This step relies on pre-agreed prefix rules between the upstream and downstream systems to ensure that the table name after removing the prefix is ​​consistent with the table name in the upstream system.

[0081] S260. Establish a mapping relationship between the upstream table name and the downstream table name to generate a lineage relationship view, and use the lineage relationship view as a result of the business data relationship analysis.

[0082] Specifically, the controller can associate the extracted upstream table names with the downstream table names, forming a mapping chain from upstream to downstream tables. For example, the table "orders" in upstream system A is mapped to the table "P_00orders" in downstream system B. System B's table "P_00orders" can then be mapped as an upstream table to the table "P_01orders" in system C, thereby establishing a cross-system data lineage relationship. Ultimately, the controller will present the mapping relationship in a visual or structured manner, forming a business data relationship analysis result, clearly showing the complete flow path of data from upstream to downstream, and supporting comprehensive monitoring and management of business data flow.

[0083] In a specific embodiment, Figure 3 A schematic diagram of a blood relationship view is provided for the second embodiment of the present invention. Figure 3 The figure shows three systems, namely System A, System B, and System C. Each system has several data tables, represented by circles, and arrows indicate the flow of data, that is, data flows from one data table to another.

[0084] The technical solution of the embodiment of the present invention can quickly establish cross-system data associations and clearly display the full-link flow of business data by performing inter-system lineage analysis on each transaction-related system based on preset regular expressions and data tables within the system to generate business data relationship analysis results. It helps managers accurately grasp the overall picture of business data flow, provides comprehensive and accurate data support for business decision-making, and improves the scientificity and effectiveness of architecture management.

[0085] Example 3

[0086] Figure 4 This is a schematic diagram of the structure of a business data relationship analysis device provided by the third embodiment of the present invention. Figure 4 As shown, the apparatus includes: an interface request data determination module 310, configured to obtain system data traffic and pre-process the system data traffic to determine interface request data;

[0087] The system and interface determination module 320 is used to determine the locator in the interface request data, and to construct a dictionary tree based on the locator. The dictionary tree is used to determine each transaction-related system and its corresponding interface;

[0088] The intra-system lineage analysis module 330 is used to perform intra-system lineage analysis on each transaction-related system to generate an intra-system data table;

[0089] The inter-system lineage analysis module 340 is used to perform inter-system lineage analysis on each transaction-related system based on a preset regular expression and a data table within the system to generate a business data relationship analysis result.

[0090] Optionally, the interface request data determination module 310 is specifically used to: determine non-interface request data in the system data traffic according to a specified field and preset filtering rules; filter the non-interface request data in the system data traffic and retain the interface request data.

[0091] Optionally, the system and interface determination module 320 is specifically used to: use the child nodes with the same parent node in the dictionary tree as target child nodes; cluster each target child node using the cosine similarity algorithm to generate each cluster node; and determine each transaction-related system and its corresponding interface by identifying the root path of each cluster node.

[0092] Optionally, the system and interface determination module 320 specifically includes: a node clustering unit, used to: determine the locator string corresponding to each target sub-node; convert each locator string into a vector, and calculate the similarity between each vector according to the cosine similarity algorithm; store the target sub-nodes whose similarity is greater than a preset similarity threshold closely to generate each clustering node.

[0093] Optionally, the intra-system lineage analysis module 330 is specifically used to: take each transaction-related system as the target system, and determine the target interface corresponding to the target system; based on static analysis technology, extract processing metadata from the SQL statement of the target interface, and determine each data table related to data processing based on the processing metadata; analyze the data flow relationship between each data table to generate the intra-system data table of the target system.

[0094] Optionally, the lineage analysis module 330 within the system specifically includes: a data table traversal unit, which is used to: use a preset parsing tool to convert the processing metadata into a tree structure; use the data after the first specified field in the tree structure as the target table for processing, and the table involved after the second specified field as the source table; traverse the tree structure based on the target table and the source table to obtain various data tables related to data processing.

[0095] Optionally, the inter-system lineage analysis module 340 is specifically used to: match the data tables within the system through preset regular expressions to determine the downstream table and the downstream table name; remove the prefix of the downstream table name to obtain the upstream table name; establish a mapping relationship between the upstream table name and the downstream table name to generate a lineage relationship view, and use the lineage relationship view as the business data relationship analysis result.

[0096] The technical solution of the present invention, through the construction of a dictionary tree and lineage analysis technology, can efficiently organize system architecture, intuitively present inter-system relationships, and reduce the cost and error of manually registering interfaces and data relationships. By performing intra-system lineage analysis on each transaction-related system to generate internal data tables, it is possible to deeply explore the internal data flow paths of the system, accurately locate the data processing process and the relationships between tables, and provide comprehensive and accurate data support for business decision-making.

[0097] A business data relationship analysis device provided by an embodiment of the present invention can execute a business data relationship analysis method provided by any embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method.

[0098] Example 4

[0099] Figure 5A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0100] like Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0101] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0102] The processor 11 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for analyzing business data relationships.

[0103] In some embodiments, a business data relationship analysis method can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the business data relationship analysis method described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute a business data relationship analysis method in any other appropriate manner (for example, by means of firmware).

[0104] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0106] In the context of the present invention, computer-readable storage media can be tangible media that can contain or store a computer program for use with an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Computer-readable storage media can include but are not limited to electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, computer-readable storage media can be machine-readable signal media. More specific examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0107] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0108] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0109] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0110] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0111] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for analyzing business data relationships, characterized in that: include: Acquiring system data traffic, and preprocessing the system data traffic to determine interface request data; Determine the locator in the interface request data, and construct a dictionary tree based on the locator, and determine each transaction-related system and its corresponding interface according to the dictionary tree; Conduct intra-system lineage analysis on each transaction-related system to generate intra-system data tables; Based on the preset regular expressions and the data tables in the system, the inter-system lineage analysis is performed on each transaction-related system to generate business data relationship analysis results.

2. The method according to claim 1, characterized in that The pre-processing of the system data traffic to determine the interface request data includes: Determine non-interface request data in system data traffic based on specified fields and preset filtering rules; Non-interface request data in the system data traffic is filtered, and interface request data is retained.

3. The method according to claim 1, characterized in that The determining of each transaction-related system and its corresponding interface according to the dictionary tree includes: Taking the child nodes with the same parent node in the dictionary tree as target child nodes; Clustering the target sub-nodes using a cosine similarity algorithm to generate cluster nodes; Each transaction-related system and its corresponding interface are determined by identifying the root path of each cluster node.

4. The method according to claim 3, characterized in that The clustering of the target sub-nodes using a cosine similarity algorithm to generate cluster nodes includes: Determine the locator character string corresponding to each target sub-node; Convert each of the locator character strings into a vector, and calculate the similarity between the vectors using a cosine similarity algorithm; The target sub-nodes whose similarity is greater than a preset similarity threshold are stored close to each other to generate cluster nodes.

5. The method according to claim 1, characterized in that The aforementioned intra-system lineage analysis of each transaction-related system is performed to generate intra-system data tables, including: Take each transaction-related system as the target system and determine the target interface corresponding to the target system; Extracting processing metadata from the SQL statements of the target interface based on static analysis technology, and determining data tables related to data processing based on the processing metadata; Analyze the data flow relationship between each data table to generate the system data table of the target system.

6. The method according to claim 5, characterized in that The determining of data tables related to data processing based on the processing metadata includes: Using a preset parsing tool to convert the processing metadata into a tree structure; The data after the first specified field in the tree structure is used as the target table for processing, and the table involved after the second specified field is used as the source table; The tree structure is traversed based on the target table and the source table to obtain various data tables related to data processing.

7. The method according to claim 1, characterized in that The inter-system lineage analysis of each transaction-related system is performed based on the preset regular expression and the data table in the system to generate the business data relationship analysis results, including: Match the data tables in the system using a preset regular expression to determine the downstream table and the downstream table name; Remove the prefix from the downstream table name to obtain the upstream table name; A mapping relationship between upstream table names and downstream table names is established to generate a lineage relationship view, and the lineage relationship view is used as a result of the business data relationship analysis.

8. A business data relationship analysis device, characterized in that: include: An interface request data determination module is used to obtain system data traffic and pre-process the system data traffic to determine interface request data; A system and interface determination module, configured to determine a locator in the interface request data, construct a dictionary tree based on the locator, and determine each transaction-related system and its corresponding interface based on the dictionary tree; The intra-system lineage analysis module is used to perform intra-system lineage analysis on each transaction-related system to generate intra-system data tables; The inter-system lineage analysis module is used to perform inter-system lineage analysis on each transaction-related system based on a preset regular expression and the data table in the system to generate a business data relationship analysis result.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that The computer storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 7 when executed.

Images