A linkage visualization method, device and medium based on multi-source heterogeneous data

Abstract

The embodiment of the specification discloses a linkage visualization method and device based on multi-source heterogeneous data and a medium, relates to the technical field of visualization analysis, and the method comprises the following steps: acquiring the visualization analysis requirement of a user, determining a plurality of heterogeneous data sources according to the visualization analysis requirement, extracting metadata based on the plurality of heterogeneous data sources to determine a target metadata set; field relationship matching information is determined by identifying the field association relationship of the target metadata set, and a dynamic data structure graph corresponding to the target metadata set is generated according to the field relationship matching information; in response to a node triggering operation of a user front end, node association information corresponding to a triggering node is determined based on the dynamic data structure graph and the node triggering operation, and a penetration type analysis result is generated on a visualization interface, thereby solving the problem that business personnel cannot build complex table association relationships with zero threshold in the analysis and management process of multi-source heterogeneous data due to the low correlation between artificial modeling and business scenarios.

Description

Technical Field

[0001] This specification relates to the field of visualization analysis technology, and in particular to a method, device and medium for linkage visualization based on multi-source heterogeneous data. Background Technology

[0002] In the process of managing data assets, existing business intelligence tools (such as Power BI and Tableau) support access to multiple data sources and can be implemented through manual field matching and connector modeling. Field linkage can also be configured in report settings. However, relying on manual configuration of field relationships requires business personnel to understand database table structures, SQL syntax, and other technical knowledge, making data analysis too difficult. Lineage graph tools (such as Apache Atlas) focus on tracing technical lineage, automatically generating upstream and downstream dependencies between tables and fields, and visually presenting ETL processes and task flow paths. However, the field dependency paths they generate lack business semantic explanations, making it difficult for business personnel to intuitively understand the logical relationships between business entities such as "customer table - order table - marketing table." Furthermore, existing visualization systems only support static displays or simple report linkages, failing to support cross-table penetration analysis through clicks and drill-down operations, resulting in low efficiency for in-depth data insights.

[0003] In the process of analyzing and managing multi-source heterogeneous data, existing technologies rely on manual modeling and have low relevance to business scenarios. This results in business personnel being unable to build complex table relationships with zero barriers, leading to a lag in data analysis. Summary of the Invention

[0004] This specification provides one or more embodiments of a linkage visualization method, device, and medium based on multi-source heterogeneous data to solve the following technical problem: In the process of analyzing and managing multi-source heterogeneous data, existing technologies rely on manual modeling, which has low relevance to business scenarios. As a result, business personnel cannot build complex table relationships with zero barriers, leading to data analysis delays.

[0005] One or more embodiments of this specification employ the following technical solutions:

[0006] This specification provides one or more embodiments of a linked visualization method based on multi-source heterogeneous data. The method includes: acquiring the user's visualization analysis requirements; determining multiple heterogeneous data sources based on the visualization analysis requirements; extracting metadata based on the multiple heterogeneous data sources to determine a target metadata set; identifying field association relationships in the target metadata set to determine field relationship matching information; generating a dynamic data structure graph corresponding to the target metadata set based on the field relationship matching information; and responding to a node triggering operation on the user's front end, determining node association information corresponding to the triggering node based on the dynamic data structure graph and the node triggering operation to generate a penetrating analysis result on the visualization interface.

[0007] This specification provides one or more embodiments of a linked visualization device based on multi-source heterogeneous data, including:

[0008] At least one processor; and,

[0009] A memory communicatively connected to the at least one processor; wherein,

[0010] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the above-described method.

[0011] This specification provides one or more embodiments of a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0012] The above-described at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0013] The technical solutions described in this specification, through their embodiments, highlight the high technical hurdle that traditional methods require business personnel to understand database table structures, field naming conventions, and SQL syntax to effectively utilize data. Intelligent semantic parsing automatically transforms data into target business types and accurately locates heterogeneous data sources, enabling business personnel to determine the required data without needing technical expertise. Simultaneously, multi-dimensional metadata extraction transforms technical fields into semantic information understandable to the business, eliminating the gap between business language and technical metadata. By identifying field relationships, a dynamic data structure graph is generated, realizing the connections between heterogeneous database tables and effectively identifying potential relationships between different data tables. This reduces the difficulty of traditional manual configuration modeling and improves modeling accuracy and efficiency. Without manual intervention, it significantly enhances the intelligence and visualization capabilities of multi-table data organization, solving problems such as low field modeling efficiency, weak graph interaction, and difficult-to-understand data structures in existing technologies. It creates a barrier-free channel from business needs to analytical conclusions, lowering the professional technical threshold for cross-system data analysis. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0015] Figure 1 A flowchart illustrating a method for linking and visualizing multi-source heterogeneous data, provided in an embodiment of this specification;

[0016] Figure 2 A schematic diagram of the system structure of a linkage visualization system based on multi-source heterogeneous data provided in the embodiments of this specification;

[0017] Figure 3 A schematic diagram of a spectral structure provided for an embodiment of this specification;

[0018] Figure 4 This is a schematic diagram of the structure of a linkage visualization device based on multi-source heterogeneous data, provided as an embodiment of this specification. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0020] This specification provides a method for interactive visualization based on multi-source heterogeneous data. It should be noted that the execution subject in this specification can be a server or any device with data processing capabilities. Figure 1 This is a flowchart illustrating a method for linking and visualizing multi-source heterogeneous data, as provided in an embodiment of this specification. Figure 1 As shown, the main steps include the following:

[0021] Step S101: Obtain the user's visualization analysis requirements, determine multiple heterogeneous data sources based on the visualization analysis requirements, extract metadata based on multiple heterogeneous data sources, and determine the target metadata set.

[0022] In cross-business data analysis scenarios, the analytical requirements proposed by front-end business personnel contain clearly defined business entities and rules. However, the back-end database contains various named tables, making it difficult for business personnel to understand the mapping relationship between these technical table names and actual business operations. For example, front-end business personnel might propose analyzing the consumption behavior of high-net-worth customers in the Yangtze River Delta region. The business entities are "customer" and "consumption," specified as "high-net-worth" and "Yangtze River Delta." The same business requirement may involve multiple systems, such as user profile databases, transaction databases, and geographic information databases. However, business personnel lack the ability to determine which tables contain the required data and can only rely on technical personnel to manually filter the data, leading to analysis delays. In the past, specific table combinations have been integrated for similar requirements (such as "East China customer repurchase rate analysis"), but due to the lack of systematic records, data source matching needs to be re-matched each time, resulting in repetitive work.

[0023] Based on the visualization analysis requirements, multiple heterogeneous data sources are identified, specifically including: extracting business semantics from the visualization analysis requirements using a pre-trained industry entity recognition model to determine the target analysis business type; collecting historical data source access information from multiple preset data sources to statistically determine at least one historical access business type corresponding to each data source based on this historical data source access information; determining at least one high-frequency access business type corresponding to each data source according to the access frequency of each historical access business type and in a specified order of the access frequency; and matching the target analysis business type with at least one high-frequency access business type corresponding to each data source to determine the corresponding multiple heterogeneous data sources.

[0024] In one embodiment of this specification, an industry entity recognition model is used to translate natural language requirements into machine-understandable business type labels, such as customer analysis and consumer behavior, reducing the gap between business language and technical metadata. By utilizing historical access frequency statistics, the most commonly used data source combinations under different business types are identified. For example, "customer analysis" frequently uses user profile tables and transaction flow tables, replacing human experience. Based on the similarity between current requirements and historically high-frequency types, the most relevant heterogeneous data sources are automatically locked to avoid invalid data access.

[0025] First, a pre-trained industry entity recognition model is used to process user input. It's important to note that this model is trained on a domain-specific labeled corpus and can identify key entities and intents within the user's request. For example, given the input "Analysis of consumption preferences of high-net-worth clients in the Yangtze River Delta region over the past six months," the model outputs structured labels: Region: Yangtze River Delta, Client Type: High Net Worth, Indicator: Consumption Preference, Time Range: Past Six Months. Based on pre-defined business type mapping rules, the target business type is determined to be customer behavior analysis. Entries containing both "customer" and "consumption" labels are categorized as customer behavior analysis. Furthermore, this industry entity recognition model requires periodic incremental training with new corpus data to cover emerging business vocabulary.

[0026] The system has a built-in data source access log library that records the triplet information for each analysis task. This triplet includes the business type, the list of accessed data sources, and the frequency of use. For each data source, historical access records are aggregated by business type. All historical tasks from data source A are extracted, and their frequency under business types such as customer analysis and marketing performance is calculated. These are then sorted in descending order of frequency, retaining the top K most frequent business types. For example, in the user profile table, the most frequent type is customer analysis (85 times), which is greater than marketing performance (32 times). Whenever a new task is executed, the system automatically records the correspondence between business type and data source and refreshes the statistical results.

[0027] If the target business type is completely identical to the high-frequency type of data source A, it is directly selected. If there is no completely matching data source item, the semantic similarity between the target type and the high-frequency types of each data source is calculated. Here, Word2Vec word vector cosine similarity can be used, and if it exceeds a set threshold, it is selected. Check whether the candidate data source set covers all key entities of the requirements. For example, "Yangtze River Delta" needs to include a geographical field, and "consumption preference" needs to include order details. If not covered, fill in the tables corresponding to the missing entities from high-frequency data sources of similar businesses. Finally, a list of heterogeneous data sources is obtained, such as user profile tables, order transaction tables, and city geographic information tables, ensuring that they not only conform to the current business semantics but are also historically verified to be efficient and usable.

[0028] Compared to traditional manual data source matching, the above technical solution employs a pre-trained entity recognition model to accurately capture business semantics and associate them with technical metadata, avoiding table selection errors caused by terminology differences. It dynamically constructs high-frequency type-data source mappings based on historical access frequencies, making the selection process data-driven and ensuring optimal table selection in common scenarios. Through combination coverage verification during the matching phase, it confirms in advance that candidate data sources contain all necessary entities, mitigating the risk of missing data from the source. Business personnel do not need to understand professional table structures; they can automatically obtain accurately matched heterogeneous data source combinations simply through natural language descriptions, compressing the technical research that originally required hours to minutes, achieving zero-threshold cross-table analysis.

[0029] Based on these multiple heterogeneous data sources, metadata extraction is performed to determine the target metadata set. Specifically, this includes: extracting table features for each database table within each heterogeneous data source to determine the multi-dimensional field features corresponding to each table. These multi-dimensional field features include field name, field data type, and field sample value. The field description text is parsed for specified fields in each database table. Database logs are monitored, and data lineage is extracted by parsing ETL task execution records in the database logs to generate a data lineage identifier. This data lineage identifier includes the source table, target table, and field mapping relationship. Based on these multi-dimensional field features, the field description text, and the data lineage identifier, the multi-dimensional field features are further refined.

[0030] In cross-system data integration scenarios, heterogeneous data sources such as MySQL relational tables, PostGIS spatial databases, and MongoDB document databases exhibit differences in field naming and data types, making it difficult for business users to intuitively understand the business meaning of fields. Furthermore, the lack of business context in database table structures leads to misuse of fields by business users. In addition, traditional metadata is a static snapshot, which cannot detect dynamically generated derived fields by ETL tasks, causing biases in subsequent analysis.

[0031] In one embodiment of this specification, a physical connection is established for each heterogeneous data source. This involves connecting to the database via JDBC, reading geographic files using GDAL, and traversing all data tables in the database. The table structure definition is read to obtain static attributes such as field names and data types. The first N rows of the data table are extracted as sample values; text fields are truncated with typical value fragments, while numeric fields retain their original values. Extended database attributes, such as COMMENT in MySQL and pg_description in PostgreSQL, are scanned to extract field description text. If no database description exists, field descriptions or file metadata tags, such as BandDescription in GeoTIFF, are parsed from the API documentation.

[0032] Deploy a lightweight log collector on the database server to capture transaction logs in real time. Configure ETL task tagging rules, such as recognizing SQL comments in the format / *ETL_TASK:user_profile_agg* / . Parse the INSERT INTO target_table SELECT...FROM source_table statements in the logs; extract the source table name, target table name, and field mapping expressions (e.g., target_col = source_col_a * 0.8); recursively decompose nested subqueries, tracing back to the original physical table fields. Integrate the basic characteristics, descriptive text, and lineage of fields into a unified structure, create a descriptive object for each field, merge all field objects, and add table-level attributes. The data volume is obtained through SELECT COUNT(*), and the update timestamp is read through file attributes or database statistics.

[0033] By proactively parsing database description text and API documentation, the field name usr_attr_7 is associated with the business term "Membership Level," enabling business personnel to directly understand the field's purpose. Through real-time parsing of field-level transformation logic in ETL logs, calculation rules are directly exposed in metadata, helping business personnel verify the correctness of metric calculation logic. By continuously monitoring database logs, sample values ​​and data types are automatically updated when structural changes occur, ensuring real-time synchronization of metadata with the production environment and eliminating data mapping errors caused by outdated metadata. Business personnel can understand that the gmv field is calculated from the order details table and the membership level field is described as "1-Diamond Member" without consulting technical personnel, significantly reducing the cognitive threshold and technical dependence of cross-table data analysis.

[0034] Step S102: Identify the field relationships of the target metadata set and determine the field relationship matching information, so as to generate a dynamic data structure graph corresponding to the target metadata set based on the field relationship matching information.

[0035] The target metadata set is subjected to field relationship identification to determine field relationship matching information. Specifically, this includes: determining the uniqueness ratio of each field based on the field sample values ​​in the target metadata set, and identifying primary key candidate fields that exceed a preset ratio threshold based on the uniqueness ratio; traversing the sample value sets of other fields, and if the sample value set corresponding to a foreign key candidate field is contained in the primary key sample value set corresponding to the primary key candidate field, generating a primary-foreign key relationship pair, wherein the primary-foreign key relationship pair includes the main table, primary key field, subordinate table, and foreign key field; performing semantic relationship matching on multiple fields through the field names and field description text in the target metadata set to determine semantic field relationship pairs; and determining static field relationship matching pairs based on the primary-foreign key relationship pairs and the semantic field relationship pairs, and correcting the static field relationships based on the data lineage identifier in the target metadata set to determine the field relationship matching information. Semantic relationship matching is performed on multiple fields using field names and field description text in the target metadata set to determine semantic field relationship pairs. Specifically, this includes: determining the field type of each field; calculating text similarity for the field name when the field type is text; constructing probability distribution models for the sample value set of the numerical field when the field type is numeric; and determining the similarity score between multiple numeric fields based on the KL divergence value between these probability distribution models. Semantic field relationship pairs are then determined based on the text similarity and / or the similarity score. Based on the data lineage identifier in the target metadata set, the static field relationship is corrected to determine field relationship matching information. This includes: parsing the dynamic relationship records in the data lineage identifier to extract the source field name, target field name, and conversion type; comparing the static field relationship pair with the source field name, target field name, and conversion type for conflict detection; and correcting the static field relationship when a conflict is detected to determine the field relationship matching information.

[0036] Traditional methods rely on field name matching, but when field names are irregular, such as cust_code and vip_no, they cannot identify related fields with the same value range. Business personnel find it difficult to determine whether account balance (account_balance) and total assets (total_asset) can be correlated for analysis, easily leading to erroneous data mixing. Furthermore, static correlations cannot detect the ETL processing chain; for example, if a mobile phone number is encrypted as an MD5 value, business personnel may mistakenly match the encrypted text with the original field, resulting in analytical bias. This solution reveals the actual correlation between the customer table's .vip_no and the order table's .cust_code through sample value overlap analysis; interprets account_balance as "available balance for the day" based on the descriptive text, distinguishing it from total_asset and avoiding erroneous correlations; and identifies user tables, mobile phone numbers, MD5 values, risk control tables, and encrypted mobile phone numbers through lineage identifiers, preventing business personnel from directly correlating original fields.

[0037] In one embodiment of this specification, the primary and foreign key relationships are first deduced based on sample values ​​of fields in the target metadata set. Specifically, this is achieved through the following process: For each field, a sample value set is extracted, and the proportion of unique values ​​to the total sample is calculated. When this proportion is significantly higher than the standard for primary key uniqueness, the field is marked as a primary key candidate field, and its complete sample value set is recorded. Subsequently, the sample value sets of other fields are traversed to check if there are any foreign key candidate fields whose value sets exhibit a high degree of inclusion relationship with the value sets of primary key candidate fields (i.e., the vast majority of values ​​in the foreign key value set also exist in the primary key value set). If this is satisfied, a primary and foreign key relationship pair is generated, explicitly annotating the primary table name, primary key field name, secondary table name, and foreign key field name. For example, after the `vip_no` field of the user table is determined as the primary key, when it is detected that 98% of the values ​​of the `cust_code` field of the order table exist in the `vip_no` value set, a primary and foreign key relationship is established.

[0038] Next, semantic relationship matching is performed, which requires different processing based on the data type of the fields. For text fields such as customer names and product descriptions, after normalizing the field names by removing special symbols and unifying lowercase, the edit distance similarity is calculated. This is then combined with semantically weighted text such as database COMMENT content or API documentation descriptions. When the overall similarity exceeds the industry's standard threshold, it is determined to be a semantically equivalent relationship. For numerical fields such as transaction amounts and inventory quantities, sample value distribution models are constructed for each. Through visual distribution pattern comparison and probability distribution difference quantitative analysis, data fields with similar statistical characteristics are identified. Here, the visual distribution pattern comparison is such as the overlap of histogram shapes. When the text similarity or statistical feature similarity reaches the preset standard, semantic field relationship pairs are generated and the relationship type is labeled. For example, "account balance" and "available credit" are marked as complementary relationships.

[0039] The process involves parsing transformation records in the data lineage identifier, such as the `INSERT INTO targetSELECT source FROM raw` statement in the ETL task log, to extract the source field, target field, and transformation operation type. These transformation operation types include aggregation calculation, encryption, and type conversion. Static relationship pairs (primary / foreign key pairs or semantic pairs) are compared item by item with the lineage records. When a discrepancy is detected between the source field of a static relationship and the lineage source field, the original related field is replaced with the lineage source field. When the lineage reveals a logical conflict in the static relationship, such as a static semantic association field actually being an encrypted relationship, the original confidence level is downgraded and a conflict flag is added. When the lineage path contains intermediate derived fields, such as `gmv = price * quantity`, a derived node is inserted into the association path and the relationship chain is updated. Finally, dynamically calibrated field relationship matching information is output, including the corrected relationship type, confidence level, and lineage version identifier. Throughout the process, incremental updates are implemented by real-time monitoring of the database logs to ensure that field relationships dynamically evolve with changes in the data environment.

[0040] The above technical solution automatically derives primary and foreign key relationships based on the overlap of sample values. This allows business users to automatically identify the consistency of value ranges and establish associations when faced with naming differences such as `vip_no` and `cust_code`, eliminating the blind spots of manual matching. By supplementing business meaning with descriptive text and cross-validating with value distribution patterns, it accurately associates business-equivalent fields, avoiding omissions in associations due to name differences. Furthermore, by parsing ETL lineage identifiers, it dynamically corrects static relationships into derivation paths, preserving both business semantic associations and clarifying technical differences, preventing association errors caused by missing dynamic logic. Business users do not need to know whether `cust_code` is a foreign key or how `md5_phone` is generated; it automatically constructs accurate association networks, supporting zero-threshold cross-table penetration analysis.

[0041] Based on the field relationship matching information, a dynamic data structure graph corresponding to the target metadata set is generated. Specifically, this includes: creating a master node for each data table and a child node for each field, wherein the child node is attached to the master node of the corresponding data table; generating connection edges between master nodes based on the field relationship pairs in the field relationship matching information, and annotating the associated field names on the connection edges; and attaching attribute information to each master node to generate a dynamic data structure graph corresponding to the target metadata set, wherein the attribute information includes the data table name, number of fields, data volume, and update time.

[0042] In cross-system data analysis scenarios, related fields scattered across heterogeneous data sources cannot be intuitively perceived by business personnel due to naming differences and technical barriers, requiring technical personnel to manually associate them one by one; business personnel cannot quickly determine the data scale or update time of a certain table, leading to the selection of expired or low-quality data; traditional list-style metadata display cannot present the business chain relationship of customers, orders, and logistics, hindering multi-table penetration analysis.

[0043] In one embodiment of this specification, when constructing a dynamic data structure graph, the node hierarchy is first created. A rectangular master node is created for each data table, its size dynamically set based on the data volume. The table's row count is obtained by querying the database statistics interface (e.g., `SELECT COUNT(*) FROM table_name`), dividing the data volume into preset intervals. For example, small tables (below 10,000 rows) use standard-sized rectangles, medium-sized tables (10,000 to millions of rows) are enlarged by 1.5 times, and large tables (above millions of rows) are enlarged by 2 times. The table name, such as "Customer Master Table," is displayed in bold at the center of the node, and a business type color is added to the node border, such as blue for basic information tables and green for transaction tables. The table structure definition is parsed to obtain a list of all fields. A circular child node is created for each field and labeled with the field name, such as "customer_id." Field types are distinguished by child node colors, such as dark blue for text fields, green for numeric fields, and orange for date fields. Child nodes are arranged in a tree layout below their respective master nodes, with indentation lines clearly indicating their affiliation. Next, inter-table join edges are generated. Relationship pairs are read from the field relationship matching information, and directed connections are drawn between the corresponding primary nodes. Primary and foreign key relationships are represented by solid arrows, such as in the `order_id` table. → In the payment table `.ref_id`, semantic relationships are represented by dashed arrows, such as `.region` ≈ `.name`. When multiple relationships point to the same target field, for example, `.phone` in the user table and `.mobile` in the member table are both related to `.contact` in the service record table, a forked connection is generated at the target node, and a floating tooltip is added to display conflict details, including the source table name, confidence level, and relationship type.

[0044] Then, dynamic attributes are injected. For example, the table name is displayed in the center of the main node, and the font size adjusts automatically with the node size; the number of fields is indicated by adding a marker in the upper right corner of the node, such as... " 28 fields; the status bar at the bottom of the node displays a formatted value to indicate the amount of data, such as " "12,850,326 rows"; the label on the left of the node displays the last update time, with data sourced from the database transaction log commit timestamp. A scheduled task is deployed to call the database metadata interface to refresh attribute values. Incremental listening is enabled for frequently updated tables, and the data volume label is updated immediately after capturing INSERT / UPDATE events via the database CDC mechanism.

[0045] Through the above technical solutions, by automatically parsing field relationship matching information and generating connection lines in real time, the newly added field relationships are synchronized to the graph immediately, eliminating the delay caused by manual maintenance. Through recursive expansion and path highlighting mechanisms, clicking on the customer table will automatically expand the complete business chain, and support drilling down data at any intermediate node, such as directly viewing the order details of a customer. This allows business personnel to perform cross-table penetration analysis without technical background, truly achieving zero-threshold data exploration.

[0046] Step S103: In response to the node triggering operation on the user's front end, based on the dynamic data structure graph and the node triggering operation, determine the node association information corresponding to the triggering node, so as to generate penetrating analysis results on the visualization interface.

[0047] Based on the dynamic data structure graph and the node triggering operation, the node association information corresponding to the triggering node is determined to generate penetrating analysis results in the visualization interface. Specifically, this includes: determining the operation type of the node triggering operation, where the operation type includes any one or more of node click, region selection, and field double-click; when the operation type is node click, recursively querying the upstream and downstream association paths in the dynamic data structure graph starting from the triggering node to generate an association path chain, highlighting the association path chain in the visualization interface, and displaying detailed data of each node in the association path chain in a separate window; when the operation type is region selection, parsing the spatial coordinates or value range corresponding to the triggering node to generate filtering conditions, and extending them to the association table, rendering the filtered data of the association table in preset parallel sub-windows to display the conditional filtering results; when the operation type is field double-click, parsing the field calculation expression corresponding to the triggering node and generating a logic flowchart, displaying the logic flowchart, and obtaining the corresponding original data value table.

[0048] When business personnel explore scenarios involving multi-source data association, traditional solutions require them to manually piece together the analysis chain of customers, orders, and logistics, making it impossible to access the complete business chain with a single click. After selecting a specific area or condition, they need to view the data in each table separately, making real-time linked analysis impossible. In addition, when faced with composite indicators, such as GMV = unit price × quantity, business personnel find it difficult to trace the original field composition.

[0049] In one embodiment of this specification, upon responding to a front-end node trigger operation, the user interaction event is first captured and the operation type is identified. If it is a node click, a recursive path query process is initiated. All upstream nodes (such as the registration system table) and downstream nodes (such as the order table) directly associated with the triggering node are retrieved through the adjacency table (stored as [source table ID, target table ID, associated fields, relationship type]). This process is then recursively applied to secondary associated nodes (such as the upstream third-party authorization table and the downstream logistics table), generating a complete path chain: third-party authorization table, registration system table, customer table, order table, and logistics table. This path is highlighted with a thick orange line in the visualization interface, while non-associated nodes are grayed out. Simultaneously, the latest data for each table in the path is loaded in a separate paginated window. Table attributes are displayed at the top of each page, including table name, data volume, and update time. The first 100 rows of sample data are rendered below, such as the customer table displaying ID, name, and contact information field values, supporting sorting by field and keyword filtering.

[0050] If the operation type is area selection, such as drawing a polygon in the Yangtze River Delta on a map, a GIS engine such as the GDAL library is invoked to convert screen coordinates into geospatial range conditions, generating WKT format POLYGON((longitude 1, latitude 1, longitude 2, latitude 2...)), or parsing the numerical range to generate SQL filter conditions, such as unit_price>5000. This condition is automatically extended to the related tables in the dynamic map. Starting with the customer table, the related order and product tables are located through field relationship matching information, constructing a multi-table join query. The results are then rendered in preset parallel sub-windows. For example, the left window displays customer data that meets the regional conditions, the middle window displays related order details, and the right window loads the corresponding product information. The title bars of each window display the number of records matched in real time, such as: Customers: 12,340 records.

[0051] When the operation type is a double-click on a field, such as double-clicking the GMV field node, a drill-down analysis process is executed. The calculation expression, GMV = price * quantity, is extracted from the data lineage identifier and broken down into operands price, quantity, and the operator * by the abstract syntax tree parser. A directed acyclic graph (DAG) is generated, with the left-hand nodes representing the source fields (e.g., circular icons labeled price and quantity), and the calculation nodes (e.g., diamond icons labeled multipliers) connecting to the target field GMV. Blue arrows indicate the data flow. Below the flowchart, a raw data table is loaded. The left table displays sample values ​​for the price field containing product ID and price, and the right table displays sample values ​​for the quantity field containing order ID and quantity. Sorting by clicking the table header or filtering by input boxes is supported.

[0052] Through the above technical solutions, a complete association path is automatically constructed via recursive queries. Users only need to click the starting node to highlight the entire chain, significantly reducing the cognitive burden and technical threshold of multi-level association analysis. Based on the association relationships in the dynamic graph, the selection conditions are intelligently extended to the association table, enabling penetrating linkage by selecting customer regions and automatically bringing up related orders and products. This allows business personnel to obtain cross-entity analysis results in a single operation. By parsing the lineage expression to generate a visual logic diagram, the abstract formula is transformed into an understandable "Price × Quantity = GMV". ” The computation flow provides synchronous verification of raw data, helping business personnel to verify the rationality of indicators and avoid decision-making biases caused by ambiguity in calculation rules.

[0053] The technical solutions described in this specification, through their embodiments, highlight the high technical hurdle that traditional methods require business personnel to understand database table structures, field naming conventions, and SQL syntax to effectively utilize data. Intelligent semantic parsing automatically transforms data into target business types and accurately locates heterogeneous data sources, enabling business personnel to determine the required data without needing technical expertise. Simultaneously, multi-dimensional metadata extraction transforms technical fields into semantic information understandable to the business, eliminating the gap between business language and technical metadata. By identifying field relationships, a dynamic data structure graph is generated, realizing the connections between heterogeneous database tables and effectively identifying potential relationships between different data tables. This reduces the difficulty of traditional manual configuration modeling and improves modeling accuracy and efficiency. Without manual intervention, it significantly enhances the intelligence and visualization capabilities of multi-table data organization, solving problems such as low field modeling efficiency, weak graph interaction, and difficult-to-understand data structures in existing technologies. It creates a barrier-free channel from business needs to analytical conclusions, lowering the professional technical threshold for cross-system data analysis.

[0054] Figure 2 This specification provides a schematic diagram of the system architecture of a linked visualization system based on multi-source heterogeneous data, as shown in the embodiments of this specification. Figure 2 As shown, the linked visualization system includes a data access module, a metadata management model, a field association identification module, a graph construction and display module, an interactive linkage analysis module, and a rule maintenance and graph update module. The data access module handles the access and preprocessing of data tables; the metadata management module manages field information and table structures uniformly; the field association identification module performs semantic matching and sample value analysis; the graph construction and display module builds a front-end visualized data graph; the interactive linkage analysis module supports user click filtering and path query operations; and the rule maintenance and graph update module dynamically manages rules and graph structures.

[0055] In one embodiment of this specification, the system supports access to structured data tables from multiple sources (such as relational databases, data platform APIs, file systems, etc.). The data structure of each table is extracted using metadata such as field names, field types, and sample values. Specifically, data access is achieved through JDBC, ODBC, HTTP APIs, or middleware. During the preprocessing stage, field names are standardized in terms of case sensitivity and symbol handling. The attribute metadata structure of each field, {field name, data type, sample value, table name}, is constructed and stored in the field metadata database for subsequent processing.

[0056] Next, a combination of semantic matching algorithms and sample data analysis was used to determine potential field correspondences between different tables, forming a preliminary field relationship network. This was first achieved through field name similarity matching, such as... " user_id ” and " uid ” Matching is performed using methods such as edit distance and root word normalization. Secondly, data value set overlap analysis is conducted; if there is significant overlap in the value sets of field A and field B, the field overlap degree is calculated. Inference can also be made based on field function type, combining uniqueness / repetition rate to infer primary key, foreign key, or indicator attributes. Accuracy is enhanced by using built-in system rules, such as the order table typically being linked to the user table via user_id. The output structure is the generated field relationship pair {Table A. Field X, Table B. Field Y, Association Confidence}.

[0057] The identified field relationships are organized into a visual graph or tree structure, forming a logical network between multiple data sources. Figure 3 A schematic diagram of a spectral structure provided for an embodiment of this specification, such as... Figure 3 As shown, nodes are organized by data table or business entity. Edges are generated from field relationships, displaying the associated fields. Node attributes include metadata such as table name, number of fields, data volume, and update time. It supports initiating upstream and downstream paths from any node.

[0058] In the front-end visualization page, users can click, drag, and filter graph nodes, automatically analyzing the actual data in each table based on the graph structure. Interactive analysis here includes path penetration analysis, conditional filtering, and field value drill-down. Path penetration analysis starts from the customer table and automatically displays their orders and activity participation. In conditional filtering, selecting a tag node automatically filters out matching users and their orders. Field value drill-down means that clicking on a specific indicator in the visualization automatically drills down to the original fields that constitute that indicator. Additionally, users can manually add or delete field relationships, and the system automatically refreshes the graph.

[0059] The system supports real-time or periodic scanning of new tables and fields, automatically assessing their impact on the current graph structure. If a potential relationship is found in a new field, the system prompts the user to expand the graph. A field change monitoring mechanism, such as monitoring database schema change logs, automatically reruns the field matching process. Users can set notification rules, such as prompting for an update when the overlap between a new field and an existing graph exceeds 70%. Through these technical solutions, the system automatically identifies and models the relationships between fields and dimensions across multiple data sources, further transforming them into interactive graphs or tree diagrams. It also supports user-side operations such as path penetration, field filtering, and drill-down analysis. By using semantic similarity matching, sample value analysis, and rule templates, the system effectively identifies potential relationships between different data tables, reducing the difficulty of traditional manual configuration modeling and improving modeling accuracy and efficiency. Therefore, without the need for manual intervention, it significantly improves the intelligence and visualization capabilities of multi-table data organization, and solves problems such as low field modeling efficiency, weak graph interaction, and difficult-to-understand data structure in existing technologies. It not only enhances the transparency of data relationships, but also improves the data analysis capabilities of business personnel, providing users with an efficient, intuitive, and flexible way to gain data insights in scenarios such as marketing, operations, and tag building.

[0060] This manual also provides examples of automatic tag system modeling and linked analysis with customer profiles. The business background involves a large internet platform that has accumulated a large amount of customer behavior data, distributed across multiple heterogeneous systems such as the registration system, event tracking log system, transaction system, and marketing platform. The operations team needs to build customer tag information (such as "high-value users," "active users," and "users at risk of churn") for audience segmentation and targeted marketing. The table field naming is inconsistent across different systems; for example, it's user_id in the registration system and uid in the transaction system. Tag fields are scattered and difficult to view centrally. Customer analysis requires cross-table joins, heavily relying on data warehouse support, which is inefficient.

[0061] Using the above technical solution, the system accesses the registry (registration information), transaction details table, and marketing log table. Field names are standardized, such as uniformly identifying user_id, uid, and user number as the same primary key semantics. Semantic matching and sample value analysis identify field relationships and construct a corresponding graph structure. When operations personnel click on the "High-Value Users" tag node in the graph, the system automatically filters users who meet the following rules: recent 30-day spending > 2000, activity score > 90. The system then displays the active activities, participation frequency, and conversion rate of this user group. This user group can be exported as the target user group for the next campaign.

[0062] This manual also provides examples of order anomaly analysis and business relationship tracing. The business background is that a SaaS supply chain platform discovered a large number of order fulfillment delays. Operations personnel need to determine the cause of the delays, but the data is scattered across the order system, customer management system, product system, and logistics system, making analysis difficult. The goal is to identify the key factor combination of delays (e.g., a certain type of customer + a certain type of product + a certain region + a certain logistics method). In this context, multiple tables have inconsistent fields; for example, the order table uses `customer_id`, and the customer table uses `cust_id`. The field logic chains are complex, requiring manual table queries to find the cause. There is a lack of intuitive, structured anomaly path visualization tools. Using the technical solution provided in the examples in this manual, when a user selects "Delivery delay > 24 hours," the system automatically links to the customers, products, and logistics methods of these orders. The user's filtering reveals: the customer level is "newly registered," the product is "cold chain," and the logistics is "third-party courier A." ” The system automatically marks these as potential abnormal combinations and supports exporting reports.

[0063] This specification also provides an embodiment of a linkage visualization device based on multi-source heterogeneous data, such as... Figure 4 As shown, the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the above-described method.

[0064] This specification also provides a non-volatile computer storage medium storing computer-executable instructions configured to perform the above-described method.

[0065] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0066] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0067] The devices, media, and methods provided in the embodiments of this specification are one-to-one correspondences. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0068] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0069] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0070] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0072] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0073] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0074] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0075] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, the phrase "comprising a…" … ” The definition of a specific element does not preclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0076] The above description is merely one or more embodiments of this specification and is not intended to limit this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of one or more embodiments of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for interactive visualization based on multi-source heterogeneous data, characterized in that, The method includes: Obtain the user's visualization analysis requirements, determine multiple heterogeneous data sources based on the visualization analysis requirements, extract metadata based on the multiple heterogeneous data sources, and determine the target metadata set; Field association relationships are identified in the target metadata set to determine field relationship matching information, and a dynamic data structure graph corresponding to the target metadata set is generated based on the field relationship matching information. In response to a node triggering operation from the user's front end, based on the dynamic data structure graph and the node triggering operation, the node association information corresponding to the triggering node is determined so as to generate penetrating analysis results in the visualization interface; The target metadata set is subjected to field association relationship identification to determine field relationship matching information, specifically including: Based on the field sample values ​​in the target metadata set, the uniqueness ratio of each field is determined, and primary key candidate fields exceeding a preset ratio threshold are determined according to the uniqueness ratio. Iterate through the sample value sets of other fields. If there is a sample value set corresponding to a foreign key candidate field that is contained in the primary key sample value set corresponding to the primary key candidate field, generate a primary-foreign key relationship pair, wherein the primary-foreign key relationship pair includes the main table, the primary key field, the subordinate table, and the foreign key field. Semantic relationship matching is performed on multiple fields using the field names and field description text in the target metadata set to determine semantic field relationship pairs; Based on the primary-foreign key relationship pair and the semantic field relationship pair, a static field relationship matching pair is determined. Then, based on the data lineage identifier in the target metadata set, the static field relationship matching pair is corrected to determine the field relationship matching information.

2. The linkage visualization method based on multi-source heterogeneous data according to claim 1, characterized in that, Based on the aforementioned visualization analysis requirements, multiple heterogeneous data sources were identified, specifically including: By using a pre-trained industry entity recognition model, business semantics are extracted from the visualization analysis requirements to determine the target analysis business type; Collect historical data source access information from multiple preset data sources, and based on the historical data source access information, count at least one historical access service type corresponding to each data source; Based on the access frequency corresponding to each of the historical access service types, at least one high-frequency access service type corresponding to each data source is determined in a specified order of the access frequencies. Based on the target analysis service type and at least one high-frequency access service type corresponding to each data source, data source matching is performed to determine multiple corresponding heterogeneous data sources.

3. A linked visualization method based on multi-source heterogeneous data according to claim 1, wherein Based on the aforementioned multiple heterogeneous data sources, metadata extraction is performed to determine the target metadata set, specifically including: In each of the heterogeneous data sources, data table features are extracted for each database table to determine the multi-dimensional field features corresponding to each data table, wherein the multi-dimensional field features include field name, field data type and field sample value; Parse the field description text in the specified fields of each of the database tables; The database logs are monitored, and data lineage is extracted by parsing the ETL task execution records in the database logs to generate data lineage identifiers. The data lineage includes source data tables, target data tables, and field mapping relationships. The target metadata set is determined based on the multi-dimensional field features, the field description text, and the data lineage identifier.

4. A linkage visualization method based on multi-source heterogeneous data according to claim 1, characterized in that, Semantic relationship matching is performed on multiple fields using field names and field description text in the target metadata set to determine semantic field relationship pairs, specifically including: Determine the field type for each field. When the field type is a text field, calculate the text similarity of the field name to determine the text similarity between multiple fields. When the field type is a numeric field, a probability distribution model is constructed for the sample value set of the numeric field, and the similarity score between multiple numeric fields is determined based on the KL divergence value between the probability distribution models. Based on the text similarity and / or the similarity score, determine the semantic field relationship pairs.

5. The linked visualization method based on multi-source heterogeneous data according to claim 1, characterized in that, Based on the data lineage identifier in the target metadata set, the static field relationship matching pairs are corrected to determine the field relationship matching information, specifically including: Parse the dynamic relationship records in the data lineage identifier to extract the source field name, target field name, and conversion type; The static field relationship matching pair is compared with the source field name, the target field name, and the conversion type to perform conflict detection. When a conflict is detected, the static field relationship matching pair is corrected to determine the field relationship matching information.

6. A linkage visualization method based on multi-source heterogeneous data according to claim 1, characterized in that, Based on the field relationship matching information, a dynamic data structure graph corresponding to the target metadata set is generated, specifically including: A master node is created for each data table, and child nodes are created for each field. The child nodes are attached to the master node of the corresponding data table. Based on the field relationship pairs in the field relationship matching information, generate connection edges between main nodes and annotate the associated field names on the connection edges; Attribute information is attached to each master node to generate a dynamic data structure graph corresponding to the target metadata set. The attribute information includes the data table name, number of fields, data volume, and update time.

7. A linked visualization method based on multi-source heterogeneous data according to claim 1, characterized in that, Based on the dynamic data structure graph and the node triggering operation, the node association information corresponding to the triggering node is determined to generate penetrating analysis results in the visualization interface, specifically including: Determine the operation type of the node-triggered operation, wherein the operation type includes any one or more of node click, region selection, and field double-click; When the operation type is node click, the upstream and downstream related paths in the dynamic data structure graph are recursively queried from the triggering node to generate a related path chain, which is then highlighted in the visualization interface and the detailed data of each node in the related path chain is displayed in a separate window. When the operation type is the area selection, the spatial coordinates or value range corresponding to the trigger node are parsed to generate filtering conditions, which are then extended to the association table. In the preset parallel sub-windows, the filtered data of the association table are rendered respectively to display the condition filtering results. When the operation type is double-clicking the field, the expression of the field corresponding to the trigger node is parsed and a logic flowchart is generated to display the logic flowchart and obtain the corresponding original data value table.

8. A linkage visualization device based on multi-source heterogeneous data, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

9. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are configured to perform the method as described in any one of claims 1-7.

Images