Data visualization method and system based on multi-source data fusion

Abstract

This invention discloses a data visualization method and system based on multi-source data fusion, relating to the field of data processing technology. The method includes: accessing multi-source heterogeneous data sources, extracting structural metadata and calculating field hash values ​​to generate a unified data fingerprint containing cross-source mapping relationships; receiving front-end interaction requests, using the unified data fingerprint as an index to generate a query execution topology graph containing operator pushdown rules and streaming connection order; streaming physical data according to the topology graph cursor and hashing and merging, outputting a standardized aggregated data stream; the front-end rendering engine dynamically downsampling the data stream based on the viewport pixel size, generating and visualizing a set of rendering feature points; when the underlying data changes, comparing the fingerprint to locate the affected topology graph, generating an incremental stream to accurately update the feature point set, triggering local redrawing. This invention improves cross-source query efficiency, avoids memory overflow risks, eliminates front-end rendering lag, and achieves millisecond-level accurate local redrawing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data visualization method and system based on multi-source data fusion. Background Technology

[0002] With the rapid development of big data technology, enterprises have accumulated a large amount of heterogeneous data from multiple sources scattered across different storage systems (such as relational databases, NoSQL databases, Excel files, and Application Programming Interfaces (APIs)). To extract business insights from this dispersed data, data visualization dashboards are widely used. However, existing multi-source data visualization technologies face several bottlenecks when dealing with massive amounts of data: 1) Difficulty in integrating multi-source data and lack of a unified routing mechanism: Traditional methods often rely on manually writing complex integration scripts or using heavyweight Extraction Transformation Loading (ETL) tools to extract all data when connecting multiple data sources. When the underlying data source structure changes, the system cannot automatically detect it, leading to query failures or data corruption.

[0003] 2) Memory Overflow and Network I / O Bottleneck: When performing cross-source join operations, traditional methods often use nested loops or fetch all data into memory before merging. When the data volume reaches millions, it can easily lead to system memory overflow (OOM); at the same time, transmitting all data over the network causes severe I / O latency.

[0004] 3) Front-end rendering stuttering and oversampling issues: When the back-end returns hundreds of thousands or even more data points to the front-end, due to the limited physical screen pixels of the front-end chart component (such as a width of only 1920px), data points far exceeding the number of pixels will overlap and cover each other, resulting in an "oversampling" phenomenon, which causes the browser rendering engine to overload and screen stuttering.

[0005] 4) Lack of incremental update mechanism or excessive cost: When the underlying data changes frequently, traditional visualization systems usually need to re-initiate a full query and redraw the entire chart, which cannot accurately locate the affected data subset for local incremental updates, resulting in a huge waste of system resources.

[0006] Therefore, there is an urgent need for a full-link data visualization method that can achieve operator pushdown and streaming hash connection, front-end dynamic downsampling, and local dirty rectangle redrawing under a unified data fingerprint index. Summary of the Invention

[0007] This invention provides a data visualization method and system based on multi-source data fusion. This invention solves the problems of existing technologies, such as difficulties in multi-source data fusion and the lack of a unified routing mechanism, memory overflow and network I / O bottlenecks, front-end rendering stuttering and oversampling issues, and the lack or excessive cost of incremental update mechanisms.

[0008] In a first aspect, embodiments of the present invention provide a data visualization method based on multi-source data fusion, the method comprising: By accessing multi-source heterogeneous data sources, extracting table structure metadata and calculating field hash values, a unified data fingerprint containing cross-source mapping relationships is generated. Receive interaction requests from the front-end dashboard, use the unified data fingerprint as an index, parse and generate a query execution topology graph that includes operator pushdown rules and streaming connection order; Based on the query execution topology, the underlying physical data is retrieved in a cursor-stream manner and hashed and merged in memory to output a standardized aggregated data stream. The front-end rendering engine receives the standardized aggregated data stream, dynamically downsamples the data stream based on the viewport pixel size of the current chart component, generates a set of rendering feature points, and visualizes them. When the underlying data source changes, the affected query execution topology is located by comparing the recalculated unified data fingerprint, the incremental standardized aggregated data stream is regenerated, and the corresponding rendering feature point set is accurately updated using the incremental stream, triggering local redrawing.

[0009] The technical solution provided in this application has at least the following beneficial effects: 1) Unified data fingerprint and status awareness: By extracting table structure metadata and data status version information, a unified data fingerprint is generated. This not only embeds physical query routing, but also accurately perceives changes in underlying data rows by comparing version information, providing a foundation for incremental updates.

[0010] 2) Memory safety and performance optimization: The streaming hash connection cost estimation method is adopted, and the side with lower memory overhead is used as the memory resident of the construction side. The other side is matched by cursor streaming probing, which constrains the peak memory usage to a specified level and avoids the risk of memory overflow from the source. At the same time, the selection and projection operators are pushed down to the native dialect, which greatly reduces network I / O.

[0011] 3) Pixel-level dynamic downsampling: Based on the physical pixel size, the pixels are directly divided into bins according to the logical X coordinate mapping, and 1-5 visual representative points such as the first, last, middle, largest, and smallest are extracted. While preserving visual features, the amount of data transmitted and rendered is reduced to a minimum, completely solving the oversampling stuttering problem.

[0012] 4) Distortion-free local incremental redraw: When the data on the construction side changes, the data on the detection side is rematched by pulling the time window again, avoiding miscalculations caused by the loss of historical data in the streaming state; the dirty rectangular area is accurately constructed based on the incremental pixel binning index, and only the local area is cleared and redrawn to achieve millisecond-level real-time data linkage.

[0013] In one optional implementation, multiple heterogeneous data sources are accessed, table structure metadata is extracted, and field hash values ​​are calculated to generate a unified data fingerprint containing cross-source mapping relationships, including: Establish a lightweight connection with each underlying data source in the multi-source heterogeneous data source through the protocol adapter, execute table structure query statements, and extract the data source physical connection information, structure metadata and data status version information of each underlying data source. The structural metadata of each underlying data source is normalized in type, cleaned in format, and concatenated to obtain the corresponding normalized field feature string; The multinomial rolling hash algorithm is used to calculate the corresponding field hash value by calculating the normalized field feature string of each underlying data source. Users can drag and drop two normalized field feature strings from different underlying data sources to establish a relationship based on the preset visual configuration interface, generate cross-source mapping pairs, and calculate the corresponding joint hash based on the corresponding field hash values; Iterate through all cross-source mapping pairs established by the user to generate a set including several cross-source mapping pairs, joint hashes and their corresponding physical routing information, i.e., the cross-source mapping pair set; The physical connection information, field hash values, and cross-source mapping pairs of all underlying data sources are serialized and packaged according to a preset structured format to generate a unified data fingerprint containing cross-source mapping relationships for multi-source heterogeneous data sources.

[0014] In one optional implementation, the structure metadata includes the physical table name, physical field name, and primitive data type; The data status version information includes the data change log location, the maximum update timestamp, or the data row count checksum.

[0015] In one optional implementation, the system receives interaction requests from the front-end dashboard, uses the unified data fingerprint as an index, parses and generates a query execution topology graph containing operator pushdown rules and streaming connection order, including: The system receives interaction requests from the front-end dashboard and uses a classic logical expression parsing method to transform the interaction request into a logical operator tree composed of logical operator nodes. The leaf nodes of the logical operator tree represent projection operations on a single underlying data source, and its non-leaf nodes represent relational algebra operations, including selection, projection, and join. Using the unified data fingerprint as an index, the logical operator tree is traversed. For the selection operator or projection operator in the logical operator tree, the physical data source and dialect type to which the field belongs are queried in the unified data fingerprint. The selection operator or projection operator is transformed into the native dialect fragment of the underlying physical data source and then stripped from the logical operator tree. Traverse all selection and projection operators to obtain the pushdown rule set consisting of all native dialect fragments pushed down to the leaf nodes and the stripped logic tree after removing all selection and projection operators; The basic statistical weights of each physical table pre-stored in the unified data fingerprint are queried. For any two nodes to be connected in the stripped logical tree, the execution cost is calculated using the streaming hash connection cost estimation method. Based on the principle of cost minimization, the probe side and the streaming connection construction side are divided. Using dynamic programming, we search for the connection path with the minimum total execution cost among all possible binary connection trees in the stripped logic tree, and generate an ordered binary connection queue, i.e., a connection order sequence. The pushdown rule set and the join order sequence are structurally assembled to generate the final directed acyclic graph, i.e., the query execution topology graph.

[0016] In one alternative implementation, the interaction request includes the selected dimension, metric, and filtering conditions for the user interaction.

[0017] In one alternative implementation, based on the query execution topology graph, underlying physical data is retrieved via a cursor stream and hashed and merged in memory to output a standardized aggregated data stream, including: Starting from the leaf node of the query execution topology graph, a cursor query request is initiated to the corresponding underlying data source based on the physical routing information and pushdown rule set encapsulated in the leaf node. Based on the cursor query request, and using a batch cursor fetching strategy, a series of continuous raw data shards are fetched in batches from the underlying data source. If the query execution topology reaches a node on the streaming connection construction side determined by the connection order sequence, a hash index is built in memory, and each row in the corresponding original data shard is inserted into the corresponding memory hash bucket to generate a set of memory hash buckets. If the query execution reaches a node on the probe side of the topology graph, a series of new raw data shards are batch-fetched from the underlying data source of the probe side based on the batch cursor fetch strategy. For each row in the new original data shard, extract its connection key value and calculate its hash address. Locate the corresponding memory hash bucket in the memory hash bucket set and scan for matching. If the match is successful, concatenate the data row with the matching row in the memory hash bucket to generate the connected intermediate data stream. The joined intermediate data streams are aggregated to the root node of the query execution topology graph. An accumulator dictionary based on the grouping key is maintained in memory. For each data tuple in the joined intermediate data stream, its grouping key and metric value are extracted and incremental aggregation calculation is performed. Finally, a standardized aggregated data stream is output.

[0018] In one alternative implementation, the front-end rendering engine receives the standardized aggregated data stream, dynamically downsamples the data stream based on the viewport pixel size of the current chart component, generates a set of rendering feature points, and visualizes them, including: The front-end rendering engine listens to the physical pixel value of the viewport width of the current chart component and receives the standardized aggregated data stream; If the total number of data points in the standardized aggregated data stream is greater than the physical pixel value, it is determined that there is severe oversampling, triggering a downsampling mechanism to extract the logical boundary extreme values ​​of the standardized aggregated data stream and generate downsampling execution parameters, which include logical boundary extreme values ​​and physical pixel values. Based on the logical boundary extreme values ​​in the downsampling execution parameters, the X-axis value range of the standardized aggregated data stream is divided into several independent pixel binning intervals according to the logical X coordinate of the data points, generating a set of pixel binning intervals with a size not exceeding the physical pixel values ​​in the downsampling execution parameters, and calculating the corresponding pixel binning index. Traverse each non-empty pixel binning interval in the pixel binning interval set and calculate the set of visual representative points for that pixel binning interval; The visual representative point sets of all non-empty pixel binning intervals are merged in X-axis order, and duplicates are removed to generate a downsampled subset. Based on the logical boundary extreme values ​​of the standardized aggregated data stream, the logical coordinates of each data point in the downsampled subset are converted into the physical screen coordinates of the front-end canvas, and then encapsulated into a structured object by combining the type identifier of the corresponding point and the pixel bin index number to generate the final set of rendering feature points. The front-end rendering engine retrieves a set of rendering feature points and generates a visual region.

[0019] In one optional implementation, when the underlying data source changes, the affected query execution topology is located by comparing the recalculated unified data fingerprint, an incrementally standardized aggregated data stream is regenerated, and the corresponding set of rendering feature points is precisely updated using the incremental stream, triggering a local redraw, including: When an add, modify, or delete operation occurs in the underlying data source, the data change log is intercepted, the change field corresponding to the physical record that has been changed in the data change log and the data status version information of the table to which it belongs are extracted, and the change field hash value of the change field is recalculated using a multinomial rolling hash algorithm. The hash value of the changed field and the latest data status version information are compared with the index table of the unified data fingerprint. If the hash value of the changed field matches the hash value of the field in the index table and the latest data status version information has changed, it is determined that the hash value of the field participates in the front-end dashboard query. The matching hash value of the field and the routing information of the physical table to which it belongs are collected to generate the affected fingerprint set. Based on the affected fingerprint set, reverse tracing and isolation are performed on the query execution topology graph to generate an affected topology subgraph; Incremental cursor query requests are initiated for leaf nodes within the affected topology subgraph to retrieve the corresponding incremental raw data shards, and an incremental probing mechanism is used to generate an incremental standardized aggregated data stream. For each incremental data point in the incremental standardized aggregated data stream, extract its incremental logical X coordinate; Based on the incremental logical X coordinate and downsampling execution parameters, calculate the incremental pixel bin index corresponding to the incremental data point in the pixel bin interval set; In the corresponding set of rendering feature points, hash lookup is used to quickly locate all target feature points whose pixel binning index is equal to the incremental pixel binning index. The incremental data points are cached and merged with the corresponding target feature points in the pixel binning interval to generate a locally updated feature point set. The logical coordinates of each incremental data point in the local update feature point set are converted into physical screen coordinates of the front canvas to generate updated coordinate points. Based on the incremental pixel binning index, a corresponding dirty rectangle region is constructed on the visualization area. The front-end rendering engine clears the dirty rectangle region and reads the updated coordinate points from the local update feature point set, and redraws the lines only within this local region. After the drawing is completed, the system will synchronously replace the new feature points in the locally updated feature point set with the rendered feature point set, thus completing this incremental linkage update.

[0020] In one alternative implementation, based on the affected fingerprint set, reverse tracing and isolation are performed on the query execution topology graph to generate an affected topology subgraph, including: Traverse the query execution topology graph, starting from the leaf node of the query execution topology graph and traversing backwards. For the pushdown rule set encapsulated by the leaf node, if the field hash value corresponding to the pushdown field contained in the pushdown rule set exists in the affected fingerprint set, then mark the leaf node as the affected source node. Propagate the labels of the affected source nodes upstream along the edges of the query execution topology graph, label all intermediate connection nodes that take the affected source nodes as input, and obtain the corresponding number of affected intermediate nodes; Continue until the root node is reached, and extract all marked affected source nodes, affected intermediate nodes and their connecting edges to generate an isolated affected topology subgraph in memory.

[0021] Secondly, embodiments of the present invention provide a data visualization system based on multi-source data fusion for implementing a data visualization method, the system comprising: The fingerprint generation unit is used to access multi-source heterogeneous data sources, extract table structure metadata and calculate field hash values ​​to generate a unified data fingerprint containing cross-source mapping relationships. The topology generation unit is used to receive interaction requests from the front-end dashboard, and, using the unified data fingerprint as an index, parse and generate a query execution topology graph that includes operator pushdown rules and streaming connection order. The streaming execution unit is used to pull the underlying physical data in a streaming manner through a cursor according to the query execution topology graph, perform hash merging in memory, and output a standardized aggregated data stream. The visualization unit is used by the front-end rendering engine to receive the standardized aggregated data stream, dynamically downsample the data stream based on the viewport pixel size of the current chart component, generate a set of rendering feature points, and visualize them. The incremental linkage unit is used to locate the affected query execution topology by comparing the recalculated unified data fingerprint when the underlying data source changes, regenerate the incremental standardized aggregated data stream, and use the incremental stream to accurately update the corresponding rendering feature point set, triggering local redrawing.

[0022] A third aspect of this invention provides an electronic device, which includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by at least one processor, such that the at least one processor can perform the method proposed in the first aspect of the present invention.

[0023] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in the first aspect of the present invention. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps of a data visualization method based on multi-source data fusion provided in an embodiment of the present invention. Figure 3This is a schematic diagram of the functional units of a data visualization system based on multi-source data fusion provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0026] The present invention will be further described below with reference to the accompanying drawings.

[0027] Reference Figure 1 , Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0028] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0029] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0030] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an electronic program for a data visualization system based on multi-source data fusion.

[0031] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the electronic device of the present invention can be set in the electronic device. The electronic device calls the electronic program of the data visualization system based on multi-source data fusion stored in the memory 1005 through the processor 1001, and executes the data visualization method based on multi-source data fusion provided in the embodiment of the present invention.

[0032] Reference Figure 2 The present invention provides a data visualization method based on multi-source data fusion, the method comprising: S201: Access multi-source heterogeneous data sources, extract table structure metadata and calculate field hash values ​​to generate a unified data fingerprint containing cross-source mapping relationships; S202: Receive the interaction request from the front-end dashboard, use the unified data fingerprint as an index, parse and generate a query execution topology graph that includes operator pushdown rules and streaming connection order; S203: Based on the query execution topology graph, the underlying physical data is retrieved in a cursor-stream manner and hashed and merged in memory to output a standardized aggregated data stream; S204: The front-end rendering engine receives the standardized aggregated data stream, dynamically downsamples the data stream based on the viewport pixel size of the current chart component, generates a set of rendering feature points, and visualizes them; S205: When the underlying data source changes, the affected query execution topology is located by comparing the recalculated unified data fingerprint, the incremental standardized aggregated data stream is regenerated, and the corresponding rendering feature point set is accurately updated using the incremental stream, triggering local redrawing.

[0033] The technical solution provided in this application has at least the following beneficial effects: 1) Unified data fingerprint and status awareness: By extracting table structure metadata and data status version information, a unified data fingerprint is generated. This not only embeds physical query routing, but also accurately perceives changes in underlying data rows by comparing version information, providing a foundation for incremental updates.

[0034] 2) Memory safety and performance optimization: The streaming hash connection cost estimation method is adopted, and the side with lower memory overhead is used as the memory resident of the construction side. The other side is matched by cursor streaming probing, which constrains the peak memory usage to a specified level and avoids the risk of memory overflow from the source. At the same time, the selection and projection operators are pushed down to the native dialect, which greatly reduces network I / O.

[0035] 3) Pixel-level dynamic downsampling: Based on the physical pixel size, the pixels are directly divided into bins according to the logical X coordinate mapping, and 1-5 visual representative points such as the first, last, middle, largest, and smallest are extracted. While preserving visual features, the amount of data transmitted and rendered is reduced to a minimum, completely solving the oversampling stuttering problem.

[0036] 4) Distortion-free local incremental redraw: When the data on the construction side changes, the data on the detection side is rematched by pulling the time window again, avoiding miscalculations caused by the loss of historical data in the streaming state; the dirty rectangular area is accurately constructed based on the incremental pixel binning index, and only the local area is cleared and redrawn to achieve millisecond-level real-time data linkage.

[0037] In one optional implementation, multiple heterogeneous data sources are accessed, table structure metadata is extracted, and field hash values ​​are calculated to generate a unified data fingerprint containing cross-source mapping relationships, including: S2011: Establish a lightweight connection with each underlying data source in the multi-source heterogeneous data source through the protocol adapter, execute table structure query statements, and extract the data source physical connection information, structure metadata and data status version information of each underlying data source. In this embodiment, the protocol adapter uses different drivers for different data sources (e.g., JDBC Driver for MySQL, HTTP Client for REST API, and POI library for local Excel). S2012: Perform type normalization, format cleaning, and concatenation on the structural metadata of each underlying data source to obtain the corresponding normalized field feature string; Type mapping is implemented as follows: To prevent discrepancies in precision definitions for the same data type across different databases (e.g., DATETIME in MySQL and TIMESTAMP in Oracle), it is uniformly mapped to DATE; all floating-point numbers and integers (e.g., INT, BIGINT, FLOAT, DECIMAL) are uniformly mapped to NUM; all character types (e.g., VARCHAR, TEXT, CLOB) are uniformly mapped to STR; field names are standardized: field names are converted to all lowercase and all underscores and spaces are removed (e.g., Cust_Name is converted to custname); the processed physical table name, normalized field name, and normalized type are concatenated into a single string to obtain the normalized field feature string; the concatenation rule uses a highly distinctive, invisible control character delimiter (preferably, the ASCII code 31 delimiter \x1F), and the final concatenation format is logically equivalent to: db_id\x1Ftable_name\x1Ffield_name\x1Ffield_type; this ensures that even if two different databases have the same user.id field, there will be no confusion. S2013: Using a multinomial rolling hash algorithm, the normalized field feature string of each underlying data source is calculated to obtain the corresponding field hash value. The formula is: In the formula, The hash value of the field is an integer, which serves as a "digital ID card" for the metadata characteristics of this field. Since it is of fixed length, subsequent steps can directly perform integer comparisons, which is much faster than string comparisons. For the normalized field feature string, the first i When a character is executed at the underlying code level, its corresponding ASCII code (or Unicode code point) integer value is automatically retrieved for calculation. For example, the ASCII code of the character 'c' is 99, so in the formula... =99; i For character indexing; The total number of characters in the normalized field feature string determines the highest-order term of the polynomial summation; The default prime constant is P=131 or P=13331 (both are prime numbers) in engineering implementation. The reason for choosing prime numbers is that prime numbers can maximize the hashing property of features in multiplication operations, avoiding different strings from calculating the same hash value (i.e. reducing hash collisions). The modulus of a large prime number is a pre-defined, extremely large prime constant, which is typically chosen in engineering implementations. =10 9 +7 (i.e., 1000000007) or =2 64 (Utilizing the natural overflow of unsigned 64-bit integers in computers), the reason for choosing large prime numbers is: as the string grows longer, the cumulative result of the preceding polynomials will increase exponentially, quickly exceeding the storage limit of the computer's CPU registers (such as 64-bit integers), causing an overflow error. Operation, forcibly limiting the result to 00 to Within the range of 1, large prime numbers can ensure that the value after taking the remainder is still evenly distributed, which greatly reduces the probability of collision; The remainder operator; S2014: Users can drag and drop two normalized field feature strings from different underlying data sources to establish a relationship based on the preset visual configuration interface, generating a cross-source mapping pair, and calculating the corresponding joint hash based on the corresponding field hash value. The formula is as follows: In the formula, For joint hashing; For the underlying data source A The field hash value of the normalized field feature string; For the underlying data source B The field hash value of the normalized field feature string; requirements and The remainder has been processed to ensure its range is [0, ...]. Within [1], by multiplying by a prime constant and adding them together, the resulting joint hash value range is [0, ..., ...]. [1] If further compression of storage space is required, the remainder operation can be performed again on the result; this formula ensures the uniqueness and collision prevention capability of cross-source mapping pairs; S2015: Traverse all cross-source mapping pairs established by the user and generate a set including several cross-source mapping pairs, joint hashes and their corresponding physical routing information, i.e., the cross-source mapping pair set; S2016: Serialize and package the physical connection information, field hash values, and cross-source mapping pairs of all underlying data sources according to a preset structured format to generate a unified data fingerprint of multi-source heterogeneous data sources containing cross-source mapping relationships. The fingerprint contains an embedded routing table of "hash value - physical query path", which not only records the logical mapping, but also contains all the physical coordinates required to perform the query.

[0038] In one optional implementation, the structure metadata includes the physical table name, physical field name, and primitive data type; The data status version information is used for subsequent incremental determination. Its value range and acquisition method are as follows: data change log position (such as MySQL's Binlog offset / GTID, a non-negative integer or UUID string), maximum update timestamp (UNIX timestamp, a 10-digit integer, ranging from 10...). 9 Level), data row count checksum (e.g., a 32-bit unsigned integer calculated using the Cyclic Redundancy Check (CRC) 32 algorithm, with a value range of 00 to 4294967295).

[0039] In one optional implementation, the system receives interaction requests from the front-end dashboard, uses the unified data fingerprint as an index, parses and generates a query execution topology graph containing operator pushdown rules and streaming connection order, including: S2021: Receive the interaction request from the front-end dashboard, and use the classic logical expression parsing method to transform the interaction request into a logical operator tree composed of logical operator nodes. The leaf nodes of the logical operator tree represent projection operations on a single underlying data source, and its non-leaf nodes represent relational algebra operations, including selection, projection, and join. S2022: Using the unified data fingerprint as an index, traverse the logical operator tree. For the selection operator or projection operator in the logical operator tree, query the physical data source and dialect type to which the field belongs in the unified data fingerprint. Transform the selection operator or projection operator into the native dialect fragment of the underlying physical data source and peel it off from the logical operator tree. In this embodiment, when converting the selection operator (Filter) or projection operator (Project) into native dialect fragments of the underlying physical data source, it is necessary to generate corresponding SQL based on the dialect type recorded in the unified data fingerprint. For example, for date filtering, if the underlying data source is MySQL, it is converted into UNIX_TIMESTAMP(date_col)>?; if the underlying data source is Elasticsearch, it is converted into a range query JSON fragment {"range": {"date_col": {"gt": ?}}}; the stripping process directly binds these native dialect fragments to the leaf nodes of the query execution topology graph, so that invalid data is directly filtered during underlying execution; S2023: Traverse all selection operators and projection operators to obtain the pushdown rule set consisting of all native dialect fragments pushed down to the leaf nodes and the stripped logic tree after removing all selection operators and projection operators; S2024: Query the basic statistical weights of each physical table pre-stored in the unified data fingerprint, calculate the execution cost of any two nodes to be connected in the stripped logical tree using the streaming hash connection cost estimation method, and divide the probe side and the streaming connection construction side based on the principle of cost minimization. The formula is: In the formula, For the first node With the node The execution cost required to perform a streaming hash join is a dimensionless value, used only to compare the merits of different join orders using a dynamic programming algorithm. The smaller the cost, the better the join order. For the first node; For the first node; This is a cardinality function for data nodes, used to estimate the number of rows in a data node. Since this invention does not retrieve the full amount of data, the estimated number of rows here is the underlying statistical metadata read from the unified data fingerprint (such as the statistical information view of a relational database, or the row index of an Excel file). To take the sign of the minimum value; This is a memory weighting coefficient, with a recommended value range of [0.1, 100.0]. It is used to quantify the impact of "unit memory usage" on the overall system stability. If deployed in a memory-constrained environment or handling extremely high-concurrency Kanban requests, it can be increased. A value (e.g., set to 10) forces the optimizer to try its best to find a connection sequence with minimal memory usage; This is the CPU and network I / O weighting coefficient, with a recommended value range of [0.1, 100.0]. It quantifies the time overhead of "reading and calculating per unit number of rows". This value reflects the query performance of the underlying physical database and the network bandwidth. If the network latency is high, it can be increased. This value makes the optimizer more inclined to reduce the total amount of data read; For memory overhead, the side with a smaller cardinality of memory overhead is designated as the streaming connection construction side, responsible for building hash bucket sets in memory. The side with a larger cardinality of memory overhead is designated as the probing side, responsible for streaming and scanning for matches in the already built hash bucket sets using cursors. This partitioning rule ensures that peak memory usage is constrained. This scale helps to mitigate the risk of memory overflow from the source. In each recursive layer of the dynamic programming search, the system enumerates all pairwise connection combinations, calculates the cost of each combination according to the streaming hash connection cost estimation formula, and simultaneously labels the roles. The combination with the lowest cost is selected as the optimal solution for that layer and recursively moves upwards until the root node, generating a connection order sequence containing complete role labels. This sequence is encapsulated in the intermediate node of the query execution topology graph, and the subsequent execution engine strictly determines which side's data resides in memory for table creation and which side's streaming probe matching is based on the role identifier. S2025: Using dynamic programming algorithm, search for the connection path with the minimum total execution cost among all possible binary connection trees in the stripped logic tree, and generate an ordered binary connection queue, i.e., a connection order sequence. S2026: The pushdown rule set and the join order sequence are structurally assembled to generate the final directed acyclic graph, i.e., the query execution topology graph; The leaf nodes of the query execution topology encapsulate the routing information and pushdown rule sets of each physical data source and are responsible for initiating the underlying physical query. The root node aggregates all connection results and points to the output interface of the final aggregated data.

[0040] In one alternative implementation, the interaction request includes the selected dimension, metric, and filtering conditions for the user interaction.

[0041] In one alternative implementation, based on the query execution topology graph, underlying physical data is retrieved via a cursor stream and hashed and merged in memory to output a standardized aggregated data stream, including: S2031: Start activation from the leaf node of the query execution topology graph, and initiate a cursor query request to the corresponding underlying data source based on the physical routing information and pushdown rule set encapsulated in the leaf node; S2032: Based on the cursor query request and the batch cursor fetching strategy, a series of continuous raw data shards are fetched in batches from the underlying data source; In this embodiment, the recommended range for the size of the original data fragment (i.e., the batch size) fetched from the underlying layer each time in the batch cursor fetch strategy is [1000, 50000] rows, with 10000 rows being the optimal value; this range balances network I / O handshake latency and memory buffer pressure. S2033: If the execution reaches a node on the streaming connection construction side determined by the connection order sequence in the query execution topology graph, a hash index is built in memory, and each row in the corresponding original data shard is inserted into the corresponding memory hash bucket to generate a memory hash bucket set; S2034: If the query execution topology reaches a node on the probe side, a series of new raw data shards are batch-fetched from the underlying data source on the probe side based on the batch cursor fetch strategy. S2035: For each row in the new original data shard, extract its connection key value and calculate its hash address. Locate the corresponding memory hash bucket in the memory hash bucket set and scan for matching. If the matching is successful, concatenate the data of that row with the matching row in the memory hash bucket to generate the connected intermediate data stream. In this embodiment, when constructing the memory hash bucket set, in order to avoid the problem of excessively long linked lists caused by hash collisions, when the number of elements in a memory hash bucket exceeds the threshold (preferably 8), the hash bucket treeification operation is triggered (that is, the linked list structure is upgraded to a red-black tree structure) to reduce the matching time complexity in the worst case. After the raw data fragments on the probe side are successfully matched in the memory hash bucket, the two rows of data are concatenated into a tuple using the zero-copy memory technique. Then, the probe side fragments are immediately released from memory to ensure that the memory usage is only the size of the hash bucket set on the construction side plus a small amount of batch processing fragment size. S2036: Aggregate the intermediate data streams after connection to the root node of the query execution topology graph, maintain an accumulator dictionary based on grouping keys in memory, extract the grouping key and metric value for each data tuple in the intermediate data stream after connection and perform incremental aggregation calculation, and finally output a standardized aggregated data stream.

[0042] In one alternative implementation, the front-end rendering engine receives the standardized aggregated data stream, dynamically downsamples the data stream based on the viewport pixel size of the current chart component, generates a set of rendering feature points, and visualizes them, including: S2041: The front-end rendering engine listens to the physical pixel value of the viewport width of the current chart component and receives the standardized aggregated data stream; S2042: If the total number of data points in the standardized aggregated data stream is greater than the physical pixel value, it is determined that there is severe oversampling, triggering a downsampling mechanism to extract the logical boundary extreme value of the standardized aggregated data stream and generate downsampling execution parameters, the downsampling execution parameters including logical boundary extreme value and physical pixel value; S2043: Based on the logical boundary extreme values ​​in the downsampling execution parameters, the X-axis value range of the standardized aggregated data stream is divided into several independent pixel binning intervals according to the logical X-coordinate of the data points. A set of pixel binning intervals with a size not exceeding the physical pixel values ​​in the downsampling execution parameters is generated, and the corresponding pixel binning index is calculated using the following formula: In the formula, The pixel bin index to which the data point belongs; This is the floor symbol; Physical pixel value; This represents the logical coordinates on the X-axis. These are the maximum and minimum logical coordinates on the X-axis in the logical boundary extremes, respectively; this formula ensures that the binning index is strictly aligned with the physical screen pixel column and prevents tail overflow; S2044: Traverse each non-empty pixel binning interval in the pixel binning interval set, and calculate the set of visual representative points for that pixel binning interval, using the following formula: In the formula, The first pixel bin interval set Non-empty pixel binning intervals (i.e., the corresponding number on the screen) The set of visual representative points generated after extracting the pixel column contains 1 to 5 key data points, which are used to replace several dense data points in the original bin (when the first, last, middle, largest and smallest extracted points overlap, the deduplication operation is used to ensure that there are no duplicate points in the set). This refers to the first data point in the current pixel binning interval, arranged in the data stream time sequence. This refers to the last data point in the current pixel binning interval, arranged in the data stream time sequence. This refers to the data point located in the middle of the current pixel binning interval; The ordinate of the data point; To find the specific data point with the largest ordinate (i.e., metric value) among all data points contained in the current pixel binning interval; To find the specific data point with the smallest ordinate (i.e., metric value) among all data points contained in the current pixel binning interval; S2045: Merge the visual representative point sets of all non-empty pixel bin intervals in X-axis order, remove duplicates, and generate a downsampled subset; S2046: Based on the logical boundary extreme values ​​of the standardized aggregated data stream, convert the logical coordinates of each data point in the downsampled subset into physical screen coordinates of the front-end canvas, and encapsulate them into a structured object by combining the corresponding point type identifier and the pixel bin index number to generate the final set of rendering feature points. The formula is as follows: In the formula, This represents the physical screen coordinates on the X-axis. This represents the physical screen coordinates on the Y-axis. This represents the logical coordinates on the X-axis. This represents the logical coordinate value on the Y-axis. These represent the maximum and minimum values ​​of the logical coordinates on the X-axis during the logical boundary extrema. These represent the maximum and minimum values ​​of the logical coordinates on the Y-axis within the logical boundary extrema. Physical pixel value; The height is the pixel value; S2047: The front-end rendering engine retrieves the set of rendering feature points and generates a visualization area.

[0043] In one optional implementation, when the underlying data source changes, the affected query execution topology is located by comparing the recalculated unified data fingerprint, an incrementally standardized aggregated data stream is regenerated, and the corresponding set of rendering feature points is precisely updated using the incremental stream, triggering a local redraw, including: S2051: When an add, modify, or delete operation occurs in the underlying data source, intercept the data change log, extract the change field corresponding to the physical record that has been changed in the data change log and the data status version information of the table to which it belongs, and use the multinomial rolling hash algorithm to recalculate the change field hash value of the change field. S2052: Compare the changed field hash value and the latest data status version information with the unified data fingerprint index table. If the changed field hash value matches the field hash value in the index table and the latest data status version information has changed, determine that the field hash value participates in the front-end dashboard query, collect the matched field hash value and its corresponding physical table routing information, and generate the affected fingerprint set. When a Data Manipulation Language (DML) operation (such as INSERT, UPDATE, DELETE) occurs in the underlying data source, the extracted data status version information is preferably the position information [file_name, position] of the database's write-ahead log or Binlog. The changed field hash value and the latest Binlog position are compared with the unified data fingerprint index table. Only when the field hash value matches and the position information is greater than the position of the previous record is it determined that the data status version information has changed and the subsequent construction of the affected topology subgraph is triggered. This dual verification mechanism completely eliminates the performance loss caused by frequent triggering of meaningless full verification due to unchanged table structure. S2053: Based on the affected fingerprint set, perform reverse tracing and isolation on the query execution topology graph to generate an affected topology subgraph; S2054: Initiate an incremental cursor query request for the leaf nodes within the affected topology subgraph, pull the corresponding incremental raw data shards, and use the incremental probing mechanism to generate an incremental standardized aggregated data stream; Specifically, if a leaf node in the affected topology subgraph is classified as a probe side, the incremental raw data shards are directly pulled and scanned for matching in the memory hash bucket set of the original construction side. If it is classified as a construction side, since the historical data of the streaming probe side has been discarded, the construction side node and its downstream probe side nodes are marked, the probe side raw data shards within the affected time window are pulled again, and they are re-matched with the memory hash bucket set updated with incremental raw data shards. The successfully matched data shards are aggregated to the local root node of the affected topology subgraph, incremental aggregation calculation is performed, and only the changed grouping keys and their new metric values ​​are output to generate an incremental normalized aggregated data stream. If the affected node is classified as the construction side, it is necessary to re-fetch the original data shards of the probe side within the affected time window. The "affected time window" here refers to a sliding window calculated backward from the current dashboard query initiation time (preferably from the most recent 1 hour to the most recent 24 hours, depending on the update delay tolerance of the specific business data). Using the filtering conditions of this time window, an incremental cursor query is initiated to the underlying data source of the probe side to fetch the incremental original data shards and re-match them with the updated memory hash bucket set, thus avoiding the huge network overhead caused by fetching all historical data of the probe side. S2055: Extract the incremental logical X coordinate of each incremental data point in the incremental standardized aggregated data stream; S2056: Based on the incremental logical X-coordinate and downsampling execution parameters, calculate the incremental pixel bin index corresponding to the incremental data point in the pixel bin interval set, using the following formula: In the formula, This is the incremental pixel bin index to which the incremental data point belongs; This is the floor symbol; Physical pixel value; The logical coordinates of the incremental data points are located on the X-axis. These are the maximum and minimum logical coordinates on the X-axis in the logical boundary extremes, respectively; this formula ensures that the binning index is strictly aligned with the physical screen pixel column and prevents tail overflow; S2057: In the corresponding set of rendered feature points, use hash lookup to quickly locate all target feature points whose pixel binning index is equal to the incremental pixel binning index, and cache and merge the incremental data points with the corresponding target feature points in the pixel binning interval to generate a locally updated feature point set. S2058: Convert the logical coordinates of each incremental data point in the local update feature point set into the physical screen coordinates of the front canvas, and generate the updated coordinate points. S2059: Based on the incremental pixel binning index, a corresponding dirty rectangle region is constructed on the visualization area. The front-end rendering engine clears the dirty rectangle region and reads the updated coordinate points from the local update feature point set, and redraws the lines only within this local region. S20510: After the drawing is completed, the system will synchronously replace the new feature points in the local update feature point set with the rendered feature point set, thus completing this incremental linkage update.

[0044] In one alternative implementation, based on the affected fingerprint set, reverse tracing and isolation are performed on the query execution topology graph to generate an affected topology subgraph, including: S20531: Traverse the query execution topology graph, starting from the leaf node of the query execution topology graph and traversing backwards. For the pushdown rule set encapsulated by the leaf node, if the field hash value corresponding to the pushdown field contained in the pushdown rule set exists in the affected fingerprint set, then mark the leaf node as the affected source node. S20532: Propagate the labels of the affected source nodes upstream along the edges of the query execution topology graph, label all intermediate connection nodes that take the affected source nodes as input, and obtain the corresponding number of affected intermediate nodes; S20533: Continue until the root node is reached, and extract all marked affected source nodes, affected intermediate nodes and their connecting edges, and generate an isolated affected topology subgraph in memory.

[0045] This invention also provides a data visualization system 300 based on multi-source data fusion, referring to... Figure 3 The system may include the following units: The fingerprint generation unit 301 is used to access multi-source heterogeneous data sources, extract table structure metadata and calculate field hash values ​​to generate a unified data fingerprint containing cross-source mapping relationships. The topology generation unit 302 is used to receive the interaction request from the front-end dashboard, and use the unified data fingerprint as an index to parse and generate a query execution topology graph that includes operator pushdown rules and streaming connection order. The streaming execution unit 303 is used to retrieve the underlying physical data in a streaming manner through a cursor and perform hash merging in memory according to the query execution topology graph, and output a standardized aggregated data stream. The visualization unit 304 is used by the front-end rendering engine to receive the standardized aggregated data stream, dynamically downsample the data stream based on the viewport pixel size of the current chart component, generate a set of rendering feature points, and visualize it. The incremental linkage unit 305 is used to locate the affected query execution topology by comparing the recalculated unified data fingerprint when the underlying data source changes, regenerate the incremental standardized aggregated data stream, and use the incremental stream to accurately update the corresponding set of rendering feature points, triggering local redrawing.

[0046] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the data visualization method based on multi-source data fusion of the present invention.

[0047] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM) or Non-Volatile Memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0048] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0049] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the data visualization method based on multi-source data fusion of the embodiments of the present invention.

[0050] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0051] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0052] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0053] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0054] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0055] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data visualization method based on multi-source data fusion, characterized in that, The method includes: By accessing multi-source heterogeneous data sources, extracting table structure metadata and calculating field hash values, a unified data fingerprint containing cross-source mapping relationships is generated. Receive interaction requests from the front-end dashboard, use the unified data fingerprint as an index, parse and generate a query execution topology graph that includes operator pushdown rules and streaming connection order; Based on the query execution topology, the underlying physical data is retrieved in a cursor-stream manner and hashed and merged in memory to output a standardized aggregated data stream. The front-end rendering engine receives the standardized aggregated data stream, dynamically downsamples the data stream based on the viewport pixel size of the current chart component, generates a set of rendering feature points, and visualizes them. When the underlying data source changes, the affected query execution topology is located by comparing the recalculated unified data fingerprint, the incremental standardized aggregated data stream is regenerated, and the corresponding rendering feature point set is accurately updated using the incremental stream, triggering local redrawing.

2. The data visualization method based on multi-source data fusion according to claim 1, characterized in that, It connects to multi-source heterogeneous data sources, extracts table structure metadata and calculates field hash values ​​to generate a unified data fingerprint containing cross-source mapping relationships, including: Establish a lightweight connection with each underlying data source in the multi-source heterogeneous data source through the protocol adapter, execute table structure query statements, and extract the data source physical connection information, structure metadata and data status version information of each underlying data source. The structural metadata of each underlying data source is normalized in type, cleaned in format, and concatenated to obtain the corresponding normalized field feature string; The multinomial rolling hash algorithm is used to calculate the corresponding field hash value by calculating the normalized field feature string of each underlying data source. Users can drag and drop two normalized field feature strings from different underlying data sources to establish a relationship based on the preset visual configuration interface, generate cross-source mapping pairs, and calculate the corresponding joint hash based on the corresponding field hash values; Iterate through all cross-source mapping pairs established by the user to generate a set including several cross-source mapping pairs, joint hashes and their corresponding physical routing information, i.e., the cross-source mapping pair set; The physical connection information, field hash values, and cross-source mapping pairs of all underlying data sources are serialized and packaged according to a preset structured format to generate a unified data fingerprint containing cross-source mapping relationships for multi-source heterogeneous data sources.

3. The data visualization method based on multi-source data fusion according to claim 2, characterized in that, The structure metadata includes physical table names, physical field names, and raw data types; The data status version information includes the data change log location, the maximum update timestamp, or the data row count checksum.

4. The data visualization method based on multi-source data fusion according to claim 3, characterized in that, Receive interaction requests from the front-end dashboard, use the unified data fingerprint as an index, parse and generate a query execution topology graph containing operator pushdown rules and streaming connection order, including: The system receives interaction requests from the front-end dashboard and uses a classic logical expression parsing method to transform the interaction request into a logical operator tree composed of logical operator nodes. The leaf nodes of the logical operator tree represent projection operations on a single underlying data source, and its non-leaf nodes represent relational algebra operations, including selection, projection, and join. Using the unified data fingerprint as an index, the logical operator tree is traversed. For the selection operator or projection operator in the logical operator tree, the physical data source and dialect type to which the field belongs are queried in the unified data fingerprint. The selection operator or projection operator is transformed into the native dialect fragment of the underlying physical data source and then stripped from the logical operator tree. Traverse all selection and projection operators to obtain the pushdown rule set consisting of all native dialect fragments pushed down to the leaf nodes and the stripped logic tree after removing all selection and projection operators; The basic statistical weights of each physical table pre-stored in the unified data fingerprint are queried. For any two nodes to be connected in the stripped logical tree, the execution cost is calculated using the streaming hash connection cost estimation method. Based on the principle of cost minimization, the probe side and the streaming connection construction side are divided. Using dynamic programming, we search for the connection path with the minimum total execution cost among all possible binary connection trees in the stripped logic tree, and generate an ordered binary connection queue, i.e., a connection order sequence. The pushdown rule set and the join order sequence are structurally assembled to generate the final directed acyclic graph, i.e., the query execution topology graph.

5. The data visualization method based on multi-source data fusion according to claim 4, characterized in that, The interaction request includes the user's selected dimensions, metrics, and filtering conditions.

6. The data visualization method based on multi-source data fusion according to claim 5, characterized in that, Based on the query execution topology, the underlying physical data is retrieved via a cursor stream and hashed and merged in memory to output a standardized aggregated data stream, including: Starting from the leaf node of the query execution topology graph, a cursor query request is initiated to the corresponding underlying data source based on the physical routing information and pushdown rule set encapsulated in the leaf node. Based on the cursor query request, and using a batch cursor fetching strategy, a series of continuous raw data shards are fetched in batches from the underlying data source. If the query execution topology reaches a node on the streaming connection construction side determined by the connection order sequence, a hash index is built in memory, and each row in the corresponding original data shard is inserted into the corresponding memory hash bucket to generate a set of memory hash buckets. If the query execution reaches a node on the probe side of the topology graph, a series of new raw data shards are batch-fetched from the underlying data source of the probe side based on the batch cursor fetch strategy. For each row in the new original data shard, extract its connection key value and calculate its hash address. Locate the corresponding memory hash bucket in the memory hash bucket set and scan for matching. If the match is successful, concatenate the data row with the matching row in the memory hash bucket to generate the connected intermediate data stream. The joined intermediate data streams are aggregated to the root node of the query execution topology graph. An accumulator dictionary based on the grouping key is maintained in memory. For each data tuple in the joined intermediate data stream, its grouping key and metric value are extracted and incremental aggregation calculation is performed. Finally, a standardized aggregated data stream is output.

7. The data visualization method based on multi-source data fusion according to claim 6, characterized in that, The front-end rendering engine receives the standardized aggregated data stream, dynamically downsamples the data stream based on the viewport pixel size of the current chart component, generates a set of rendering feature points, and visualizes them, including: The front-end rendering engine listens to the physical pixel value of the viewport width of the current chart component and receives the standardized aggregated data stream; If the total number of data points in the standardized aggregated data stream is greater than the physical pixel value, it is determined that there is severe oversampling, triggering a downsampling mechanism to extract the logical boundary extreme values ​​of the standardized aggregated data stream and generate downsampling execution parameters, which include logical boundary extreme values ​​and physical pixel values. Based on the logical boundary extreme values ​​in the downsampling execution parameters, the X-axis value range of the standardized aggregated data stream is divided into several independent pixel binning intervals according to the logical X coordinate of the data points, generating a set of pixel binning intervals with a size not exceeding the physical pixel values ​​in the downsampling execution parameters, and calculating the corresponding pixel binning index. Traverse each non-empty pixel binning interval in the pixel binning interval set and calculate the set of visual representative points for that pixel binning interval; The visual representative point sets of all non-empty pixel binning intervals are merged in X-axis order, and duplicates are removed to generate a downsampled subset. Based on the logical boundary extreme values ​​of the standardized aggregated data stream, the logical coordinates of each data point in the downsampled subset are converted into the physical screen coordinates of the front-end canvas, and then encapsulated into a structured object by combining the type identifier of the corresponding point and the pixel bin index number to generate the final set of rendering feature points. The front-end rendering engine retrieves a set of rendering feature points and generates a visual region.

8. The data visualization method based on multi-source data fusion according to claim 7, characterized in that, When the underlying data source changes, the affected query execution topology is located by comparing the recalculated unified data fingerprint, an incrementally standardized aggregated data stream is regenerated, and the corresponding rendering feature point set is precisely updated using the incremental stream, triggering a local redraw, including: When an add, modify, or delete operation occurs in the underlying data source, the data change log is intercepted, the change field corresponding to the physical record that has been changed in the data change log and the data status version information of the table to which it belongs are extracted, and the change field hash value of the change field is recalculated using a multinomial rolling hash algorithm. The hash value of the changed field and the latest data status version information are compared with the index table of the unified data fingerprint. If the hash value of the changed field matches the hash value of the field in the index table and the latest data status version information has changed, it is determined that the hash value of the field participates in the front-end dashboard query. The matching hash value of the field and the routing information of the physical table to which it belongs are collected to generate the affected fingerprint set. Based on the affected fingerprint set, reverse tracing and isolation are performed on the query execution topology graph to generate an affected topology subgraph; Incremental cursor query requests are initiated for leaf nodes within the affected topology subgraph to retrieve the corresponding incremental raw data shards, and an incremental probing mechanism is used to generate an incremental standardized aggregated data stream. For each incremental data point in the incremental standardized aggregated data stream, extract its incremental logical X coordinate; Based on the incremental logical X coordinate and downsampling execution parameters, calculate the incremental pixel bin index corresponding to the incremental data point in the pixel bin interval set; In the corresponding set of rendering feature points, hash lookup is used to quickly locate all target feature points whose pixel binning index is equal to the incremental pixel binning index. The incremental data points are cached and merged with the corresponding target feature points in the pixel binning interval to generate a locally updated feature point set. The logical coordinates of each incremental data point in the local update feature point set are converted into physical screen coordinates of the front canvas to generate updated coordinate points. Based on the incremental pixel binning index, a corresponding dirty rectangle region is constructed on the visualization area. The front-end rendering engine clears the dirty rectangle region and reads the updated coordinate points from the local update feature point set, and redraws the lines only within this local region. After the drawing is completed, the system will synchronously replace the new feature points in the locally updated feature point set with the rendered feature point set, thus completing this incremental linkage update.

9. The data visualization method based on multi-source data fusion according to claim 8, characterized in that, Based on the affected fingerprint set, reverse tracing and isolation are performed on the query execution topology graph to generate an affected topology subgraph, including: Traverse the query execution topology graph, starting from the leaf node of the query execution topology graph and traversing backwards. For the pushdown rule set encapsulated by the leaf node, if the field hash value corresponding to the pushdown field contained in the pushdown rule set exists in the affected fingerprint set, then mark the leaf node as the affected source node. Propagate the labels of the affected source nodes upstream along the edges of the query execution topology graph, label all intermediate connection nodes that take the affected source nodes as input, and obtain the corresponding number of affected intermediate nodes; Continue until the root node is reached, and extract all marked affected source nodes, affected intermediate nodes and their connecting edges to generate an isolated affected topology subgraph in memory.

10. A data visualization system based on multi-source data fusion, used to implement the data visualization method as described in any one of claims 1-9, characterized in that, The system includes: The fingerprint generation unit is used to access multi-source heterogeneous data sources, extract table structure metadata and calculate field hash values ​​to generate a unified data fingerprint containing cross-source mapping relationships. The topology generation unit is used to receive interaction requests from the front-end dashboard, and use the unified data fingerprint as an index to parse and generate a query execution topology graph that includes operator pushdown rules and streaming connection order. The streaming execution unit is used to pull the underlying physical data in a streaming manner through a cursor according to the query execution topology graph, perform hash merging in memory, and output a standardized aggregated data stream. The visualization unit is used by the front-end rendering engine to receive the standardized aggregated data stream, dynamically downsample the data stream based on the viewport pixel size of the current chart component, generate a set of rendering feature points, and visualize them. The incremental linkage unit is used to locate the affected query execution topology by comparing the recalculated unified data fingerprint when the underlying data source changes, regenerate the incremental standardized aggregated data stream, and use the incremental stream to accurately update the corresponding rendering feature point set, triggering local redrawing.

Images