Low-code data analysis platform and method based on multi-source data real-time access

Abstract

The invention relates to the technical field of data analysis, and discloses a low-code data analysis platform and method based on multi-source data real-time access, and the platform comprises a data access module, a modeling module, a rendering engine and an early warning module. The method corresponds to the platform. According to the method and the device, the problems of real-time integration and standardization of the multi-source heterogeneous data are solved by obtaining the standardized data entity, and a high-quality data basis is provided for subsequent analysis; a dynamically constructed knowledge graph and a graph embedding technology are deeply fused into a data analysis process, so that semantic association among data entities is intelligently understood, an optimized query statement is automatically generated, intelligent recommendation and rationality verification are provided for establishing a data association relationship for a user, and the technical threshold of data modeling is reduced; a second-level response closed loop from data access to visual presentation is formed based on intelligent chart recommendation of data features and early warning rules and instant calculation and triggering in a real-time data stream.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, specifically a low-code data analysis platform and method based on real-time access to multi-source data. Background Technology

[0002] In the process of enterprise informatization construction, multiple independent business systems such as PMS, WMS, OMS, and TMS are commonly deployed. These systems have accumulated a large amount of business data over long-term operation, but due to independent construction and inconsistent data standards, serious data silos have formed. Business personnel must rely on technical personnel to write complex SQL queries or develop customized reports to obtain comprehensive data analysis results across systems. This traditional approach has several prominent drawbacks: technical personnel need to spend a significant amount of time understanding business requirements and writing query code, resulting in response cycles of several days; the generated fixed reports cannot flexibly adapt to rapidly changing business analysis needs; traditional ETL processes cause data delays, failing to support real-time decision-making; and the repeated development of similar reports leads to a significant waste of human and computing resources. While some existing data analysis solutions have made improvements in individual aspects, they still have significant shortcomings in achieving overall efficiency in real-time integration of multi-source data, intelligent visualization modeling, and business early warning linkage. Summary of the Invention

[0003] The purpose of this application is to provide a low-code data analysis platform and method based on real-time access to multi-source data, so as to solve the technical problems mentioned in the background.

[0004] To achieve the above objectives, this application discloses the following technical solutions: In a first aspect, this application discloses a low-code data analysis platform based on real-time access to multi-source data, the platform comprising: The data access module is configured to: connect to multiple heterogeneous data sources, perform consistency processing on the data sources and automatically map heterogeneous fields with the same name, obtain standardized data entities and output them; The modeling module, connected to the data access module, is configured to: receive the data entities, establish relationships between the data entities through a visual interface, generate a unified data model, automatically generate data query statements based on a dynamically constructed knowledge graph, and perform syntactic and logical validity checks on the relationships; wherein, the knowledge graph is constructed by parsing the data entities and the relationships. The rendering engine, connected to the modeling module, is configured to: receive the data model, recommend and generate visual charts based on the data characteristics of the data model and the user-configured warning rules, and implement a responsive layout that adapts to cross-terminals. The early warning module, connected to the modeling module and the rendering engine, is configured to: configure threshold rules based on the data model, and when the data access module performs real-time data synchronization, calculate the incoming data in real time and trigger an early warning based on the pre-defined rules. The knowledge graph uses graph embedding technology to map data entities and relationships to a vector space to generate vector representations. The modeling module uses these vector representations to provide semantically similar relationship recommendations for establishing the relationships.

[0005] Optionally, the data access module includes a stream-batch integrated processing engine and a field mapping engine; the stream-batch integrated processing engine is configured as follows: Data is accessed through standardized API interfaces, and real-time data synchronization is achieved using incremental snapshot technology for real-time data streams. Historical batch data is verified for data consistency using a version number-based consistency check mechanism.

[0006] Optionally, the incremental snapshot technique includes: during each real-time data stream synchronization, only capturing and persisting data records that have changed since the previous snapshot point and their change timestamps.

[0007] Optionally, the field mapping engine is configured as follows: The system extracts sample data of fields with the same name, infers the field type based on a predefined rule set, and performs type conversion according to the source system priority or user decision to complete the automatic mapping of heterogeneous fields with the same name; wherein, the predefined rule set includes rules for performing data format regular expression matching, null value rate statistics, and numerical distribution analysis on the sample data.

[0008] Optionally, the modeling module establishes relationships between data entities through connections and automatically generates JOIN statements based on the knowledge graph. The process of automatically generating JOIN statements includes: traversing existing relationship paths between data entities in the knowledge graph to avoid generating invalid relationships that would produce a Cartesian product, and prioritizing paths with high relationship confidence.

[0009] Optionally, the modeling module performs logical rationality verification on the association relationship, including: The knowledge graph is used to verify whether there are predefined semantic types that can be associated between connected data entities, or the graph embedding technology is used to calculate the vector similarity between data entities to determine the rationality of the association.

[0010] Optionally, the graph embedding technology uses the Node2Vec or TransE algorithm as the graph embedding model, and trains the data by using successful user association modeling operations accumulated on the platform as positive samples to learn semantic associations with business significance between data entities.

[0011] Optionally, the recommendation logic for the visualization charts of the rendering engine includes: When the data feature of the data model is a time series and threshold warning rules are configured, a line chart containing threshold reference lines is preferred; when the data feature is geographically distributed and regional anomaly warnings are configured, a heat map is preferred.

[0012] Optionally, the dynamic construction process of the knowledge graph includes: By monitoring data flow change events in the data access module, and according to predefined business entity types and relationship templates, the key fields in the data entities are automatically extracted as entity attributes, and semantic relationship edges between data entities are inferred and created in real time based on foreign key relationships or co-occurrence analysis.

[0013] Secondly, this application discloses a low-code data analysis method based on real-time access to multi-source data, applied to the low-code data analysis platform based on real-time access to multi-source data as described above. The method includes: Data access steps: Connect to multiple heterogeneous data sources, perform consistency processing on the data sources and automatic mapping of heterogeneous fields with the same name to obtain standardized data entities; Modeling steps: Receive the data entities, establish the relationships between the data entities through a visual interface, generate a unified data model, automatically generate data query statements based on a dynamically constructed knowledge graph, and perform syntactic and logical validity checks on the relationships; wherein, the knowledge graph is constructed by parsing the data entities and the relationships. Rendering steps: Receive the data model, recommend and generate visual charts based on the data characteristics of the data model and the warning rules configured by the user, and implement a responsive layout that adapts to cross-terminals; Early warning step: Based on the data model, configure threshold rules, and during the real-time data synchronization process of the data access step, calculate the incoming data in real time and trigger an early warning based on the threshold rules; The method further includes a knowledge graph processing step: mapping the data entities and the relationships in the knowledge graph to a vector space using graph embedding technology to generate vector representations, and using the vector representations in the modeling step to provide semantically similar relationship recommendations for establishing the relationships.

[0014] Beneficial Effects: This application's low-code data analysis platform and method based on real-time access to multi-source data overcomes the challenges of real-time integration and standardization of heterogeneous multi-source data by acquiring standardized data entities, providing a high-quality data foundation for subsequent analysis. It deeply integrates dynamically constructed knowledge graphs and graph embedding technology into the data analysis process, enabling intelligent understanding of semantic relationships between data entities, automatic generation of optimized query statements, and intelligent recommendations and rationality checks for users to establish data relationships, thus lowering the technical threshold for data modeling. Intelligent chart recommendations based on data features and early warning rules, along with instant computation and triggering in real-time data streams, form a second-level response closed loop from data access to visualization. Furthermore, through the organic synergy of these technologies, the complex data analysis tasks are effectively transferred from technical professionals to business users, improving the efficiency of data analysis and decision support while ensuring data accuracy. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a structural block diagram of a low-code data analysis platform based on real-time access to multi-source data, provided in an embodiment of this application. Figure 2 A flowchart illustrating a low-code data analysis method based on real-time access to multi-source data, provided in an embodiment of this application. Detailed Implementation

[0017] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0018] In the process of enterprise informatization, to meet the needs of different business scenarios such as project management, warehouse control, order processing, and transportation scheduling, most enterprises deploy multiple independently operating business systems, such as PMS (Project Management System), WMS (Warehouse Management System), OMS (Order Management System), and TMS (Transportation Management System). Over the long term, these systems have accumulated massive amounts of data closely related to business operations. However, due to the inconsistent construction cycles, technical architectures, and data standards of each system, the data exhibits significant heterogeneity. Not only do the data formats and storage methods differ, but even fields with the same business meaning (such as "order number") may exist as strings, numeric values, or other different types in different systems, ultimately forming serious data silos.

[0019] For business personnel, in practice, they often need to obtain comprehensive data from across systems to support analysis and decision-making. However, limited by data silos and their own technical capabilities, they cannot directly manipulate multi-source heterogeneous data and must rely on professional technicians to write complex SQL queries or develop customized reports to meet their needs. This traditional model has several prominent pain points: First, it has low response efficiency. Technicians need to spend a lot of time understanding business requirements before writing query code or developing reports, resulting in a response cycle of several days, which is seriously lagging behind the immediacy requirements of business decisions. Second, it has insufficient adaptability. The functions and dimensions of customized reports are fixed and cannot flexibly respond to dynamic changes in business scenarios, making it difficult to meet the personalized and immediacy analysis needs of business personnel. Third, it lacks real-time performance. Traditional ETL processes mostly adopt batch processing mode, and there is a delay of hours or even longer from data collection and cleaning to availability, which cannot provide effective support for decision-making scenarios that require real-time data support. Fourth, it wastes resources severely. Similar analysis needs under different business scenarios often require the repeated development of reports, resulting in unnecessary consumption of human and computing resources.

[0020] Although some existing data analytics solutions attempt to optimize single aspects such as data access and report presentation, they have failed to fundamentally overcome core issues such as the real-time integration and standardization of multi-source heterogeneous data, the high technical threshold of data modeling, and the efficient collaboration of the entire process of data processing and visualization. In terms of achieving efficient end-to-end operation of "data access-modeling analysis-visualization presentation-decision support", the overall efficiency is still significantly insufficient, making it difficult to meet enterprises' core demands for "low threshold, high timeliness, and accurate data" in data analytics.

[0021] For this reason, this embodiment provides a low-code data analysis platform and method based on real-time access to multi-source data.

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application. Secondly, in this document, the term "comprising" is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0023] Firstly, this embodiment provides a low-code data analysis platform based on real-time access to multi-source data, such as... Figure 1 As shown, it includes a data access module, a modeling module, a rendering engine, and an early warning module, which are described in detail below.

[0024] The data access module is configured to: connect to multiple heterogeneous data sources, perform consistency processing on the data sources and automatically map heterogeneous fields with the same name, obtain standardized data entities and output them.

[0025] In practical implementation, the heterogeneous data sources of the data access module include internal enterprise business systems such as PMS, WMS, OMS, and TMS, as well as relational databases like MySQL and TIDB, static CSV / Excel files, and Kafka streaming data queues. Consistency processing includes data format standardization (dates are standardized to YYYY-MM-DDHH:MM:SS format, and numerical precision is retained to two decimal places) and data cleaning (removing special characters through regular expression matching and eliminating duplicate records based on field uniqueness rules). Automatic mapping of heterogeneous fields with the same name is achieved through field semantic recognition and type adaptation algorithms.

[0026] The modeling module, connected to the data access module, is configured to: receive data entities, establish relationships between data entities through a visual interface, generate a unified data model, automatically generate data query statements based on a dynamically constructed knowledge graph, and perform syntactic and logical validity checks on the relationships; wherein, the knowledge graph is constructed by parsing data entities and relationships.

[0027] In practical implementation, the modeling module's visual interface adopts a left-right split layout. The left side is the data entity list area (displaying standardized data entities connected by various systems, such as orders, inventory, and logistics orders), and the right side is the association editing area. Users can drag and drop entities into the editing area and click to establish relationships. The knowledge graph stores data in the form of "entity-attribute-association" triples. Entities are standardized data entities, attributes are the key fields of the entities (such as order number and product ID), and associations are the business logic relationships between entities (such as an order containing products).

[0028] The rendering engine, connected to the modeling module, is configured to: receive the data model, recommend and generate visual charts based on the data characteristics of the data model and the user-configured warning rules, and implement responsive layouts that adapt to different devices.

[0029] In practice, the rendering engine automatically identifies data characteristics through statistical analysis, including time series (such as daily sales data), geographical distribution (such as regional inventory data), and category proportion (such as the sales proportion of product categories). Responsive layout is achieved through CSS media queries, dynamically adjusting the arrangement and size of charts for different screen sizes.

[0030] The early warning module, connected to the modeling module and the rendering engine, is configured to: configure threshold rules based on the data model, and calculate and trigger early warnings based on the pre-defined rules when the data access module performs real-time data synchronization.

[0031] In practice, the threshold rules of the early warning module support three types: numerical (e.g., inventory quantity ≤ 100), range (e.g., order completion rate 50%-80%), and trend (e.g., sales decline for 3 consecutive days). The early warning methods include platform pop-up prompts and WeChat / DingTalk message pushes. The message content includes the early warning indicator, the current value, the threshold range, and related data links.

[0032] Furthermore, knowledge graphs use graph embedding technology to map data entities and relationships into a vector space, generating vector representations. The modeling module uses these vector representations to provide semantically similar relationship recommendations for establishing relationships.

[0033] In practical implementation, graph embedding technology maps data entities and relationships to a 256-dimensional Euclidean vector space. After the vector representation is generated, the relationship recommendation confidence is... Calculated using the following formula: In the formula, Harmony The weighting coefficients are set to 0.5 and 0.5 respectively, to balance the business semantics and attribute matching. The vector cosine similarity between data entities i and j represents the degree of semantic association, and its value ranges from 0 to 1. The overlap rate of key fields between entities i and j (value 0-1). Key fields are predefined by the business scenario (e.g., order number of order entity, user ID of user entity). The association strength (value 0-1) between entities i and j within their respective business scenarios is determined based on an industry business process knowledge base (e.g., the association strength between an order and a logistics order is 0.98). This formula uses a combination of exponential and logarithmic functions to achieve non-linear weight adjustment, resulting in a more significant increase in confidence for associations with high semantic matching and high business relevance. The final confidence level... The value range is 0-1.

[0034] In this implementation, the data access module integrates and standardizes multi-source heterogeneous data, achieving code-free integration and breaking down data silos to provide a high-quality data foundation for subsequent analysis. The modeling module lowers the modeling threshold through visualization, combining knowledge graphs to achieve intelligent recommendation of relationships and automatic generation of query statements, reducing the technical barrier to data modeling and analysis and enabling business personnel to independently complete data analysis tasks. The rendering engine accurately recommends visual charts based on data characteristics and early warning requirements, and cross-terminal adaptation enhances usability flexibility, ensuring that visual charts accurately match data characteristics and improving data interpretation efficiency. The early warning module monitors data changes in real time, triggering early warnings to form a closed loop, ensuring timely detection of business anomalies and supporting rapid decision-making. The collaboration of these modules achieves full-process automation from data access to decision support, comprehensively improving the convenience and timeliness of data analysis.

[0035] Existing data access solutions are difficult to adapt to both real-time data streams and historical batch data at the same time. The access interface has poor compatibility, and historical data synchronization is prone to duplication or loss, resulting in low data access efficiency and inability to guarantee consistency.

[0036] Therefore, as an optional implementation of this embodiment, the data access module includes a stream-batch integrated processing engine and a field mapping engine; the stream-batch integrated processing engine is configured as follows: Data is accessed through standardized API interfaces, and real-time data synchronization is achieved using incremental snapshot technology for real-time data streams. Historical batch data is verified for data consistency using a version number-based consistency check mechanism.

[0037] In practical implementation, the integrated stream and batch processing engine is a unified component that combines real-time and batch processing. Developed in Java and based on the Spring Cloud framework, it achieves high availability and supports horizontal scaling to handle varying data volumes. Standardized API interfaces cover RESTful API, JDBC, ODBC, and Kafka message queue interfaces. Interface adaptation logic is implemented using a dynamic proxy pattern, automatically selecting the matching interface for data access based on the data source type.

[0038] Secondly, the version number adopts a combination format of "data source code-data type identifier-time stamp-batch sequence number", such as "WMS-REAL-20250610153000-001". The data source code is a unique identifier for each system (such as WMS corresponding to the warehouse management system), the data type identifier distinguishes between real-time data (REAL) and historical data (HIST), the timestamp is accurate to the minute, and the batch sequence number is an auto-incrementing sequence of the day (001-999).

[0039] Furthermore, the consistency verification mechanism's execution process is as follows: First, obtain the version number of the historical data to be synchronized and the latest version number of the data already stored on the platform; Second, compare the data source encoding, data type identifier, and timestamp of the two version numbers. If they are completely consistent, it is determined that the data has not changed, and synchronization is skipped; if there are differences, extract the difference data fragments; Third, perform field integrity verification (ensuring that no required fields are missing) and data format verification (matching standardized format requirements) on the difference data; Fourth, synchronize the difference data and update the platform data version number.

[0040] In addition, the field mapping engine is a core sub-component of the data access module, responsible for the identification and type conversion of heterogeneous fields with the same name. The specific implementation details will be described in detail in the subsequent claims.

[0041] In this implementation, the integrated stream and batch processing engine adapts to different data sources through multiple standardized API interfaces, enabling unified access to real-time data streams and historical batch data, thus improving the compatibility and scalability of data access. Differentiated processing strategies are employed for real-time and historical data: real-time data streams utilize incremental snapshot technology to ensure synchronization timeliness, while historical batch data uses a version number verification mechanism to ensure consistency, avoiding duplicate synchronization and loss of historical data. Standardized API interfaces reduce the development workload for data source adaptation and lower system maintenance costs. Simultaneously, the field mapping engine assists in completing data standardization transformation, jointly achieving efficient and accurate access to multi-source data, providing reliable input for subsequent data processing.

[0042] Current real-time data synchronization methods mostly use full-volume transmission, which involves large data transmission volume and consumes a lot of bandwidth resources, resulting in low synchronization efficiency. This cannot meet the business requirements of second-level data updates and lacks effective recording of data change trajectories.

[0043] Therefore, based on the aforementioned integrated stream and batch processing engine, as a further optional implementation method of this embodiment, the incremental snapshot technology includes: capturing and persisting only the data records that have changed since the last snapshot point and their change timestamps during each real-time data stream synchronization.

[0044] In practice, incremental snapshot technology is based on log parsing principles, acquiring data change records by monitoring the binlog (for relational databases) or message queue offset (for streaming data) of the data source. Snapshot point configuration supports two modes: scheduled mode and triggered mode, allowing users to choose based on the data source's change frequency. The scheduled mode allows snapshot intervals from 1 minute to 60 minutes, with a default configuration of 5 minutes, suitable for scenarios with stable data change frequencies. The triggered mode has a trigger threshold set at 1000 data changes, suitable for scenarios with significant data change fluctuations.

[0045] Secondly, the change timestamp uses the YYYY-MM-DDHH:MM:SS.fff format, accurate to the millisecond level, to record the specific time of data addition, modification, or deletion. During data capture, the operation type identifier is recorded synchronously: "INSERT" for adding data, "UPDATE" for modifying data, and "DELETE" for deleting data. The operation type identifier, along with the data record and change timestamp, is stored in the platform's incremental data storage area.

[0046] In addition, the incremental data storage area adopts a hybrid storage architecture of Redis and MySQL. Redis is used to cache the incremental data of the most recent hour (supporting fast query), and MySQL is used to persistently store historical incremental data (supporting long-term traceability). Data storage is partitioned and managed according to the data source code and snapshot timestamp.

[0047] In this implementation, each time a snapshot is triggered, the incremental snapshot technology only captures the changed data from the previous snapshot point to the current time, significantly reducing the amount of data transmission during real-time data synchronization, reducing bandwidth consumption, and improving synchronization efficiency. By recording data change details through operation type identifiers and millisecond-level timestamps, accurate tracking of data change trajectories is achieved, facilitating subsequent troubleshooting and data backtracking. The hybrid storage architecture ensures fast access and long-term retention of incremental data, and the two snapshot point configuration modes adapt to the change characteristics of different data sources, improving technical applicability and meeting the business requirements for second-level data updates.

[0048] The existing mapping of heterogeneous fields with the same name in multi-source data relies on manual configuration, which is inefficient and prone to errors. Furthermore, field type inference relies on a single feature judgment, which is not accurate enough and cannot meet the automation requirements of data standardization and integration.

[0049] Therefore, based on the aforementioned field mapping engine, as a further optional implementation of this embodiment, the field mapping engine is configured as follows: The system extracts sample data of fields with the same name, infers the field type based on a predefined rule set, and performs type conversion according to the source system priority or user decision to complete the automatic mapping of heterogeneous fields with the same name. The predefined rule set includes rules for performing data format regular expression matching, null value rate statistics, and numerical distribution analysis on the sample data.

[0050] In practice, the core execution process of the field mapping engine consists of three steps: sample extraction, type inference, and type conversion.

[0051] During the sample extraction phase, sample data is extracted at 8% of the total data volume of the field. The sample adopts a stratified sampling method to ensure coverage of data sources with different value ranges and formats. For example, for the "order number" field, samples containing alphanumeric combinations and pure numeric formats are extracted at the same time.

[0052] Secondly, the predefined rule set uses a series of predefined regular expression patterns to match field sample data to infer its potential data type. It mainly includes three core rules: First, data format regular expression matching rules, covering more than 10 common formats such as mobile phone numbers (matching mobile phone numbers starting with the number 1, the second digit being 3-9, and a total length of 11 digits), email addresses (matching strings conforming to standard email format (including the "@" symbol and a valid domain name structure)), and dates (matching standardized date strings conforming to the "year-month-day" format (e.g., YYYY-MM-DD)); Second, null value rate statistics rules, calculating the proportion of null values ​​to the total number of samples, marking fields as low-quality and prompting the user when the null value rate exceeds 50%; Third, numerical distribution analysis rules, determining whether a field is numerical by calculating the coefficient of variation (the ratio of the standard deviation to the mean) of the sample data. The formula for calculating the coefficient of variation is: In the formula, is the coefficient of variation, with a value ≥ 0, dimensionless, used to characterize the degree of dispersion of the data; n is the number of sample data points, which is a positive integer; For the kth sample data, it is a numerical variable; The mean of the sample data is a numerical variable. For a minimum value, such as 10 -6 This is used to avoid the problem of the denominator being 0 when the mean is 0.

[0053] During the type inference phase, the output field type (string, numeric, date, boolean) is determined by combining the results of the three types of rules. The inference confidence is calculated using a weighted voting algorithm, with a weight of 0.4 for regular expression matching, 0.2 for null value rate statistics, and 0.4 for numerical distribution analysis. When the confidence is ≥0.7, the field type is determined directly; when it is below 0.7, a user decision is triggered.

[0054] During the type conversion phase, the source system priority is pre-configured by the user according to business importance (e.g., core business systems have higher priority than auxiliary systems). The field types of high-priority systems are used as the target conversion types. During the conversion process, type-compatible algorithms are used to avoid data loss (e.g., numeric to string conversion retains the original precision, string to numeric conversion filters non-numeric characters).

[0055] In this implementation method, the field mapping engine obtains representative sample data through stratified sampling, performs field type inference based on a multi-dimensional predefined rule set, and improves the accuracy of field type inference; it completes type conversion by combining source system priority or user decision, realizes automated mapping of heterogeneous fields with the same name, reduces manual intervention, and improves data access efficiency; the type compatibility algorithm ensures the integrity of data during the conversion process, ensures the reliability of numerical distribution analysis, and provides solid support for data standardization.

[0056] In the current data modeling process, establishing relationships relies on users manually writing JOIN statements, which can easily lead to invalid relationships such as Cartesian products, resulting in low query efficiency, distorted analysis results, and a lack of intelligent recommendations for relationship paths, increasing the operational difficulty for business personnel.

[0057] Therefore, as an optional implementation method of this embodiment, the modeling module establishes the relationship between data entities by connecting them and automatically generates JOIN statements based on the knowledge graph. The process of automatically generating JOIN statements includes: traversing the existing relationship paths between data entities in the knowledge graph to avoid generating invalid relationships that would produce a Cartesian product, and prioritizing the recommendation of paths with high relationship confidence.

[0058] In practice, the modeling module uses a visual interactive design for connection. Users select two data entities in the entity list on the left, drag them to the editing area, and click the "Establish Connection" button. The system automatically generates a connection and pops up a connection attribute configuration window (to set the connection fields and connection type).

[0059] Secondly, the knowledge graph's association path traversal adopts a bidirectional breadth-first search (BFS) algorithm, with a traversal depth limited to 3 levels (to avoid association failure due to excessively long paths). During the traversal, three core indicators are recorded: the number of entities in the path, the weight of the associated edges, and the historical usage frequency. The invalid association avoidance logic includes: a path is considered invalid if it has more than 4 entities, if the average weight of the associated edges is less than 0.3, and if the historical usage frequency is 0 and there is no business semantic support.

[0060] A feasible approach is to calculate the association confidence using a nonlinear fusion model, with the following formula: In the formula, The association confidence level, with a value of 0-1, is used to evaluate the rationality of the association path; , which is the activation function, to normalize the confidence level and ensure uniformity of units; , , The weighting coefficients should balance path length, association strength, and usage frequency, ensuring a unique equilibrium among these three factors. (The value should be...) , , ; The length of the associated path (the number of associated edges in the path) is a positive integer. This is the average weight of the associated edges in the path, ranging from 0 to 1, and is set based on business semantics; The path's historical usage frequency (the number of times it is selected by the user per unit of time) is a non-negative integer.

[0061] Furthermore, when generating a JOIN statement, the JOIN type is automatically determined based on the field matching relationships in the association path: one-to-one associations use inner join (INNER JOIN), one-to-many associations use left join (LEFT JOIN), and many-to-many associations use cross join (CROSS JOIN), prompting the user for confirmation. JOIN order optimization uses a dynamic programming algorithm, prioritizing table join operations with smaller amounts of associated data to minimize query execution time.

[0062] In this implementation, the modeling module simplifies the relationship establishment process through visual connections, reducing the operational difficulty for business personnel. Based on a bidirectional BFS algorithm, it traverses the knowledge graph's relationship paths and uses an innovative confidence formula to filter highly reasonable paths, avoiding invalid relationships such as Cartesian products, thus improving query execution efficiency and the accuracy of analysis results. Intelligent recommendation of relationship confidence reduces the cost for users to select relationship paths, lowering the modeling threshold. Automatically generated optimized JOIN statements enable non-technical personnel to quickly complete data modeling, improving the efficiency and reliability of data modeling.

[0063] Existing data modeling relationship validation only focuses on syntactic correctness and ignores logical rationality, resulting in distorted analysis results based on incorrect logical relationships. Furthermore, the validation method is too simplistic to cover relationship judgment in complex business scenarios.

[0064] Therefore, based on the aforementioned establishment of association relationships, as a further optional implementation method in this embodiment, the modeling module performs logical rationality verification on the association relationships, including: Knowledge graphs can be used to verify whether there are predefined semantic types that can be associated between connected data entities, or graph embedding techniques can be used to calculate the vector similarity between data entities to determine the rationality of the association.

[0065] In practice, the predefined associative semantic types are built on an industry business process knowledge base, covering 15 core association types, including “order-product”, “purchase order-inventory”, “user-order”, “logistics order-warehouse”, etc. Each semantic type contains explicit association field constraints (such as “order-product” association must be based on the “product ID” field).

[0066] Secondly, the formula for calculating vector similarity using graph embedding technology is: In the formula, This represents the similarity of the mixed vectors, with a value between 0 and 1, used to quantify the degree of semantic association between data entities; The vector representations of data entities i and j are Euclidean vectors of the same dimension. It is the dot product of vectors; They are vectors The L2 norm is used for normalization to ensure dimensional consistency. For vectors The Manhattan distance, with a value ≥ 0.

[0067] In addition, the verification process is as follows: First, perform predefined semantic type verification. If a matching semantic type exists, the association is directly determined to be reasonable. Second, if no predefined semantic type exists, calculate the similarity of the mixed vectors. If the similarity is ≥0.6, the association is determined to be reasonable. Otherwise, it is determined to be unreasonable and the user is prompted to adjust the association.

[0068] In this implementation, the modeling module employs a dual mechanism of "semantic type verification + hybrid vector similarity verification." First, it quickly matches common and reasonable associations to improve verification efficiency. Then, it uses similarity to determine the reasonableness of complex associations, ensuring the accuracy and comprehensiveness of the verification. The similarity algorithm uses L2 norm normalization to ensure uniformity of the dimensions of various indicators and improve computational accuracy. This dual verification mechanism comprehensively covers the verification of associations in both common and complex business scenarios, avoiding distortion of analysis results caused by erroneous associations. Verification result prompts help users quickly adjust associations, reducing the cost of trial and error in modeling and improving the reliability of data modeling.

[0069] The existing graph embedding model training samples lack business relevance, and the parameter configuration uses general default values, which makes the generated vector representation unable to accurately reflect the business semantic relationship between data entities, thus affecting the effect of association recommendation and verification.

[0070] Therefore, as an optional implementation method in this embodiment, the graph embedding technology uses the Node2Vec or TransE algorithm as the graph embedding model, and uses the successful association modeling operations accumulated by users on the platform as positive samples for training, so as to learn the semantic association relationships with business significance between data entities.

[0071] In practical implementation, the core parameter configuration of the Node2Vec algorithm is as follows: traversal strategy parameters. (Return probability, controls the probability of backtracking to visited nodes) (Entry / exit probability, controlling the probability of exploring unvisited areas), walk length is 12 (number of nodes in each path), window size is 6 (width of the context window during training), and embedding dimension is 256. This parameter configuration was determined through multiple business scenario tests and can balance the capture of local and global correlation information. Using this configuration can improve the business semantic relevance of vector representation. In practical applications, the parameters can be adjusted according to the business scenario, but this application embodiment does not limit this.

[0072] The core parameters of the TransE algorithm are configured as follows: embedding dimension of 256, learning rate of 0.0015 (adopting an adaptive learning rate strategy, with a high learning rate in the early stages of training to accelerate convergence and a low learning rate in the later stages to optimize accuracy), number of iterations of 1200, and marginal parameter of 1.0 (the distance between positive and negative samples). This parameter configuration ensures that the model converges to the optimal solution within a finite number of iterations, improving training efficiency and vector representation quality. In practical applications, other parameter combinations can be selected, but this embodiment does not limit them.

[0073] Secondly, the positive sample selection criteria are: user-created relationships are used to generate analysis reports, and these reports are marked as "valid" by business personnel (i.e., the analysis results meet business expectations). Test-based relationship operations and erroneous operations are excluded during the selection process. The sample library adopts a rolling update mechanism, adding valid samples for the current week and removing outdated samples older than one year (to avoid sample invalidation due to business changes). The sample library size is maintained in the range of 100,000 to 1 million records.

[0074] Furthermore, the model training employs the batch gradient descent algorithm, with the loss function being: In the formula, This is the total loss value, with a value ≥ 0, used to measure the deviation between the model's predicted value and the actual value; This is a set of positive sample triples, containing all valid (head entities) verified by business operations. ,relation Tail entity A triple is a set of triples that represent a real business relationship between data entities. For negative sample triples, the head entity in the positive sample triples is randomly replaced. generate" "and replace the head entity" Furthermore, the generated triples do not have any real business relationship, that is, a set of triples that represent meaningless relationships between data entities; This represents the margin between positive and negative samples, used to differentiate the loss between them. (e.g., value...) ; This represents the distance loss for the triplet, with a value ≥ 0.

[0075] This implementation method selects either the Node2Vec or TransE algorithm as the base model. It trains the model using positive samples validated in business contexts and an innovative loss function. This allows the model to learn semantic relationships between data entities that are meaningful to business contexts, improving the business relevance of vector representations. The loss function enhances the model's ability to distinguish between reasonable and meaningless associations by setting margins between positive and negative samples. Optimized parameter configuration balances the capture of local and global association information, ensuring rapid model convergence and obtaining high-quality vector representations. A continuously updated sample library allows the model to adapt to business changes, continuously ensuring the accuracy of association recommendations and validations, and providing reliable semantic support for data modeling.

[0076] Existing visualization chart recommendations are based solely on the number of data dimensions or numerical types, without considering alert rules and business scenarios. This results in a mismatch between recommended charts and data presentation requirements, and issues such as layout errors and incomplete display of chart elements occur when displayed across different devices, impacting the user experience.

[0077] Therefore, as a feasible alternative implementation method in this embodiment, the recommendation logic for the visualization chart of the rendering engine includes: When the data model's data features are time series and threshold warning rules are configured, a line chart containing threshold reference lines is preferred; when the data features are geographically distributed and regional anomaly warnings are configured, a heat map is preferred.

[0078] In practical implementation, the rendering engine's chart recommendation logic is based on a dual matching of data features and early warning rules. The specific matching relationships are as follows: Time series data + threshold early warning rules → line chart with threshold reference lines (the reference lines are displayed as red dashed lines, indicating the threshold values); Geographical distribution data + regional anomaly early warning → heat map (abnormal areas are displayed in dark tones, with color legends indicating the anomaly level); Categorical data + category proportion requirements → pie chart / donut chart (displaying the proportion and specific values ​​of each category); Multi-dimensional comparison data + difference analysis requirements → bar chart / combination chart (supporting side-by-side comparison of multiple sets of data); Trend comparison data + rate of change analysis requirements → area chart / line chart (highlighting data change trends).

[0079] Secondly, the responsive layout is implemented using CSS media queries, setting three screen size thresholds: large screen (width ≥ 1200px), medium screen (768px ≤ width < 1200px), and small screen (width < 768px). Large screens use a 2-3 column layout, with each column displaying 1-2 charts, chart width 600px-800px, and font size 14px; medium screens use a 1-2 column layout, chart width 400px-600px, and font size 12px; small screens use a single-column vertical scrolling layout, with chart width adapted to the screen width (leaving 20px margins on the left and right), and font size 10px.

[0080] In addition, the chart element adaptation logic includes: automatic line wrapping of chart titles (up to 2 lines), horizontal arrangement of legends (collapsed into a drop-down menu on small screens), and automatic hiding of data labels when space is insufficient (displayed when the mouse hovers over them), ensuring the readability of charts on different devices.

[0081] In this implementation, the rendering engine first identifies the data characteristics of the data model and the configured early warning rules, and matches the corresponding visualization chart type to ensure that the chart can accurately present key information and anomaly warnings, thereby improving the efficiency of data interpretation. Then, it adjusts the layout and chart elements according to the terminal screen size through CSS media queries to avoid layout errors and incomplete element display issues when displaying across terminals. The chart element adaptation logic ensures the readability of charts on different terminals, improving the user experience. Overall, it realizes intelligent recommendation and cross-terminal adaptive display of charts, meeting the data analysis needs of business personnel in different scenarios.

[0082] Existing knowledge graphs are mostly built statically, which cannot respond to changes in data sources in a timely manner (such as the addition of fields or the removal of data sources). Furthermore, the inference of related edges relies solely on foreign key relationships, ignoring potential business connections. This results in insufficient completeness and timeliness of the knowledge graph, failing to provide effective support for data modeling.

[0083] Therefore, as a feasible alternative implementation method in this embodiment, the dynamic construction process of the knowledge graph includes: By listening to data flow change events in the data access module, and according to predefined business entity types and relationship templates, the key fields in the data entities are automatically extracted as entity attributes, and semantic relationship edges between data entities are inferred and created in real time based on foreign key relationships or co-occurrence analysis.

[0084] In practice, the monitoring of data flow change events adopts an event-driven model. The types of events monitored include: data source addition events, data entity field addition / modification / deletion events, data batch update events, and data source offline events. The monitoring frequency is once every 1 second to ensure timely capture of data changes.

[0085] Secondly, the predefined business entity types cover common business scenarios for enterprises, including 12 core entities such as orders, users, products, inventory, purchase orders, and logistics orders. Each entity type is associated with a corresponding key field template (e.g., the key fields for the order entity include order number, order time, order amount, and user ID). The relationship templates are based on industry business process definitions and include 20 core relationships such as "order-attribute-user", "order-includes-product", and "purchase order-replenishment-inventory". The templates clearly define the matching rules for the associated entity types and fields.

[0086] A feasible approach is to use a feature importance algorithm for key field extraction. The formula for calculating the business importance score of a field is as follows: In the formula, A business importance score, ranging from 0 to 1, is used to assess the degree to which a field is core to the business. , , The weighting coefficient is used to highlight the core function of field uniqueness. These three factors are balanced, and their values ​​can be as follows: , , ; To ensure field uniqueness (primary key fields take a value of 1, ordinary fields take a value of 0-1, calculated based on the deduplication rate); This represents the frequency of field occurrence (the percentage of non-nullable fields in a data entity), and its value ranges from 0 to 1. This is the field association rate (the percentage of times this field is associated with other entity fields), with a value between 0 and 1. Fields with a score of 0.6 or higher are identified as key fields and automatically added to the entity attributes of the knowledge graph.

[0087] Secondly, the association edge inference adopts a dual-mechanism fusion strategy: a foreign key relationship inference mechanism (if field F of entity A is the primary key of entity B, then an association edge from A to B is created, and the association edge weight is set to 1.0); and a co-occurrence analysis inference mechanism, which calculates the co-occurrence frequency of two entities in the same business transaction. In the formula, This represents the co-occurrence frequency of entities A and B, ranging from 0 to 1, and is used to measure the degree of business association between the two entities. The number of business transactions where A and B both occur is a non-negative integer. The number of business transactions involving entity A is a positive integer. The number of business transactions involving entity B is a positive integer. When the co-occurrence frequency is not less than 0.5, an association edge between A and B is created, with the weight of the association edge set to the co-occurrence frequency. After the association edge is created, meaningless association edges are filtered out using business semantic validation rules (matching the association relationship template).

[0088] In this implementation, the knowledge graph monitors data stream changes in real time through an event-driven model, ensuring its timeliness and responding promptly to data changes. Key fields are extracted based on predefined templates and an innovation importance scoring algorithm, ensuring the business relevance of entity attributes and providing high-quality semantic support for association recommendation and modeling. A dual mechanism of foreign key relationships and co-occurrence analysis is used to infer association edges, ensuring the accuracy of direct associations while also uncovering potential business associations, thus improving the completeness of the knowledge graph. Business semantic verification after the creation of association edges filters out meaningless association edges, ensuring the reliability of the knowledge graph and providing effective support for data modeling.

[0089] Secondly, this embodiment provides a low-code data analysis method based on real-time access to multi-source data, applied to the low-code data analysis platform based on real-time access to multi-source data as described above, such as... Figure 2 As shown, the method includes: Data access steps: Connect to multiple heterogeneous data sources, perform consistency processing on the data sources and automatic mapping of heterogeneous fields with the same name to obtain standardized data entities; Modeling steps: Receive data entities, establish relationships between data entities through a visual interface, generate a unified data model, and automatically generate data query statements based on a dynamically constructed knowledge graph, and perform syntactic and logical validity checks on the relationships; the knowledge graph is constructed by parsing data entities and their relationships. Rendering steps: Receive the data model, recommend and generate visual charts based on the data characteristics of the data model and the user-configured warning rules, and implement a responsive layout that adapts to cross-terminals; Early warning steps: Based on the data model, configure threshold rules, and during the real-time data synchronization process of the data access step, calculate the incoming data in real time and trigger an early warning based on the threshold rules; The method also includes a knowledge graph processing step: mapping data entities and relationships in the knowledge graph to a vector space using graph embedding technology to generate vector representations, and using the vector representations in the modeling step to provide semantically similar relationship recommendations for establishing relationships.

[0090] It should be noted that the low-code data analysis method based on real-time access to multi-source data in this embodiment corresponds to the aforementioned low-code data analysis platform based on real-time access to multi-source data. Therefore, the parts of the low-code data analysis method based on real-time access to multi-source data in this embodiment that are not described in detail (including but not limited to specific implementation methods and technical effects) can be referred to the relevant descriptions in the aforementioned low-code data analysis platform based on real-time access to multi-source data, and will not be repeated here.

[0091] In the embodiments provided in this application, it should be understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, code, or any suitable combination thereof. For hardware implementation, the processor may be implemented in one or more of the following: application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to implement the functions described herein, or combinations thereof. For software implementation, some or all of the processes of the embodiments may be performed by a computer program instructing the associated hardware. During implementation, the program may be stored in a computer-readable storage medium or transmitted as one or more instructions or code on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of a computer program from one place to another. Storage media may be any available medium accessible to a computer. Computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code having the form of instructions or data structures and accessible to a computer.

[0092] Finally, it should be noted that the above description is only a preferred embodiment of this application and is not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A low-code data analysis platform based on real-time access to multi-source data, characterized in that, The platform includes: The data access module is configured to: connect to multiple heterogeneous data sources, perform consistency processing on the data sources and automatically map heterogeneous fields with the same name, obtain standardized data entities and output them; The modeling module, connected to the data access module, is configured to: receive the data entities, establish relationships between the data entities through a visual interface, generate a unified data model, automatically generate data query statements based on a dynamically constructed knowledge graph, and perform syntactic and logical validity checks on the relationships; wherein, the knowledge graph is constructed by parsing the data entities and the relationships. The rendering engine, connected to the modeling module, is configured to: receive the data model, recommend and generate visual charts based on the data characteristics of the data model and the user-configured warning rules, and implement a responsive layout that adapts to cross-terminals. The early warning module, connected to the modeling module and the rendering engine, is configured to: configure threshold rules based on the data model, and when the data access module performs real-time data synchronization, calculate the incoming data in real time and trigger an early warning based on the pre-defined rules. The knowledge graph uses graph embedding technology to map data entities and relationships to a vector space to generate vector representations. The modeling module uses these vector representations to provide semantically similar relationship recommendations for establishing the relationships.

2. The low-code data analysis platform based on real-time access to multi-source data as described in claim 1, characterized in that, The data access module includes a stream-batch integrated processing engine and a field mapping engine; the stream-batch integrated processing engine is configured as follows: Data is accessed through standardized API interfaces, and real-time data synchronization is achieved using incremental snapshot technology for real-time data streams. Historical batch data is verified for data consistency using a version number-based consistency check mechanism.

3. The low-code data analysis platform based on real-time access to multi-source data according to claim 2, characterized in that, The incremental snapshot technology includes: during each real-time data stream synchronization, only capturing and persisting data records that have changed since the previous snapshot point and their change timestamps.

4. The low-code data analysis platform based on real-time access to multi-source data according to claim 2, characterized in that, The field mapping engine is configured as follows: The system extracts sample data of fields with the same name, infers the field type based on a predefined rule set, and performs type conversion according to the source system priority or user decision to complete the automatic mapping of heterogeneous fields with the same name; wherein, the predefined rule set includes rules for performing data format regular expression matching, null value rate statistics, and numerical distribution analysis on the sample data.

5. The low-code data analysis platform based on real-time access to multi-source data according to claim 1, characterized in that, The modeling module establishes relationships between data entities through connections and automatically generates JOIN statements based on the knowledge graph. The process of automatically generating JOIN statements includes: traversing existing relationship paths between data entities in the knowledge graph to avoid generating invalid relationships that would result in a Cartesian product, and prioritizing paths with high relationship confidence.

6. The low-code data analysis platform based on real-time access to multi-source data according to claim 5, characterized in that, The modeling module performs logical validity checks on the association relationships, including: The knowledge graph is used to verify whether there are predefined semantic types that can be associated between connected data entities, or the graph embedding technology is used to calculate the vector similarity between data entities to determine the rationality of the association.

7. The low-code data analysis platform based on real-time access to multi-source data according to claim 1, characterized in that, The graph embedding technology uses the Node2Vec or TransE algorithm as the graph embedding model, and trains the data by using successful user association modeling operations accumulated on the platform as positive samples to learn the semantic association relationships between data entities that have business significance.

8. The low-code data analysis platform based on real-time access to multi-source data according to claim 1, characterized in that, The recommendation logic for the visualization charts of the rendering engine includes: When the data feature of the data model is a time series and threshold warning rules are configured, a line chart containing threshold reference lines is preferred; when the data feature is geographically distributed and regional anomaly warnings are configured, a heat map is preferred.

9. The low-code data analysis platform based on real-time access to multi-source data according to claim 1, characterized in that, The dynamic construction process of the knowledge graph includes: By monitoring data flow change events in the data access module, and according to predefined business entity types and relationship templates, the key fields in the data entities are automatically extracted as entity attributes, and semantic relationship edges between data entities are inferred and created in real time based on foreign key relationships or co-occurrence analysis.

10. A low-code data analysis method based on real-time access to multi-source data, applied to the low-code data analysis platform based on real-time access to multi-source data as described in any one of claims 1-9, characterized in that, The method includes: Data access steps: Connect to multiple heterogeneous data sources, perform consistency processing on the data sources and automatic mapping of heterogeneous fields with the same name to obtain standardized data entities; Modeling steps: Receive the data entities, establish the relationships between the data entities through a visual interface, generate a unified data model, automatically generate data query statements based on a dynamically constructed knowledge graph, and perform syntactic and logical validity checks on the relationships; wherein, the knowledge graph is constructed by parsing the data entities and the relationships. Rendering steps: Receive the data model, recommend and generate visual charts based on the data characteristics of the data model and the warning rules configured by the user, and implement a responsive layout that adapts to cross-terminals; Early warning step: Based on the data model, configure threshold rules, and during the real-time data synchronization process of the data access step, calculate the incoming data in real time and trigger an early warning based on the threshold rules; The method further includes a knowledge graph processing step: mapping the data entities and the relationships in the knowledge graph to a vector space using graph embedding technology to generate vector representations, and using the vector representations in the modeling step to provide semantically similar relationship recommendations for establishing the relationships.

Images