Multi-source data analysis method, device, equipment and storage medium

Abstract

The application relates to the technical field of data analysis and natural language processing, and discloses a multi-source data analysis method, device, equipment and storage medium, the method comprising the following steps: integrating metadata of a pre-accessed heterogeneous data source, and constructing a global logical data model; identifying semantic elements of a pre-acquired natural language query statement, mapping the semantic elements to corresponding logical entities in the global logical data model, and generating a structured intermediate representation; generating a unified SQL query scheme comprising multiple SQL sub-query statements based on the intermediate representation; distributing the SQL sub-query statements to corresponding data sources in the heterogeneous data source for execution, and acquiring execution results of each data source; uniformly calculating and processing the execution results of each data source, generating a unified result set, and outputting the unified result set in a preset format. The application realizes the simplification of multi-source heterogeneous data analysis through natural language interaction and SQL generation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis and natural language processing, and particularly relates to a multi-source data analysis method and device, equipment and a storage medium. BACKGROUND

[0002] In the field of enterprise data analysis, data assets are usually scattered in multiple heterogeneous data management systems. In the face of such multi-source heterogeneous data environment, if business personnel need to perform cross-system correlation analysis, they must rely on professional data engineers to write complex SQL query scripts involving multiple database connections, or perform data extraction, conversion and centralization through a cumbersome ETL process. This process not only has high technical threshold and long cycle, but also is difficult to quickly respond to flexible business analysis needs.

[0003] On the other hand, although natural language to SQL technology allows users to use natural language for querying to some extent, it simplifies data access of a single data source, but in a complex multi-source scenario, the existing scheme is usually difficult to accurately understand the semantics involving business entities in different data sources, resulting in low query efficiency or even incorrect execution. Therefore, how to effectively reduce the technical complexity of multi-source heterogeneous data query analysis and improve the accuracy and execution efficiency of the query is a technical problem to be solved. SUMMARY

[0004] The main purpose of the present application is to provide a multi-source data analysis method, device, equipment and storage medium, aiming to solve the technical problems of high complexity and high technical threshold of enterprise multi-source heterogeneous data query analysis.

[0005] To achieve the above-mentioned purpose, the present application provides a multi-source data analysis method, which comprises the following steps:

[0006] Integrating the metadata of the pre-accessed heterogeneous data sources to construct a global logical data model;

[0007] Identifying the semantic elements of the pre-acquired natural language query statement, mapping the semantic elements to the corresponding logical entities in the global logical data model, and generating a structured intermediate representation;

[0008] Based on the intermediate representation and the SQL syntax characteristics supported by the heterogeneous data sources, a unified SQL query scheme containing multiple SQL subquery statements is generated;

[0009] The SQL subquery statements in the unified SQL query scheme are distributed to the corresponding data sources for execution, and the execution results of each data source are obtained;

[0010] The execution results of the data sources are uniformly calculated to generate a uniform result set, and the uniform result set is output in a preset format.

[0011] In an embodiment, the step of integrating metadata of the pre-accessed heterogeneous data sources and constructing a global logical data model comprises:

[0012] The meta-information of the heterogeneous data sources is scanned, extracted and registered to obtain a meta-information set including table structure, field type and inter-table relationship;

[0013] Based on the meta-information set, a logical association relationship between different tables in the heterogeneous data sources is established to obtain the global logical data model.

[0014] In an embodiment, the step of identifying semantic elements of a pre-obtained natural language query statement and mapping the semantic elements to corresponding logical entities in the global logical data model to generate a structured intermediate representation comprises:

[0015] The natural language query statement is segmented, entity-recognized and semantic role labeled to analyze the query intent, filtering condition and aggregation operation of the natural language query statement;

[0016] The semantic elements contained in the query intent, filtering condition and aggregation operation are matched and mapped to logical tables and fields in the global logical data model to obtain logical entities;

[0017] When the semantic elements involve multiple logical tables, the required table connection logic is determined based on the logical association relationship defined in the global logical data model;

[0018] Based on the query intent, filtering condition, aggregation operation, logical entity and table connection logic, a structured intermediate representation is generated.

[0019] In an embodiment, the step of generating a uniform SQL query scheme comprising multiple SQL sub-query statements based on the intermediate representation and the SQL syntax features supported by the heterogeneous data sources comprises:

[0020] The intermediate representation is converted into an abstract query plan independent of the types of the heterogeneous data sources;

[0021] According to the SQL syntax and functions supported by each data source in the heterogeneous data sources, the abstract query plan is adapted to generate executable SQL sub-query statements for each corresponding data source;

[0022] Based on the abstract query plan, the execution order and coordination logic of each SQL sub-query statement are determined to form the uniform SQL query scheme.

[0023] In an embodiment, the step of adapting the abstract query plan to generate executable SQL subquery statements for each respective data source according to the SQL syntax and functions supported by each data source in the heterogeneous data sources comprises:

[0024] query rewriting optimization and / or predicate pushdown optimization are performed on the executable SQL subquery statements to form optimized SQL subquery statements.

[0025] In an embodiment, the unified SQL query scheme includes execution order and coordination logic of each SQL subquery statement, and the step of distributing the SQL subquery statements in the unified SQL query scheme to corresponding data sources in the heterogeneous data sources for execution and obtaining execution results of each data source comprises:

[0026] each SQL subquery statement is dispatched to its corresponding data source according to the execution order and coordination logic;

[0027] the received SQL subquery statements are executed in parallel on the data sources and the execution status of each SQL subquery statement is monitored;

[0028] when it is detected that the execution status of the data source is successful, the execution result returned by the data source is obtained.

[0029] In an embodiment, the step of performing unified calculation processing on the execution results of each data source, generating a unified result set, and outputting the unified result set in a preset format comprises:

[0030] the execution results of each data source are connected, aggregated, and sorted to generate the unified result set;

[0031] the unified result set is converted into structured data conforming to the requirements of a preset format, wherein the preset format includes JSON format, CSV format, and API data interaction format;

[0032] the converted unified result set is output through a preset data interface.

[0033] In addition, to achieve the above-mentioned purposes, the present application also provides a multi-source data analysis device, which comprises:

[0034] a metadata management module for integrating metadata of pre-accessed heterogeneous data sources and constructing a global logical data model;

[0035] a semantic mapping module for identifying semantic elements of a pre-obtained natural language query statement, mapping the semantic elements to corresponding logical entities in the global logical data model, and generating a structured intermediate representation.

[0036] a unified generation module, configured to generate a unified SQL query scheme containing a plurality of SQL subquery statements based on the intermediate representation and SQL syntax characteristics supported by the heterogeneous data sources;

[0037] an execution module, configured to distribute the SQL subquery statements in the unified SQL query scheme to corresponding data sources in the heterogeneous data sources for execution, and obtain execution results of the data sources;

[0038] a structured output module, configured to perform unified calculation and processing on the execution results of the data sources, generate a unified result set, and output the unified result set in a preset format.

[0039] In addition, to achieve the above-mentioned purposes, the present application also provides a terminal device, which comprises a memory, a processor, and a multi-source data analysis program stored in the memory and executable on the processor. The multi-source data analysis program, when executed by the processor, implements the steps of the multi-source data analysis method as described above.

[0040] In addition, to achieve the above-mentioned purposes, the present application also provides a computer readable storage medium, which stores a multi-source data analysis program. The multi-source data analysis program, when executed by a processor, implements the steps of the multi-source data analysis method as described above.

[0041] The one or more technical solutions provided by the present application have at least the following technical effects:

[0042] Firstly, by pre-accessing the heterogeneous data sources and integrating the metadata, a global logical data model is constructed, and the scattered and structurally different data sources are abstracted into a consistent logical view, so that the user does not need to deeply understand the specific details of each data source, and the complexity of data access is directly reduced. Secondly, the natural language query statement is received and the semantic elements are identified, and the semantic elements are mapped to the global model to generate a structured intermediate representation, which allows non-technical users to use intuitive natural language for querying, avoiding the need to write complex SQL or learn specific query syntax. Thirdly, based on the intermediate representation and the SQL characteristics of the heterogeneous data sources, a unified SQL query scheme is generated, reducing the workload of manually writing and debugging queries and improving the query efficiency. Finally, the execution results of the data sources are uniformly calculated and processed and formatted, ensuring the consistency and readability of the results, and the user does not need to integrate the multi-source results, simplifying the analysis process. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 Flowchart of the first embodiment of the multi-source data analysis method of the present application;

[0044] Figure 2A flowchart of a second embodiment of the multi-source data analysis method of the present application;

[0045] Figure 3 A flowchart of a third embodiment of the multi-source data analysis method of the present application;

[0046] Figure 4 A flowchart of a fourth embodiment of the multi-source data analysis method of the present application;

[0047] Figure 5 A flowchart of a fifth embodiment of the multi-source data analysis method of the present application;

[0048] Figure 6 A flowchart of a sixth embodiment of the multi-source data analysis method of the present application;

[0049] Figure 7 A module structure diagram of the multi-source data analysis device of the embodiments of the present application;

[0050] Figure 8 A device structure diagram of the hardware running environment involved in the multi-source data analysis method of the embodiments of the present application.

[0051] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are merely intended to explain the present application and not to limit the present application.

[0053] The main technical solution of the present application is as follows: metadata of pre-accessed heterogeneous data sources is integrated to build a global logical data model; semantic elements of a pre-acquired natural language query statement are identified, and the semantic elements are mapped to corresponding logical entities in the global logical data model to generate a structured intermediate representation; based on the intermediate representation and SQL syntax characteristics supported by the heterogeneous data sources, a unified SQL query scheme containing multiple SQL sub-query statements is generated; the SQL sub-query statements in the unified SQL query scheme are distributed to corresponding data sources for execution, and execution results of each data source are acquired; the execution results of each data source are uniformly calculated and processed to generate a unified result set, and the unified result set is output in a preset format.

[0054] This application takes into account that, in the field of enterprise data analytics, data assets are typically stored across multiple heterogeneous data management systems. Faced with this multi-source, heterogeneous data environment, business personnel who need to perform cross-system correlation analysis must rely on professional data engineers to write complex SQL query scripts involving multiple database connections, or undergo cumbersome ETL processes for data extraction, transformation, and centralization. This process is not only technically demanding and time-consuming, but also makes it difficult to quickly respond to flexible and ever-changing business analysis needs. On the other hand, although natural language to SQL technology allows users to query using natural language to some extent, simplifying data access from a single data source, in complex multi-source scenarios, existing solutions often struggle to accurately understand the semantics of business entities involved in different data sources, leading to low query efficiency or even incorrect execution.

[0055] Based on this, the main purpose of this application is to provide multi-source data analysis methods, devices, equipment and storage media, aiming to solve the technical problems of high complexity and high technical threshold in enterprise multi-source heterogeneous data query and analysis.

[0056] Specifically, the following are the detailed steps of the first embodiment of the multi-source data analysis method of this application:

[0057] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multi-source data analysis method of this application. In this embodiment, the multi-source data analysis method includes steps S1001 to S1005:

[0058] Step S1001: Integrate metadata from pre-connected heterogeneous data sources to build a global logical data model.

[0059] Heterogeneous data sources refer to data sources with different physical locations and formats within an enterprise, such as relational databases (MySQL, PostgreSQL), data warehouses (Snowflake), and object storage (OSS). These data sources differ in their storage structure and access methods.

[0060] Metadata refers to data that describes data, including information such as table structure, field types, and relationships between tables in the data source. It is the foundation for data management and understanding.

[0061] The global logical data model is a virtual, unified data view built by integrating metadata from heterogeneous data sources. It does not physically centralize data storage; instead, it associates distributed data sources through metadata, treating multiple heterogeneous data sources as a logically unified database.

[0062] Specifically, the system first pre-connects to distributed, heterogeneous data sources within the enterprise via built-in data connectors, enabling real-time or near real-time connectivity to different types of data sources. Then, these data sources are automatically scanned, their metadata extracted and uniformly registered. Through metadata integration, the system constructs a global logical data model. It's important to note that this global logical data model does not change the physical storage location of the data; instead, it forms a virtual data directory through metadata associations, allowing subsequent steps to operate on the underlying distributed data based on a unified logical view.

[0063] More specifically, the global logical data model shields the physical differences between heterogeneous data sources. Users and systems do not need to worry about the data storage location and format, but only need to operate based on a unified logical view, thereby reducing the complexity of cross-source data queries and reducing the workload of manually handling differences in data sources.

[0064] Step S1002: Identify the semantic elements of the pre-acquired natural language query statement, map the semantic elements to the corresponding logical entities in the global logical data model, and generate a structured intermediate representation.

[0065] Natural language queries refer to data analysis requirements entered by users in everyday language, without needing to follow specific syntax or code formats; semantic elements are the components in natural language queries that carry core meaning, including core intent, filtering conditions, aggregation and sorting requirements, etc.

[0066] Among them, logical entities are abstract concepts in the global logical data model, corresponding to specific tables or fields in the underlying data source; structured intermediate representation (IR) is a structured format that is independent of the specific data model and is used to fully express the semantic logic of natural language query statements, including the mapping relationship between semantic elements and logical entities and query logic (such as filtering, aggregation, and table join).

[0067] Specifically, first, the system receives the user's natural language query statement, which is then processed by the NL2SQL engine. This involves word segmentation to break the statement down into its smallest linguistic units, entity recognition to locate key information, and semantic role labeling to clarify the logical relationships between elements. Subsequently, based on the constructed global logical data model, the identified semantic elements are automatically mapped to logical entities within the model. If a semantic element involves cross-table joins, the join logic is automatically identified based on the inter-table relationships defined in the metadata. Finally, these mappings and logical relationships are integrated into a structured intermediate representation (IR), serving as a semantic blueprint for generating cross-source SQL queries.

[0068] In one specific embodiment, a user of an e-commerce company inputs a natural language query: "Top three brands in mobile phone sales in Region C during the fourth quarter of 2023". First, the query is segmented and entity recognized to extract semantic elements: the core intent is "query sales figures", the filtering conditions include "fourth quarter of 2023" (time), "Region C" (region), and "mobile phone category" (product type), and the aggregation and sorting requirements are "top three" (TopN sorting) and "brands" (grouping objects).

[0069] Next, the global logical data model is queried to map semantic elements to corresponding logical entities: "Sales revenue" is mapped to the `sales_fact.sales_amount` field, "Q4 2023" is mapped to the `time_dim.quarter` field (value "2023Q4"), "Region C" is mapped to the `region_dim.region` field (value "C"), "Mobile phone category" is mapped to the `product_dim.category` field (value "Mobile phone"), and "Brand" is mapped to the `product_dim.brand` field. Since the `sales_fact` table is associated with the `product_dim` table via `product_id` and with the `time_dim` table via `time_id`, the join logic of these tables is automatically identified, and the above mapping relationships and logic are integrated into a structured IR. This IR describes the semantic logic of "filtering sales data for the mobile phone category in Region C in Q4 2023, grouping and summarizing sales revenue by brand, and taking the top three."

[0070] In this process, the accurate identification of semantic elements ensures that the query intent is unbiased, the mapping capability of the global logical data model enables the system to associate multi-source data without user intervention, and the final generated IR provides clear guidance for subsequent cross-source SQL generation, thereby enabling non-technical users to efficiently initiate complex cross-source queries and significantly reducing the operational threshold.

[0071] Step S1003: Based on the intermediate representation and the SQL syntax features supported by the heterogeneous data source, generate a unified SQL query scheme containing multiple SQL subquery statements.

[0072] Among them, SQL syntax features refer to the differences in SQL syntax rules and functions supported by different database systems (such as MySQL, PostgreSQL, Snowflake), such as whether specific functions, data type handling methods, or query structures are supported.

[0073] Among them, SQL subquery statements are auxiliary query statements nested in the main query, used to obtain intermediate results from one or more data sources, and are usually used as filtering conditions, data sources or calculation basis of the main query.

[0074] The unified SQL query solution integrates multiple SQL subquery statements targeting different heterogeneous data sources into a logically coherent and collaboratively executable overall query plan. This solution must simultaneously adapt to the syntax characteristics of each data source and satisfy the global query logic.

[0075] Specifically, the unified computing engine first receives the intermediate representation (IR) generated in step two. This IR clearly defines the semantic logic of the query. The engine first analyzes the data source types involved in the IR and calls the metadata management module to obtain the SQL syntax features supported by each data source. Then, the query task is split according to the logic of the IR. Specifically, the operation that needs to access different data sources is decomposed into multiple SQL subqueries. The syntax of each subquery strictly follows the SQL features of the corresponding data source.

[0076] Meanwhile, subquery logic is optimized through query rewriting technology, pushing filtering conditions down to the data source for execution as much as possible to reduce data transmission volume. Finally, these subqueries are integrated into a unified SQL query scheme, which includes data flow dependencies between subqueries (such as the result of one subquery serving as the input of another) and the global execution order, ensuring that all subqueries execute collaboratively to satisfy the original query intent.

[0077] In one specific embodiment, a manufacturing company needs to query "the top two product types in terms of production pass rate for each factory in the Q3C region in 2023". The data distribution is as follows: MySQL stores production records (production_records table, containing factory_id, product_type, and pass_rate fields), and PostgreSQL stores factory information (factories table, containing factory_id and region fields). MySQL does not support window functions, while PostgreSQL supports ROW_NUMBER(). The intermediate representation (IR) clearly requires joining the two tables, filtering by region='C', aggregating and calculating the pass rate by product_type, and selecting the top two types. After analysis by the unified computing engine, the following unified SQL query scheme is generated:

[0078] One approach is using PostgreSQL subqueries: leveraging the window function feature, this method calculates the pass rate ranking of each factory by grouping by factory_id, while simultaneously filtering for region='C'. The syntax is: SELECT factory_id, product_type, pass_rate, ROW_NUMBER() OVER (PARTITION BY product_type ORDER BY pass_rateDESC) AS rn FROM factories JOIN production_records USING(factory_id) WHEREregion='C'.

[0079] Secondly, MySQL subquery statements: Since window functions are not supported, local sorting is achieved through LIMIT. The syntax is: SELECT product_type, AVG(pass_rate) AS avg_pass FROM production_records WHERE factory_id IN (SELECT factory_id FROM factories WHERE region='C') GROUP BY product_type ORDER BY avg_pass DESC LIMIT 2.

[0080] Thirdly, there is the global integration logic: the engine merges the results of the two subqueries in a unified calculation layer, verifies the data consistency through the factory_id relationship defined in the metadata, and finally returns the merged ranking result.

[0081] In this process, subquery splitting adapts to the syntax differences between MySQL and PostgreSQL, ensuring that the query is executable; predicate pushdown (such as region='C' being pre-filtered in the PostgreSQL subquery statement) reduces the amount of data transferred across sources; users do not need to pay attention to the underlying syntax details and can initiate complex cross-source queries simply through natural language, demonstrating the improved compatibility, efficiency and ease of use brought about by the technical features.

[0082] Step S1004: Distribute the SQL subquery statements in the unified SQL query scheme to the corresponding data sources in the heterogeneous data sources for execution, and obtain the execution results of each data source.

[0083] Specifically, the unified SQL query scheme is first analyzed to identify the multiple SQL subqueries and their corresponding target data sources. For example, MySQL subqueries correspond to MySQL databases, and PostgreSQL subqueries correspond to PostgreSQL databases. Then, relying on a storage-compute separation and multi-tenant resource isolation architecture, the unified computing engine distributes each subquery to the corresponding data source's computing nodes for execution. For instance, OLTP subqueries involving real-time transaction data are distributed to MySQL for execution, while OLAP subqueries involving complex statistical analysis are distributed to PostgreSQL for execution.

[0084] During this process, resources are dynamically allocated based on the data source type and query load characteristics to ensure that different subqueries are processed in parallel within independent resource spaces, avoiding mutual interference. After each data source executes its subquery, it only returns the filtered intermediate result set to the unified computing engine, rather than transmitting the full original data.

[0085] More specifically, by distributing subqueries to their respective data sources for execution, the native computing capabilities of each data source can be fully utilized, such as MySQL's index optimization and PostgreSQL's parallel query capabilities. This avoids the traditional model of extracting all data to a central node for computation, significantly reducing cross-network data transmission volume. Simultaneously, multi-tenant resource isolation ensures that high-priority business queries are not blocked by analytical queries, improving system stability.

[0086] Step S1005: Perform unified calculation processing on the execution results of each data source to generate a unified result set, and output the unified result set according to a preset format.

[0087] Unified computing processing refers to the integration operations performed by the unified computing engine on the intermediate result sets returned by various data sources, including data format conversion, cross-source data association, global aggregation, sorting and filtering, in order to eliminate the structural differences of multi-source data and generate complete results that meet the query intent.

[0088] The unified result set is a structured data set formed after unified computation and processing. It contains all the information required by natural language query statements. The data format and logical relationships are consistent at the global level and can be directly used for subsequent output or analysis.

[0089] Specifically, the unified computing engine first receives intermediate result sets returned by various data sources after executing subqueries. These results may have different data structures, such as tabular data returned by MySQL or JSON arrays returned by PostgreSQL. The engine standardizes the heterogeneous result sets into a unified format through its built-in data transformation module, including but not limited to unified field names and data types. Then, it performs computational processing based on the global logic defined in the intermediate representation (IR).

[0090] For example, the engine can correlate the "sales revenue of stores in region A" returned by MySQL with the "national average sales revenue" returned by PostgreSQL by date field, calculate the daily differences, and sort the results in ascending order by date. If duplicate data or outliers are found in the result set, the engine will automatically perform deduplication, validation, and other cleaning operations to ensure data accuracy.

[0091] After completing the calculation, the engine generates a unified result set containing all query results, and then outputs the results to the target application or storage location through a standard interface according to the user or preset output format.

[0092] In one specific embodiment, a user on an e-commerce platform queries "the percentage and year-on-year growth rate of GMV for each category in region B during the 2023 Double 11 shopping festival," which involves three sub-query results: "GMV for each category in region B" (including the category and gmv_2023 fields) returned by MySQL, "GMV for each category in region B in 2022" (including the category and gmv_2022 fields) returned by PostgreSQL, and "National Category Comparison Dictionary" (including the category_id and category_name fields) returned by OSS.

[0093] Subsequently, the unified calculation engine first standardized the category field of the three result sets to a string type, and then used the category_id to associate with the category dictionary to complete the category name. Next, it calculated the year-on-year growth rate and sorted the results in descending order by GMV share, generating a unified result set containing the fields category_name, gmv_2023, gmv share, and year-on-year growth rate. Finally, according to the preset "BI report" format requirements, the results were output to Tableau in CSV format, automatically generating pie charts and line charts.

[0094] In one feasible implementation, refer to Figure 2 In the second embodiment of this application, step S1001, which is the step of integrating metadata from pre-accessed heterogeneous data sources and constructing a global logical data model, includes steps S2001~S2002:

[0095] Step S2001: Scan, extract and register the metadata of the heterogeneous data source to obtain a set of metadata including table structure, field types and inter-table relationships.

[0096] The metadata set refers to the sum of metadata collected from heterogeneous data sources through scanning and extraction operations. It includes structured information such as the table structure, field types, and inter-table relationships of each data source, and serves as the basic material for building a global logical data model.

[0097] Specifically, the metadata management module initiates automated scanning of pre-connected heterogeneous data sources. For database data sources, it reads system tables via JDBC interface to obtain table structure, field types, and index information; for files in object storage, it infers field types and delimiters by parsing file headers and sample data; for data warehouses, it calls their metadata API to obtain information such as table partitions and field constraints. After scanning, the extracted metadata is standardized and registered, unifying field naming rules and data type mappings to form a structured metadata set.

[0098] Step S2002: Based on the metadata set, establish the logical relationship between different tables in the heterogeneous data source to obtain the global logical data model.

[0099] Among them, logical relationships are abstract connection rules established based on the relationships between tables in the metadata set. They are used to describe the dependencies between different data source tables at the global logical level without changing the physical storage location of the data.

[0100] Specifically, logical relationships are automatically established based on the table relationship descriptions in the metadata set (e.g., the `product_id` field in a MySQL table is marked as a foreign key, relating to the `product_id` field in a PostgreSQL table). More specifically, semantic relationships between tables across data sources are identified by matching field names, data types, and constraints (e.g., primary key-foreign key relationships). For tables where relationships are not explicitly defined, potential relationships are supplemented through field content similarity analysis. Ultimately, this metadata and logical relationships are integrated into a global logical data model, presented as a virtual directory, enabling dispersed heterogeneous data sources to form a logically interconnected whole.

[0101] More specifically, automated metadata scanning and extraction avoids the tedious work of manually organizing metadata, reduces the risk of metadata inconsistencies caused by human error, and supports real-time access to new data sources, improving model scalability. Logical relationships are established based on the metadata set, automatically identifying dependencies between cross-source tables without requiring users to manually write join code, providing accurate logical basis for subsequent semantic mapping of natural language queries and cross-source SQL generation. Furthermore, the dynamic maintenance of logical relationships and the mechanism for automatic updates when metadata changes ensure the real-time nature of the global logical data model, avoiding model failure issues caused by metadata changes in traditional ETL, further improving the stability and reliability of multi-source data analysis.

[0102] In one feasible implementation, refer to Figure 3 In the third embodiment of this application, step S1002, namely, identifying the semantic elements of the pre-acquired natural language query statement, mapping the semantic elements to the corresponding logical entities in the global logical data model, and generating a structured intermediate representation, includes steps S3001 to S3004:

[0103] Step S3001: Perform word segmentation, entity recognition, and semantic role labeling on the natural language query statement, and parse the query intent, filtering conditions, and aggregation operations of the natural language query statement.

[0104] Semantic role labeling refers to the process of labeling the semantic functions of each component in a natural language query statement, thereby clarifying the role of each semantic element in the query logic.

[0105] Aggregation operations are logical actions that perform statistical calculations on data, such as "summation", "average", "maximum value", and "sorting", which correspond to expressions such as "sum", "average", "highest", and "ranking" in natural language.

[0106] Specifically, the NL2SQL engine is used to perform natural language processing on the user-input natural language query. The query is broken down into its smallest semantic units through word segmentation, and then key data objects and constraints are located through entity recognition. Semantic role labeling further clarifies the logical relationships between the elements. For example, labeling "highest" indicates an aggregation operation, "product" indicates an aggregation object, "D zone" indicates the spatial dimension of the filtering conditions, and "this quarter" indicates the time dimension.

[0107] Step S3002: Match and map the semantic elements contained in the query intent, filtering conditions and aggregation operations with the logical tables and fields in the global logical data model to obtain logical entities.

[0108] Specifically, based on the parsed query intent, filtering conditions, and aggregation operations, the system queries the global logical data model and maps semantic elements to corresponding logical entities. For example, "sales amount" is mapped to the sales_fact.amount field in the logical model, "product" is mapped to the product_dim.product_name field, and "region D" is mapped to the region_dim.region_name field.

[0109] Step S3003: When the semantic element involves multiple logical tables, the required table connection logic is determined based on the logical relationships defined in the global logical data model.

[0110] Among them, table join logic refers to the table join rules automatically generated based on the logical relationships defined in the global logical data model when a query involves multiple logical tables, which are used to merge data from different tables.

[0111] Specifically, if a semantic element involves multiple logical tables, the predefined logical relationships in the global logical data model are invoked to determine the table connection logic.

[0112] Step S3004: Based on the query intent, filtering conditions, aggregation operations, logical entities, and the table join logic, generate a structured intermediate representation.

[0113] Specifically, the query intent, filtering conditions, aggregation operations, logical entities, and table join logic mentioned above are integrated into a structured intermediate representation (IR). This IR is stored in a data model-independent format and includes field mapping relationships (such as "sales amount" → amount), filtering condition expressions (such as region_name='Region D' AND quarter='This Quarter'), aggregation functions, and table join rules, such as sales_fact JOIN product_dim ON sales_fact.product_id = product_dim.product_id, fully expressing the semantic logic of the query.

[0114] In one feasible implementation, refer to Figure 4 In the fourth embodiment of this application, step S1003, namely, generating a unified SQL query scheme containing multiple SQL subqueries based on the intermediate representation and the SQL syntax features supported by the heterogeneous data source, includes steps S4001 to S4003:

[0115] Step S4001: Convert the intermediate representation into an abstract query plan that is independent of the heterogeneous data source type.

[0116] An abstract query plan is a logical query blueprint that is independent of the specific data source type. It is based on intermediate representation (IR) transformation and includes the logical operations of the query (such as filtering, aggregation, and joining) and the data flow, but does not involve the SQL syntax details of the specific database.

[0117] Specifically, the process of generating a unified SQL query plan is the core step in transforming abstract semantic logic into a concrete query plan that can be executed across sources. It is divided into three levels: abstract plan generation, syntax adaptation, and collaborative logic determination. First, the unified computing engine receives the intermediate representation (IR) and converts it into an abstract query plan.

[0118] More specifically, semantic elements in IR are mapped to general logical operators, and the dependencies between operators form a directed acyclic graph. At this point, the plan does not involve the specific syntax of any data source.

[0119] Step S4002: Based on the SQL syntax and functions supported by each data source in the heterogeneous data source, adapt the abstract query plan to generate executable SQL subquery statements for each corresponding data source.

[0120] Among them, the executable SQL subquery statement is a query statement that conforms to the SQL syntax specification of a specific data source. It is adapted from the abstract query plan and can be executed directly on the corresponding data source.

[0121] Specifically, based on the type of heterogeneous data source, a list of supported SQL syntax and functions for each data source is obtained (e.g., MySQL supports the GROUP_CONCAT function, and PostgreSQL supports array operators). Based on this, the general operators in the abstract query plan are adapted into executable SQL subqueries for each data source.

[0122] For example, the "TopN sort" operator in the abstract plan is converted into a LIMIT clause in the MySQL subquery statement and into the ROW_NUMBER() window function in the PostgreSQL subquery statement; operators involving cross-source joins are split into multiple subqueries targeting a single source, such as splitting "joining the sales table and the product table" into a MySQL-side sales data subquery and a PostgreSQL-side product data subquery.

[0123] In one feasible embodiment, step S4002, which is the step of adapting the abstract query plan to generate an executable SQL subquery statement for each corresponding data source according to the SQL syntax and functions supported by each data source in the heterogeneous data source, includes step S4004:

[0124] Step S4004: Perform query rewriting optimization and / or predicate pushdown optimization on the executable SQL subquery statement to form an optimized SQL subquery statement.

[0125] Query rewriting optimization is an optimization method that improves execution efficiency by adjusting the logical structure of SQL subqueries (such as replacing subqueries with JOIN, merging duplicate conditions, and eliminating redundant calculations) without changing the query results, making them more in line with the execution preferences of the database optimizer.

[0126] Among them, predicate pushdown optimization is an optimization method that pushes down the filtering conditions in the SQL subquery statement to the storage or computing layer of the data source as far as possible, so that the data source filters out irrelevant data before returning the results, thereby reducing the size of the intermediate result set.

[0127] Specifically, the optimization process for executable SQL subqueries involves generating subqueries that conform to the data source syntax, and then further improving execution efficiency through logical adjustments and conditional pushovers. In detail, the parser identifies the logical operations within the subquery and, in conjunction with the data source's metadata, determines whether there are any structures that can be optimized.

[0128] For query rewriting optimization, predefined rewriting rules are applied to adjust the subquery structure: for example, nested subqueries are converted into JOIN operations, and the database's optimization support for JOIN (such as hash join and merge join) is used to improve efficiency; or expressions containing repeated calculations are merged to reduce redundant calculations.

[0129] For predicate pushdown optimization, identify the filtering predicates in the subquery and determine whether these predicates can be pushed down to the data source for execution: if the fields involved in the predicate have indexes in the data source, then push the predicate down to the WHERE clause of the data source, so that the data source directly filters out data that does not meet the conditions when scanning the table, and only returns intermediate results that meet the conditions. If the subquery contains aggregation operations, push down the predicates that can be pushed down to be executed before the aggregation, reducing the amount of data processed by the aggregation.

[0130] Ultimately, the subqueries, after being rewritten and / or optimized by predicate pushdown, achieve better execution paths and smaller data processing volumes while maintaining semantic integrity, resulting in optimized SQL subquery statements.

[0131] Step S4003: Based on the abstract query plan, determine the execution order and coordination logic of each SQL subquery statement to form the unified SQL query scheme.

[0132] Specifically, based on the DAG dependencies of the abstract query plan, the execution order and collaborative logic of subqueries are determined: if subquery B depends on the result of subquery A, then A is marked as a prerequisite task; if subqueries C and D have no dependency relationship, they are set to be executed in parallel. Simultaneously, by adding collaborative logic such as temporary result storage and data transfer interfaces, it is ensured that the subquery results flow as expected. Ultimately, these subqueries, execution order, and collaborative logic are integrated into a unified SQL query scheme.

[0133] In one feasible implementation, refer to Figure 5 In the fifth embodiment of this application, the unified SQL query scheme includes the execution order and collaborative logic of each SQL subquery statement. Step S1004, namely, the step of distributing the SQL subquery statements in the unified SQL query scheme to the corresponding data sources in the heterogeneous data sources for execution and obtaining the execution results of each data source, includes steps S5001~S5003:

[0134] Step S5001: According to the execution order and coordination logic, each SQL subquery statement is scheduled and distributed to its corresponding data source;

[0135] The scheduling and distribution process is the process of allocating each SQL subquery statement to a corresponding heterogeneous data source based on the execution order and collaborative logic defined in the unified SQL query scheme. It is necessary to match the data source type and computing resources to ensure that the subquery is executed in the correct environment.

[0136] Specifically, the engine identifies dependencies between subqueries (e.g., subquery B needs to wait for subquery A to complete) and the possibility of parallel processing (e.g., subqueries C and D have no dependency), and generates a scheduling plan based on the load status of the data source. Subsequently, the engine distributes the subqueries to the corresponding data sources through the data source connector.

[0137] For example, MySQL subquery statements can be sent to a MySQL database via a JDBC connection, while PostgreSQL subquery statements can be sent to a PostgreSQL cluster via their dedicated drivers.

[0138] Step S5002: Execute the received SQL subquery statements in parallel on the data source and monitor the execution status of each SQL subquery statement;

[0139] Among them, the execution status is the real-time running status of the SQL subquery statement on the data source, including statuses such as "pending execution", "in execution", "execution successful", and "execution failed", which are used to monitor the query progress and the validity of the results.

[0140] Specifically, after distribution, independent subqueries are run simultaneously in resource-isolated containers to avoid excessive load on a single data source. Meanwhile, the monitoring module collects the execution status of each subquery in real time, obtaining progress through the data source's execution log interface and using heartbeat detection to determine connection status. If a timeout or error is detected, a retry mechanism is triggered or an exception message is returned.

[0141] Step S5003: When the execution status of the data source is detected as successful, the execution result returned by the data source is obtained.

[0142] Specifically, when the monitoring module detects that the execution status of a subquery has changed to "execution successful," the engine retrieves an intermediate result set from the data source through a preset interface (containing only the necessary data filtered by the subquery, not the entire table data). If the subquery fails to execute, detailed logs are returned based on the error type for users or administrators to troubleshoot.

[0143] In one feasible implementation, refer to Figure 6 In the sixth embodiment of this application, step S1005, which involves uniformly calculating and processing the execution results of each data source to generate a unified result set and outputting the unified result set according to a preset format, includes steps S6001 to S6003:

[0144] Step S6001: Connect, aggregate, and sort the execution results of each data source to generate the unified result set;

[0145] Specifically, the unified computing engine receives intermediate result sets returned by various data sources and performs connection operations based on the logical relationships defined in the global logical data model.

[0146] For example, perform an inner join between the "User ID" field in the sales table and the "User ID" field in the user table to ensure that the data rows correspond one-to-one.

[0147] Subsequently, based on the aggregation operation requirements in the intermediate representation (IR), such as "summing sales" and "calculating the average," aggregation calculations are performed on the joined data. For example, the SUM() function is executed on the "sales" field, and the COUNT() function is executed on the "order quantity" field. Finally, according to the sorting requirements in the IR, the aggregation results are globally sorted to form a unified result set containing complete query logic.

[0148] Step S6002: Convert the unified result set into structured data that meets the preset format requirements, wherein the preset format includes JSON format, CSV format and API data interaction format.

[0149] Specifically, based on the output format preset by the user or application scenario, the unified result set is formatted: for example, when converting structured data to JSON format, field names are used as keys and field values ​​are used as values, generating nested or flat JSON objects; when converting to CSV format, field values ​​are separated by commas, and field names are written in the first line.

[0150] For example, the preset output formats include, but are not limited to, JSON format required by the front-end system, CSV format required for Excel import, and API format required for third-party system integration.

[0151] Step S6003: Output the converted unified result set through a preset data interface.

[0152] The preset data interface refers to the predefined output channel specification, such as HTTP interface, JDBC interface or file transfer protocol, which is used to direct the converted result to the target application.

[0153] Specifically, if the target is a BI tool, the data is written directly to its data connection pool via JDBC interface; if the target is a front-end page, it is returned in JSON format via HTTP interface; if it needs to be stored in a data warehouse, a CSV file is transferred via FTP protocol.

[0154] In addition, refer to Figure 7 This application also proposes a multi-source data analysis device, the multi-source data analysis device comprising:

[0155] Metadata management module 10 is used to integrate metadata from pre-connected heterogeneous data sources and build a global logical data model;

[0156] The semantic mapping module 20 is used to identify the semantic elements of the pre-acquired natural language query statement, map the semantic elements to the corresponding logical entities in the global logical data model, and generate a structured intermediate representation.

[0157] The unified generation module 30 is used to generate a unified SQL query scheme containing multiple SQL subquery statements based on the intermediate representation and the SQL syntax features supported by the heterogeneous data source.

[0158] The execution module 40 is used to distribute the SQL subquery statements in the unified SQL query scheme to the corresponding data sources in the heterogeneous data sources for execution, and obtain the execution results of each data source;

[0159] The structured output module 50 is used to perform unified calculation processing on the execution results of the various data sources, generate a unified result set, and output the unified result set according to a preset format.

[0160] The multi-source data analysis device provided in this application employs the multi-source data analysis method described in the above embodiments, aiming to solve the technical problems of high complexity and high technical threshold in querying and analyzing multi-source heterogeneous data in enterprises. Compared with the prior art, the beneficial effects of the multi-source data analysis device provided in this application are the same as those of the multi-source data analysis method provided in the above embodiments, and other technical features in the multi-source data analysis device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0161] This application provides a multi-source data analysis device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-source data analysis method in the above embodiment 1.

[0162] The multi-source data analysis device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The multi-source data analysis device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0163] like Figure 8As shown, the multi-source data analysis device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the multi-source data analysis device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the multi-source data analysis device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows a multi-source data analysis device with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0164] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0165] The multi-source data analysis device provided in this application employs the multi-source data analysis method described in the above embodiments, aiming to solve the technical problems of high complexity and high technical threshold in querying and analyzing multi-source heterogeneous data in enterprises. Compared with the prior art, the beneficial effects of the multi-source data analysis device provided in this application are the same as those of the multi-source data analysis method provided in the above embodiments, and other technical features in this multi-source data analysis device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0166] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0167] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0168] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the multi-source data analysis method described in the above embodiments.

[0169] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0170] The aforementioned computer-readable storage medium may be included in a multi-source data analysis device; or it may exist independently and not assembled into a multi-source data analysis device.

[0171] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0172] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0173] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0174] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described multi-source data analysis method. It aims to solve the technical problems of high complexity and high technical threshold in querying and analyzing multi-source heterogeneous data in enterprises. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the multi-source data analysis method provided in the above embodiments, and will not be repeated here.

[0175] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the multi-source data analysis method described above.

[0176] The computer program product provided in this application aims to solve the technical problems of high complexity and high technical threshold in querying and analyzing multi-source heterogeneous data in enterprises. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the multi-source data analysis method provided in the above embodiments, and will not be repeated here.

[0177] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0178] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.

[0180] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multi-source data analysis method, characterized in that, The multi-source data analysis method includes: Integrate metadata from pre-connected heterogeneous data sources to build a global logical data model; Identify the semantic elements of a pre-acquired natural language query statement, map the semantic elements to the corresponding logical entities in the global logical data model, and generate a structured intermediate representation; Based on the intermediate representation and the SQL syntax features supported by the heterogeneous data source, a unified SQL query scheme containing multiple SQL subquery statements is generated. The SQL subquery statements in the unified SQL query scheme are distributed to the corresponding data sources in the heterogeneous data sources for execution, and the execution results of each data source are obtained; The execution results from each data source are processed uniformly to generate a unified result set, which is then output in a preset format.

2. The multi-source data analysis method as described in claim 1, characterized in that, The steps of integrating metadata from pre-connected heterogeneous data sources and constructing a global logical data model include: The metadata of the heterogeneous data source is scanned, extracted, and registered to obtain a set of metadata including table structure, field types, and inter-table relationships; Based on the metadata set, logical relationships between different tables in the heterogeneous data source are established to obtain the global logical data model.

3. The multi-source data analysis method as described in claim 1, characterized in that, The steps of identifying semantic elements of a pre-acquired natural language query statement, mapping the semantic elements to corresponding logical entities in the global logical data model, and generating a structured intermediate representation include: The natural language query statement is segmented, entity recognized, and semantic role labeled to parse the query intent, filtering conditions, and aggregation operations of the natural language query statement. The semantic elements contained in the query intent, filtering conditions, and aggregation operations are matched and mapped with the logical tables and fields in the global logical data model to obtain logical entities; When the semantic element involves multiple logical tables, the required table join logic is determined based on the logical relationships defined in the global logical data model. Based on the query intent, filtering conditions, aggregation operations, logical entities, and the table join logic, a structured intermediate representation is generated.

4. The multi-source data analysis method as described in claim 1, characterized in that, The step of generating a unified SQL query scheme containing multiple SQL subqueries based on the intermediate representation and the SQL syntax features supported by the heterogeneous data source includes: The intermediate representation is converted into an abstract query plan that is independent of the heterogeneous data source type; Based on the SQL syntax and functions supported by each data source in the heterogeneous data source, the abstract query plan is adapted to generate executable SQL subquery statements for each corresponding data source; Based on the abstract query plan, the execution order and coordination logic of each SQL subquery statement are determined to form the unified SQL query scheme.

5. The multi-source data analysis method as described in claim 4, characterized in that, The step of adapting the abstract query plan to generate executable SQL subquery statements for each corresponding data source based on the SQL syntax and functions supported by each data source in the heterogeneous data source includes: The executable SQL subquery statement is optimized by query rewriting and / or predicate pushdown to form an optimized SQL subquery statement.

6. The multi-source data analysis method as described in claim 1, characterized in that, The unified SQL query scheme includes the execution order and coordination logic of each SQL subquery statement. The step of distributing the SQL subquery statements in the unified SQL query scheme to the corresponding data sources in the heterogeneous data sources for execution and obtaining the execution results of each data source includes: Based on the execution order and coordination logic, each SQL subquery statement is scheduled and distributed to its corresponding data source; The received SQL subqueries are executed in parallel on the data source, and the execution status of each SQL subquery is monitored. When the execution status of the data source is detected as successful, the execution result returned by the data source is obtained.

7. The multi-source data analysis method as described in claim 1, characterized in that, The step of performing unified calculations on the execution results of each data source to generate a unified result set, and outputting the unified result set according to a preset format, includes: The execution results from each data source are connected, aggregated, and sorted to generate the unified result set; The unified result set is converted into structured data that conforms to a preset format, wherein the preset format includes JSON format, CSV format and API data interaction format; The unified result set after conversion is output through a preset data interface.

8. A multi-source data analysis device, characterized in that, The device includes: The metadata management module is used to integrate metadata from pre-connected heterogeneous data sources and build a global logical data model. The semantic mapping module is used to identify the semantic elements of the pre-acquired natural language query statement, map the semantic elements to the corresponding logical entities in the global logical data model, and generate a structured intermediate representation. A unified generation module is used to generate a unified SQL query scheme containing multiple SQL subquery statements based on the intermediate representation and the SQL syntax features supported by the heterogeneous data source. The execution module is used to distribute the SQL subquery statements in the unified SQL query scheme to the corresponding data sources in the heterogeneous data sources for execution, and obtain the execution results of each data source; The structured output module is used to perform unified calculation and processing on the execution results of the various data sources, generate a unified result set, and output the unified result set according to a preset format.

9. A multi-source data analysis device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-source data analysis method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the multi-source data analysis method as described in any one of claims 1 to 7.

Images