Multi-source data analysis method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data analysis and natural language processing, and discloses a multi-source data analysis method and device, equipment and a storage medium, and the method comprises the following steps: carrying out metadata integration on a pre-accessed heterogeneous data source, and constructing a global logic data model; identifying semantic elements of a pre-acquired natural language query statement, mapping the semantic elements to corresponding logic entities in the global logic data model, and generating a structured intermediate representation; generating a unified SQL query scheme containing a plurality of SQL sub-query statements based on the intermediate representation; the SQL sub-query statements are distributed to corresponding data sources in the heterogeneous data sources to be executed, and execution results of all the data sources are obtained; and performing unified calculation processing on the execution results of the data sources to generate a unified result set, and outputting the unified result set according to a preset format. By means of natural language interaction and SQL generation, simplification of multi-source heterogeneous data analysis is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data analysis and natural language processing, and particularly relates to a multi-source data analysis method and device, equipment and a storage medium. BACKGROUND

[0002] In the field of enterprise data analysis, data assets are usually scattered in multiple heterogeneous data management systems. In the face of such multi-source heterogeneous data environment, if business personnel need to perform cross-system correlation analysis, they must rely on professional data engineers to write complex SQL query scripts involving multiple database connections, or perform data extraction, conversion and centralization through a cumbersome ETL process. This process not only has high technical threshold and long cycle, but also is difficult to quickly respond to flexible business analysis needs.

[0003] On the other hand, although natural language to SQL technology allows users to use natural language for querying to some extent, it simplifies data access of a single data source, but in a complex multi-source scenario, the existing scheme is usually difficult to accurately understand the semantics of business entities involving different data sources, resulting in low query efficiency or even incorrect execution. Therefore, how to effectively reduce the technical complexity of multi-source heterogeneous data query analysis and improve the accuracy and execution efficiency of the query is a technical problem to be solved. SUMMARY

[0004] The main purpose of the present application is to provide a multi-source data analysis method, device, equipment and storage medium, aiming to solve the technical problems of high complexity and high technical threshold of enterprise multi-source heterogeneous data query analysis.

[0005] To achieve the above purpose, the present application provides a multi-source data analysis method, which comprises the following steps: Integrating the metadata of the pre-accessed heterogeneous data sources to construct a global logical data model; Identifying the semantic elements of the pre-acquired natural language query statement, mapping the semantic elements to the corresponding logical entities in the global logical data model, and generating a structured intermediate representation; Based on the intermediate representation and the SQL syntax characteristics supported by the heterogeneous data sources, generating a unified SQL query scheme containing multiple SQL subquery statements; Distributing the SQL subquery statements in the unified SQL query scheme to the corresponding data sources for execution, and obtaining the execution results of each data source; Performing unified calculation processing on the execution results of each data source to generate a unified result set, and outputting the unified result set in a preset format.

[0006] In an embodiment, the step of performing metadata integration on the pre-accessed heterogeneous data sources to construct a global logical data model comprises: scanning, extracting and registering meta-information of the heterogeneous data sources to obtain a meta-information set including table structure, field type and inter-table relationship; establishing logical association relationship between different tables in the heterogeneous data sources based on the meta-information set to obtain the global logical data model.

[0007] In an embodiment, the step of identifying semantic elements of a pre-acquired natural language query statement, mapping the semantic elements to corresponding logical entities in the global logical data model, and generating a structured intermediate representation comprises: performing word segmentation, entity recognition and semantic role labeling on the natural language query statement, analyzing query intent, filtering condition and aggregation operation of the natural language query statement; matching and mapping semantic elements contained in the query intent, filtering condition and aggregation operation with logical tables and fields in the global logical data model to obtain logical entities; when the semantic elements involve multiple logical tables, determining required table connection logic based on logical association relationship defined in the global logical data model; generating a structured intermediate representation based on the query intent, filtering condition, aggregation operation, logical entities and the table connection logic.

[0008] In an embodiment, the step of generating a unified SQL query scheme comprising multiple SQL sub-query statements based on the intermediate representation and SQL syntax features supported by the heterogeneous data sources comprises: converting the intermediate representation into an abstract query plan independent of the type of the heterogeneous data sources; adapting the abstract query plan to generate executable SQL sub-query statements for each corresponding data source according to the SQL syntax and functions supported by each data source in the heterogeneous data sources; determining execution order and coordination logic of each SQL sub-query statement based on the abstract query plan to form the unified SQL query scheme.

[0009] In an embodiment, the step of adapting the abstract query plan to generate executable SQL sub-query statements for each corresponding data source according to the SQL syntax and functions supported by each data source in the heterogeneous data sources comprises: performing query rewriting optimization and / or predicate pushdown optimization on the executable SQL sub-query statements to form optimized SQL sub-query statements.

[0010] In an embodiment, the unified SQL query scheme includes execution order and coordination logic of each SQL subquery statement, the step of distributing the SQL subquery statements in the unified SQL query scheme to corresponding data sources in the heterogeneous data sources for execution and obtaining execution results of each data source includes: According to the execution order and coordination logic, each SQL subquery statement is dispatched to its corresponding data source; The received SQL subquery statements are executed on the data source in parallel, and the execution status of each SQL subquery statement is monitored; When it is detected that the execution status of the data source is successful, the execution result returned by the data source is obtained.

[0011] In an embodiment, the step of uniformly computing and processing the execution results of each data source, generating a unified result set, and outputting the unified result set in a preset format includes: The execution results of each data source are connected, aggregated, and sorted to generate the unified result set; The unified result set is converted into structured data conforming to the requirements of a preset format, wherein the preset format includes JSON format, CSV format, and API data interaction format; The converted unified result set is output through a preset data interface.

[0012] In addition, to achieve the above-mentioned purposes, the present application also provides a multi-source data analysis device, which comprises: A metadata management module is configured to integrate metadata of pre-accessed heterogeneous data sources and construct a global logical data model; A semantic mapping module is configured to identify semantic elements of a pre-obtained natural language query statement, map the semantic elements to corresponding logical entities in the global logical data model, and generate a structured intermediate representation; A unified generation module is configured to generate a unified SQL query scheme including multiple SQL subquery statements based on the intermediate representation and SQL syntax characteristics supported by the heterogeneous data sources; An execution module is configured to distribute the SQL subquery statements in the unified SQL query scheme to corresponding data sources in the heterogeneous data sources for execution and obtain execution results of each data source; A structured output module is configured to uniformly compute and process the execution results of each data source, generate a unified result set, and output the unified result set in a preset format.

[0013] In addition, to achieve the above object, the application further provides a terminal device, comprising a memory, a processor, and a multi-source data analysis program stored in the memory and executable on the processor, and the multi-source data analysis program realizes the steps of the multi-source data analysis method as described above when executed by the processor.

[0014] In addition, to achieve the above object, the application further provides a computer readable storage medium, wherein the computer readable storage medium stores a multi-source data analysis program, and the multi-source data analysis program realizes the steps of the multi-source data analysis method as described above when executed by a processor.

[0015] The one or more technical solutions provided by the application have at least the following technical effects: Firstly, by pre-accessing heterogeneous data sources and integrating metadata, a global logical data model is constructed, and scattered and various-structured data sources are abstracted into a consistent logical view, so that the user does not need to deeply understand the specific details of each data source, and the complexity of data access is directly reduced. Secondly, a natural language query statement is received and semantic elements are identified, and the semantic elements are mapped to a global model to generate a structured intermediate representation, which allows non-technical users to use intuitive natural language for querying, avoiding the writing of complex SQL or the learning of specific query syntax. Then, a unified SQL query scheme is generated based on the intermediate representation and the SQL characteristics of the heterogeneous data sources, reducing the workload of manually writing and debugging queries and improving the query efficiency. Finally, the results of each data source are uniformly calculated and formatted, ensuring the consistency and readability of the results, and the user does not need to integrate the multi-source results, simplifying the analysis process. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 The flowchart of the first embodiment of the multi-source data analysis method of the application is shown in the figure. Figure 2 The flowchart of the second embodiment of the multi-source data analysis method of the application is shown in the figure. Figure 3 The flowchart of the third embodiment of the multi-source data analysis method of the application is shown in the figure. Figure 4 The flowchart of the fourth embodiment of the multi-source data analysis method of the application is shown in the figure. Figure 5 The flowchart of the fifth embodiment of the multi-source data analysis method of the application is shown in the figure. Figure 6 The flowchart of the sixth embodiment of the multi-source data analysis method of the application is shown in the figure. Figure 7 The module structure diagram of the multi-source data analysis device of the embodiment of the application is shown in the figure. Figure 8A device structure schematic diagram of a hardware running environment involved in a multi-source data analysis method in embodiments of the present application.

[0017] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0018] It should be understood that the specific embodiments described herein are merely intended to explain the present application and not to limit the present application.

[0019] The main technical solution of the present application is: integrating metadata of pre-accessed heterogeneous data sources, constructing a global logical data model; identifying semantic elements of a pre-acquired natural language query statement, mapping the semantic elements to corresponding logical entities in the global logical data model, and generating a structured intermediate representation; generating a unified SQL query scheme containing multiple SQL subquery statements based on the intermediate representation and the SQL syntax characteristics supported by the heterogeneous data sources; distributing the SQL subquery statements in the unified SQL query scheme to the corresponding data sources for execution, and obtaining the execution results of each data source; performing unified calculation processing on the execution results of each data source, generating a unified result set, and outputting the unified result set in a preset format.

[0020] The present application actually takes into account that in the field of enterprise data analysis, data assets are usually scattered in multiple heterogeneous data management systems. In the face of such multi-source heterogeneous data environment, if business personnel need to perform cross-system correlation analysis, they must rely on professional data engineers to write complex SQL query scripts involving multiple database connections, or perform data extraction, conversion and centralization through a tedious ETL process. This process not only has high technical threshold and long cycle, but also is difficult to quickly respond to flexible and variable business analysis needs. On the other hand, although natural language to SQL technology allows users to use natural language for querying to some extent, it simplifies data access of a single data source, but in a complex multi-source scenario, existing solutions are usually difficult to accurately understand the semantics involving business entities in different data sources, resulting in low query efficiency or even incorrect execution.

[0021] Based on this, the main purpose of the present application is to provide a multi-source data analysis method, device, equipment and storage medium, aiming to solve the technical problems of high complexity and high technical threshold of enterprise multi-source heterogeneous data query analysis.

[0022] Specifically, the following are the detailed steps of the first embodiment of the multi-source data analysis method of the present application: Referring to Figure 1 , Figure 1FIG. 1 is a flowchart of a first embodiment of the multi-source data analysis method of the present application. In this embodiment, the multi-source data analysis method comprises steps S1001-S1005: Step S1001, integrating metadata of pre-accessed heterogeneous data sources, and constructing a global logical data model.

[0023] Among them, the heterogeneous data source refers to the data source with different physical positions and formats in the enterprise, such as relational database (MySQL, PostgreSQL), data warehouse (Snowflake), object storage (OSS), etc. These data sources have differences in storage structure and access method.

[0024] Among them, the metadata refers to the data describing the data, including table structure, field type, and inter-table relationship in the data source, etc. It is the basis of data management and understanding.

[0025] Among them, the global logical data model is a virtual and unified data view constructed by integrating the metadata of heterogeneous data sources. It does not physically store data in a centralized manner, but associates scattered data sources through metadata, and regards multiple heterogeneous data sources as a logically unified database.

[0026] Specifically, first, through the built-in data connector, the scattered heterogeneous data sources in the enterprise are pre-accessed to realize real-time or quasi-real-time connection of different types of data sources. Then, these data sources are automatically scanned, and the metadata therein is extracted and uniformly registered. Through the integration of metadata, the system constructs a global logical data model. It is worth noting that the global logical data model does not change the physical storage location of the data, but forms a virtual data directory through metadata association, so that the subsequent steps can operate the underlying scattered data based on a unified logical view.

[0027] More specifically, the global logical data model masks the physical differences of heterogeneous data sources, and users and systems do not need to pay attention to the data storage location and format, but only need to operate based on a unified logical view, thereby reducing the complexity of cross-source data query and reducing the workload of manually handling data source differences.

[0028] Step S1002, identifying semantic elements of a pre-acquired natural language query statement, mapping the semantic elements to corresponding logical entities in the global logical data model, and generating a structured intermediate representation.

[0029] Among them, the natural language query statement refers to the data analysis requirement input by the user in the form of daily language, without following a specific syntax or code format; the semantic element is the component part of the natural language query statement that carries the core meaning, including the core intent, the filtering condition, the aggregation and sorting requirement, etc.

[0030] Among them, the logical entity is an abstract concept in the global logical data model, corresponding to the specific table or field of the underlying data source; the structured intermediate representation (Intermediate Representation, IR) is a structured format independent of the specific data model, which is used to fully express the semantic logic of the natural language query statement, including the mapping relationship between semantic elements and logical entities and query logic (such as filtering, aggregation, and table connection).

[0031] Specifically, first, the natural language query statement input by the user is received, and the query statement is processed by the NL2SQL engine. Specifically, the statement is disassembled into the smallest language unit by word segmentation, and then the key information is located by entity recognition, and the logical relationship of each element is clarified by semantic role labeling. Subsequently, based on the constructed global logical data model, the identified semantic elements are automatically mapped to the logical entities in the model. If the semantic elements involve cross-table association, the connection logic is automatically identified according to the inter-table relationship defined in the metadata. Finally, these mapping relationships and logical relationships are integrated into a structured intermediate representation (IR) as a semantic blueprint for subsequent cross-source SQL generation.

[0032] In a specific embodiment, a certain e-commerce enterprise user inputs a natural language query statement "2023Q4 C region mobile phone category sales top three brands". First, the query is segmented and entity recognized, and the semantic elements are extracted: the core intent is "query sales", the filtering conditions include "2023Q4" (time), "C region" (region), "mobile phone category" (product type), and the aggregation sorting requirements are "top three" (TopN sorting) and "brand" (grouping object).

[0033] Next, the global logical data model is queried, and the semantic elements are mapped to the corresponding logical entities: "sales" is mapped to the sales_fact.sales_amount field, "2023Q4" is mapped to the time_dim.quarter field (value "2023Q4"), "C region" is mapped to the region_dim.region field (value "C"), "mobile phone category" is mapped to the product_dim.category field (value "mobile phone"), and "brand" is mapped to the product_dim.brand field. Since the sales_fact table is associated with the product_dim table through product_id and with the time_dim table through time_id, the table connection logic is automatically identified, and the above mapping relationships and logic are integrated into a structured IR. The IR describes the semantic logic of "filtering the sales data of 2023Q4 C region mobile phone category, grouping by brand, and aggregating sales by brand to get the top three".

[0034] During this process, the accurate identification of semantic elements ensures unbiased query intent, the mapping ability of the global logical data model enables the system to correlate multi-source data without user intervention, and the generated IR provides clear guidance for subsequent cross-source SQL generation, enabling non-technical users to efficiently initiate complex cross-source queries and significantly reducing the operation threshold.

[0035] Step S1003, based on the intermediate representation and the SQL syntax characteristics supported by the heterogeneous data sources, a unified SQL query scheme containing multiple SQL subquery statements is generated.

[0036] Among them, SQL syntax characteristics refer to the differences in SQL syntax rules and functions supported by different database systems (such as MySQL, PostgreSQL, Snowflake), such as whether to support specific functions, data type processing methods or query structure.

[0037] Among them, SQL subquery statements are auxiliary query statements nested in the main query, used to obtain intermediate results from one or more data sources, usually as filtering conditions, data sources or calculation basis for the main query.

[0038] Among them, the unified SQL query scheme is an integrated SQL subquery statement for different heterogeneous data sources, forming a logically coherent and collaboratively executable overall query plan. This scheme needs to adapt to the syntax characteristics of each data source and meet the global query logic.

[0039] Specifically, first, the unified computing engine receives the intermediate representation (IR) generated in step two, which has explicitly defined the semantic logic of the query. Analyze the data source types involved in the IR, and call the metadata management module to obtain the SQL syntax characteristics supported by each data source. Then, according to the logical splitting of the IR, the query task is divided into multiple SQL subquery statements, each of which strictly follows the SQL characteristics of the corresponding data source.

[0040] At the same time, through query rewriting technology, the subquery logic is optimized to push the filtering conditions to the data source execution as much as possible to reduce data transmission. Finally, these subqueries are integrated into a unified SQL query scheme, which contains the data flow dependency between subqueries (such as the results of a subquery as input to another subquery) and the global execution order, ensuring that each subquery is executed collaboratively to meet the original query intent.

[0041] In a specific embodiment, a manufacturing enterprise needs to query "the top 2 product types in terms of production pass rate in Q3 2023 in factories in region C". The data distribution is as follows: production records are stored in MySQL (production_records table, containing factory_id, product_type, and pass_rate fields), and factory information is stored in PostgreSQL (factories table, containing factory_id and region fields). MySQL does not support window functions, while PostgreSQL supports ROW_NUMBER(). The intermediate representation (IR) has clearly indicated that the two tables need to be associated, filtered by region='C', aggregated by product_type to calculate the pass rate, and the top 2 types are taken. After analysis by the unified computing engine, the following unified SQL query scheme is generated: The first is a PostgreSQL subquery statement: using its window function feature, the pass rate ranking of each factory is calculated by grouping by factory_id, and region='C' is filtered. The syntax is SELECT factory_id, product_type, pass_rate, ROW_NUMBER() OVER (PARTITION BY product_type ORDER BY pass_rate DESC) AS rn FROM factories JOIN production_records USING(factory_id) WHERE region='C'.

[0042] The second is a MySQL subquery statement: due to the lack of support for window functions, local sorting is achieved through LIMIT. The syntax is SELECT product_type, AVG(pass_rate) AS avg_pass FROM production_records WHERE factory_id IN (SELECT factory_id FROM factories WHERE region='C') GROUP BY product_type ORDER BY avg_pass DESC LIMIT 2.

[0043] The third is the global integration logic: the engine merges the results of the two subqueries in the unified computing layer, verifies the data consistency through the factory_id association relationship defined in the metadata, and finally returns the merged ranking results.

[0044] In this process, the subquery splitting adapts to the syntax differences between MySQL and PostgreSQL, ensuring that the query is executable; predicate pushdown (such as region='C' in the PostgreSQL subquery statement is filtered in advance) reduces the amount of data transmitted across sources; users do not need to pay attention to the underlying syntax details, and can initiate complex cross-source queries through natural language, embodying the compatibility, efficiency, and ease of use brought by the technical features.

[0045] In step S1004, the SQL subquery statements in the unified SQL query scheme are distributed to the corresponding data source execution of the heterogeneous data sources, and the execution results of each data source are obtained.

[0046] Specifically, first, the unified SQL query scheme is parsed to identify the multiple SQL subquery statements and their corresponding target data sources, such as MySQL subquery statements corresponding to MySQL databases and PostgreSQL subquery statements corresponding to PostgreSQL databases. Subsequently, relying on the storage-computing separation and multi-tenant resource isolation architecture, the unified computing engine distributes each subquery to the computing node of the corresponding data source for execution, such as distributing OLTP-type subqueries involving real-time transaction data to MySQL for execution and distributing OLAP-type subqueries for complex statistical analysis to PostgreSQL for execution.

[0047] In this process, resources are dynamically allocated according to data source types and query load characteristics to ensure that different subqueries are processed in parallel in independent resource spaces, avoiding mutual interference. After each data source executes the subquery, only the filtered intermediate result set is returned to the unified computing engine, rather than transmitting the full amount of raw data.

[0048] More specifically, by distributing subqueries to corresponding data sources for execution, the native computing capabilities of each data source can be fully utilized, such as MySQL's index optimization and PostgreSQL's parallel query, thereby avoiding the traditional mode of extracting full data to a central node for recalculation, significantly reducing the amount of data transmission across the network. At the same time, multi-tenant resource isolation ensures that high-priority business queries are not blocked by analysis-type queries, improving system stability.

[0049] In step S1005, the execution results of each data source are uniformly computed and processed to generate a unified result set, and the unified result set is output in a pre-set format.

[0050] Among them, the unified computing processing refers to the integration operation of the intermediate result set returned by each data source by the unified computing engine, including data format conversion, cross-source data association, global aggregation, sorting, and filtering, etc., to eliminate the structural differences of multi-source data and generate complete results that meet the query intent.

[0051] The unified result set is a structured data set formed after unified computing processing, contains all information required by the natural language query statement, and the data format and logical relationship are consistent at the global level, which can be directly used for subsequent output or analysis.

[0052] Specifically, first, the unified computing engine receives the intermediate result sets returned after each data source executes the subquery. These results may have different data structures, such as table format data returned by MySQL and JSON array returned by PostgreSQL. The engine standardizes the heterogeneous result sets into a unified format through the built-in data conversion module, including but not limited to unified field names and data types, and then performs computing processing based on the global logic defined in the intermediate representation (IR).

[0053] For example, the "A region store sales" returned by MySQL and the "national average sales" returned by PostgreSQL are associated by date field, the daily difference value is calculated, and the result is sorted in ascending order of date. If there are duplicate data or abnormal values in the result set, the engine will automatically perform deduplication, verification and other cleaning operations to ensure data accuracy.

[0054] After completing the calculation, the engine generates a unified result set containing all query results, and then outputs the results to the target application or storage location according to the user or preset output format through the standard interface.

[0055] In a specific embodiment, a user of an e-commerce platform queries "B region GMV proportion of each category and year-on-year growth rate during Double 11 in 2023", which involves three subquery results: "B region GMV of each category" returned by MySQL (containing category, gmv_2023 fields), "B region GMV of each category in 2022" returned by PostgreSQL (containing category, gmv_2022 fields), and "national category dictionary" returned by OSS (containing category_id, category_name fields).

[0056] Then, the unified computing engine first standardizes the category field of the three result sets to string type, associates the category dictionary through category_id, and supplements the complete category name; then calculates the year-on-year growth rate and sorts it in descending order of GMV proportion, generating a unified result set containing category_name, gmv_2023, gmv proportion, and year-on-year growth rate fields. Finally, according to the preset "BI report" format requirements, the results are output to Tableau in CSV format to automatically generate pie charts and line charts.

[0057] In a possible implementation, refer to Figure 2In the second embodiment of the present application, the step S1001, i.e., the step of integrating the metadata of the pre-accessed heterogeneous data sources and constructing a global logical data model, comprises steps S2001-S2002: Step S2001, scanning, extracting and registering the meta-information of the heterogeneous data sources, and obtaining a meta-information set comprising table structure, field type and inter-table relationship.

[0058] The meta-information set refers to the sum of metadata collected from the heterogeneous data sources through scanning and extraction operations, and comprises structured information such as table structure, field type, inter-table relationship, etc. of each data source, and is the basic material for constructing a global logical data model.

[0059] Specifically, the metadata management module initiates an automatic scan of the pre-accessed heterogeneous data sources. For database-type data sources, the system table is read through the JDBC interface to obtain table structure, field type and index information. For files in object storage, the field type and delimiter are inferred by parsing the file header and sample data. For data warehouses, the metadata API is called to obtain table partitioning, field constraints, etc. After scanning, the extracted meta-information is standardized and registered, the field naming rules and data type mapping are unified, and a structured meta-information set is formed.

[0060] Step S2002, based on the meta-information set, establishing a logical association relationship between different tables in the heterogeneous data sources, and obtaining the global logical data model.

[0061] The logical association relationship is an abstract connection rule established based on the inter-table relationship in the meta-information set, and is used to describe the dependency relationship between different data source tables at the global logical level without changing the physical storage location of the data.

[0062] Specifically, based on the inter-table relationship description in the meta-information set (such as the product_id field of the MySQL table being marked as a foreign key and being associated with the product_id of the PostgreSQL table), the logical association relationship is automatically established. Specifically, by matching the field name, data type and constraint condition (such as primary key-foreign key relationship), the semantic association between cross-data source tables (such as the association between the sales table and the product table through product_id, and the association between the order table and the user table through user_id) is identified. For tables without explicitly defined relationships, potential associations are supplemented through field content similarity analysis. Finally, these meta-information and logical association relationships are integrated into a global logical data model, which is presented in the form of a virtual directory, so that the dispersed heterogeneous data sources form a mutually associated whole at the logical level.

[0063] More specifically, the automated meta-information scanning and extraction avoids the tedious work of manually organizing metadata, reduces the risk of metadata inconsistency caused by human operation errors, and supports real-time access of new data sources, improving the scalability of the model. Based on the logical association relationship established by the meta-information set, the dependencies between cross-source tables are automatically identified, and there is no need for users to manually write association code, providing accurate logical basis for subsequent semantic mapping of natural language query statements and cross-source SQL generation. In addition, the dynamic maintenance of logical association relationships ensures the real-time nature of the global logical data model by automatically updating when metadata changes, avoiding the model failure problem caused by metadata changes in traditional ETL, further improving the stability and reliability of multi-source data analysis.

[0064] In a feasible implementation, with reference to Figure 3 In the third embodiment of the present application, the step S1002 of identifying the semantic elements of the pre-acquired natural language query statement, mapping the semantic elements to the corresponding logical entities in the global logical data model, and generating a structured intermediate representation includes steps S3001-S3004: Step S3001, tokenizing, entity recognition, and semantic role labeling are performed on the natural language query statement to analyze the query intent, filtering conditions, and aggregation operations of the natural language query statement.

[0065] Wherein, semantic role labeling refers to the process of labeling the semantic function of each component in the natural language query statement, thereby clarifying the role of each semantic element in the query logic.

[0066] Wherein, the aggregation operation is a logical action for statistical calculation of data, such as "sum", "average", "maximum", "sorting", etc., corresponding to "total", "average", "highest", "ranking" and other expressions in natural language.

[0067] Specifically, the NL2SQL engine is used to process the natural language query statement input by the user. The sentence is split into the smallest semantic unit through tokenization, and then the key data objects and constraints are located through entity recognition. Subsequently, semantic role labeling further clarifies the logical relationship of each element. For example, "highest" is labeled as an aggregation operation, "product" is an aggregation object, "D zone" is a spatial dimension of the filtering condition, and "this quarter" is a time dimension.

[0068] Step S3002, match and map the semantic elements contained in the query intent, filtering conditions, and aggregation operations with the logical tables and fields in the global logical data model to obtain logical entities.

[0069] Specifically, based on the parsed query intent, filter conditions and aggregation operations, the system queries the global logical data model, and maps the semantic elements to corresponding logical entities. For example, "sales" is mapped to the sales_fact.amount field in the logical model, "product" is mapped to the product_dim.product_name field, and "D area" is mapped to the region_dim.region_name field.

[0070] Step S3003, when the semantic elements involve multiple logical tables, the required table join logic is determined based on the logical association relationship defined in the global logical data model.

[0071] wherein the table join logic refers to the inter-table association rule automatically generated based on the logical association relationship defined in the global logical data model, for merging data of different tables, when the query involves multiple logical tables.

[0072] Specifically, if the semantic elements involve multiple logical tables, the predefined logical association relationship in the global logical data model is called to determine the table join logic.

[0073] Step S3004, based on the query intent, filter conditions, aggregation operations, logical entities and the table join logic, a structured intermediate representation is generated.

[0074] Specifically, the above query intent, filter conditions, aggregation operations, logical entities and table join logic are integrated into a structured intermediate representation (IR). The IR is stored in a data model-independent format, and contains field mapping relationships (such as "sales"→amount), filter condition expressions (such as region_name='D area' AND quarter='this quarter'), aggregation functions and table join rules, for example, sales_fact JOIN product_dim ON sales_fact.product_id =product_dim.product_id, which fully express the semantic logic of the query.

[0075] In a feasible implementation, with reference to Figure 4 In the fourth embodiment of the present application, the step S1003, i.e., the step of generating a unified SQL query scheme containing multiple SQL subquery statements based on the intermediate representation and the SQL syntax features supported by the heterogeneous data sources, includes steps S4001-S4003: Step S4001, the intermediate representation is converted into an abstract query plan independent of the type of heterogeneous data sources.

[0076] The abstract query plan is a logical query blueprint independent of specific data source types, and is converted based on an intermediate representation (IR) and includes logical operations (such as filtering, aggregation, and connection) and data flow direction of the query, but does not involve specific SQL syntax details of the database.

[0077] Specifically, the process of generating a unified SQL query scheme is the core link of converting abstract semantic logic into a specific query plan that can be executed across sources, and is divided into three levels of abstract plan generation, syntax adaptation, and cooperative logic determination. First, the unified computing engine receives the intermediate representation (IR) and converts it into an abstract query plan.

[0078] More specifically, the semantic elements in the IR are mapped to general logical operators, and the dependency relationship between the operators forms a directed acyclic graph. At this time, the plan does not involve any specific syntax of the data source.

[0079] In step S4002, the abstract query plan is adapted to generate executable SQL subquery statements for each corresponding data source according to the SQL syntax and functions supported by each data source in the heterogeneous data source.

[0080] The executable SQL subquery statement is a query statement generated for a specific data source that conforms to its SQL syntax specification, and is adapted from the abstract query plan and can be directly executed on the corresponding data source.

[0081] Specifically, according to the type of the heterogeneous data source, the list of SQL syntax and functions supported by each data source is obtained (for example, MySQL supports the GROUP_CONCAT function, and PostgreSQL supports array operators). Based on this, the general operators in the abstract query plan are adapted to executable SQL subquery statements for each data source.

[0082] For example, the "TopN sorting" operator in the abstract plan is converted to the LIMIT clause in the MySQL subquery statement and the ROW_NUMBER() window function in the PostgreSQL subquery statement. Operators involving cross-source connection are split into multiple subqueries for single sources, such as splitting the "sales table and product table connection" into a MySQL-side sales data subquery and a PostgreSQL-side product data subquery.

[0083] In one possible embodiment, the step S4002, i.e., the step of adapting the abstract query plan to generate executable SQL subquery statements for each corresponding data source according to the SQL syntax and functions supported by each data source in the heterogeneous data source, is followed by step S4004: Step S4004, query rewriting optimization and / or predicate pushdown optimization are performed on the executable SQL subquery statement to form an optimized SQL subquery statement.

[0084] The query rewriting optimization is an optimization method that adjusts the logical structure of the SQL subquery statement (such as replacing the subquery with JOIN, merging repeated conditions, and eliminating redundant calculations) without changing the query result, so as to make it more in line with the execution preferences of the database optimizer and improve the execution efficiency.

[0085] The predicate pushdown optimization is an optimization method that pushes the filtering conditions in the SQL subquery statement to the storage layer or the calculation layer of the data source as much as possible to make the data source filter out irrelevant data before returning the result, thereby reducing the size of the intermediate result set.

[0086] Specifically, after generating a subquery that conforms to the syntax of the data source, the execution efficiency is further improved through logical adjustment and condition pushdown. Specifically, the logical operations in the subquery are identified by the parser, and the metadata of the data source is combined to determine whether there is an optimizable structure.

[0087] For query rewriting optimization, predefined rewriting rules are applied to adjust the subquery structure: for example, nested subqueries are converted to JOIN operations to take advantage of the database's optimization support for JOIN (such as hash join and merge join) to improve efficiency; or expressions containing redundant calculations are merged to reduce redundant calculations.

[0088] For predicate pushdown optimization, filtering predicates in the subquery are identified, and it is determined whether these predicates can be pushed down to the data source for execution: if the fields involved in the predicate have an index in the data source, the predicate is pushed down to the WHERE clause of the data source, so that the data source can filter out data that does not meet the conditions when scanning the table, and only return the intermediate results that meet the conditions. If the subquery contains aggregation operations, push down the predicates that can be pushed down to execute before aggregation to reduce the amount of data processed by aggregation.

[0089] Finally, the subquery after rewriting and / or predicate pushdown optimization has a more optimal execution path and smaller data processing amount while maintaining the same semantics, forming an optimized SQL subquery statement.

[0090] Step S4003, based on the abstract query plan, the execution order and coordination logic of each SQL subquery statement are determined to form the unified SQL query scheme.

[0091] Specifically, based on the DAG dependency of the abstract query plan, the execution order of the subqueries and the coordination logic are determined: if subquery B depends on the result of subquery A, mark A as a prerequisite task; if subqueries C and D have no dependency relationship, set them to be executed in parallel. At the same time, by adding temporary result storage, data transmission interface and other coordination logic, it is ensured that the subquery results can flow as expected. Finally, these subqueries, execution orders and coordination logic are integrated into a unified SQL query plan.

[0092] In a feasible implementation, with reference to Figure 5 In the fifth embodiment of the present application, the unified SQL query plan includes the execution order and coordination logic of each SQL subquery statement, and the step S1004, i.e., the step of distributing the SQL subquery statements in the unified SQL query plan to the corresponding data sources for execution and obtaining the execution results of each data source, includes steps S5001-S5003: Step S5001, according to the execution order and coordination logic, each SQL subquery statement is dispatched and distributed to its corresponding data source; Among them, the dispatching and distribution is a process of allocating each SQL subquery statement to the corresponding heterogeneous data source according to the execution order and coordination logic defined in the unified SQL query plan, which needs to match the data source type and computing resources to ensure that the subquery is executed in the correct environment.

[0093] Specifically, the dependency relationship between subqueries (such as subquery B needs to wait for subquery A to complete) and the parallel possibility (such as subqueries C and D have no dependency) are identified, and a scheduling plan is generated in combination with the load state of the data source. Subsequently, the engine distributes the subqueries to the corresponding data sources through the data source connector.

[0094] For example, the MySQL subquery statement is sent to the MySQL database through the JDBC connection, and the PostgreSQL subquery statement is sent to the PostgreSQL cluster through its dedicated driver.

[0095] Step S5002, execute the received SQL subquery statements in parallel on the data source and monitor the execution status of each SQL subquery statement; Among them, the execution status: the real-time running status of the SQL subquery statement on the data source, including the states of "to be executed", "executing", "execution success", "execution failure" and the like, which is used to monitor the query progress and the validity of the result.

[0096] Specifically, after the distribution is completed, the sub-queries without dependency are simultaneously run in the resource-isolated container, avoiding overloading of a single data source. Meanwhile, the monitoring module collects the execution status of each sub-query in real time, i.e., the progress is obtained through the execution log interface of the data source, and the connection is judged to be normal through the heartbeat detection, if the timeout or error is found, the retry mechanism is triggered or the abnormal prompt is returned.

[0097] Step S5003, when the execution status of the data source is detected as execution success, the execution result returned by the data source is obtained.

[0098] Specifically, when the monitoring module detects that the execution status of a sub-query becomes "execution success", the engine pulls the intermediate result set (only containing the necessary data after the sub-query filtering, rather than the full table data) from the data source through a preset interface. If the sub-query execution fails, detailed logs are returned according to the error type for user or administrator to troubleshoot.

[0099] In a feasible implementation, with reference to Figure 6 In the sixth embodiment of the present application, the step S1005, i.e., the step of uniformly calculating and processing the execution results of the data sources, generating a uniform result set, and outputting the uniform result set in a preset format, includes steps S6001-S6003: Step S6001, connecting, aggregating and sorting the execution results of the data sources to generate the uniform result set; Specifically, the uniform calculation engine receives the intermediate result set returned by each data source, and performs a connection operation according to the logical association relationship defined in the global logical data model.

[0100] For example, the "user ID" of the sales table is internally connected with the "user ID" of the user table, ensuring one-to-one correspondence of data rows.

[0101] Subsequently, based on the aggregation operation requirements in the intermediate representation (IR), such as "sum of sales" and "calculate average value", the aggregated calculation is performed on the connected data, such as performing the SUM() function on the "sales" field and performing the COUNT() function on the "order quantity". Finally, according to the sorting requirements in the IR, the aggregated results are globally sorted to form a uniform result set containing complete query logic.

[0102] Step S6002, converting the uniform result set into structured data conforming to the preset format requirements, wherein the preset format includes JSON format, CSV format and API data interaction format.

[0103] Specifically, the output format preset according to the user or application scenario is used to format convert the unified result set: for example, when converting structured data into JSON format, the field name is used as the key and the field value is used as the value to generate a nested or flat JSON object; when converting into CSV format, the field values are separated by commas and the field names are written in the first line.

[0104] For example, the preset output format includes but is not limited to JSON format required by the front-end system, CSV format required by Excel import, and API format required by third-party system docking.

[0105] In step S6003, the converted unified result set is output through a preset data interface.

[0106] The preset data interface refers to a pre-defined output channel specification, such as an HTTP interface, a JDBC interface, or a file transfer protocol, which is used to direct the output of the converted result to a target application.

[0107] Specifically, if the target is a BI tool, the JDBC interface is used to directly write to its data connection pool; if the target is a front-end page, the HTTP interface is used to return in JSON format; if storage to a data warehouse is required, the FTP protocol is used to transfer a CSV file.

[0108] In addition, with reference to Figure 7 The application also provides a multi-source data analysis device, which comprises: A metadata management module 10 is configured to integrate metadata of pre-accessed heterogeneous data sources and construct a global logical data model. A semantic mapping module 20 is configured to identify semantic elements of a pre-acquired natural language query statement, map the semantic elements to corresponding logical entities in the global logical data model, and generate a structured intermediate representation. A unified generation module 30 is configured to generate a unified SQL query scheme comprising a plurality of SQL subquery statements based on the intermediate representation and SQL syntax characteristics supported by the heterogeneous data sources. An execution module 40 is configured to distribute the SQL subquery statements in the unified SQL query scheme to corresponding data sources for execution and obtain execution results of each data source. A structured output module 50 is configured to perform unified calculation and processing on the execution results of each data source, generate a unified result set, and output the unified result set in a preset format.

[0109] The multi-source data analysis device provided in the application adopts the multi-source data analysis method in the above embodiment, and aims to solve the technical problems of high complexity and high technical threshold of enterprise multi-source heterogeneous data query analysis. Compared with the prior art, the multi-source data analysis device provided in the application has the same beneficial effects as the multi-source data analysis method provided in the above embodiment, and other technical features in the multi-source data analysis device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0110] The application provides a multi-source data analysis device, which comprises at least one processor and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multi-source data analysis method in the above embodiment one.

[0111] The multi-source data analysis device in the embodiment of the application can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 8 The multi-source data analysis device shown is only an example, and should not bring any limitation to the function and use range of the embodiment of the application.

[0112] As Figure 8As shown, the multi-source data analysis device can include a processing apparatus 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a read-only memory 1002 or loaded from a storage apparatus 1003 into a random access memory 1004. Various programs and data required for operation of the multi-source data analysis device are also stored in the random access memory 1004. The processing apparatus 1001, the read-only memory 1002, and the random access memory 1004 are connected to each other by a bus 1005. An input / output interface 1006 is also connected to the bus. Generally, the following systems can be connected to the input / output interface 1006: input apparatuses 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output apparatuses 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the multi-source data analysis device to communicate wirelessly or by wire with other devices to exchange data. Although the multi-source data analysis device having various systems is shown in the figure, it should be understood that all the shown systems are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0113] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication apparatus, or installed from the storage apparatus 1003, or installed from the read-only memory 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0114] The multi-source data analysis device provided by the present disclosure adopts the multi-source data analysis method in the above embodiments, and aims to solve the technical problems of high complexity and high technical threshold of enterprise multi-source heterogeneous data query analysis. Compared with the prior art, the multi-source data analysis device provided by the present disclosure has the same beneficial effects as the multi-source data analysis method provided by the above embodiments, and other technical features in the multi-source data analysis device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0115] It should be understood that various aspects of the disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any appropriate manner in any one or more embodiments or examples.

[0116] The above description is merely illustrative of the application and is not intended to limit the scope of the application. Any variations and modifications that can be made by any person skilled in the art within the spirit and scope of the application are intended to be encompassed by the application. Therefore, the scope of the application should be determined by the appended claims.

[0117] The application provides a computer readable storage medium having stored thereon computer readable program instructions (i.e., a computer program) for performing the multi-source data analysis method in the above embodiments.

[0118] The computer readable storage medium provided by the application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system or device, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electric connection having one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), etc., or any appropriate combination thereof.

[0119] The above computer readable storage medium can be contained in the multi-source data analysis device; or can exist separately and not be assembled into the multi-source data analysis device.

[0120] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0121] The flow diagrams and the block diagrams in the drawings are meant as methodological and functional description of implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0122] The modules involved in the embodiments of the present application can be implemented in software or in hardware. In some cases, the names of the modules do not constitute a limitation on the modules themselves.

[0123] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e., computer programs) for executing the above-mentioned multi-source data analysis method, and aims to solve the technical problems of high complexity and high technical threshold of enterprise multi-source heterogeneous data query analysis. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the multi-source data analysis method provided by the above-mentioned embodiments, and will not be described here.

[0124] The application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the multi-source data analysis method as described above.

[0125] The computer program product provided by the application aims to solve the technical problems of high complexity and high technical threshold of enterprise multi-source heterogeneous data query analysis. Compared with the prior art, the beneficial effects of the computer program product provided by the application are the same as those of the multi-source data analysis method provided by the above-mentioned embodiments, and are not described here.

[0126] It should be noted that in this document, the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that processes, methods, articles or systems including a series of elements not only include those elements, but also include other elements not explicitly listed, or further include elements inherent to such processes, methods, articles or systems. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or system including the element.

[0127] The above-mentioned application example serial numbers are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) as described above, and includes a plurality of instructions for making a terminal device (which can be a mobile phone, a computer, a server, a controlled terminal, or a network device, etc.) execute the method of each embodiment of the application.

[0129] The above is only the preferred embodiment of the application, and does not limit the patent scope of the application, and any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the application.

Claims

1. A multi-source data analysis method, characterized in that, The multi-source data analysis method includes: Integrate metadata from pre-connected heterogeneous data sources to build a global logical data model; Identify the semantic elements of a pre-acquired natural language query statement, map the semantic elements to the corresponding logical entities in the global logical data model, and generate a structured intermediate representation; Based on the intermediate representation and the SQL syntax features supported by the heterogeneous data source, a unified SQL query scheme containing multiple SQL subqueries is generated. The SQL subquery statements in the unified SQL query scheme are distributed to the corresponding data sources in the heterogeneous data sources for execution, and the execution results of each data source are obtained; The execution results from each data source are processed uniformly to generate a unified result set, which is then output in a preset format.

2. The multi-source data analysis method as described in claim 1, characterized in that, The steps of integrating metadata from pre-connected heterogeneous data sources and constructing a global logical data model include: The metadata of the heterogeneous data source is scanned, extracted, and registered to obtain a set of metadata including table structure, field types, and inter-table relationships; Based on the metadata set, logical relationships between different tables in the heterogeneous data source are established to obtain the global logical data model.

3. The multi-source data analysis method as described in claim 1, characterized in that, The steps of identifying semantic elements of a pre-acquired natural language query statement, mapping the semantic elements to corresponding logical entities in the global logical data model, and generating a structured intermediate representation include: The natural language query statement is segmented, entity recognized, and semantic role labeled to parse the query intent, filtering conditions, and aggregation operations of the natural language query statement. The semantic elements contained in the query intent, filtering conditions, and aggregation operations are matched and mapped with the logical tables and fields in the global logical data model to obtain logical entities; When the semantic element involves multiple logical tables, the required table join logic is determined based on the logical relationships defined in the global logical data model. Based on the query intent, filtering conditions, aggregation operations, logical entities, and the table join logic, a structured intermediate representation is generated.

4. The multi-source data analysis method as described in claim 1, characterized in that, The step of generating a unified SQL query scheme containing multiple SQL subqueries based on the intermediate representation and the SQL syntax features supported by the heterogeneous data source includes: The intermediate representation is converted into an abstract query plan that is independent of the heterogeneous data source type; Based on the SQL syntax and functions supported by each data source in the heterogeneous data source, the abstract query plan is adapted to generate executable SQL subquery statements for each corresponding data source; Based on the abstract query plan, the execution order and coordination logic of each SQL subquery statement are determined to form the unified SQL query scheme.

5. The multi-source data analysis method as described in claim 4, characterized in that, The step of adapting the abstract query plan to generate executable SQL subquery statements for each corresponding data source based on the SQL syntax and functions supported by each data source in the heterogeneous data source includes: The executable SQL subquery statement is optimized by query rewriting and / or predicate pushdown to form an optimized SQL subquery statement.

6. The multi-source data analysis method as described in claim 1, characterized in that, The unified SQL query scheme includes the execution order and coordination logic of each SQL subquery statement. The step of distributing the SQL subquery statements in the unified SQL query scheme to the corresponding data sources in the heterogeneous data sources for execution and obtaining the execution results of each data source includes: Based on the execution order and coordination logic, each SQL subquery statement is scheduled and distributed to its corresponding data source; The received SQL subqueries are executed in parallel on the data source, and the execution status of each SQL subquery is monitored. When the execution status of the data source is detected as successful, the execution result returned by the data source is obtained.

7. The multi-source data analysis method as described in claim 1, characterized in that, The step of performing unified calculations on the execution results of each data source to generate a unified result set, and outputting the unified result set according to a preset format, includes: The execution results from each data source are connected, aggregated, and sorted to generate the unified result set; The unified result set is converted into structured data that conforms to a preset format, wherein the preset format includes JSON format, CSV format and API data interaction format; The unified result set after conversion is output through a preset data interface.

8. A multi-source data analysis device, characterized in that, The device includes: The metadata management module is used to integrate metadata from pre-connected heterogeneous data sources and build a global logical data model. The semantic mapping module is used to identify the semantic elements of the pre-acquired natural language query statement, map the semantic elements to the corresponding logical entities in the global logical data model, and generate a structured intermediate representation. A unified generation module is used to generate a unified SQL query scheme containing multiple SQL subquery statements based on the intermediate representation and the SQL syntax features supported by the heterogeneous data source. The execution module is used to distribute the SQL subquery statements in the unified SQL query scheme to the corresponding data sources in the heterogeneous data sources for execution, and obtain the execution results of each data source; The structured output module is used to perform unified calculation and processing on the execution results of the various data sources, generate a unified result set, and output the unified result set according to a preset format.

9. A multi-source data analysis device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the multi-source data analysis method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the multi-source data analysis method as described in any one of claims 1 to 7.

Images