Multimodal query unified processing system based on semantic analysis
The unified multimodal query processing system based on semantic parsing supports multiple query methods and automatically derives the optimal table join, solving the problems of low efficiency and heavy user burden in existing multi-table join queries, and achieving efficient and convenient query processing.
Patent Information
- Application Number
- CN202510986926.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-17
- Publication Date
- 2025-10-31
AI Technical Summary
Existing technologies lack a system that can uniformly handle multiple query methods, especially in multi-table join queries, where users need to explicitly specify join conditions, increasing the burden and potentially leading to suboptimal query efficiency.
A unified multimodal query processing system based on semantic parsing is adopted, including a multimodal query input module, a semantic parsing module, a multi-table join derivation module, a script generation module, and a query execution module. It supports DSL, VSL, and NL queries, automatically derives the optimal table join method, calculates the shortest connected path using the Steiner Tree algorithm, and generates the execution script.
It achieves unified processing of various query input methods, automatically derives the optimal table join, improves query efficiency, reduces user learning costs, and enhances user experience.
Smart Images
Figure CN120873166A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and more specifically, to a unified processing system for multimodal queries based on semantic parsing. Background Technology
[0002] With the increasing demand for data analysis, users are diversifying their data query methods. Common query input methods currently include Domain-Specific Languages (DSLs), Visual Query Languages (VSLs), and Natural Languages (NLs). Each query method has its advantages: DSLs are suitable for professional users, offering precise expression but with a high learning curve; VSLs are intuitive and easy to use but have limited expressive power; NLs best align with human habits but are difficult to parse.
[0003] In existing technologies, systems for handling single query methods are relatively mature, but systems capable of uniformly handling multiple query methods are lacking. Especially when dealing with multi-table joins, automatically deriving the optimal table join method becomes a technical challenge. Traditional methods require users to explicitly specify join conditions, which not only increases the user's burden but may also lead to suboptimal query efficiency. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a unified multimodal query processing system based on semantic parsing.
[0005] To address the aforementioned technical problems, the present invention adopts the following technical solution:
[0006] A unified multimodal query processing system based on semantic parsing includes:
[0007] The multimodal query input module is used to receive query requests input by users in various ways, including domain-specific language (DSL) queries, visual language (VSL) queries, and natural language (NL) queries.
[0008] The semantic parsing module is used to convert query requests from different input methods into a unified structure that represents the semantics of the query.
[0009] The multi-table join deduction module is used to deduce the join method between tables in multi-table scenarios based on the single-level logic in the query semantics, including collecting statistical information, calculating the shortest connection path, and determining the join method.
[0010] The script generation module is used to generate the final execution script based on the query semantic structure and association methods.
[0011] The query execution module is used to execute the generated script and return the query results.
[0012] As a further aspect of the present invention, the multimodal query input module also supports voice input query requests.
[0013] As a further aspect of the present invention: the semantic parsing module includes:
[0014] The DSL parsing submodule is used to parse domain-specific language queries and convert them into query semantic structures.
[0015] The VSL parsing submodule is used to parse visual language query operations and convert them into query semantic structures.
[0016] The NL parsing submodule is used to parse natural language queries and convert them into query semantic structures.
[0017] As a further aspect of the present invention: the multi-table association derivation module includes:
[0018] The statistics collection submodule is used to collect information from the filter tables, dimension tables, and measure tables involved in the query.
[0019] The shortest connected path calculation submodule is used to construct a graph or tree model based on the relationships between tables and calculate the shortest connected path using the Steiner Tree algorithm.
[0020] The association method determination submodule is used to determine the table join method based on the shortest connection path and apply deduplication operation when necessary.
[0021] As a further aspect of the present invention: the association method determines that the submodule performs deduplication in the N:* scenario when the filter table and the measure table are N:*, and the deduplication operation includes implementation using GROUP BY or DISTINCT syntax.
[0022] As a further aspect of the present invention: the script generation module supports the integration of multi-level query logic and generates the final execution script based on the template.
[0023] As a further aspect of the present invention, the execution engine module supports multiple database engines and can adapt to different data storage environments.
[0024] Compared with the prior art, the advantages of this invention are:
[0025] This solution supports multiple query input methods to meet the needs of different users; it automatically derives the optimal table join method to improve query efficiency; it uses a unified semantic representation to facilitate system expansion and maintenance; and it reduces the learning cost for users and enhances the user experience. Attached Figure Description
[0026] Figure 1 This is a system diagram of the present invention;
[0027] Figure 2 This is a system diagram of the semantic parsing module of the present invention;
[0028] Figure 3 This is a system diagram of the multi-table association derivation module of the present invention;
[0029] Figure 4 This is a flowchart of the present invention;
[0030] Figure 5 This is an example diagram of an embodiment of the present invention.
[0031] Explanation of the labels in the diagram: Detailed Implementation
[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0033] Please see Figure 1-3 A unified multimodal query processing system based on semantic parsing includes a multimodal query input module, a semantic parsing module, a multi-table join derivation module, a script generation module, and a query execution module. The multimodal query input module receives various forms of query requests; the semantic parsing module converts different queries into a unified semantic structure; the multi-table join derivation module automatically determines the optimal table join method; and the script generation module and query execution module generate and execute the final query script.
[0034] Specifically: The multimodal query input module is used to receive query requests from users through various methods, including:
[0035] a. Queries can be performed using simple expressions input via a domain-specific parsing language (DSL);
[0036] b. Input queries visually (VSL), such as by dragging and dropping, configuring reports, and retrieving data based on the reports;
[0037] c. Querying via natural language input (NL), such as NL querying via Chinese, i.e., NLQ;
[0038] The input methods for data queries include, but are not limited to, the above-mentioned methods. For example, voice input can be used as the input for data retrieval.
[0039] The semantic parsing module is used to convert query requests from different input methods into a unified structure representing the semantics of the query. The semantic parsing module includes:
[0040] The DSL parsing submodule is used to parse domain-specific language queries and convert them into query semantic structures.
[0041] The VSL parsing submodule is used to parse visual language query operations and convert them into query semantic structures.
[0042] The NL parsing submodule is used to parse natural language queries and convert them into query semantic structures.
[0043] The multi-table join derivation module is used in multi-table scenarios to deduce the join methods between tables based on single-level logic in the query semantics. This includes collecting statistical information, calculating the shortest connection path, and determining the join method. It should be noted that the smallest storage unit of data may be a table; for ease of understanding, this term will be used to refer to the storage unit as a table. Query semantics typically involve both multi-level and single-level logic, but multi-level logic can be broken down into single-level logic and then merged, ultimately resulting in single-level logic. Therefore, this article focuses on the derivation of single-level logic. The multi-table join derivation module includes:
[0044] The statistics collection submodule is used to collect information from the filter tables, dimension tables, and measure tables involved in the query.
[0045] Example:
[0046] The tables involved in the filtering:
[0047] Value filtering involves tables, for example: average transaction amount for cities equal to Shanghai -> table containing the city;
[0048] The table to be filtered by dimension is, for example: average transaction amount in each city -> table of the city.
[0049] The tables involved in the measurement:
[0050] The table used for measurement design, for example: the average transaction amount for cities equal to Shanghai -> the table containing the transaction amount.
[0051] The shortest connected path calculation submodule is used to construct a graph or tree model based on the relationships between tables and calculate the shortest connected path using the Steiner Tree algorithm.
[0052] The process for obtaining the shortest connected path between the tables involved in the query is as follows:
[0053] Suppose there are n tables containing "data to be queried," and these tables may have relationships or related fields. Based on these relationships, a "model" is constructed, which can be a graph (with cycles) or a tree (without cycles). The relationships between tables may include, but are not limited to: ① Association fields describing the intersection of the two tables, such as the customer ID being the association field between the customer table and the order table. ② Relationships describing the relationship between corresponding entities in the two tables, which can be one of four types (1:1, 1:N, N:1, N:N); for example, the relationship between the customer table and the order table is 1:N, meaning one customer can purchase multiple orders.
[0054] The shortest connected path is obtained based on the graph / tree constructed from tables and the relationships between them, and the tables involved in the query semantics. Methods for obtaining the shortest connected path include, but are not limited to: ① using the Steiner Tree algorithm to find a spanning tree in the graph, connecting a specified number of points with the minimum total edge weight; ② using the table containing the metric as the main table and finding the shortest connection path. Furthermore, there may be only one shortest connected path, or multiple shortest connected paths, which the user can choose.
[0055] The subtree formed by the shortest connected path contains tables and the relationships between tables, which constitute the context of the "metric" query. In a single query, the context of all metrics is consistent.
[0056] The join method determination submodule is used to determine the table join method based on the shortest connection path and apply deduplication operations when necessary. When the table containing the filter and the table containing the measure are N:*(N:1 / N:N), deduplication operations are required. Deduplication operations include, but are not limited to, using the GROUP BY or DISTINCT syntax.
[0057] The script generation module is used to generate the final execution script based on the query semantic structure and association methods. Specifically, the script generation module supports the integration of multi-level query logic and generates the final execution script based on a template.
[0058] The query execution module executes the generated scripts and returns the query results. The execution engine module supports multiple database engines, enabling it to adapt to different data storage environments.
[0059] like Figure 4 As shown, based on the above, the present invention also provides a unified processing method for multimodal queries based on semantic parsing, including the following steps:
[0060] I. Data Retrieval Scenario Input: Users input and query "data to be retrieved". The query methods include, but are not limited to, DSL query, VSL query and NL query.
[0061] II. Transforming user input into a structure that can represent the semantics of the query, including but not limited to:
[0062] The DSL will parse the DSL syntax and then transform it into a structure that represents the semantics of the query.
[0063] VSL will transform the semantics revealed by visualization into a structure that represents the semantics of the query;
[0064] NL will convert the parsed semantics into a structure that represents the query semantics.
[0065] III. Derivation of JOIN Method in Single-Layer Scenarios with Multiple Tables: In multi-table scenarios, the SemiJoin method is adopted to derive the optimal table join method based on the inter-table relationships and query semantics.
[0066] IV. Integrate multi-level queries and generate the final execution script based on the version.
[0067] 5. Generate and execute the final query script, and return the query results.
[0068] Example:
[0069] like Figure 5 As shown, taking "Checking the account balance of a customer who purchased a computer" as an example:
[0070] Users input queries using natural language;
[0071] The NL parsing submodule identifies "computer" as the filter condition, "customer" as the dimension, and "account balance" as the metric.
[0072] The statistical information collection submodule identifies the order table (filter table) and customer table (metric table) involved;
[0073] The shortest path calculation unit found that the two tables are related by customer ID, with a relationship of 1:N;
[0074] The association method determination submodule uses the SemiJoin method to connect the two tables and deduplicates customer IDs.
[0075] Generate and execute the SQL script, and return the result. The SQL is as follows:
[0076] SELECT SUM(customer table.account balance) FROM;
[0077] (SELECT Customer FROM Order Table Where Product = 'Computer' GROUP BY Customer)TMP, Customer Table;
[0078] Where customer_table.customer = TMP.customer;
[0079] The above description is merely a preferred embodiment of the present invention; however, the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and its improved concepts, should be covered within the scope of protection of the present invention.
Claims
1. A unified multimodal query processing system based on semantic parsing, characterized in that, include: A multimodal query input module is used to receive query requests input by users in various ways, including DSL queries, VSL queries, and NL queries; The semantic parsing module is used to convert query requests from different input methods into a unified structure that represents the semantics of the query. The multi-table join deduction module is used to deduce the join method between tables in multi-table scenarios based on the single-level logic in the query semantics, including collecting statistical information, calculating the shortest connection path, and determining the join method. The script generation module is used to generate the final execution script based on the query semantic structure and association methods. The query execution module is used to execute the generated script and return the query results.
2. The multimodal query unified processing system based on semantic parsing according to claim 1, characterized in that: The multimodal query input module also supports voice input query requests.
3. The multimodal query unified processing system based on semantic parsing according to claim 1, characterized in that: The semantic parsing module includes: The DSL parsing submodule is used to parse domain-specific language queries and convert them into query semantic structures. The VSL parsing submodule is used to parse visual language query operations and convert them into query semantic structures. The NL parsing submodule is used to parse natural language queries and convert them into query semantic structures.
4. The multimodal query unified processing system based on semantic parsing according to claim 1, characterized in that: The multi-table association derivation module includes: The statistics collection submodule is used to collect information from the filter tables, dimension tables, and measure tables involved in the query. The shortest connected path calculation submodule is used to construct a graph or tree model based on the relationships between tables and calculate the shortest connected path using the Steiner Tree algorithm. The association method determination submodule is used to determine the table join method based on the shortest connection path and apply deduplication operation when necessary.
5. The multimodal query unified processing system based on semantic parsing according to claim 4, characterized in that: The association method determines that the submodule performs deduplication in the N:* scenario when the filter table and the measure table are N:*. The deduplication operation includes using the GROUP BY or DISTINCT syntax.
6. The multimodal query unified processing system based on semantic parsing according to claim 1, characterized in that: The script generation module supports the integration of multi-level query logic and generates the final execution script based on the template.
7. The multimodal query unified processing system based on semantic parsing according to claim 1, characterized in that: The execution engine module supports multiple database engines and can adapt to different data storage environments.