Method and system for realizing intelligent number asking based on large model

Abstract

The invention relates to the technical field of natural language processing, in particular to a method and system for achieving intelligent number asking based on a large model, and the method comprises the following steps: integrating data from various data sources including a database, a data warehouse, WEB service and Excel files; the data is cleaned, converted, standardized and normalized through the data management center, so that the data quality is ensured, and basic data support is provided for subsequent intelligent question analysis; the beneficial effects are that the method for realizing intelligent number asking based on the large model provided by the invention responds to complex and changeable data requirements in real time, helps an enterprise to quickly make a data-based decision, and reduces the time cost of waiting for the assistance of an IT department or a data analysis team; through effective integration and intelligent search of mass data resources, various data in an enterprise can be fully mined and utilized, and idleness and waste of the data resources are avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and system for implementing intelligent questioning based on a large model. Background Art

[0002] With the advent of the big data era and the rapid advancement of artificial intelligence technology, a large amount of data resources have accumulated within enterprises, but how to effectively extract and analyze this data and convert it into valuable business insights has become a challenge.

[0003] Although traditional business intelligence (BI) tools can provide data visualization and basic data analysis functions, they have limitations when dealing with complex and changeable natural language query requirements raised by non-professionals. Users often need to have certain data analysis skills to make full use of them. Summary of the invention

[0004] The purpose of the present invention is to provide a method and system for implementing intelligent numerical calculation based on a large model to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for realizing intelligent number-finding based on a large model, the method comprising the following steps:

[0006] Integrate data from multiple data sources, including databases, data warehouses, web services, and Excel files;

[0007] The data governance center cleans, converts, standardizes and normalizes the data to ensure data quality and provide basic data support for subsequent intelligent data analysis.

[0008] Preferably, it also includes semantic modeling and knowledge graph construction based on the big model:

[0009] Conduct dimensional modeling for specific subject domains, build summary models, and set dimensions, indicators, and date fields;

[0010] Graph databases and vector databases are used to store data. By constructing a set of triples in the form of <entity, relationship, entity>, the system's semantic understanding ability is enhanced to build a knowledge graph under the subject domain.

[0011] Preferably, it also includes intelligent question rule parsing and SQL generation based on a large model:

[0012] A rich set of rules is used to parse user questions, extract key elements such as indicators, dimensions, and time conditions, and calculate intent scores;

[0013] If the intent score is higher than the preset threshold, a SQL query statement is directly generated through rule parsing; otherwise, the embedded large model based on the Transformer architecture is called for further processing to generate a SQL query statement.

[0014] Preferably, it also includes an intelligent question-answering engine and data processing based on a large model:

[0015] Deploy one or more intelligent question-answering engines based on large models to receive natural language questions input by users;

[0016] The large model in the intelligent question-answering engine understands and infers the query intent of the user's question and generates general logical SQL.

[0017] According to the type of the underlying database, the general logical SQL is converted into the corresponding physical SQL, and the data interface is called to perform the query operation;

[0018] Establish an efficient data indexing system and use vector retrieval technology to improve query efficiency on large-scale data sets;

[0019] The query results are converted into visual charts and text summaries, and fed back to users through dynamic visualization components.

[0020] Preferably, it also includes: a dynamic feedback and learning mechanism, which continuously optimizes the performance and accuracy of the large model by collecting user feedback to adapt to different user habits and business scenarios;

[0021] When processing sensitive data, encryption technology is used to desensitize the data and strict access control policies are implemented to ensure data security and effective protection of user privacy in the intelligent question-answering service;

[0022] Provides flexible configuration options, allowing administrators to adjust the system's security policies and privacy protection settings according to actual needs.

[0023] An intelligent number-asking system based on a large model, the system comprising:

[0024] Data integration module, used to introduce external data sources and integrate data from various data sources such as databases, data warehouses, WEB services, and Excel;

[0025] The data governance center is used to clean, convert, standardize and normalize the integrated data to ensure data quality and provide a reliable data foundation for subsequent intelligent data analysis.

[0026] Preferably, it also includes:

[0027] The semantic modeling module is used to perform dimensional modeling on data tables under a specific subject domain, build summary models, and set dimensions, indicators, and date fields;

[0028] The knowledge graph construction module is used to construct the association relationship between models. It uses graph database and vector database to store data. By constructing a set of triples in the form of <entity, relationship, entity>, it enhances the system's semantic understanding ability and thus builds a knowledge graph under the subject domain.

[0029] Preferably, it also includes:

[0030] The rule parsing module uses a rich set of rules to extract key elements of user questions and parse their intent. The extracted key elements include indicators, dimensions, time conditions, dimension values, aggregation conditions, and sorting conditions.

[0031] The SQL generation module calculates the intent score based on the rule parsing results. If the score is higher than the preset threshold, an SQL query statement is generated through rule parsing; otherwise, the embedded big model is called to generate the query SQL. The big model is based on the Transformer architecture and is fine-tuned for specific government data query contexts.

[0032] Preferably, it also includes:

[0033] The natural language understanding module uses an advanced and large-scale trained language model, and adapts to the intelligent question-asking scenario through fine-tuning technology. It uses NLU technology to accurately understand the natural language questions input by users and extract keywords and entities.

[0034] Data processing and indexing module, which establishes an efficient data indexing system, pre-processes and organizes the original data to improve query efficiency;

[0035] The intelligent question-answering engine contains one or more large models, which are used to receive user questions, identify query intent through understanding and reasoning, generate logical SQL, and convert it into physical SQL to execute queries based on the database type. Finally, the query results are converted into visual charts and text summaries and fed back to the user.

[0036] Preferably, it also includes:

[0037] Dynamic feedback and learning mechanism, which continuously optimizes the performance and accuracy of large models through user feedback to improve question-answering accuracy and user experience;

[0038] Security and privacy protection mechanism: encryption technology is used for desensitization when processing sensitive data, and strict access control policies are implemented to ensure that data security and user privacy are effectively protected.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] The method and system for realizing intelligent number-finding based on a large model proposed in the present invention have the following effects:

[0041] (1) Improve work efficiency: Users do not need to master complex query syntax. They can obtain the required data by simply asking questions in natural language, which greatly reduces the threshold for data query and improves the work efficiency of employees at all levels.

[0042] (2) Data-driven decision acceleration: Respond to complex and ever-changing data needs in real time, help enterprises make data-based decisions quickly, and reduce the time cost of waiting for assistance from the IT department or data analysis team;

[0043] (3) Improved data resource utilization: Through the effective integration and intelligent search of massive data resources, various types of data within the enterprise can be fully mined and utilized, avoiding idleness and waste of data resources;

[0044] (4) Enhanced accuracy and consistency: Through the understanding ability of large models and precise query technology, the accuracy of data retrieval can be improved and the consistency of data interpretation across departments and business scenarios can be ensured;

[0045] (5) Adaptive learning and optimization: The system has self-learning capabilities. With the use and feedback of users, the large model can continuously optimize its own understanding and generation capabilities, and continuously improve service quality;

[0046] (6) Lowering the threshold for data analysis: New employees or those who are not familiar with the data system can quickly get started with query work, reducing the time and cost of training specifically for the data query system. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] For example, see Figure 1 The present invention provides a technical solution: a method for realizing intelligent numerical calculation based on a large model, the method comprising the following steps:

[0050] S1. Data integration. Introduce external data sources and integrate data from multiple data sources, including databases, data warehouses, WEB services, Excel, etc., and perform data cleaning, conversion, standardization and normalization through the data governance center to ensure data quality;

[0051] S2. Semantic modeling, building a subject domain knowledge graph. Perform dimension modeling on the data table under the subject domain, build a summary model, set dimensions, indicators, and date fields, construct the relationship between models, and use graph databases and vector databases for storage to enhance the system's semantic understanding capabilities. The formula uses a set of constructed triples: <entity, relationship, entity>, for example: (data directory A, belongs to department, department B);

[0052] S3. Rule parsing. The intelligent question system uses a rich set of rules to extract key elements and analyze intent. The extracted key elements include indicators, dimensions, time conditions, dimension values, aggregation conditions, sorting conditions, etc. Intent recognition is performed according to the matched key elements, and the intent score is calculated. If the score is higher than the 0.8 threshold, the rule recognition is considered to be passed, and SQL is generated through rule parsing. Otherwise, the query SQL is generated by calling the embedded large model.

[0053] S4. Embedding and fine-tuning of large models. The advanced pre-trained model based on the Transformer architecture, which performs well in natural language processing tasks, is embedded in the Hairuo large model and fine-tuned for specific government data query contexts. This step ensures that the large model can understand and accurately answer natural language questions related to internal government data. Knowledge enhancement is performed on the large model, and the model understands the data table structure, field meaning, and business logic by injecting structured and unstructured data from the internal government database;

[0054] S5, Natural Language Understanding Module. The system uses the advanced and large-scale trained Hairuo language model, and uses Fine-tuning technology to adapt it to the intelligent question-asking scenario. It uses NLU technology to accurately understand the natural language questions input by users, identify user query intentions, extract keywords and entities from user questions, such as indicators, time, location, dimensions, limiting conditions and other elements, and convert them into structured SQL query statements or data analysis instructions.

[0055] S6, data processing and indexing module. The system has established an efficient data indexing system to pre-process and organize the original data, which is convenient for fast retrieval and analysis of large models. At the same time, Havenask vector retrieval technology is used to improve the query efficiency on large-scale data sets.

[0056] S7. Intelligent question-answering engine driven by large models. The engine contains one or more large models. After receiving the user's natural language questions, the model identifies the query intent of the user's questions through understanding and reasoning, and generates general logical SQL. The intelligent question-answering system supports external databases of various database types. According to the different database types, the general logical SQL is converted into physical SQL in the syntax format of the indicator library database, and the underlying data interface is called to perform query operations. The query results are further processed and interpreted by the model, and converted into easy-to-understand visual charts + text summaries, which are fed back to users through dynamic visualization components.

[0057] S8. Dynamic feedback and learning mechanism: The system has the ability to learn through user feedback loops. It continuously optimizes model performance and accuracy through user feedback, making it more adaptable to user habits and business scenarios, and improving question-answering accuracy and user experience.

[0058] S9. Security and privacy protection mechanism: When processing sensitive data, the system uses encryption technology to desensitize the data and implements strict access control policies to ensure that data security and user privacy are effectively protected while providing intelligent question and answer services.

[0059] Example 2, based on Example 1, proposes that: users can ask questions such as "What is the GDP trend of Jinan in 2024" through natural language. The intelligent question system will first extract key elements: [2024 (time condition), Jinan (dimension, dimension value), GDP (indicator), trend (time dimension)], perform intent recognition based on the matched key elements, and parse this question into an SQL query statement.

[0060] For example: `select MONTH(create_time)as month,city,sum(GDP)as GDP from tbl_city_gdp where city='Jinan City'and YEAR(create_time)=2024group by MONTH(create_time),city`

[0061] The query SQL is executed by the real-time analysis data warehouse and the results are calculated, which are finally presented to the user in the form of charts or tables. At the same time, the system will continuously optimize its ability to understand and answer similar questions based on user feedback, and recommend similar historical questions or generate new questions.

[0062] Embodiment 3, based on embodiment 1, proposes a system for implementing intelligent number-asking based on a large model, the system comprising:

[0063] Data integration module, used to introduce external data sources and integrate data from various data sources such as databases, data warehouses, WEB services, and Excel;

[0064] The data governance center is used to clean, convert, standardize and normalize the integrated data to ensure data quality and provide a reliable data foundation for subsequent intelligent data analysis.

[0065] The semantic modeling module is used to perform dimensional modeling on data tables under a specific subject domain, build summary models, and set dimensions, indicators, and date fields;

[0066] The knowledge graph construction module is used to construct the association relationship between models. It uses graph database and vector database to store data. By constructing a set of triples in the form of <entity, relationship, entity>, it enhances the system's semantic understanding ability and thus builds a knowledge graph under the subject domain.

[0067] The rule parsing module uses a rich set of rules to extract key elements of user questions and parse their intent. The extracted key elements include indicators, dimensions, time conditions, dimension values, aggregation conditions, and sorting conditions.

[0068] The SQL generation module calculates the intent score based on the rule parsing results. If the score is higher than the preset threshold, an SQL query statement is generated through rule parsing; otherwise, the embedded big model is called to generate the query SQL. The big model is based on the Transformer architecture and is fine-tuned for specific government data query contexts.

[0069] The natural language understanding module uses an advanced and large-scale trained language model, and adapts to the intelligent question-asking scenario through fine-tuning technology. It uses NLU technology to accurately understand the natural language questions input by users and extract keywords and entities.

[0070] Data processing and indexing module, which establishes an efficient data indexing system, pre-processes and organizes the original data to improve query efficiency;

[0071] The intelligent question-answering engine contains one or more large models, which are used to receive user questions, identify query intent through understanding and reasoning, generate logical SQL, and convert it into physical SQL to execute queries based on the database type. Finally, the query results are converted into visual charts and text summaries and fed back to the user.

[0072] Dynamic feedback and learning mechanism, which continuously optimizes the performance and accuracy of large models through user feedback to improve question-answering accuracy and user experience;

[0073] Security and privacy protection mechanism: encryption technology is used for desensitization when processing sensitive data, and strict access control policies are implemented to ensure that data security and user privacy are effectively protected.

[0074] Embodiment 4, based on embodiment 3, proposes an intelligent number-asking system based on a large model, the system comprising:

[0075] Large model embedding and optimization, supporting pluggable embedding of third-party large models;

[0076] a natural language processing module configured to receive a question about a database query posed by a user in a natural language form;

[0077] Supports correction and optimization of SQL converted from natural language of large models to form queryable physical SQL;

[0078] A database interface module, used to execute the SQL query statement and retrieve data from the database;

[0079] A result display module presents the search results to users in a user-friendly format.

[0080] The large model uses pre-training and fine-tuning techniques to enhance the ability to understand domain-specific vocabulary and sentence patterns in natural language queries.

[0081] The SQL generation module includes an SQL template matching mechanism and a dynamic SQL construction algorithm. The algorithm adapts to various SQL query modes according to different database types based on the semantic analysis results provided by the large model.

[0082] A feedback loop mechanism is included, which allows the system to continuously learn and optimize the large model through user feedback on query results, so as to improve the accuracy and performance of natural language to SQL conversion.

[0083] Embodiment 5, refer to the attached Figure 1 As shown in the figure, the user sends a natural language query request, the natural language understanding is matched with the knowledge base rules, the system knowledge base is self-trained, the knowledge base rule matching degree is calculated, and SQL is generated according to the system built-in rules. If the rule parsing score is lower than the score, the LLM large model natural language is converted to SQL, and the underlying OLAP engine is called to execute the query SQL, and the query results and visualization charts are constructed.

[0084] The specific process is as follows:

[0085] 1) The user sends a natural language data query request through a conversational window;

[0086] 2) The system refreshes the knowledge base in real time and converts the index model word segmentation into word vectors for comparison with the natural language input by the user to calculate the rule matching degree;

[0087] 3) When the knowledge base rule matching degree exceeds 80%, the system built-in rules are used to generate SQL, and the query SQL is assembled according to the matching indicators, dimensions, and table names;

[0088] 4) When the knowledge base rule matching degree does not exceed 80%, the integrated LLM big model is used to convert natural language into SQL. The big model technology is used to conduct in-depth semantic analysis of user questions, identify key elements such as query entities (such as table names, column names), relationships, restrictions and query intent, and the SQL generation module dynamically constructs the corresponding SQL query statement based on the analysis results and the context association ability and logical reasoning ability of the big model;

[0089] 5) Correct the SQL in combination with the system's own knowledge base and convert it into physical SQL that can be executed in the OLAP engine;

[0090] 6) Execute the corrected query SQL with the OLAP database system, parse the query results, build natural language, visual charts or lists according to the result type, and respond to the conversational BI front-end page.

[0091] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for implementing intelligent number-finding based on a large model, characterized in that: The method comprises the following steps: Integrate data from multiple data sources, including databases, data warehouses, web services, and Excel files; The data governance center cleans, converts, standardizes and normalizes the data to ensure data quality and provide basic data support for subsequent intelligent data analysis.

2. The method for realizing intelligent numerical calculation based on a large model according to claim 1, characterized in that: It also includes semantic modeling and knowledge graph construction based on large models: Conduct dimensional modeling for specific subject domains, build summary models, and set dimensions, indicators, and date fields; Graph databases and vector databases are used to store data. By constructing a set of triples in the form of <entity, relationship, entity>, the system's semantic understanding ability is enhanced to build a knowledge graph under the subject domain.

3. The method for realizing intelligent numerical calculation based on a large model according to claim 1, characterized in that: It also includes intelligent question rule parsing and SQL generation based on large models: A rich set of rules is used to parse user questions, extract key elements such as indicators, dimensions, and time conditions, and calculate intent scores; If the intent score is higher than the preset threshold, a SQL query statement is directly generated through rule parsing; otherwise, the embedded large model based on the Transformer architecture is called for further processing to generate a SQL query statement.

4. The method for realizing intelligent numerical calculation based on a large model according to claim 1, characterized in that: It also includes intelligent question-answering engine and data processing based on large models: Deploy one or more intelligent question-answering engines based on large models to receive natural language questions input by users; The large model in the intelligent question-answering engine understands and infers the query intent of the user's question and generates general logical SQL. According to the type of the underlying database, the general logical SQL is converted into the corresponding physical SQL, and the data interface is called to perform the query operation; Establish an efficient data indexing system and use vector retrieval technology to improve query efficiency on large-scale data sets; The query results are converted into visual charts and text summaries, and fed back to users through dynamic visualization components.

5. The method for realizing intelligent numerical calculation based on a large model according to claim 1, characterized in that: Also includes: Dynamic feedback and learning mechanism, by collecting user feedback, continuously optimizes the performance and accuracy of large models to adapt to different user habits and business scenarios; When processing sensitive data, encryption technology is used to desensitize the data and strict access control policies are implemented to ensure data security and effective protection of user privacy in the intelligent question-answering service; Provides flexible configuration options, allowing administrators to adjust the system's security policies and privacy protection settings according to actual needs.

6. A system for realizing intelligent mathematical calculation based on a large model according to the method for realizing intelligent mathematical calculation based on a large model according to any one of claims 1 to 5, characterized in that: The system comprises: Data integration module, used to introduce external data sources and integrate data from various data sources such as databases, data warehouses, WEB services, and Excel; The data governance center is used to clean, convert, standardize and normalize the integrated data to ensure data quality and provide a reliable data foundation for subsequent intelligent data analysis.

7. The intelligent questioning system based on a large model according to claim 6 is characterized by: Also includes: The semantic modeling module is used to perform dimensional modeling on data tables under a specific subject domain, build summary models, and set dimensions, indicators, and date fields; The knowledge graph construction module is used to construct the association relationship between models. It uses graph database and vector database to store data. By constructing a set of triples in the form of <entity, relationship, entity>, it enhances the system's semantic understanding ability and thus builds a knowledge graph under the subject domain.

8. The intelligent questioning system based on a large model according to claim 6 is characterized by: Also includes: The rule parsing module uses a rich set of rules to extract key elements of user questions and parse their intent. The extracted key elements include indicators, dimensions, time conditions, dimension values, aggregation conditions, and sorting conditions. The SQL generation module calculates the intent score based on the rule parsing results. If the score is higher than the preset threshold, an SQL query statement is generated through rule parsing; otherwise, the embedded big model is called to generate the query SQL. The big model is based on the Transformer architecture and is fine-tuned for specific government data query contexts.

9. The intelligent questioning system based on a large model according to claim 6 is characterized by: Also includes: The natural language understanding module uses an advanced and large-scale trained language model, and adapts to the intelligent question-asking scenario through fine-tuning technology. It uses NLU technology to accurately understand the natural language questions input by users and extract keywords and entities. Data processing and indexing module, which establishes an efficient data indexing system, pre-processes and organizes the original data to improve query efficiency; The intelligent question-answering engine contains one or more large models, which are used to receive user questions, identify query intent through understanding and reasoning, generate logical SQL, and convert it into physical SQL to execute queries based on the database type. Finally, the query results are converted into visual charts and text summaries and fed back to the user.

10. The intelligent questioning system based on a large model according to claim 6, characterized in that: Also includes: Dynamic feedback and learning mechanism, which continuously optimizes the performance and accuracy of large models through user feedback to improve question-answering accuracy and user experience; Security and privacy protection mechanism: encryption technology is used for desensitization when processing sensitive data, and strict access control policies are implemented to ensure that data security and user privacy are effectively protected.

Images