Intelligent data analysis method and device fusing AI and BI

Abstract

The invention discloses an intelligent data analysis method and device fusing AI and BI. The method comprises the steps that a question input by a user in a natural language through a BI tool is received; supervising and finely adjusting the large model; carrying out preference learning on the large model after supervision and fine tuning; the large model based on preference learning can generate SQL statements by inputting user questions and database table structures; and querying according to the generated SQL statement to generate a visual chart. The device can run all processes of the AI and BI fused data intelligent analysis method. According to the method, some bottleneck problems in traditional BI are solved, an intelligent SQL generation tool is adopted, the use threshold is lowered, a user can ask questions through a simple natural language and rapidly obtain a required data analysis result, and the flexibility and the intelligent level are improved.

Description

Technical Field

[0001] The present invention relates to the field of BI data analysis technology, and in particular to a data intelligent analysis method integrating AI and BI. Background Art

[0002] In today's data-driven business environment, enterprises have a growing demand for data analysis, and Business Intelligence (BI) is a bridge between data and decision-making, and its importance is self-evident. However, although traditional BI systems play an important role in enterprise data management and analysis, they also have some limitations and challenges:

[0003] (1) Static reports and lack of flexibility

[0004] Traditional BI visualization tools often rely on cumbersome processes such as users dragging components, creating dashboards and reports, and these contents are usually static and difficult to quickly adjust or customize according to user needs, which limits the efficiency and convenience of data analysis.

[0005] (2) Complexity and high barriers to entry

[0006] Traditional BI tools usually require users to have a certain technical background, including SQL query skills, data modeling knowledge, etc., which leads to a high threshold for non-technical personnel to use and limits the popularity and daily use of BI tools.

[0007] (3) Data delay and lack of real-time performance

[0008] There may be a significant time lag in the transmission, processing and updating of data from the source system to the BI platform, resulting in untimely data for decision-making. For business scenarios that require immediate response, such as financial markets or customer service, the delay has a great impact.

[0009] In recent years, artificial intelligence (AI) has developed rapidly, especially big model technology, which has attracted wide attention in the industry for its higher controllability, wider application scenarios, and rapid growth in the number and quality of models. For example, ChatGPT, iFlytek big model, Tongyi Qianwen, etc., have not only changed the way of human-computer interaction, but also redefined the use and computing form of products by vectorizing data, code and language, enabling users to directly interact with products in natural language and reduce their dependence on traditional mouse operations. The purpose of using big model technology goes beyond simple text generation. The key is to use these models to promote the fundamental reconstruction of BI infrastructure. However, big model technology also has certain limitations. For example, untrained big models lack adaptability and cannot deeply understand business logic changes. In addition, big models have certain risks of inducing illusions and malicious behavior. Therefore, when applying big models to the BI field, it is particularly necessary to pay attention to the training of models in terms of industry knowledge and product knowledge, to ensure that the model can strengthen its understanding of the industry based on the industry scenario corpus data of specific customers, and deeply integrate with BI products, such as in report configuration, selection of operation controls, and selection of data. Summary of the invention

[0010] The purpose of the present invention is to provide a data intelligent analysis method and device that integrates AI and BI, aiming to use big model technology to enhance traditional BI tools, realize the effective integration of BI and AI technologies, convert input natural language into intelligent SQL statements, enhance the degree of intelligence, and enable users to directly interact with the product using natural language.

[0011] To achieve the purpose of the present invention, the present invention provides a data intelligent analysis method integrating AI and BI, the method comprising the following steps:

[0012] Step 1: Receive questions input by users in natural language through BI tools;

[0013] Step 2: Fine-tune the large model;

[0014] According to the user question Q, the corresponding database table structure S and the manually constructed fine-tuning data set D, the original large model is fine-tuned to obtain a large model after supervised fine-tuning. The large model after supervised fine-tuning can generate SQL query statements according to the user's input;

[0015] Step 3: Perform preference learning based on the supervised fine-tuned large model obtained in step 2;

[0016] Improve the supervised fine-tuned large model by aligning the feedback of the SQL query execution generated by the supervised fine-tuned model with the contribution of the ground truth SQL;

[0017] Step 4: The large model based on preference learning can adapt to relevant business problems and generate SQL statements by inputting new user questions and database table structures.

[0018] Step 5: Query based on the generated SQL statement and generate a visual chart.

[0019] Furthermore, the most relevant database table structure S is extracted by semantic analysis, and the database table structure S is defined as:

[0020]

[0021] Where T i represents the i-th table in the database, C ij Representation table T i The jth column in m i Representation table T i The number of columns to include.

[0022] Furthermore, the step 3 carries out preference learning in the following manner:

[0023] Step (1) Generate SQL queries using the fine-tuned large model. For each instance in the training set, obtain a predicted SQL. These predicted SQL query sets are used to generate SQL queries using R gen express;

[0024] Step (2) Use manual annotation to obtain the ground truth of SQL in the existing text real ;

[0025] Step (3) defines an indicator function to evaluate the execution result. The indicator function is defined as follows:

[0026]

[0027] in, Predict the value R for SQL gen , R is the SQL base truth value R in the ready-made text real ;

[0028] Step (4) uses the indicator function As a condition for labeling preferences, preference learning is performed to improve the large model after supervised fine-tuning.

[0029] According to another aspect of the present invention, a data intelligent analysis device integrating AI and BI is provided, which mainly comprises: a BI demand receiving module, an intelligent SQL generation module based on a large model, and an intelligent data visualization display module;

[0030] The BI requirement receiving module is used to receive questions input by the user in natural language through the BI tool;

[0031] The intelligent SQL generation module based on the big model mainly includes three submodules: a big model supervised fine-tuning submodule, a preference learning submodule and an SQL statement generation submodule. The big model supervised fine-tuning submodule takes the user question Q, the database table structure S and the manually constructed fine-tuning data set D as input, and supervises and fine-tunes the original big model to obtain a supervised fine-tuned big model. The supervised fine-tuned big model can generate SQL query statements according to the user's input; the preference learning submodule improves the supervised fine-tuned big model by ensuring that the feedback of the execution of the SQL query statement generated by the supervised fine-tuned big model according to the user's input is consistent with the contribution of the basic fact SQL; the SQL statement generation submodule can adapt to relevant business problems based on the big model after preference learning, and can generate SQL statements by inputting new user questions and database table structures;

[0032] The intelligent data visualization display module performs queries based on the generated SQL statements to generate visualization charts.

[0033] Furthermore, in the large model supervised fine-tuning submodule, semantic parsing is used to extract the most relevant database table structure S through the questions asked by the user. The database table structure S is defined as:

[0034]

[0035] Where T i represents the i-th table in the database, C ij Representation table T i The jth column in m i Representation table T i The number of columns to include.

[0036] Furthermore, the specific implementation method of the preference learning submodule is as follows:

[0037] Step 1. A fine-tuned large model is obtained through the supervised fine-tuning submodule. The trained supervised fine-tuning model is used to generate SQL queries. For each instance in the training set, a predicted SQL is obtained. The set of these SQL queries is used in R gen express;

[0038] Step 2. Obtain the ground truth of SQL in the ready-made text using manual annotations real ;

[0039] Step 3. Define an indicator function to evaluate the execution result. The indicator function is defined as follows:

[0040]

[0041] in, Predict the value R for SQL gen , R is the SQL base truth value R in the ready-made text real ;

[0042] Step 4. Use the indicator function As a condition for labeling preferences, preference learning is performed to improve the large model after supervised fine-tuning.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) Realizing AI & BI integration functions

[0045] The AI&BI (artificial intelligence and business intelligence) function combines AI technology and BI tools to realize intelligent and visualized data analysis and provide strong support for business decision-making. It mainly realizes two functions: a generative BI system enhanced by large models and a conversational data report analysis service.

[0046] The big model-enhanced generative BI system realizes the intelligent generation of large-screen dashboards based on the generation capability of the big language model, reducing the workload of large-screen production and improving work efficiency. The conversational data report analysis service uses big model technology to enable users to describe business needs through simple natural language and directly generate corresponding charts and data reports, effectively lowering the threshold for data analysis.

[0047] (2) Implementing intelligent SQL functions

[0048] The smart SQL function mainly realizes the function of converting the input natural language into SQL statements, which can simplify the process of data processing and analysis. It allows users to describe the required data in natural language without having to master complex SQL syntax, and the platform can combine the large model and existing data sources to obtain the corresponding SQL query statements. This function can greatly improve work efficiency, especially for those who are not familiar with SQL syntax. The core advantage of the smart SQL function lies in its convenience and efficiency. Traditionally, data analysis and query require professional technicians to write and optimize SQL statements, which is not only time-consuming and labor-intensive, but also prone to errors. With the smart SQL function, any user only needs to express their needs in natural language, such as "Please show me the sales data for the past month" or "Query the list of all customers over 30 years old", and the system will automatically generate the corresponding SQL statement and execute the query.

[0049] The intelligent SQL function not only improves the efficiency of data processing, but also lowers the threshold for use, allowing more people to participate in data analysis and decision-making. Especially today when data-driven decision-making is becoming more and more important, the intelligent SQL function provides enterprises with a powerful tool to obtain and use data more quickly and make wise business decisions.

[0050] In addition, the query results generated by intelligent SQL can be directly used to create dynamic reports and charts to form a generative dashboard. The dashboard can be updated in real time to display the latest data analysis results, allowing management and other decision makers to grasp the business situation at any time and make timely adjustments.

[0051] In terms of data interface, the intelligent SQL function also plays an important role. By converting natural language queries into standard SQL statements, the system can quickly build data interfaces so that other systems or applications can easily call these interfaces to obtain the required data. This not only improves the efficiency of data sharing and utilization, but also enhances the interoperability between systems. Get the required data easily. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The included drawings are used to provide a further understanding of the embodiments of the present invention, which constitute a part of the specification, are used to illustrate the embodiments of the present invention, and together with the text description, explain the principles of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A schematic diagram of the principle structure of a data intelligent analysis device integrating AI and BI provided according to a specific embodiment of the present invention is shown.

[0054] Figure 2 A schematic diagram of the principle structure of an intelligent SQL generation module based on a large model provided according to a specific embodiment of the present invention is shown.

[0055] Figure 3 A schematic diagram showing visualization of the output of a large model using a BI tool provided according to a specific embodiment of the present invention is shown. DETAILED DESCRIPTION

[0056] It should be noted that, in the absence of conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present invention and its application or use. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] As a data intelligent analysis method integrating AI and BI provided by an embodiment of the present invention, the following steps are taken:

[0058] Step 1: Receive questions input by users in natural language through BI tools.

[0059] Step 2: Use semantic parsing to extract the most relevant database table structure S.

[0060] The database table structure S is defined as:

[0061]

[0062] Where T i represents the i-th table in the database, C ij Representation table T i The jth column in m i Representation table T i The number of columns to include.

[0063] Step 3: Fine-tune the large model

[0064] According to the user question Q, the corresponding database table structure S and the manually constructed fine-tuning dataset D, the original large model is fine-tuned to obtain a large model after supervised fine-tuning. The supervised fine-tuning model can generate SQL query statements based on the user's input, which can be expressed as:

[0065] R gen =g(Q,S,D)

[0066] g(·) denotes the text generation model used as the backbone model for the supervised fine-tuning process.

[0067] Step 4: Perform preference learning based on the supervised fine-tuned large model obtained in step 3, and improve the large model by aligning the feedback of the execution of SQL query statements generated by the supervised fine-tuned model with the contribution of the basic fact SQL.

[0068] Step (1) Generate SQL queries using the fine-tuned large model. For each instance in the training set, obtain a predicted SQL. These predicted SQL query sets are used to generate SQL queries using R gen express;

[0069] Step (2) Use manual annotation to obtain the ground truth of SQL in the existing text real ;

[0070] Step (3) defines an indicator function to evaluate the execution result. The indicator function is defined as follows:

[0071]

[0072] in, Predict the value R for SQL gen , R is the SQL base truth value R in the ready-made text real ;

[0073] Step (4) uses the indicator function As a condition for labeling preferences, preference learning is performed to improve the large model after supervised fine-tuning.

[0074] Step 5: Generate SQL statements based on the large model after preference learning by inputting user questions and database table structure.

[0075] Step 6: Query based on the generated SQL statement, generate the visualization chart component configuration requirements, and output the unstructured analysis results.

[0076] Based on the above-mentioned data intelligent analysis method integrating AI and BI, according to another aspect of the present invention, a data intelligent analysis device integrating AI and BI is provided, which can run all steps of the data intelligent analysis method integrating AI and BI.

[0077] As an embodiment of the present invention, a data intelligent analysis device integrating AI and BI is provided, which mainly includes: a BI demand receiving module, a large model-based intelligent SQL generation module and an intelligent data visualization display module. Figure 1 shown.

[0078] The BI requirement receiving module is used to receive questions input by users in natural language through BI tools.

[0079] The present invention proposes two standards for intelligent SQL generation: 1) enhancing accurate database execution, and 2) improving accurate SQL generation. The large model-based intelligent SQL generation module aims to develop an intelligent SQL generation system that meets the above standards.

[0080] like Figure 2 As shown in the figure, the intelligent SQL generation module based on the large model mainly includes three sub-modules: a large model supervised fine-tuning sub-module, a preference learning sub-module and a SQL statement generation sub-module.

[0081] The large model supervised fine-tuning submodule takes user questions, database table structures, and manually constructed fine-tuning datasets as inputs, performs supervised fine-tuning on the original large model, and obtains a supervised fine-tuned large model, which can generate SQL query statements based on user input.

[0082] Assuming that the user question and the corresponding database table structure are Q and S respectively, supervised fine-tuning of the original large model is performed as follows:

[0083] 1) Reading of relevant database table structures. When processing large databases, inputting a complete database table structure poses challenges to input length restrictions and redundancy. To solve this problem, the present invention uses semantic parsing to match relevant tables. The most relevant database table structure S is extracted through the questions asked by the user as the input prompt for the subsequent supervised fine-tuning model. The database table structure S is defined as:

[0084]

[0085] Where T i represents the i-th table in the database, C ij Representation table T i The jth column in m i Representation table T i The number of columns to include.

[0086] 2) Supervised fine-tuning. According to the user question Q, the corresponding database table structure S, and the manually constructed fine-tuning dataset D, the original large model is supervised fine-tuned to obtain a supervised fine-tuned large model. The supervised fine-tuning model can generate SQL query statements based on the user input, which can be expressed as:

[0087] R gen =g(Q,S,D)

[0088] g(·) denotes the text generation model used as the backbone model for the supervised fine-tuning process.

[0089] In order for the model to generate useful knowledge for accurate database execution and effectively contribute to SQL queries, preference learning is required. The preference learning submodule improves the large model by aligning the feedback of SQL query statements generated by the supervised fine-tuned model with the contribution of the ground truth SQL. The method is as follows:

[0090] Step 1. A fine-tuned large model is obtained through the supervised fine-tuning submodule. The trained supervised fine-tuning model is used to generate SQL queries. For each instance in the training set, a predicted SQL is obtained. The set of these SQL queries is used in R gen express.

[0091] Step 2. Obtain the ground truth of SQL in the ready-made text using manual annotations real .

[0092] Step 3. Define an indicator function to evaluate the execution result. The indicator function is defined as follows:

[0093]

[0094] in, Predict the value R for SQLgen , R is the SQL base truth value R in the ready-made text rea

[0095] Step 4. Use the indicator function As a condition for labeling preferences, preference learning is performed to improve the large model after supervised fine-tuning.

[0096] The SQL statement generation submodule generates SQL statements based on the large model after preference learning by inputting user questions and database table structure.

[0097] In the test phase, user questions and database table structures are given. The big model after using preference learning is based on the collected and generated expert knowledge, and is combined with questions, database table structures and big model instruction prompts to generate a ready-made text-to-SQL model and generate SQL statements.

[0098] The intelligent data visualization module performs queries based on the generated SQL statements, generates visualization chart component configuration requirements, and outputs unstructured analysis results. The BI tool completes the visualization based on the output of the large model, such as Figure 3 shown.

[0099] The data intelligent analysis method integrating AI and BI provided by the present invention mainly uses large model technology, solves some bottleneck problems in traditional BI, lowers the usage threshold, and improves flexibility and intelligence level.

[0100] (1) Intelligent understanding of business needs

[0101] Instantly understand and respond to business needs. Users can ask simple questions in natural language and quickly obtain the required data analysis results, achieving an instant question-and-answer data analysis experience.

[0102] (2) Intelligent SQL Generation

[0103] The present invention uses big model technology to intelligently convert human natural language into SQL statements, which can significantly reduce the time required to write complex statements, optimize database index queries, and rectify query logic, so that the database can better return query results. For scenarios involving complex operations such as multi-table connections, subqueries, and aggregate functions, intelligent SQL generation tools can greatly simplify these tasks, making even complex queries easy to create.

[0104] (3) Intelligent Data Visualization

[0105] The present invention not only supports natural language queries, but also can automatically generate corresponding data visualization charts based on user questions. Users can obtain data through natural language questions and get corresponding visualization charts to help them better understand the trends and relationships behind the data. This intelligent data visualization function makes the data analysis process more intuitive and vivid, helping users quickly discover patterns and anomalies in the data.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data intelligent analysis method integrating AI and BI, characterized in that: The method comprises the following steps: Step 1: Receive questions input by users in natural language through BI tools; Step 2: Fine-tune the large model; According to the user question Q, the corresponding database table structure S and the manually constructed fine-tuning data set D, the original large model is fine-tuned to obtain a large model after supervised fine-tuning. The large model after supervised fine-tuning can generate SQL query statements according to the user's input; Step 3: Perform preference learning based on the supervised fine-tuned large model obtained in step 2; Improve the supervised fine-tuned large model by aligning the feedback of the SQL query execution generated by the supervised fine-tuned model with the contribution of the ground truth SQL; Step 4: The large model after preference learning can generate SQL statements by inputting user questions and database table structures; Step 5: Query based on the generated SQL statement and generate a visual chart.

2. According to claim 1, a data intelligent analysis method integrating AI and BI is characterized in that: The most relevant database table structure S is extracted by semantic analysis, and the database table structure S is defined as: Where T i represents the i-th table in the database, C ij Representation table T i The jth column in m i Representation table T i The number of columns to include.

3. According to claim 1, a data intelligent analysis method integrating AI and BI is characterized in that: Step 3 carries out preference learning as follows: Step (1) Generate SQL queries using the fine-tuned large model. For each instance in the training set, obtain a predicted SQL. These predicted SQL query sets are used to generate SQL queries using R gen express; Step (2) Use manual annotation to obtain the ground truth of SQL in the existing text real ; Step (3) defines an indicator function to evaluate the execution result. The indicator function is defined as follows: in, Predict the value R for SQL gen , R is the SQL base truth value R in the ready-made text real ; Step (4) uses the indicator function As a condition for labeling preferences, preference learning is performed to improve the large model after supervised fine-tuning.

4. A data intelligent analysis device integrating AI and BI, characterized in that: It includes BI demand receiving module, intelligent SQL generation module based on big model and intelligent data visualization display module; The BI requirement receiving module is used to receive questions input by the user in natural language through the BI tool; The large model-based intelligent SQL generation module mainly includes three submodules: a large model supervised fine-tuning submodule, a preference learning submodule and an SQL statement generation submodule. The large model supervised fine-tuning submodule takes the user question Q, the database table structure S and the manually constructed fine-tuning data set D as input, and performs supervised fine-tuning on the original large model to obtain a supervised fine-tuned large model. The supervised fine-tuned large model can generate SQL query statements according to the user's input; The preference learning submodule improves the supervised fine-tuned large model by ensuring that the feedback of the execution of SQL query statements generated by the supervised fine-tuned large model according to the user's input is consistent with the contribution of the basic fact SQL; The SQL statement generation submodule can generate SQL statements by inputting user questions and database table structures based on the large model after preference learning; The intelligent data visualization display module performs queries based on the generated SQL statements to generate visualization charts.

5. The data intelligent analysis device integrating AI and BI according to claim 4 is characterized in that: In the large model supervised fine-tuning submodule, semantic parsing is used to extract the most relevant database table structure S through the questions asked by the user. The database table structure S is defined as: Where T i represents the i-th table in the database, C ij Representation table T i The jth column in m i Representation table T i The number of columns to include.

6. The data intelligent analysis device integrating AI and BI according to claim 4, characterized in that: The specific implementation method of the preference learning submodule is as follows: Step 1. A fine-tuned large model is obtained through the supervised fine-tuning submodule. The trained supervised fine-tuning model is used to generate SQL queries. For each instance in the training set, a predicted SQL is obtained. The set of these SQL queries is used in R gen express; Step 2. Obtain the ground truth of SQL in the ready-made text using manual annotations real ; Step 3. Define an indicator function to evaluate the execution result. The indicator function is defined as follows: in, Predict the value R for SQL gen , R is the SQL base truth value R in the ready-made text real ; Step 4. Use the indicator function As a condition for labeling preferences, preference learning is performed to improve the large model after supervised fine-tuning.

Images