Data analysis method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a data analysis method and device, electronic equipment and a computer readable storage medium. The method comprises the steps of receiving a data analysis request; matching the data analysis request with multiple pieces of table information stored in a vector database, and determining at least one piece of target table information associated with the data analysis request; through a target model, based on the data analysis request and the at least one piece of target table information, generating a target structured query language (SQL) statement for at least one target data table corresponding to the at least one piece of target table information; querying the at least one target data table based on the target SQL statement to obtain a query result; and performing data analysis based on the query result to obtain a data analysis result for the data analysis request. According to the invention, the data analysis efficiency can be improved.

Description

Technical Field

[0001] The present application relates to computer technology, and in particular to a data analysis method, device, electronic device and computer-readable storage medium. Background Art

[0002] With the rapid development of artificial intelligence, more and more application scenarios use artificial intelligence, which brings us efficient and convenient processing methods and improves people's efficiency in handling various affairs.

[0003] However, in data analysis scenarios, the data that needs to be analyzed is often not publicly available, and the model cannot obtain this data on its own for data analysis. Therefore, this data needs to be input into the model for analysis. When the amount of data is large, the data analysis process will be very long and inefficient. Summary of the invention

[0004] The embodiments of the present application provide a data analysis method, device, electronic device and computer-readable storage medium, which can improve the efficiency of data analysis.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a data analysis method, including:

[0007] Receive data analysis requests;

[0008] Matching the data analysis request with multiple pieces of table information stored in a vector database to determine at least one piece of target table information associated with the data analysis request;

[0009] Generate, through the target model, a target structured query language SQL statement for at least one target data table corresponding to the at least one target table information based on the data analysis request and the at least one target table information;

[0010] Querying the at least one target data table based on the target SQL statement to obtain a query result;

[0011] Data analysis is performed based on the query result to obtain a data analysis result for the data analysis request.

[0012] In the above solution, matching the data analysis request with the multiple pieces of table information stored in the vector database to determine at least one piece of target table information associated with the data analysis request includes:

[0013] The data analysis request is semantically matched with a plurality of table information stored in a vector database to obtain at least one target table information whose semantic similarity with the data analysis request meets a first similarity condition.

[0014] In the above solution, the target model is a large language model, and the target structured query language SQL statement for at least one target data table corresponding to the at least one target table information is generated based on the data analysis request and the at least one target table information through the target model, including:

[0015] Based on the data analysis request and the at least one target table information, generating first prompt information, wherein the first prompt information is used to prompt the generation of a structured query language SQL statement;

[0016] Outputting a target SQL statement based on the first prompt information through the large language model.

[0017] In the above scheme, the method further comprises:

[0018] Matching the data analysis request with reference SQL statements stored in a vector database to obtain at least one matching target reference SQL statement;

[0019] Based on the data analysis request and the at least one target table information, generating first prompt information includes:

[0020] First prompt information is generated based on the data analysis request, the at least one target table information and the at least one target reference SQL statement.

[0021] In the above scheme, the method further includes:

[0022] Matching the data analysis request with reference data records stored in a vector database to obtain at least one matching target reference data record;

[0023] The generating first prompt information based on the data analysis request, the at least one target table information and the at least one target reference SQL statement includes:

[0024] First prompt information is generated based on the data analysis request, the at least one target table information, the at least one target reference SQL statement and the at least one target reference data record.

[0025] In the above solution, the data analysis is performed based on the query result to obtain the data analysis result for the data analysis request, including:

[0026] Based on the data analysis request and the query result, generating second prompt information, wherein the second prompt information is used to prompt the generation of the data analysis result;

[0027] The query result is analyzed through the large language model based on the data analysis method suggested by the prompt information, and the data analysis result is output.

[0028] In the above scheme, the method further includes:

[0029] Matching the data analysis request with the data analysis strategy stored in the vector database to obtain a matching target data analysis strategy;

[0030] The generating second prompt information based on the data analysis request and the query result includes:

[0031] Based on the query result and the target data analysis strategy, second prompt information is generated.

[0032] The present application provides a data analysis device, including:

[0033] A receiving module, used for receiving a data analysis request;

[0034] a matching module, configured to match the data analysis request with a plurality of table information stored in a vector database, and determine at least one target table information associated with the data analysis request;

[0035] A generating module, configured to generate, through a target model, a target structured query language SQL statement for at least one target data table corresponding to the at least one target table information based on the data analysis request and the at least one target table information;

[0036] A query module, configured to query the at least one target data table based on the target SQL statement to obtain a query result;

[0037] The data analysis module is used to perform data analysis based on the query result to obtain a data analysis result for the data analysis request.

[0038] An embodiment of the present application provides an electronic device, including:

[0039] A memory for storing executable instructions;

[0040] The processor is used to implement the data analysis method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0041] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the data analysis method provided in the embodiment of the present application.

[0042] An embodiment of the present application provides a computer program product storing a computer program for implementing the data analysis method provided in the embodiment of the present application when executed by a processor.

[0043] The embodiment of the present application receives a data analysis request, matches the data analysis request with multiple table information stored in a vector database, determines at least one target table information associated with the data analysis request, and generates a target structured query language SQL statement for at least one target data table corresponding to the at least one target table information based on the data analysis request and the at least one target table information through a target model; queries the at least one target data table based on the target SQL statement to obtain a query result; performs data analysis based on the query result to obtain a data analysis result for the data analysis request, thereby improving the efficiency of data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is an optional structural diagram of the data analysis system provided in the embodiment of the present application;

[0045] Figure 2 is an optional structural diagram of an electronic device provided in an embodiment of the present application;

[0046] Figure 3 It is an optional flow chart of the data analysis method provided in the embodiment of the present application;

[0047] Figure 4 It is an optional structural diagram of the data analysis system provided in the embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0049] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0052] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0053] 1) Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which enables the model to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on various topics by training on huge data sets. The core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, and to simulate the human language cognition and generation process to a certain extent.

[0054] The embodiments of the present application provide a data analysis method, device, electronic device and computer-readable storage medium, which can improve the efficiency of data analysis.

[0055] First, the data analysis system provided in the embodiment of the present application is described. Figure 1 , Figure 1It is an optional structural diagram of the data analysis system 100 provided in the embodiment of the present application, and the terminal 103 is connected to the server 101 through the network 102. In some embodiments, the terminal 103 can be a laptop, a tablet computer, a desktop computer, a smart phone, a dedicated messaging device, a portable gaming device, a smart speaker, a smart watch, etc., but is not limited to this. The server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server 402 that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network) services, and big data and artificial intelligence platforms. The network 102 can be a wide area network or a local area network, or a combination of the two. The terminal 103 and the server 101 can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiment of the present application.

[0056] The data analysis method provided in the embodiment of the present application can be Figure 1 The terminal 103 or the server 101 in the embodiment may be implemented, or the two may be implemented in coordination. In some embodiments, the terminal 103 may be used to receive a data analysis request and send the data analysis request to the server 101. The server 101 is used to implement the data analysis method provided in the embodiment of the present application, and send the obtained data analysis results to the terminal 103, and display the data analysis results through the terminal 103.

[0057] Next, the electronic device for implementing the above data analysis method provided in the embodiment of the present application is described, see Figure 2 , Figure 2 is an optional structural diagram of an electronic device 200 provided in an embodiment of the present application. In practical applications, the electronic device 200 can be implemented as Figure 1 The terminal 103 or server 101 in the embodiment, where the electronic device is Figure 1 Taking the server 101 shown as an example, an electronic device for implementing the data analysis method of the embodiment of the present application is described.

[0058] Figure 2 The electronic device 200 shown includes: at least one processor 201, a memory 205, at least one network interface 202 and a user interface 203. The various components in the electronic device 200 are coupled together via a bus system 204. It is understood that the bus system 204 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 204 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 204 is not described in detail. Figure 2Various buses are labeled as bus system 204 .

[0059] The processor 201 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0060] The user interface 203 includes one or more output devices 2031 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 203 also includes one or more input devices 2032, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0061] The memory 205 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 205 may optionally include one or more storage devices that are physically remote from the processor 201.

[0062] The memory 205 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 205 described in the embodiment of the present application is intended to include any suitable type of memory.

[0063] In some embodiments, the memory 205 can store data to support various operations. Examples of these data include programs, modules, and data structures, or subsets or supersets thereof. In the embodiment of the present application, the memory 205 stores an operating system 2051, a network communication module 2052, a presentation module 2053, an input processing module 2054, and a data analysis device 2055; specifically,

[0064] Operating system 2051, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0065] A network communication module 2052, for reaching other computing devices via one or more (wired or wireless) network interfaces 202, exemplary network interfaces 202 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0066] a presentation module 2053 for enabling presentation of information (e.g., a user interface for operating peripherals and displaying content and information) via one or more output devices 2031 (e.g., a display screen, a speaker, etc.) associated with the user interface 203;

[0067] The input processing module 2054 is used to detect one or more user inputs or interactions from one of the one or more input devices 2032 and translate the detected inputs or interactions.

[0068] In some embodiments, the data analysis device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 The data analysis device 2055 stored in the memory 205 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a receiving module 20551, a matching module 20552, a generating module 20553, a query module 20554 and a data analysis module, these modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0069] In other embodiments, the data analysis device provided in the embodiments of the present application can be implemented in hardware. As an example, the data analysis device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data analysis method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0070] The data analysis method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0071] See also Figure 3 , Figure 3 is an optional flow chart of the data analysis method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are explained.

[0072] Step 301, receiving a data analysis request;

[0073] Step 302, matching the data analysis request with multiple pieces of table information stored in a vector database, and determining at least one piece of target table information associated with the data analysis request;

[0074] Step 303: Generate a target structured query language SQL statement for at least one target data table corresponding to the at least one target table information through the target model based on the data analysis request and the at least one target table information;

[0075] Step 304, querying the at least one target data table based on the target SQL statement to obtain a query result;

[0076] Step 305: Perform data analysis based on the query result to obtain a data analysis result for the data analysis request.

[0077] In actual implementation, the data analysis request can be issued by the user through the front-end system, and then sent to the server by the front-end system. Here, the front-end system uses web pages, mobile apps or applets as the human-computer interaction medium between the user and the server. The front-end system is used to record the questions raised by the user, forward them to the server, receive the data analysis results from the server and display them to the user. Here, the data analysis results can be presented in the form of text or charts. Specifically, the user can issue a user question related to data analysis, and the front-end system will use the user question as a data analysis request. Exemplarily, a user question may be, for example: "What is the ratio of men to women in Wuhan?"

[0078] In an embodiment of the present application, after receiving a data analysis request, the server matches the data analysis request with multiple table information stored in the vector database, and determines at least one target table information associated with the data analysis request. Here, the table information can be data definition language (DDL, Data Definition Language) data, which is used to define and modify the structure of the database and database objects. In actual implementation, when a new data table needs to be included in the scope of data analysis, first connect to the database, connect to the data table, and execute the DDL query statement. The statement varies depending on the different databases used. For example, for the mysql database, it is: show create table{table_name}. Next, assemble the queried DDL results into the format that needs to be stored in the vector index database, such as:

[0079] table{table_name}has column xxx(field note 1); column xxx(field note 2)....; foreign key: XXXX; metadata:{'table_name':{table_name}};

[0080] The above format mainly describes the field name stored in the table, the field's remarks, and the table name. The assembled data is then inserted into the vector database, specifically into the DDL vector index of the vector database. After completing this processing, the vector database can be used to retrieve the DDL data associated with the user's question.

[0081] In the actual scenario, multiple data tables are stored in the database, and at least one target table information is obtained by matching, so that multiple data tables are screened to reduce the amount of data. After obtaining at least one target table information, a target structured query language SQL statement for at least one target data table corresponding to at least one target table information is generated through the target model based on the data analysis request and at least one target table information. Here, the target model can be a large language model, and prompt information is formed through at least one target table information and a data analysis request, and the prompt information is input into the large language model. After a dialogue with the large language model, the large language model is made to generate a target SQL statement for at least one target data table according to the prompt information. Then, through the target SQL statement, at least one target data table is queried to obtain the query result obtained from the query in at least one target data table. It should be understood that the query result includes data related to the data analysis request. Exemplarily, if the data analysis request is "What is the ratio of men to women in Wuhan?", the query result may include the number of male residents and the number of female residents living in Wuhan.

[0082] In actual implementation, after obtaining the query result, data analysis is performed based on the query result to obtain the data analysis result for the data analysis request. Here, the analysis target requested in the data analysis request, such as calculating the male-female ratio, is calculated through the query result to obtain the data analysis result.

[0083] The embodiment of the present application receives a data analysis request, matches the data analysis request with multiple table information stored in a vector database, and determines at least one target table information associated with the data analysis request, thereby avoiding excessive amount of data that the model needs to process when the multiple table information is directly input into the target model, reducing the amount of data processing and thus improving efficiency. Through the target model, based on the data analysis request and the at least one target table information, a target structured query language SQL statement is generated for at least one target data table corresponding to the at least one target table information. Generating the target SQL statement through the target model can replace manual writing, and has high accuracy and efficiency. Finally, based on the target SQL statement, the at least one target data table is queried to obtain a query result, and data analysis is performed based on the query result to obtain a data analysis result for the data analysis request, thereby improving the efficiency of data analysis.

[0084] In some embodiments, step 302 includes: semantically matching the data analysis request with multiple table information stored in a vector database to obtain at least one target table information whose semantic similarity with the data analysis request meets a first similarity condition.

[0085] In actual implementation, the data analysis request is semantically matched with each of the multiple table information stored in the vector database to obtain the semantic similarity between the data analysis request and each table information. Here, the semantic matching method can be to extract the first semantic feature of the data analysis request and the second semantic feature of each table information, calculate the distance between the first semantic feature and each second semantic feature, and determine the calculated distance as the semantic similarity. In actual implementation, the first similarity condition can be to reach a similarity threshold, or to be ranked in the top N in the order of semantic similarity from large to small, where N is a positive integer. Exemplarily, the similarity threshold can be 80%, and the table information with a semantic similarity of 80% with the data analysis request is determined as the target table information. N can be 5, and the table information with a semantic similarity of 5 in the order of semantic similarity from large to small with the data analysis request is determined as the target table information, that is, the 5 table information with the closest semantics to the data analysis request.

[0086] In some embodiments, the target model is a large language model, and step 303 includes: generating first prompt information based on the data analysis request and the at least one target table information, wherein the first prompt information is used to prompt the generation of a structured query language SQL statement; and outputting a target SQL statement based on the first prompt information through the large language model.

[0087] In an embodiment of the present application, the target module is a large language model, such as ChatGPT. In actual implementation, after obtaining at least one target table information, a first prompt information is generated according to a data analysis request and at least one target table information. Then, the first prompt information is input into the large language model, and the large language model outputs a target SQL statement based on the first prompt information. Here, the first prompt information is used to prompt the large language model to output an SQL statement related to the data analysis request for querying at least one target data table. In actual implementation, the large language model performs text-to-SQL conversion on the first prompt information.

[0088] In some embodiments, the method also includes: matching the data analysis request with the reference SQL statement stored in the vector database to obtain at least one matching target reference SQL statement; generating first prompt information based on the data analysis request and the at least one target table information, including: generating first prompt information based on the data analysis request, the at least one target table information and the at least one target reference SQL statement.

[0089] In actual implementation, a plurality of reference SQL statements are stored in the vector database. The reference SQL statements are SQL statements corresponding to some reference questions. These reference questions and corresponding SQL statements can be collected in advance and stored in the vector database. When SQL generation for new user questions (data analysis requests) is required, the reference SQL statements can be matched first, and the first prompt information can be generated in combination with the matched target reference SQL, so that the first prompt information can be more accurate.

[0090] In actual implementation, specifically, the data analysis request is matched with at least one reference question stored in the vector database for similarity, and at least one target reference question whose similarity to the data analysis request reaches a second similarity condition is obtained, and the reference SQL statement corresponding to the target reference question is determined as the target reference SQL statement. Here, the reference question and the reference SQL are stored in the vector database in correspondence. The setting of the second similarity condition can refer to the first similarity condition, and the second similarity condition can be the same as or different from the first similarity condition. By matching the target reference SQL statement, the generation of the first prompt information can refer to the reference SQL statement corresponding to the reference question similar to the data analysis request, so that the target model can refer to the reference SQL statement to generate a more accurate target SQL statement.

[0091] In actual implementation, the reference SQL can be pre-stored in the vector database. The reference SQL is usually a complex SQL that needs to be generated for the user question. The input format can be {"user question": XXXXX, "reference SQL": "selectxxxxxxxxxx"}. Each reference SQL is vector-indexed according to the user question and written into the reference SQL vector index of the vector database.

[0092] In some embodiments, the method further includes: matching the data analysis request with the reference data records stored in the vector database to obtain at least one matching target reference data record; generating first prompt information based on the data analysis request, the at least one target table information and the at least one target reference SQL statement, including: generating first prompt information based on the data analysis request, the at least one target table information, the at least one target reference SQL statement and the at least one target reference data record.

[0093] Here, the reference data record is a pre-stored data record, such as "the male population in Wuhan is xxx". In actual implementation, the data analysis request can be matched with at least one reference data record stored in the vector database for similarity, and at least one target reference data record whose similarity meets the third similarity condition can be obtained. Then, the first prompt information can be generated based on the data analysis request, at least one target table information, at least one target reference SQL statement and at least one target reference data record. The target model can generate the corresponding target SQL in combination with the target reference data record.

[0094] In actual implementation, the reference data records can be pre-stored in the vector database. Specifically, the following SQL query is performed on each data table to obtain the fields with the field type of VARCHAR and store different data records in the database. The process is as follows:

[0095] 1. The time when the scheduled task arrives and the reference data is started to record the time of data query;

[0096] 2. Select the first data table for processing;

[0097] 3. Select the varchar type field in the first data table, and query up to 3 records with different fields. For example, if the field name is colum_1 and the table name is table_1, the SQL is:

[0098]

[0099]

[0100] 4. After completing the query of one field, query the next field until the varchar type fields of the table are traversed. Then, write the query results into the vector index corresponding to the table in rows.

[0101] 5. Execute the query of the data table in the next chapter until all data tables are processed.

[0102] In some embodiments, the data analysis is performed based on the query result to obtain the data analysis result for the data analysis request, including: generating a second prompt information based on the data analysis request and the query result, the second prompt information being used to prompt the generation of the data analysis result; performing data analysis on the query result based on the data analysis method prompted by the second prompt information through the large language model, and outputting the data analysis result.

[0103] In actual implementation, after the target model generates the target SQL, the server queries at least one target data table based on the target SQL to obtain the query result. Then, based on the data analysis request and the query result, a second prompt information is generated. Here, the second prompt information is used to prompt the generation of the data analysis result. Then, the second prompt information is input into the target model, and the query result is analyzed based on the second prompt information by the target model, and the data analysis result is output. Here, the user can also input the corresponding prompt information according to the required display type, so that the target model outputs the data analysis result in the corresponding display type. The display type is, for example, text or chart type.

[0104] In some embodiments, the method also includes: matching the data analysis request with the data analysis strategy stored in the vector database to obtain a matching target data analysis strategy; generating a second prompt information based on the data analysis request and the query result, including: generating a second prompt information based on the query result and the target data analysis strategy.

[0105] Here, the data analysis strategy is a pre-stored data record, such as a method for calculating sales conversion rate. In actual implementation, the data analysis request can be matched with at least one data analysis strategy stored in the vector database for similarity, and at least one target data analysis strategy whose similarity reaches the fourth similarity condition can be obtained. Then, based on the query result and the target data analysis strategy, a second prompt information can be generated. The target model can generate data analysis results based on the second prompt information. Specifically, the process of generating data analysis results based on the second prompt information by the target model is combined with the target data analysis strategy.

[0106] In actual implementation, the data analysis strategy can be pre-stored in the vector database. Specifically, the data analysis strategy can be pre-stored in the vector database. In one embodiment, data processing related knowledge can also be stored, including but not limited to synonyms, indicator calculation formulas, noun concept explanations, etc. Each piece of knowledge is written into the data processing knowledge vector index of the vector database.

[0107] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0108] See also Figure 4 , Figure 4 It is an optional structural diagram of the data analysis system provided in the embodiment of the present application. Here, the data analysis system includes a front-end system 401, cloud servers 402402 and a database 403. The front-end system 401 uses web pages, mobile APPs or applets as a human-computer interaction medium between users and cloud servers 402; the front-end system 401 is used to record user questions (data analysis requests) raised by users, forward them to the cloud server 402, receive the answer results of the data analysis from the cloud server 402, and draw charts to show to the front-end users. The cloud server 402 includes a user question module, a data enhancement module, a text-to-SQL statement prompt word module, a large model module, an answer summary prompt word module, a chart selection prompt word module and a model answer module. Among them, the user question module is used to receive user questions sent by the front-end system 401; the data enhancement module is used to store reference data records, data processing knowledge, table information, and reference SQL statements; it is responsible for using user questions to retrieve reference data records, data processing knowledge, table information, and reference SQL statements that are similar to user questions; the text-to-SQL statement prompt word module is used to assemble user questions, related reference data records, data processing knowledge, table information, and reference SQL statements into complete prompt words for generating SQL statements; the large model module includes a large language model, which is used to generate relevant texts according to the input prompt words. In the embodiment of the present application, SQL statements are generated through text-to-SQL statement prompt words, data analysis summaries are generated through answer summary prompt words, and chart selection results are generated through chart selection prompt words; the answer summary prompt word module is used to assemble SQL statement execution results, user questions, and SQL statement statements into complete data analysis prompt words; the chart selection prompt word module is used to assemble user questions, SQL statement execution results, SQL statement statements, and chart descriptions into complete chart selection prompt words; the model answer module is used to organize the data analysis summary, chart selection results, and SQL statement execution results output by the large model into a data format that can be received by the front-end system.

[0109] In actual implementation, after the user enters the user question, it is first necessary to generate an SQL statement that can query the corresponding data of the user data analysis, and query the corresponding data from the database to obtain the data. The process is as follows:

[0110] 1. In the DDL vector index, query the DDL records most relevant to the user's question;

[0111] 2. Query the DDL of the associated table based on the foreign key relationship recorded in the queried DDL record;

[0112] 3. Query the records related to the user's question in the reference data record vector index corresponding to the related table and the associated table;

[0113] 4. In the data analysis knowledge vector index, query the records related to the user's question;

[0114] 5. Search the reference SQL statement vector index for records related to the user's question.

[0115] 6. In the text-to-SQL statement prompt word module, assemble the first prompt information for generating the SQL statement.

[0116] Exemplarily, the first prompt information may be as follows:

[0117] DEFAULT_TEXT_TO_SQL_STATEMENT_TMPL=(

[0118] "Given an input problem, first create a grammatically correct {dialect}"

[0119] "A query statement, then looks at the query results and returns an answer."

[0120] "You can sort the results by relevant columns to return the most interesting examples in your database.\n\n"

[0121] "Never query all columns in a particular table, only the few columns that are relevant to your problem.\n\n"

[0122] "Note that you can only use column names that you see in the schema description."

[0123] "Be careful not to query columns that don't exist."

[0124] "Note which columns belong to which table."

[0125] "Qualify column names with table names when necessary."

[0126] "For fields of type ID that are associated with other tables through foreign keys, it is necessary to query the entity name of the corresponding ID in the associated table when generating SQL statements.\n"

[0127] "You must use the following format, one entry per line:\n\n"

[0128] "Problem: Here is the problem\n"

[0129] "SQL statement query: SQL statement query to be run\n"

[0130] "SQL statement result: SQL statement query result\n"

[0131] "Answer: The final answer is here\n\n"

[0132] "Only use the tables listed below.\n"

[0133] "{DDLs}\n\n"

[0134] "{reference_SQL statement s}\n\n"

[0135] "{reference_docs}\n\n"

[0136] "{reference_records}\n\n"

[0137] "Question:{query_str}\n"

[0138] "SQL statement query:")

[0139] DDLs is the DDL description of related tables and associated tables, referenc_record is the related reference data record, reference_docs is the related data processing knowledge, and reference_SQLs is the related reference SQL statement.

[0140] 7. Provide the target SQL statement generated by the large model to the SQL statement execution module for execution;

[0141] 8. The SQL statement execution module uses the target SQL statement to query the database and obtain the query results.

[0142] In actual implementation, after completing the data query, the answer summary prompt word module is used to assemble the acquired query results, user questions and SQL statements into a complete data analysis summary prompt word (second prompt information) and submit it to the big model for execution. For example, the second prompt information can be as follows:

[0143] DEFAULT_RESPONSE_SYNTHESIS_PROMPT_TMPL_V2=(

[0144] "Given an input question, synthesize a Chinese answer based on the query results.\n"

[0145] "Question:{query_str}\n"

[0146] "SQL statement: {SQL statement_query}\n"

[0147] "SQL statement result: {context_str}\n"

[0148] "answer:" )

[0150] Where query_str is the user question, SQL statement_query is the generated SQL statement, and context_str is the result of the SQL statement query.

[0151] In addition, after completing the data query, you can also use the big language model to form a chart analysis result. Specifically, use the data summary prompt word module to assemble the acquired data, user questions, SQL statements and optional charts into a complete data analysis summary prompt word, and submit it to the big model for execution. The reference prompt words are as follows:

[0152] CHART_SELECTION_PROMPT=(

[0153] "Please select the most appropriate chart type from the following chart types to display your data based on your data characteristics and analysis objectives:\n\n"

[0154] "1. **Bar Chart**-Use when you need to compare the quantity of different categories. Suitable for showing the comparative relationship between categories.\n"

[0155] "2.**Line Chart**-Use when you need to show the trend of data over time. Suitable for time series data analysis.\n"

[0156] "3. **Pie Chart** - Use when you need to show the proportion of each part in the whole. Suitable for showing composition structure.\n"

[0157] "4. **Scatter Plot**-Use when you need to explore the relationship between two variables.

[0158] Good for finding correlations. \n"

[0159] "5.**Heatmap**-Use when you need to show the distribution of data in two dimensions. Suitable for complex data set analysis.\n"

[0160] "6.**Box Plot**-Use when you need to understand the distribution of data, including statistics such as median and quartiles. Suitable for viewing data distribution and outliers.\n"

[0161] "7. **Bubble Chart** - Use when you need to compare three variables at the same time. Suitable for visualizing three-dimensional data.\n"

[0162] "8.**Radar Chart**-Use when you need to compare the comprehensive performance of multiple variables. Suitable for comparison of multi-dimensional data.\n"

[0163] "9.**Tree Map**-Use when you need to display hierarchical data. Suitable for classification comparison when space is limited.\n"

[0164] "10. **Histogram**-Use when you need to display the frequency distribution of continuous data. Suitable for understanding the distribution pattern of data.\n\n"

[0165] "Question:{query_str}\n"

[0166] "SQL statement: {SQL statement_query}\n"

[0167] "SQL statement result: {context_str}\n"

[0168] "Chart Type:" )

[0170] In the embodiment of the present application, the front-end module receives the data analysis summary results returned by the large model, and uses the query data in the answer to draw the chart according to the chart type returned by the chart selection module. Through flexible and automatic collection of DDL information in the data table, dynamic selection of the data table is realized to avoid the DDL of all tables being passed in to cause the large model input to exceed the limit; through flexible collection of reference data record information, the problem of the inconsistency between the expression in the user's question and the database record, resulting in incorrect SQL statement generation, can be greatly optimized; by introducing data analysis knowledge and reference SQL statement knowledge, the system is injected with information such as data analysis methods and complex SQL statement writing methods to improve the accuracy of SQL statement generation; in addition, through the large model to select the chart type, it is possible for the user front end to dynamically draw charts according to the query data, which improves the intuitiveness of data analysis.

[0171] The following is a description of an exemplary structure of the data analysis device 2055 provided in the present application as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the data analysis device 2055 of the memory 205 may include:

[0172] Receiving module 20551, used for receiving a data analysis request;

[0173] A matching module 20552, configured to match the data analysis request with multiple pieces of table information stored in a vector database, and determine at least one piece of target table information associated with the data analysis request;

[0174] A generating module 20553 is used to generate a target structured query language SQL statement for at least one target data table corresponding to the at least one target table information through a target model based on the data analysis request and the at least one target table information;

[0175] A query module 20554 is used to query the at least one target data table based on the target SQL statement to obtain a query result;

[0176] The data analysis module 20555 is used to perform data analysis based on the query result to obtain a data analysis result for the data analysis request.

[0177] In some embodiments, the matching module 20552 is further used to perform semantic matching between the data analysis request and multiple table information stored in the vector database to obtain at least one target table information whose semantic similarity with the data analysis request meets a first similarity condition.

[0178] In some embodiments, the target model is a large language model, and the generation module 20553 is further used to generate a first prompt information based on the data analysis request and the at least one target table information, wherein the first prompt information is used to prompt the generation of a structured query language SQL statement; and output a target SQL statement based on the first prompt information through the large language model.

[0179] In some embodiments, the device also includes: a SQL statement matching module, used to match the data analysis request with the reference SQL statement stored in the vector database to obtain at least one matching target reference SQL statement; the generation module 20553 is also used to generate a first prompt information based on the data analysis request, the at least one target table information and the at least one target reference SQL statement.

[0180] In some embodiments, the device also includes: a record matching module, used to match the data analysis request with the reference data records stored in the vector database to obtain at least one matching target reference data record; the generation module 20553 is also used to generate a first prompt information based on the data analysis request, the at least one target table information, the at least one target reference SQL statement and the at least one target reference data record.

[0181] In some embodiments, the data analysis module 20555 is also used to generate a second prompt message based on the data analysis request and the query result, and the second prompt message is used to prompt the generation of the data analysis result; through the large language model, the query result is analyzed based on the data analysis method prompted by the second prompt message, and the data analysis result is output.

[0182] In some embodiments, the device also includes: a strategy matching module, used to match the data analysis request with the data analysis strategy stored in the vector database to obtain a matching target data analysis strategy; the data analysis module 20555 is also used to generate a second prompt information based on the query result and the target data analysis strategy.

[0183] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above-mentioned method embodiment, and has similar beneficial effects as the method embodiment, so it will not be repeated.

[0184] The embodiment of the present application provides a computer program product, which includes computer instructions, which are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the data analysis method described above in the embodiment of the present application.

[0185] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the data analysis method provided by the embodiment of the present application.

[0186] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0187] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0188] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0189] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0190] In summary, the embodiments of the present application can improve the efficiency of data analysis.

[0191] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data analysis method, characterized in that: include: Receive data analysis requests; Matching the data analysis request with multiple pieces of table information stored in a vector database to determine at least one piece of target table information associated with the data analysis request; Generate, through the target model, a target structured query language SQL statement for at least one target data table corresponding to the at least one target table information based on the data analysis request and the at least one target table information; Querying the at least one target data table based on the target SQL statement to obtain a query result; Data analysis is performed based on the query result to obtain a data analysis result for the data analysis request.

2. The method according to claim 1, characterized in that The step of matching the data analysis request with the plurality of table information stored in the vector database to determine at least one target table information associated with the data analysis request includes: The data analysis request is semantically matched with a plurality of table information stored in a vector database to obtain at least one target table information whose semantic similarity with the data analysis request meets a first similarity condition.

3. The method according to claim 1, characterized in that: The target model is a large language model, and the target structured query language SQL statement for at least one target data table corresponding to the at least one target table information is generated based on the data analysis request and the at least one target table information through the target model, including: Based on the data analysis request and the at least one target table information, generating first prompt information, wherein the first prompt information is used to prompt the generation of a structured query language SQL statement; Outputting a target SQL statement based on the first prompt information through the large language model.

4. The method according to claim 3, characterized in that The method further comprises: Matching the data analysis request with reference SQL statements stored in a vector database to obtain at least one matching target reference SQL statement; Based on the data analysis request and the at least one target table information, generating first prompt information includes: First prompt information is generated based on the data analysis request, the at least one target table information and the at least one target reference SQL statement.

5. The method according to claim 4, characterized in that The method further comprises: Matching the data analysis request with reference data records stored in a vector database to obtain at least one matching target reference data record; The generating first prompt information based on the data analysis request, the at least one target table information and the at least one target reference SQL statement includes: First prompt information is generated based on the data analysis request, the at least one target table information, the at least one target reference SQL statement and the at least one target reference data record.

6. The method according to claim 3, characterized in that The performing data analysis based on the query result to obtain a data analysis result for the data analysis request includes: Based on the data analysis request and the query result, generating second prompt information, wherein the second prompt information is used to prompt the generation of the data analysis result; The query result is analyzed through the large language model based on the data analysis method suggested by the prompt information, and the data analysis result is output.

7. The method according to claim 6, characterized in that The method further comprises: Matching the data analysis request with the data analysis strategy stored in the vector database to obtain a matching target data analysis strategy; The generating second prompt information based on the data analysis request and the query result includes: Based on the query result and the target data analysis strategy, second prompt information is generated.

8. A data analysis device, characterized in that: include: A receiving module, used for receiving a data analysis request; a matching module, configured to match the data analysis request with a plurality of table information stored in a vector database, and determine at least one target table information associated with the data analysis request; A generating module, configured to generate, through a target model, a target structured query language SQL statement for at least one target data table corresponding to the at least one target table information based on the data analysis request and the at least one target table information; A query module, used to query the at least one target data table based on the target SQL statement to obtain a query result; The data analysis module is used to perform data analysis based on the query result to obtain a data analysis result for the data analysis request.

9. An electronic device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the data analysis method according to any one of claims 1 to 7 when executing the executable instructions stored in the memory.

10. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the data analysis method described in any one of claims 1 to 7 when executed by a processor.

Images