LLM-based data analysis methods and electronic devices

Abstract

This specification provides an LLM-based data analysis method, comprising: acquiring user-inputted questions; wherein the user questions are used to initiate data analysis; responding to the user questions, acquiring user features corresponding to the user; wherein the user features are used to represent the user's data analysis preferences; the user features include question features corresponding to the user questions; creating prompt words based on the user features and a set of user question templates; inputting the prompt words into the LLM, so that the LLM generates at least one additional question that has a progressive semantic relationship with the user questions based on the prompt words, and outputting and displaying the at least one additional question to the user through a visualization interface to guide the user to continue inputting the additional questions, and further performing data analysis based on the data analysis results corresponding to the user questions.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to an LLM-based data analysis method and electronic device. Background Technology

[0002] With the continuous development of LLM (Large Language Model) technology, more and more data analysis systems are beginning to enhance and expand their data analysis capabilities by integrating LLM. For example, after integrating LLM into a data analysis system, users can initiate data analysis by asking questions.

[0003] However, while integrating LLM into a data analysis system can enhance and expand the system's data analysis capabilities, users typically need to construct their own user questions based on their specific data analysis needs when initiating data analysis. For users unfamiliar with data, it can be difficult to construct effective user questions, potentially leading to low efficiency in data analysis. Summary of the Invention

[0004] In view of the above, one or more embodiments of this specification provide the following technical solutions:

[0005] According to a first aspect of one or more embodiments of this specification, an LLM-based data analysis method is proposed and applied to an LLM-based data analysis system; wherein, the data analysis system supports initiating data analysis by means of questions; the data analysis system maintains a set of user question templates that have a progressive relationship in semantics; including:

[0006] Obtain user-inputted questions; wherein, the user-inputted questions are used to initiate data analysis;

[0007] In response to the user's question, user characteristics corresponding to the user are obtained; wherein, the user characteristics are used to represent the user's data analysis preferences; the user characteristics include question characteristics corresponding to the user's question;

[0008] Prompt words are created based on the user characteristics and the user question template set; wherein, the prompt words are used to instruct the LLM to generate at least one supplementary question corresponding to the user question; the supplementary question has a progressive relationship with the user question in terms of semantics;

[0009] The prompt words are input into the LLM, which generates at least one additional question based on the prompt words. The at least one additional question is then output and displayed to the user through a visualization interface corresponding to the data analysis system, guiding the user to continue inputting additional questions. Further data analysis is then performed based on the data analysis results corresponding to the user's questions.

[0010] Optionally, the user question templates in the user question template set represent the user's follow-up question format;

[0011] The method further includes:

[0012] Obtain a sample set of historical questions from all users who have initiated data analysis in the data analysis system using the question-and-answer method;

[0013] Semantic analysis is performed on the question samples contained in the historical question sample set to determine the target historical question samples that have a progressive relationship in the semantics of the questions.

[0014] Learn the user's follow-up questioning methods from the target historical question samples, create user question templates corresponding to the follow-up questioning methods, and save the generated user question templates to a pre-created user question template set.

[0015] Optionally, the user question template set includes a knowledge graph;

[0016] In this context, nodes in the knowledge graph represent user question templates; and edges in the knowledge graph represent semantic progression relationships between user question templates.

[0017] Optionally, the content of a node in the knowledge graph includes a description instruction corresponding to the user question template represented by that node;

[0018] Based on the user characteristics and the user question template set, prompt words are created, including:

[0019] Starting from the starting node of the knowledge graph, traverse each node in the knowledge graph that is related to the user characteristics, and obtain the description instructions contained in the content of the traversed nodes.

[0020] Based on the user characteristics and the obtained description instructions, further prompt words are created.

[0021] Optionally, before traversing each node in the knowledge graph from its starting node, the method further includes:

[0022] Calculate the relevance between each node in the knowledge graph and the user features;

[0023] Identify target nodes in the knowledge graph whose relevance to the user characteristics is below a threshold.

[0024] The target node and the edge corresponding to the target node contained in the knowledge graph are pruned.

[0025] Optionally, the at least one additional question is a set of additional questions generated by the LLM based on the prompt words;

[0026] The data analysis system outputs and displays data to the user through a visual interface, including:

[0027] Calculate the relevance score between the additional questions in the set of additional questions and the user characteristics;

[0028] The additional questions in the set of additional questions are sorted based on the calculated relevance scores;

[0029] From the sorted set of additional questions, determine the N additional questions with the highest relevance scores to the user characteristics, and display the N additional questions to the user through the visualization interface corresponding to the data analysis system; where N is a preset threshold.

[0030] Optionally, the user features include multiple types of user features; wherein, the multiple types of user features correspond to different weight values; the weight value indicates the importance of the user feature to the data analysis initiated by the user's question;

[0031] Calculating the relevance score between the additional questions in the set of additional questions and the user characteristics includes:

[0032] Calculate the relevance scores between the additional questions in the additional question set and the various types of user features to obtain the relevance scores corresponding to each type of user feature among the various types of user features;

[0033] Based on the weight values ​​corresponding to the user characteristics of each type, the calculated relevance scores corresponding to the user characteristics of each type are weighted to obtain the relevance score between the additional question and the user characteristics.

[0034] Optionally, the data analysis system supports initiating various types of data analysis by asking questions; the weight values ​​corresponding to each type of user feature among the various types of user features are dynamically determined weight values ​​based on the type of data analysis initiated by the user's question.

[0035] Optionally, the various types of data analysis include:

[0036] The first type of data analysis used for data querying;

[0037] The second type of data analysis is used to perform data queries and data analysis and calculations on the retrieved data;

[0038] The third type of data analysis is used to perform data queries and visualize the retrieved data.

[0039] A fourth type of data analysis used to define data metrics and perform metric calculations related to the data metrics. Optionally, the various types of user characteristics include question characteristics corresponding to the user's question, and any one or more of the following types of user characteristics:

[0040] The first type of user characteristic represents the user's real-time data analysis preferences;

[0041] The second type of user characteristic represents the user's historical data analysis preferences;

[0042] The third type of user characteristic represents the data analysis preferences of the user group to which the user belongs.

[0043] Optionally, the user characteristics include any one or a combination of the following characteristics:

[0044] Data analysis dimensions related to data analysis;

[0045] Data analysis metrics related to data analysis;

[0046] The time period related to data analysis.

[0047] Optionally, the data analysis system may include a BI system that supports data analysis functions.

[0048] According to a second aspect of one or more embodiments of this specification, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor performs the executable instructions to implement the steps of the method as described in any of the first aspects above.

[0049] According to a third aspect of one or more embodiments of this specification, a computer-readable storage medium is also provided, having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as described in any of the first aspects above.

[0050] According to a fourth aspect of one or more embodiments of this specification, a computer program product is also provided, comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as described in any of the first aspects above.

[0051] In the above embodiments, personalized supplementary questions can be generated for users without deviating from the semantics of user-input questions. These supplementary questions can guide users to conduct further data analysis based on the data analysis results corresponding to the user questions. This not only guides users to explore data in depth according to the thinking patterns in the user question template set and uncover more valuable information, but also improves the depth and quality of data analysis. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the architecture of a data analysis service system provided in an exemplary embodiment.

[0053] Figure 2 This is a flowchart illustrating an exemplary embodiment of a data analysis method based on LLM.

[0054] Figure 3 This is a schematic diagram of a data analysis system provided in an exemplary embodiment.

[0055] Figure 4 This is an exemplary embodiment of a diagram showing how a set of user question templates is organized into a knowledge graph.

[0056] Figure 5 This is an exemplary embodiment illustrating the output of generated additional questions to a user.

[0057] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an exemplary embodiment.

[0058] Figure 7 This is a block diagram of an LLM-based data analysis device provided in an exemplary embodiment. Detailed Implementation

[0059] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0060] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0061] This specification aims to propose a technical solution in an LLM-based data analysis system that leverages the reasoning capabilities of LLM to generate personalized supplementary questions that do not deviate from the semantics of user-input questions, and guides users to conduct further data analysis based on the data analysis results corresponding to the user-input questions.

[0062] Figure 1 This is a schematic diagram of the architecture of a data analysis service system provided in an exemplary embodiment. For example... Figure 1 As shown, the system may include a server 11, a network 12, and several electronic devices, such as a PC (Personal Computer) 13, a mobile phone 14, etc.

[0063] Server 11 can be a physical server containing an independent host, or it can be a virtual server hosted in a host cluster. During operation, server 11 can run a server-side program to implement the relevant functions of the application. For example, when server 11 runs a data analysis service program, it can act as the server for that data analysis service.

[0064] PC13 and mobile phone14 are just some of the types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. During operation, the electronic device can run a client-side program to implement the relevant functions of the application. For example, when the electronic device runs a data analysis service program, it can act as a client for that data analysis service. The aforementioned data analysis service client application can be launched and run on the electronic device. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form. Of course, when using web technologies such as HTML5 or similar, the relevant functions can be implemented through a page displayed by a browser. This browser can be a standalone browser application or a browser module embedded in some applications.

[0065] As for the network 12 that enables interaction between electronic devices such as PC13 and mobile phone 14 and server 11, communication can be achieved using either wired or wireless networks, depending on the communication methods supported by the respective electronic devices. This specification does not impose any restrictions on this. For example, PC13 can support both wired and wireless communication, so it can use either wired or wireless networks as needed. Mobile phone 14 typically only supports wireless communication, so it can use a wireless network for communication.

[0066] The technical solution of this specification will be described in detail below with reference to the accompanying drawings.

[0067] Please see Figure 2 , Figure 2 This document presents a flowchart of an LLM-based data analysis method, which can be specifically applied to an LLM-based data analysis system. The system supports initiating data analysis through questioning; it maintains a set of user question templates with semantically progressive relationships. The method includes the following execution process:

[0068] Step 202: Obtain user-inputted questions; wherein, the user-inputted questions are used to initiate data analysis;

[0069] The aforementioned data analysis system may specifically include a data analysis system built on LLM.

[0070] For example, in some embodiments, the aforementioned data analysis system may specifically include a BI (Business Intelligence) system that is connected to an LLM and can support data analysis functions.

[0071] In practical applications, by integrating LLM reasoning capabilities into a data analysis system, the system's data analysis capabilities can be enhanced and expanded. In particular, it allows users to initiate data analysis by asking questions using natural language, thus enabling the system to support more complex data analysis.

[0072] For example, see Figure 3 In some embodiments, the aforementioned data analysis system may specifically be a data analysis intelligent assistant built on an LLM (Liquid Management System).

[0073] In some embodiments, data analysis systems built on an LLM basis can typically support multiple types of data analysis initiated through questioning.

[0074] For example, please continue to see Figure 3 Taking the aforementioned data analysis system as an example of a data analysis intelligent assistant built on LLM, this data analysis intelligent assistant can display various types of supported data analysis to users through a visual interface. For example... Figure 3 As shown, this data analysis intelligent assistant supports various types of data analysis functions, which may specifically include:

[0075] The first type of data analysis used for data querying (i.e.) Figure 3 The data query function shown is also known as the data retrieval function; the second type of data analysis (i.e., data retrieval) is used to perform data queries and data analysis calculations on the retrieved data. Figure 3 The data analysis functions shown); the third type of data analysis (i.e., data querying and visualization of the retrieved data). Figure 3 The data visualization function shown); the fourth type of data analysis (i.e., the function of defining data indicators and performing indicator calculations related to the data indicators). Figure 3 The functions defined by the indicators are shown.

[0076] It is important to emphasize that the specific data query methods supported by the first type of data analysis mentioned above can include various types of data query methods; the specific data analysis methods supported by the second type of data analysis mentioned above can also include various types of data analysis methods; the visualization presentation methods supported by the third type of data analysis mentioned above can also include various types of visualization presentation methods; and the indicator calculation methods supported by the third type of data analysis can also include various types of indicator calculation methods. In practical applications, these methods usually depend on the data analysis capabilities of the data analysis system itself, and will not be listed one by one in this specification. In addition to outputting the various types of supported data analysis functions to the user through a visual interface, the aforementioned intelligent data analysis assistant can also display some example questions corresponding to these types of data analysis functions through the same visual interface.

[0077] For example, in some cases, data query examples corresponding to the first type of data analysis mentioned above could specifically include "What are the top ten product subcategories in terms of unit cost over the past 90 days?"; data query examples corresponding to the second type of data analysis mentioned above could specifically include "Analyze the histogram of unit cost"; data query examples corresponding to the third type of data analysis mentioned above could specifically include "Query the unit cost of product XX and visualize it in the form of a histogram"; and data query examples corresponding to the fourth type of data analysis mentioned above could specifically include "Define indicator: Calculate the average unit cost of each product subcategory".

[0078] The user question template set maintained by the aforementioned data analysis system may specifically include several user question templates. These user question templates may include any form of templated user questions. These user question templates may have a semantic progression relationship.

[0079] For example, in practical applications, the aforementioned set of user question templates could specifically be a database of user question templates pre-created based on expert experience.

[0080] In some embodiments, in application scenarios where users initiate data analysis by asking questions, there is usually a progressive relationship in the semantics between the initial user question and the follow-up questions. Therefore, the user question templates in the above-mentioned user question template set can specifically be user question templates that can represent the user's follow-up question method.

[0081] For example, suppose a user's initial question is "What is the transaction amount?", and further follow-up questions are "What is the transaction amount in each city?" or "What is the trend of transaction amount in the past 30 days?". These follow-up questions are semantically progressive from the initial question. For instance, "What is the transaction amount in each city?" builds upon the initial question of "What is the transaction amount?" by adding "city" as a data analysis dimension, thus demonstrating semantic progression. Similarly, "What is the trend of transaction amount in the past 30 days?" builds upon the initial question of "What is the transaction amount?" by adding "time range" as a query condition, also demonstrating semantic progression.

[0082] In this scenario, the aforementioned data analysis system can acquire a historical question sample set of all users who have initiated data analysis using the question-asking method, and perform semantic analysis on the question samples contained in the historical question sample set to determine the target historical question samples that have a progressive relationship in the semantics of the questions. Then, it can learn the user's follow-up questioning methods from the aforementioned target historical question samples, create user question templates corresponding to the learned follow-up questioning methods, and save the generated user question templates to a pre-created user question template set.

[0083] It should be noted that when learning users' follow-up questioning methods from the aforementioned target historical question samples, specific artificial intelligence technologies can be used for data modeling; for example, training a neural network to learn users' follow-up questioning methods from the aforementioned target historical question samples will not be detailed in this specification.

[0084] In some embodiments, when the data analysis system creates and learns the user question template corresponding to the additional question method, it may first determine the data analysis methods supported by the data analysis system, and then match the data analysis semantics of the user's additional question method learned from the target historical question sample with the data analysis semantics of the data analysis methods supported by the data analysis system.

[0085] If, among the user append questions learned from the aforementioned target historical question samples, there are user append questions with data analysis semantics that match the data analysis methods supported by the data analysis system, these append question methods can be filtered out (e.g., directly deleted), and then user question templates corresponding to the remaining append question methods learned from the aforementioned target historical question samples can be generated.

[0086] In this way, we can avoid creating invalid user question templates in the user question template set that do not support additional question formats in the data analysis system and are not semantically matched in the data analysis.

[0087] Of course, in addition to learning how users ask follow-up questions from historical question sample sets and creating a set of user question templates based on the learned follow-up question patterns, the above data analysis system can also manually create the set of user question templates in practical applications; for example, in some embodiments, a user question template database can be manually created based on expert experience.

[0088] It should be noted that the specific data format used by the data analysis system in maintaining the aforementioned user question template set will not be specifically limited in this specification. In practical applications, it can be flexibly selected based on specific data analysis needs.

[0089] Please see Figure 4 In some embodiments, the aforementioned set of user question templates may specifically take the form of a knowledge graph. The data analysis system described above can organize this set of user question templates into a knowledge graph and maintain it within the system.

[0090] In this context, the nodes in the aforementioned knowledge graph can specifically represent user question templates. For example, the aforementioned knowledge graph can contain a starting node representing a basic question template, and several other nodes that have a direct or indirect progressive relationship with the basic question template represented by the starting node in terms of the semantics of the question. The edges between the nodes in the aforementioned knowledge graph can specifically represent the semantic progressive relationship between the user question templates represented by the nodes connected by the edge.

[0091] In practical applications, the content of the nodes in the knowledge graph mentioned above can specifically include the content of the user question template represented by that node.

[0092] For example, such as Figure 4 As shown, in some embodiments, the node content in the knowledge graph may specifically include descriptive instructions corresponding to the user question template represented by the node. That is, the node content in the knowledge graph may specifically be a user question template in the form of an instruction. In practical applications, the descriptive instructions contained in the node content of the knowledge graph may be a single descriptive instruction or a set of instruction templates consisting of multiple descriptive instructions.

[0093] It should be noted that, since the aforementioned data analysis system can support various types of data analysis initiated through questioning, and different types of data analysis can typically correspond to different user question templates, in practical applications, the nodes in the aforementioned knowledge graph can contain multiple types of nodes, and different types of nodes can correspond to different types of data analysis; correspondingly, the edges in the aforementioned knowledge graph can also contain multiple types of edges, and different types of edges can correspond to different types of progressive relationships.

[0094] In this specification, when a user needs to perform data analysis through the aforementioned data analysis system, they can use natural language to edit a user question to initiate the data analysis based on their own data analysis needs, and then input the user question into the aforementioned data analysis system; for example, in some embodiments, the user can... Figure 3 The user enters their question into the visual interface shown.

[0095] The aforementioned data analysis system can acquire user-input questions, then perform semantic analysis on the user questions based on the accessed LLM to identify the user's data analysis needs, and finally initiate data analysis based on the user's data analysis needs.

[0096] In some embodiments, the data analysis system can perform semantic analysis on the user's question based on the accessed LLM to extract data analysis entities related to the user's data analysis needs contained in the user's question; for example, these data analysis entities may specifically include data analysis entities corresponding to the data indicators, data analysis dimensions, and data analysis time periods specified by the user; and then, data analysis can be initiated based on the extracted data analysis entities.

[0097] For example, in one scenario, suppose a user enters the question "Do the average amount and average unit cost have a correlation across cities?" The data analysis system can perform semantic analysis on the user's question based on the connected LLM, and based on the semantic analysis results, parse data indicators such as "precise unit price" and "unit cost" from the user's question, as well as data analysis dimensions such as "city". Then, it can initiate a correlation analysis of the data indicators "precise unit price" and "unit cost" on the "city" dimension.

[0098] It should be noted that the specific type of data analysis initiated by the above-mentioned data analysis system based on user-input questions usually depends on the data analysis semantics of the user question, and will not be listed one by one in this specification.

[0099] Step 204: In response to the user's question, obtain the user features corresponding to the user; wherein, the user features are used to represent the user's data analysis preferences; the user features include question features corresponding to the user's question;

[0100] After receiving a user's question, the aforementioned data analysis system can not only initiate data analysis based on the question, but also further obtain user characteristics corresponding to that user.

[0101] Specifically, the user characteristics corresponding to the user can include any form of user characteristics that can be used to represent the user's data analysis preferences.

[0102] On the one hand, in order to ensure that the additional questions generated by the LLM for the user are semantically consistent with the user's input questions and do not deviate from the main body of the user's input questions, the user features corresponding to the user can specifically include the question features corresponding to the user's questions;

[0103] For example, in some embodiments, the user-inputted question can be used as a seed question for embedding encoding, and the user question can be converted into a vector form as a question feature related to the user question.

[0104] On the other hand, the user characteristics corresponding to the user can include not only the question characteristics corresponding to the user's question, but also user characteristics that can reflect the user's personalized questioning needs.

[0105] For example, in some embodiments, the user features described above may also include any one or more of the following types of user features:

[0106] The first type of user characteristic represents the user's real-time data analysis preferences;

[0107] The second type of user characteristic represents the user's historical data analysis preferences;

[0108] The third type of user characteristic represents the data analysis preferences of the user group to which the user belongs.

[0109] It should be noted that the first type of user characteristics mentioned above can specifically include user characteristics that can reflect the user's immediate data analysis needs (such as the user's data analysis interests and data focus); for example, in one example, the first type of user characteristics mentioned above can specifically be user characteristics obtained by feature extraction based on the user's behavioral data in the last hour.

[0110] The second type of user characteristics mentioned above can specifically include user characteristics that can reflect the user's short-term data analysis needs; for example, in one example, the second type of user characteristics mentioned above can specifically be user characteristics obtained by feature extraction based on the user's behavioral data over the past 7 days.

[0111] The third type of user characteristics mentioned above can specifically include user characteristics that reflect the data analysis needs of the user group to which the user belongs; for example, the third type of user characteristics mentioned above can specifically be user characteristics obtained by extracting features based on the behavioral data of the user's department over the past 30 days.

[0112] The specific details of the various user characteristics mentioned above will not be further specified in this description.

[0113] For example, in some embodiments, any one of the above-mentioned user characteristics may specifically include a combination of any one or more of the characteristics shown below:

[0114] Data analysis dimensions related to data analysis;

[0115] Data analysis metrics related to data analysis;

[0116] The time period related to data analysis.

[0117] Step 206: Create prompt words based on the user characteristics and the user question template set; wherein, the prompt words are used to instruct the LLM to generate at least one additional question corresponding to the user question; the additional question has a progressive relationship with the user question in terms of semantics;

[0118] After obtaining the user characteristics corresponding to the user, the data analysis system can further create prompts based on the user characteristics and the user question template set.

[0119] The prompt word is used to instruct the LLM to generate at least one additional question corresponding to the user's question; specifically, the additional question may be a user question that has a progressive relationship with the aforementioned user question in terms of semantics.

[0120] In some embodiments, taking the aforementioned set of user question templates as a knowledge graph, and the node content in the knowledge graph including descriptive instructions corresponding to the user question template represented by the node, as an example, when the data analysis system creates prompt words based on the acquired user features and the aforementioned set of user question templates, it can specifically start from the starting node of the aforementioned knowledge graph, traverse each node in the knowledge graph related to the user features, and obtain the descriptive instructions contained in the content of the traversed nodes. Then, it can further create prompt words based on the user's user features and all the obtained descriptive instructions. It is easy to understand that the prompt words created at this time can specifically include the user's user features as the reasoning context and the obtained descriptive instructions.

[0121] By directly incorporating user question templates as instructions into the constructed prompts, the interpretability of the context contained in the prompts can be enhanced, and the content of the user question templates can be described more accurately, thus improving the inference performance of LLM based on the prompts.

[0122] In some embodiments, before traversing each node in the knowledge graph from its starting node, the data analysis system may calculate the relevance of each node in the knowledge graph to the user features and identify target nodes in the knowledge graph whose relevance to the user features is below a threshold. Then, the system performs pruning on the target node and its corresponding edge in the knowledge graph. Specifically, this pruning process may involve setting the target node and its corresponding edge in the knowledge graph to a state where traversal is impossible (e.g., marking them as a specific state or deleting them directly).

[0123] In this way, on the one hand, the traversal efficiency of the data analysis system when traversing the knowledge graph can be improved; on the other hand, the data analysis system can also prevent the data analysis system from traversing the nodes in the knowledge graph that are not related to the user's user characteristics, so that the user question template represented by only the nodes related to the user's user characteristics can be used as the reasoning context.

[0124] It should be noted that, in addition to the user characteristics and descriptive instructions obtained, which serve as the inference context, the above-mentioned prompts can also include other forms of information as the inference context based on specific needs in practical applications.

[0125] For example, in some embodiments, the above-mentioned prompt words may specifically include the following fields:

[0126] Introductory text: Specifically, it can include some introductory texts pre-configured in the background of the data analysis system.

[0127] The aforementioned set of user question templates refers to the set of user question templates maintained by the data analysis system (such as the expert experience database).

[0128] The feature set refers to the user characteristics obtained by the data analysis system that correspond to the aforementioned users. These user characteristics can include multiple types of user features; for example, the first type, the second type, and the third type of user features mentioned above.

[0129] Chain-of-Thought (COT): Specifically, this can include a chain of thought pre-configured by the data analysis system for the inference computations that the LLM needs to perform later to generate additional questions. The chain of thought is a common method in Prompt Engineering, used to guide the LLM to output intermediate reasoning steps before generating the final answer, much like a human "thinking" step by step. The chain of thought contained in the prompt typically includes multiple pre-defined sub-steps related to the inference computations that generate additional questions; these sub-steps can be described in the form of instructions.

[0130] One-shot: Specifically, this can include a concrete example pre-configured by the data analysis system for the inference computation that the LLM needs to perform subsequently to generate additional questions. In prompting engineering, "one-shot" refers to a method that guides the LLM to understand the task objectives and output format of the inference task by providing an example. This approach lies between "zero-shot" and "multi-shot," and is suitable for application scenarios where the task format is clear and a single example is sufficient to guide the LLM in generating the required output.

[0131] Output constraints: These can include pre-configured constraints by the data analysis system for the additional queries generated by the LLM. The LLM will ultimately output the additional queries according to these constraints. In practice, these constraints can be flexibly specified based on specific needs; for example, the LLM can be instructed to output the generated additional queries in a specific data format.

[0132] Step 208: Input the prompt words into the LLM so that the LLM generates at least one additional question based on the prompt words, and outputs and displays the at least one additional question to the user through a visualization interface corresponding to the data analysis system, so as to guide the user to continue to input the additional questions, and further perform data analysis based on the data analysis results corresponding to the user's questions.

[0133] After the data analysis system creates prompt words based on the user's characteristics and the user question template set, it can further input the prompt words into the LLM so that the LLM can perform inference calculations based on the prompt words and generate at least one additional question; for example, the LLM can perform inference calculations based on the inference context contained in the prompt words.

[0134] Then, at least one additional question generated by LLM can be output and displayed to the user through the visualization interface corresponding to the data analysis system to guide the user to continue to input the additional question. Based on the data analysis results corresponding to the user's input question, further data analysis can be carried out.

[0135] For example, see Figure 5 Taking the aforementioned data analysis system as an example of a data analysis intelligent assistant built on an LLM (Local Management Model), suppose a user enters a question in the assistant's visualization interface: "Are the average amount and average unit cost correlated across cities?" The data analysis system can then perform semantic analysis on this question based on the integrated LLM, and extract data indicators such as "precise unit price" and "unit cost," as well as data analysis dimensions such as "city," from the question based on the semantic analysis results. It then initiates a correlation analysis of the data indicators "precise unit price" and "unit cost" across the "city" dimension, and finally displays the data analysis results to the user through the visualization interface. In addition, the data analysis intelligent assistant can also generate two additional questions for the user, following the process described in the previous embodiment, such as... Figure 5 The system displays additional questions such as "Analyze the distribution of yesterday's maximum unit cost" and "Analyze the histogram of yesterday's precise unit price." Below the data analysis results, these additional questions are displayed to the user to guide them to input further questions. Based on the data analysis results corresponding to the user's input questions, further data analysis is performed. At this point, the user can continue to initiate data analysis with the intelligent data analysis assistant by operating within the visualization interface (e.g., clicking) on ​​these additional questions, further analyzing the data based on the data analysis results corresponding to the user's input questions.

[0136] In some embodiments, the at least one additional question generated by the LLM based on the above prompt words can specifically be a set of additional questions; in this case, when the data analysis system outputs and displays the results to the user through a visualization interface corresponding to the data analysis system, it can specifically calculate the relevance score between the additional questions in the set of additional questions and the user's user characteristics, and sort the additional questions in the set of additional questions based on the calculated relevance score.

[0137] After the sorting is completed, the N additional questions with the highest relevance scores to the user characteristics of the above users are determined from the sorted set of additional questions. These N additional questions are then displayed to the user through the visualization interface corresponding to the data analysis system; where N is a preset threshold.

[0138] In this way, the N additional questions that are most relevant to the user's characteristics from the set of additional questions generated by LLM can be recommended to the user.

[0139] In some embodiments, if the user's user characteristics include multiple types of user characteristics, different weight values ​​can be set for each type of user characteristic; the weight value can specifically be used to represent the importance of the user characteristic to the data analysis initiated by the user's input question.

[0140] In this scenario, when the data analysis system calculates the relevance score between the additional questions in the set of additional questions and the user's user characteristics, it can specifically calculate the relevance score between the additional questions in the set of additional questions and the various types of user characteristics mentioned above, obtaining the relevance score corresponding to each type of user characteristic. Then, based on the weight values ​​corresponding to each type of user characteristic, the system can perform a weighted calculation on the calculated relevance scores corresponding to each type of user characteristic to obtain the final relevance score between the additional question and the user's user characteristics.

[0141] It should be noted that, in practical applications, the weight values ​​corresponding to these various user characteristics can be flexibly set based on specific needs.

[0142] For example, in some implementations where the data analysis system supports initiating multiple types of data analysis via questions, the weight value corresponding to each type of user characteristic can be dynamically determined based on the type of data analysis initiated by the user's input question. In other words, the weight value corresponding to each type of user characteristic can be dynamically determined based on the type of data analysis initiated by the user's input question. The weight values ​​corresponding to different types of user characteristics can differ across different types of data analysis. For instance, if a certain type of data analysis focuses more on a specific user characteristic, then that user characteristic can be assigned a higher weight value than other user characteristics.

[0143] In this way, weight values ​​can be flexibly set for user characteristics based on the specific type of data analysis initiated by the user. This allows for different types of user characteristics to be emphasized in different types of data analysis, thereby meeting the personalized requirements of data analysis.

[0144] In the above embodiments, on the one hand, personalized supplementary questions can be generated for users without deviating from the semantic analysis of user questions input by users;

[0145] For example, by including user characteristics that reflect the user's data analysis preferences in the prompts, and incorporating the question characteristics of the user's input into these user characteristics as context, the LLM can ensure that the supplementary questions generated during the inference phase are semantically consistent with the user's original question, and can also generate supplementary questions that match the user's data analysis preferences.

[0146] On the other hand, by asking additional questions to guide users to conduct further data analysis based on the data analysis results corresponding to the user's question, it is possible not only to guide users to explore the data in depth according to the thinking patterns in the user question template set and to dig out more valuable information, but also to improve the depth and quality of data analysis.

[0147] For example, by transforming expert experience into a set of user question templates and organizing them into a knowledge graph, expert experience in a relevant field can be converted into a tree-like questioning approach. Each node in the knowledge graph represents a further questioning direction based on the previous node. This logical and hierarchical questioning method can guide users to explore data according to the expert's thinking pattern, uncovering more valuable information and improving the depth and quality of data analysis.

[0148] Figure 6 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment. For example... Figure 6As shown, device 600 mainly consists of a communication interface 602, a user interface 604, a processor 606, and a data storage 608. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 610. The communication interface 602 enables device 600 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 602 may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface 602 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 602 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 602 may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0149] User interface 604 includes receiving user input and providing output to the user. Therefore, user interface 604 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 604 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 604 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 600 may support remote access from other devices via communication interface 602 or another physical interface (not shown). User interface 604 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 604 may also be configured as a display device for rendering or displaying text fragments.

[0150] Processor 606 may contain one or more general-purpose processors and / or special-purpose processors.

[0151] Data storage 608 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 606. Data storage 608 may include removable and non-removable components.

[0152] Processor 606 is capable of executing program instructions 618 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 608 to perform the various functions described herein. Data storage 608 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 600, enable device 600 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Processor 606 executing program instructions 618 may result in processor 606 using data 612.

[0153] For example, program instructions 618 may include an operating system 622 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 600 and one or more applications 620 (e.g., a browser, social application, or game application). Similarly, data 612 may include operating system data 616 and application data 614. Operating system data 616 is primarily accessible to the operating system 622, while application data 614 is primarily accessible to one or more applications 620. Application data 614 may reside in a file system visible or hidden from the user of device 600.

[0154] Application 620 can communicate with operating system 622 through one or more application programming interfaces (APIs). These APIs help application 620 read and / or write application data 614, transmit or receive information via communication interface 602, receive or display information on user interface 604, etc.

[0155] In some terminology, application 620 may be simply referred to as "app". Furthermore, application 620 can be downloaded to device 600 through one or more online app stores or app markets. However, applications can also be installed on device 600 in other ways, such as through a web browser or a physical interface on device 600 (e.g., a USB port).

[0156] Please refer to Figure 7 This specification also proposes an LLM-based data analysis device, which can be applied to, for example... Figure 6 The device shown implements the technical solution of this specification. Specifically, the device supports data analysis initiated through questioning; the device maintains a set of user question templates with a progressive semantic relationship; the device may include:

[0157] The first acquisition module 701 acquires user-input questions; wherein, the user-input questions are used to initiate data analysis.

[0158] The second acquisition module 702, in response to the user's question, acquires user features corresponding to the user; wherein, the user features are used to represent the user's data analysis preferences; the user features include question features corresponding to the user's question;

[0159] The creation module 703 creates prompt words based on the user characteristics and the user question template set; wherein, the prompt words are used to instruct the LLM to generate at least one additional question corresponding to the user question; the additional question has a progressive relationship with the user question in terms of semantics;

[0160] The input module 704 inputs the prompt words into the LLM, so that the LLM generates at least one additional question based on the prompt words, and outputs and displays the at least one additional question to the user through a visualization interface corresponding to the data analysis system, so as to guide the user to continue to input the additional questions, and further perform data analysis based on the data analysis results corresponding to the user's questions.

[0161] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0162] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0163] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0164] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0165] What those skilled in the art will understand is:

[0166] In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0167] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0168] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0169] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0170] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0171] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0172] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A LLM-based data analysis method applied to a LLM-based data analysis system; wherein, The data analysis system supports initiating data analysis in the form of a question; the data analysis system maintains a set of user question templates that have a progressive relationship in the semantics of the questions; comprising: obtaining a user question input by a user; wherein the user question is used to initiate data analysis; obtaining user characteristics corresponding to the user in response to the user question; wherein the user characteristics are used to represent the data analysis preferences of the user; the user characteristics include question characteristics corresponding to the user question; creating a prompt based on the user characteristics and the set of user question templates; wherein the prompt is used to instruct the LLM to generate at least one additional question corresponding to the user question; the additional question has a progressive relationship in the semantics of the question with the user question; inputting the prompt to the LLM to generate the at least one additional question based on the prompt by the LLM, and outputting the at least one additional question to the user through a visualization interface corresponding to the data analysis system to guide the user to continue inputting the additional question, and further perform data analysis based on the data analysis results corresponding to the user question.

2. The method of claim 1, wherein the user question templates in the set of user question templates represent the user's additional question manner; The method further comprises: obtaining a set of historical question samples of all users who have initiated data analysis in the form of a question in the data analysis system; performing semantic analysis on the question samples contained in the historical question samples to determine target historical question samples that have a progressive relationship in the semantics of the questions in the set of historical question samples; learning the user's additional question manner from the target historical question samples, creating a user question template corresponding to the additional question manner, and saving the generated user question template to the pre-created set of user question templates.

3. The method of claim 2, wherein the set of user question templates comprises a knowledge graph; wherein The nodes in the knowledge graph represent user question templates; the edges in the knowledge graph represent the progressive relationship in semantics between user question templates.

4. The method of claim 3, wherein the content of the nodes in the knowledge graph contains description instructions corresponding to the user question templates represented by the nodes; Creating a prompt based on the user characteristics and the set of user question templates comprises: starting from a starting node of the knowledge graph, traversing each node in the knowledge graph that is related to the user characteristics, and obtaining the description instructions contained in the content of the traversed nodes; further creating a prompt based on the user characteristics and the obtained description instructions.

5. The method of claim 4, before starting from a starting node of the knowledge graph, traversing each node in the knowledge graph, the method further comprises: calculating the relevance of each node in the knowledge graph to the user characteristics; determining target nodes in the knowledge graph that have a relevance to the user characteristics lower than a threshold value; Pruning the target node and edges corresponding to the target node included in the knowledge graph.

6. The method of claim 1, wherein the at least one additional question is an additional question set generated by the LLM based on the prompt word; outputting and displaying, to the user through a visualization interface corresponding to the data analysis system, the N additional questions with the highest correlation scores with the user features; calculating a correlation score of each additional question in the additional question set with the user features; sorting the additional questions in the additional question set based on the calculated correlation scores; determining N additional questions with the highest correlation scores with the user features from the sorted additional question set, and outputting and displaying the N additional questions to the user through the visualization interface corresponding to the data analysis system; wherein N is a preset threshold.

7. The method of claim 6, the user characteristics comprising a plurality of types of user characteristics; wherein, The plurality of types of user features correspond to different weight values respectively; the weight values represent the importance of the user features to the data analysis initiated by the user questions; calculating a correlation score of each additional question in the additional question set with the user features, comprises: calculating a correlation score of each additional question in the additional question set with each type of user feature in the plurality of types of user features, to obtain a correlation score corresponding to each type of user feature in the plurality of types of user features; weighting the calculated correlation score corresponding to each type of user feature based on the weight value corresponding to the type of user feature, to obtain the correlation score of the additional question with the user features.

8. The method of claim 7, wherein the data analysis system supports initiating a plurality of types of data analysis in the form of questions; and the weight value corresponding to each type of user feature in the plurality of types of user features is a weight value dynamically determined based on the type of data analysis initiated by the user question.

9. The method of claim 8, wherein the plurality of types of data analysis comprises: a first type of data analysis for data query; a second type of data analysis for data query and data analysis calculation on the queried data; a third type of data analysis for data query and visualization presentation of the queried data; a fourth type of data analysis for defining a data indicator and performing indicator calculation related to the data indicator.

10. The method of claim 7, wherein the plurality of types of user features comprises a question feature corresponding to the user question, and any one or more of the following types of user features: a first type of user feature representing a real-time data analysis preference of a user; a second type of user feature representing a historical data analysis preference of a user; a third type of user feature representing a data analysis preference of a user group to which the user belongs.

11. The method of claim 10, wherein the user features comprise a combination of any one or more of the following features: a data analysis dimension related to data analysis; a data analysis indicator related to data analysis; a data analysis time period related to data analysis.

12. The method of claim 1, the data analysis system comprising a BI system that supports data analysis functions.

13. An electronic device, comprising: including: a processor; a memory for storing processor-executable instructions; wherein the processor implements the steps of the method of any of claims 1-12 by executing the executable instructions.

14. A computer-readable storage medium, characterized in that, a computer program product tangibly embodied on a computer readable medium having instructions that, when executed, implement the steps of the method of any of claims 1-12.

15. A computer program product, characterised in that, a computer program product tangibly embodied on a computer readable medium having instructions that, when executed, implement the steps of the method of any of claims 1-12. a computer program product tangibly embodied on a computer readable medium having instructions that, when executed, implement the steps of the method of any of claims 1-12.

Images