Data insight exploration and visual analysis method and device based on large model

Abstract

The invention discloses a data insight exploration and visual analysis method based on a large model, and the method comprises the following steps: constructing a database which comprises a data name and corresponding data content, and carrying out the encoding processing of the data name and the corresponding data content through an encoder; constructing a corresponding interaction model based on the database and the large language model framework, wherein the interaction model comprises a data analysis module, an insight extraction module and an insight classification module; and analyzing the data through the interaction model to obtain a visual analysis result. The invention further provides a data insight exploration and visual analysis device. According to the method provided by the invention, the efficiency of data analysis can be improved, and the rheological condition of the data and the analysis process can be presented by a visual view, so that more perfect data analysis support is provided for a user.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence data analysis, and particularly relates to a data insight exploration and visualization analysis method and device based on a large model. Background Art

[0002] Data analysis is an indispensable tool in modern enterprises and personal life. It helps us discover patterns, identify trends, and make data-based decisions by collecting, processing, and interpreting large amounts of information.

[0003] However, there is still a relatively high threshold for data analysis work. Professional data analysts need to undergo long-term training to master various data analysis methods and have good coding skills. The scarcity of data analysts caused by this high threshold can no longer well match the growing social demand for data analysis.

[0004] Traditional data analysis processes use spreadsheet tools such as Excel to display and process data. The freedom of such tools is limited, and they cannot process data flexibly, and users need to be highly proficient in the tools.

[0005] Modern data analysts use programming languages such as Python to process data. These programming languages are more powerful, but they also require higher requirements for analysts and need to have excellent coding skills.

[0006] Emerging large language models already have strong capabilities and can read data files provided by users and write data processing code according to user requirements. However, their linear dialogue interface does not allow users to have a good overview of the overall analysis process, nor can they quickly find previous analysis results.

[0007] Patent document CN118981495A discloses an artificial intelligence-based digital visual decision-making assistance information system, including a data processing unit, a data model training module, and a data information terminal; the data processing unit includes a data collection module, a data conversion module, a data summary module, a data screening module, a domain authorization module, a data storage module, and a data analysis module; the data collection unit is unidirectionally connected to the data conversion module, the data conversion module is unidirectionally connected to the data summary module, the data summary module is unidirectionally connected to the data screening module, the data screening module is respectively interconnected with the domain authorization module and the data storage module, and the data screening module is interconnected with the data analysis module, by setting the data model training module.

[0008] Patent document CN118735373A discloses a visual data analysis method, system, storage medium and computer program product. Step 1: Construct at least one business data analysis model that matches the business requirements; the model includes, but is not limited to, a power generation volume visual analysis model, a power generation income visual analysis model, a project power generation cost visual analysis model, a platform fee visual analysis model, and a project and resource visual display model; Step 2: Use the multi-dimensional table function of the Feishu platform to receive and store the underlying business data sources; Step 3: Automatically process the underlying business data stored in the Feishu platform according to the preset business logic, including steps such as data cleaning, data transformation, and data aggregation, to generate processed data; Step 4: Automatically generate corresponding visual analysis charts based on the processed data and the constructed business data analysis models, and display the visual analysis charts on the dashboard of the Feishu platform; Step 5: Provide a user interaction interface that allows users to customize data combinations, adjust analysis model parameters, and modify the display methods of visual charts to meet different analysis needs. Summary of the Invention

[0009] The purpose of the present invention is to provide a data insight exploration and visual analysis method and device based on a large model. This method can improve the efficiency of data analysis and can present the rheological situation of data and the analysis process in a visual view, thereby providing more complete data analysis support for users.

[0010] In order to achieve the first object of the present invention, the following technical solutions are provided: A data insight exploration and visual analysis method based on a large model includes the following steps: Construct a database that includes data names and corresponding data contents, and encode the data names and corresponding data contents through an encoder; Construct a corresponding interaction model based on the database and the large language model framework. The interaction model includes a data analysis module, an insight extraction module, and an insight classification module: The data analysis module encodes the content in each round of interaction dialogue, matches it with the database based on the encoding result, and generates a corresponding reply text based on the matching result; The insight extraction module is used to collect keywords in each reply text to obtain the corresponding data names and data contents as insight results, and construct an insight block diagram for this round of dialogue based on the insight results; The insight classification module uses each round of dialogue as the root node, the insight block diagram corresponding to the reply text as the sub-node, to construct a tree-like topological structure of the theme tree view, and splices the insight block diagrams at the same level from beginning to end to output the insight view.

[0011] The present invention uses a large language model to replace the user in performing data analysis and summarization tasks, decomposes the analysis process with insights as the smallest granularity, and presents the evolution of data context and semantic context simultaneously with visual views, enabling the user to quickly understand the overall analysis process and solving the problem that the linear chat interface of current large model products makes it difficult for users to review the analysis process from a higher perspective.

[0012] Specifically, preprocessing is required before encoding the data name and the corresponding data content. The preprocessing includes collecting a preset number of rows of digital type data, as well as taking the mean and extreme value operations to ensure the uniformity of data in the subsequent processing.

[0013] Specifically, the range of the preset number of rows is 1 - 5.

[0014] Specifically, the insight block diagram includes a long rectangle and a grid formed by dividing the long rectangle. The length direction of the long rectangle is used as the time axis, and the data name and the corresponding data content are sequentially set in the width direction of the long rectangle.

[0015] Specifically, the background color of the insight block diagram is uniformly set according to the root node it belongs to, so as to facilitate the distinction of different types of theme contents.

[0016] Specifically, when viewing any insight block diagram in the insight view, a relevant insight view will also be generated. The relevant insight view performs semantic similarity matching by traversing the content in the current insight block diagram with the selected insight, and sorts the matching results in reverse order based on the time sequence to output the corresponding view.

[0017] Specifically, the encoding process is encoded in a Python environment.

[0018] To achieve the second object of the present invention, the following technical solution is provided: A data insight exploration and visualization analysis device for performing the steps of the above-mentioned visualization data analysis method based on a large model to obtain the final visualized insight view.

[0019] Compared with the prior art, the beneficial effects of the present invention are: It is proposed to decompose the analysis process with insights as the smallest granularity, and present the evolution of data context and semantic context simultaneously with visual views, enabling the user to quickly understand the overall analysis process and solving the problem that the linear chat interface of current large model products makes it difficult for users to review the analysis process from a higher perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is the flowchart of the visualization analysis method provided in this embodiment; Figure 2 It is the schematic diagram of the data analysis module provided in this embodiment; Figure 3 Schematic diagram of the insight extraction module provided in this embodiment; Figure 4 Schematic diagram of the insight classification module provided in this embodiment; Figure 5 Schematic diagram of the insight view provided in this embodiment; Figure 6 Schematic diagram of the tree - shaped topological structure provided in this embodiment; Figure 7 Schematic diagram of the data insight exploration and visualization analysis device provided in this embodiment; Figure 8 Schematic diagram of the movie data type provided in this embodiment; Figure 9 Schematic diagram of the encoding and analysis process provided in this embodiment; Figure 10 Insight block diagram corresponding to the analysis result provided in this embodiment; Figure 11 Schematic diagram of the main interface after multi - round conversations provided in this embodiment. Detailed implementation manners

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, rather than all, of the embodiments of the present invention. Usually, the components of the embodiments of the present invention described and illustrated herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the protection scope of the present invention.

[0022] As Figure 1 shown, a data insight exploration and visualization analysis method based on a large model provided in this embodiment includes the following steps: Construct a database, which includes data names and corresponding data contents, and encode the data names and corresponding data contents through an encoder; Based on the database and the large - language model framework, construct a corresponding interaction model, and the interaction model includes a data analysis module, an insight extraction module, and an insight classification module: The data analysis module encodes the content in each round of interaction conversation, matches based on the encoding result with the database, and generates a corresponding reply text based on the matching result; The insight extraction module is used to collect keywords in each response text to obtain the corresponding data names and data contents as insight results, and construct an insight block diagram for the current round of conversation based on the insight results; The insight classification module takes each round of conversation as the root node, and the insight block diagram corresponding to the response text as the child node to construct a tree-like topological structure of the theme tree view, and splices the insight block diagrams at the same level end to end to output the insight view.

[0023] More specifically, the above three modules are all played by a large language model. The large language model is pre-set with prompt words and equipped with special tools and memories to complete their respective tasks. When the user inputs a question, the data analysis module first receives the input, writes and executes code according to the user's intention, and generates a response; then, the insight extraction module extracts insights and their related information from the response, and updates them to the insight block diagram, and the relevant insight view also changes with the change of the insight block diagram; finally, the insight classification module classifies the newly generated insights under a certain theme and sub-theme, and updates the tree-like topological structure to obtain the corresponding insight view.

[0024] As Figure 2 shown, the data analysis module provided in this embodiment is as follows: The data analysis module receives the user's question, combines the previous data set and historical conversations, writes the corresponding data analysis Python code, and then executes the code through the Python interpreter to obtain the result, and generates a response according to the result.

[0025] As Figure 3 shown, the insight extraction module provided in this embodiment, whenever the data analysis model outputs a response, the system will input the new round of conversation (that is, the latest user question + data analysis model response) into the insight extraction model, and the model will combine the existing insights and background knowledge about data analysis (such as what is an insight, what are the relevant information of an insight, etc.) to extract the insights in the conversation and update them to the insight details and insight block diagram.

[0026] As Figure 4 shown, the insight classification module provided in this embodiment is as follows: The insight classification module summarizes the new insights, and the system will input them into the insight classification model. The model classifies the new insights in combination with the previous classification situation. If there is a suitable theme for the new insight in the existing themes, it will be directly classified into that theme; otherwise, a new theme will be generated and the new insight will be classified under that theme.

[0027] As Figure 5As shown, it is the insight view provided by this embodiment. A corresponding tree topology is constructed with each round of conversation as the root node and the insight block diagram corresponding to the reply text as the sub-node, and the insight block diagrams at the same level are spliced end to end to output the insight view.

[0028] More specifically, each cell in the insight view represents an insight. The cells are arranged from top to bottom, and the arrangement order is the chronological order in which the insights are extracted. The system assigns a theme color to each theme, and the background color of each insight cell is the theme color of its insight theme (different textures in the figure represent different colors). The names of all data attributes are arranged at the top of the view, and the data attributes corresponding to the dots in each insight cell upwards are the data attributes involved in the insight. The insight view shows the insights that appear in the conversation process in chronological order for the user, and the user can click on the insight block to make the chat interface jump to the conversation where the insight appears.

[0029] In addition, when viewing any insight block diagram in the insight view, a relevant insight view will also be generated. The relevant insight view traverses the content in the current insight block diagram to perform semantic similarity matching with the selected insight, and arranges the matching results in reverse order based on the chronological order to output the corresponding view; Based on the current insight displayed in the insight view, the system will display the insights related to the current insight in the relevant insight view arranged by data and semantic similarity (if the similarities are the same, they will be arranged in reverse chronological order). To save screen space, only the insight summary and its visualization are shown in each insight card. In the data-related list, the system will display the three insights with the highest data attribute coincidence degree with the current insight according to the number of coincident data attributes (for example, if the data attributes of the current insight are (A, B, C) and the data attributes of another insight are (A, B, D), then the coincident data attributes of these two insights are (A, B), and the number is two); in the semantic-related list, the three insights with the highest similarity to the current insight will be shown according to the semantic similarity between each insight and the current insight (calculated by the cosine similarity of their embeddings). The user can click on each insight card in the list to view its detailed information for comparison or reference.

[0030] As Figure 6 shown, it is the tree topology provided by this embodiment, that is, in the theme tree view, the existing themes and sub-themes are organized in a two-layer tree form. The root node of each tree is a theme, and all its sub-themes are connected below. Each insight cell in the insight list on the left side of the theme tree is connected to its corresponding sub-theme. Different trees are given different colors, and the color source is the theme color of the tree root theme.

[0031] This embodiment also provides a data insight exploration and visualization analysis device for performing the steps of the visualization analysis method provided in the above embodiment. Specifically, it includes: on the far left is the dialogue area where users pose analysis questions and the system gives responses; in the upper part of the middle column is the insight block diagram, which shows the detailed content of the insights contained in the current content of the left dialogue area, including information such as the summary of the insight, the data attributes involved, and the visualization charts; in the lower part of the middle is the relevant insight view, and on the right are the insight view and the tree-like topological structure.

[0032] To better illustrate the technical effects of this embodiment, the following data processing process is provided: First, a data set is constructed for this task, and a set of data sets about movies is selected for collation. The data types are as Figure 8 shown.

[0033] The collated data set is uploaded to the data insight exploration and visualization analysis device, and the first conversation is carried out. After entering the question "What is the trend of the total revenue of the film industry in recent years?" into the device, the device will write code to explore the data set and give an analysis result. The process is as Figure 9 shown.

[0034] After the answer is given, the insight extraction module will extract the insights and the associated information of the insights, that is, the data attributes involved (i.e., column names), key codes, code outputs, and visualization charts.

[0035] In the conversation regarding the question "What is the trend of the total revenue of the film industry in recent years?", the insight extracted by the insight extraction module is "The total revenue of movies from 1996 to 2010 showed a fluctuating trend, reaching peaks in 2000, 2002, 2004, and 2007, and showing a significant decline in 2010.", and the data attributes involved are Release Year and Worldwide Gross. These contents are displayed in the insight details view.

[0036] As Figure 10 shown, in the insight block diagram, each column in the top chart is a data attribute (column name) of the data set. The height of the bar chart represents the number of times this data attribute is involved in all explored insights. Since the data attributes involved in the first insight are Release Year and Worldwide Gross, the bar charts of these two columns are the highest, and the bar charts of other attributes are at the lowest level. The first insight is added to this view in the form of a long rectangle. Since this insight involves these two attributes of Release Year and Worldwide Gross, two dots appear in the rectangle, directly below the names of these two attributes.

[0037] After that, the insight classification module will classify this insight. Since it is the first insight, the insight classification model generates a new topic name for this insight - "Revenue Trend", and assigns yellow to this topic. The system connects the insight block diagram with the topic in the topic tree view with a line to indicate that this insight belongs to this topic.

[0038] After that, continue to input the subsequent questions "What is the impact of different movie ratings on revenue?" and "Which are the top 10 most profitable movies?", and repeat the above process to obtain the final main interface, as Figure 11 shown. Since the topic of "Which are the top 10 most profitable movies?" is not suitable to be classified under "Revenue Trend", a new branch topic "Category Performance Analysis" is generated for this topic, and red is assigned as the topic color.

[0039] In addition, the terms "up", "down", "inside", "outside", "front", "back" are for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless specifically stated otherwise, the relative steps, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0040] Of course, the above are only specific embodiments of the present invention and are not intended to limit the scope of the present invention. Any equivalent changes or modifications made according to the structure, features, and principles described in the scope of the patent application of the present invention should be included in the scope of the patent application of the present invention.

[0041] Finally, it should be noted that: the above embodiments are only specific implementation manners of the present invention, used to illustrate the technical solutions of the present invention, rather than limiting it. The protection scope of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: any person skilled in the art within the technical scope disclosed by the present invention can still modify the technical solutions recorded in the foregoing embodiments or can easily think of changes, or perform equivalent replacements on some of the technical features; and these modifications, changes, or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A data insight exploration and visualization analysis method based on a large model, characterized in that: The following steps are involved: Constructing a database, which includes data names and corresponding data contents, and encoding the data names and the corresponding data contents through an encoder; A corresponding interaction model is constructed based on the database and the large language model framework, and the interaction model includes a data analysis module, an insight extraction module and an insight classification module: The data analysis module encodes the content of each round of interactive dialogue, matches the encoding result with the database, and generates a corresponding reply text based on the matching result; The insight extraction module is used to collect keywords in each reply text to obtain corresponding data names and data contents as insight results, and construct an insight diagram for this round of conversation based on the insight results; The insight classification module takes each round of dialogue as the root node and the insight diagram corresponding to the reply text as the child node to construct a topic tree view with a tree topology structure, and splices the insight diagrams at the same level from beginning to end to output an insight view.

2. The data insight exploration and visualization analysis method based on a large model according to claim 1 is characterized in that: Before encoding the data name and the corresponding data content, preprocessing is required, and the preprocessing includes collecting a preset number of rows of digital type data, and taking the average and extreme value operations.

3. The data insight exploration and visualization analysis method based on a large model according to claim 2 is characterized in that: The preset number of rows ranges from 1 to 5.

4. The data insight exploration and visualization analysis method based on a large model according to claim 1 is characterized in that: The insight box diagram includes a rectangular box and a grid divided by the rectangular box. The length direction of the rectangular box is used as the time axis, and the data name and the corresponding data content are set in sequence along the width direction of the rectangular box.

5. The data insight exploration and visualization analysis method based on a large model according to claim 4 is characterized in that: The background color of the insight diagram is uniformly set according to the root node to which it belongs.

6. The data insight exploration and visualization analysis method based on a large model according to claim 1 is characterized in that: When viewing any insight diagram in the insight view, a related insight view will also be generated. The related insight view traverses the content in the current insight diagram to match the semantic similarity with the selected insight, and arranges the matching results in reverse order based on chronological order to output the corresponding view.

7. The data insight exploration and visualization analysis method based on a large model according to claim 1 is characterized in that: The encoding process is performed in a python environment.

8. A data insight exploration and visual analysis device, characterized in that: Used to execute the steps of the large model-based visual data analysis method as described in any one of claims 1 to 7.

Images