Visual analysis method and system based on big language model enhancement

Abstract

The invention discloses a visual analysis method and system based on large language model enhancement. The method comprises the following steps: (1) providing a data set to be visually analyzed; (2) analyzing data of the data set by adopting a large language model, generating one or more insight recommendations, and providing the insight recommendations for a user to select; (3) aiming at the selected insight type, analyzing the data of the data set by adopting a large language model, and performing insight exploration so as to obtain an insight result; (4) optionally, repeating the steps (2) and (3); and (5) generating a visual analysis summary report based on the analyzed insight type and insight result. The invention provides a unique and intelligent instructive architecture, so that the large language model is fused into the flow of each stage of visual analysis of people, and the exploration of people is intelligently assisted in an interaction mode, thereby greatly improving the efficiency and quality of visual analysis.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a visual analysis method and system based on large-scale language model enhancement. Background Art

[0002] Visual analytics combines data analysis techniques with visualization to enable effective understanding, reasoning, and decision-making based on large and complex data sets. However, the visual analysis process often requires a lot of effort.

[0003] On the user side, the data interpretation and analysis process often leads to inefficiencies. Several challenges contribute to this inefficiency: a steep learning curve for visual analytics systems, the ease of getting lost in data exploration, and the difficulty in generating insights.

[0004] These issues highlight the need for a more informed approach to visual analytics. Specifically, first, system learning challenges stem from unfamiliarity with visual analytics systems, resulting in reduced efficiency in mastering visual mapping and interaction. Second, during data exploration, the multitude of approaches to analyzing data and the lack of structured guidance and direction make it difficult to effectively discover insights. Finally, tracking and summarizing key findings is often difficult, and summarizing insights is a time-consuming task that requires significant effort and attention to detail.

[0005] To address these issues, researchers have focused on developing tutorials to assist learning in visual analytics systems, predicting insights based on interaction histories to aid user exploration, and providing editing tools to summarize exploration records into reports. However, these approaches often fail to fully leverage the innovative nature of intelligent algorithms and lack the flexibility to adapt to various visual analytics (VA) systems. Furthermore, there is a lack of generalizable intelligent methods that work across diverse VA systems.

[0006] Therefore, there is an urgent need in this field to develop more efficient visual analysis methods and systems for effective understanding, reasoning and decision-making of large or complex data sets. Summary of the Invention

[0007] The purpose of the present invention is to provide a more efficient visual analysis method and system, namely, a large language model-enhanced interactive visual analysis system (LEVA).

[0008] Another object of the present invention is to provide an application of the large language model enhanced interactive visual analysis system of the present invention, especially for performing visual analysis on large or complex data sets, thereby providing more effective and higher quality understanding, reasoning and decision-making.

[0009] In a first aspect of the present invention, a large language model enhanced visual analysis method is provided, comprising the steps of:

[0010] (1) Provide a data set to be visually analyzed;

[0011] (2) analyzing the data of the dataset using a large language model to generate one or more insight recommendations, and providing the insight recommendations to the user for selection, thereby determining a first insight type, wherein the first insight type is selected by the user from the insight recommendations, or is not selected from the insight recommendations but is input by the user;

[0012] (3) for the selected first insight type, using a large language model to analyze the data of the dataset and perform insight exploration, thereby obtaining an insight result for the first insight type;

[0013] (4) optionally, based on the insight result in the previous step, repeating steps (2) and (3) to obtain an insight result of the Nth insight type, where N is a positive integer ≥ 2; and

[0014] (5) Generate a summary report of the visual analysis based on the analyzed insight types and insight results.

[0015] In another preferred embodiment, the visual analysis is performed in a visual analysis system based on the LEVA architecture.

[0016] In another preferred embodiment, in step (2), insight recommendations are provided based on the understanding of the system, analysis tasks, user interactions and data through large language model analysis.

[0017] In another preferred embodiment, in step (2), candidate insights are determined through large language model analysis, and insight evaluation is performed on the candidate insights to obtain a weighted score for each candidate insight.

[0018] In another preferred embodiment, the weighted score is calculated based on the respective scores and weights of significance, task relevance and influence.

[0019] In another preferred embodiment, in step (2), the insight recommendations are sorted by weighted score of each candidate insight obtained from the insight evaluation and provided to the user for selection.

[0020] In another preferred embodiment, step (3) further includes the step of integrating the insight results into the visual analysis system in the form of annotations and highlights.

[0021] In another preferred embodiment, step (5) includes: automatically combing and summarizing the user's exploration history through LLM technology, accurately identifying and highlighting key insights and decision points in the analysis process.

[0022] In another preferred embodiment, step (5) includes: converting the automatically generated insight results into a report with both pictures and texts.

[0023] In another preferred embodiment, step (5) includes: visualizing historical exploration records, automatically summarizing the user's exploration history, converting these summaries into code generation reports, or a combination thereof.

[0024] In another preferred embodiment, in step (5), LLM is used to synthesize the visual images and explanations to automatically generate a detailed summary report.

[0025] In another preferred embodiment, the summary report is an analysis report containing or using LaTeX code.

[0026] In another preferred embodiment, the insight is defined by the following four attributes: insight type, insight parameter, insight topic, and insight score.

[0027] 7. The method of claim 1, wherein the insight topic is defined by four attributes: subspace, dimension, measurement, and context.

[0028] In another preferred embodiment, the insight score is obtained based on the influence score, the significance score and the relevance score.

[0029] In another preferred embodiment, the insight score is a weighted score.

[0030] In another preferred embodiment, the insight score is calculated using the following formula:

[0031] Score=∑ k∈{significance ,impact,relevance}w k ×score k

[0032] in,

[0033] Score indicates the insight score;

[0034] k∈{significance,impact,relevance} indicates that the evaluation factors include significance, impact, and relevance;

[0035] score k Represent the scores of significance score, influence score and relevance score respectively;

[0036] w k Represent the weights of significance score, influence score and relevance score respectively.

[0037] In another preferred embodiment, the method further comprises:

[0038] (i) Provide users with interactive teaching of the Large Language Model Enhancement Visual Analysis method (LEVA method).

[0039] In another preferred embodiment, the step (i) is performed before, during and / or after steps (1) to (5).

[0040] In another preferred embodiment, the interactive teaching of the visual analysis method for large language model enhancement provided to the user includes: constructing an interactive tutorial, and providing the constructed interactive tutorial to the user for learning.

[0041] In another preferred embodiment, the interactive tutorial includes: creating a grammar, generating tutorial prompts, and integrating these tutorials into the content of the original system.

[0042] In a second aspect of the present invention, a device for performing large language model enhanced visual analysis is provided, comprising:

[0043] (Z1) a data module, the data module being configured to store a data set to be analyzed;

[0044] (Z2) a LEVA analysis module, wherein the LEVA analysis module is configured to perform a large language model-enhanced visual analysis method, wherein the large language model-enhanced visual analysis method comprises the steps of:

[0045] (Y1) Provide insights and recommendations to users based on data insights automatically generated by LLM;

[0046] (Y2) performing corresponding visual analysis on the dataset based on the insight type in the insight recommendation selected by the user; and

[0047] (Y3) Organize the user's exploration process and generate a summary report;

[0048] (Z3) an input module configured to input information and / or data related to the visualization analysis; and

[0049] (Z4) An output module, configured to output: insight recommendations, a summary report of the visual analysis, or a combination thereof.

[0050] In another preferred example, the data module is further configured to acquire data and form a data set to be analyzed or a portion of the data in the data set to be analyzed.

[0051] In another preferred example, the data module further includes a data preprocessing module.

[0052] In another preferred embodiment, the data preprocessing module is configured to convert the collected metadata into a form that is easy for visual encoding and insight mining in a visual analysis system.

[0053] In another preferred embodiment, the input module is configured to allow the user to input information and / or data related to the visual analysis.

[0054] In another preferred embodiment, the input module is configured to interactively allow the user to input selected insight recommendations.

[0055] In another preferred embodiment, the insight recommendation includes: one or more insight recommendation types automatically generated based on the analysis results of the LLM.

[0056] In another preferred embodiment, the insight recommendation includes: after implementing an understanding-based insight recommendation strategy, personalized insight recommendations are provided by the system based on user behavior and analysis needs.

[0057] In another preferred embodiment, the summary report is synthesized by LLM.

[0058] In another preferred embodiment, the summary report includes visual images and explanations.

[0059] In another preferred embodiment, the summary report includes (preferably highlights) key insights and decision points in the analysis process.

[0060] In another preferred embodiment, the summary report utilizes the interactive flow visualization function of LEVA to help users effectively trace and understand the analysis process and decision path.

[0061] In a third aspect of the present invention, a visual analysis method is provided, comprising the steps of:

[0062] During the visual analysis of the data set, insights and recommendations are provided based on large language model analysis and presented interactively to users; and

[0063] Based on the selected insight, the recommended insight type is analyzed visually to obtain insight results.

[0064] It should be understood that within the scope of the present invention, the above-mentioned technical features of the present invention and the technical features described in detail below (such as in the embodiments) can be combined with each other to form new or preferred technical solutions. Due to space limitations, they will not be listed here one by one. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 An architectural flow chart of large language model enhanced visual analysis provided by one embodiment of the present invention is shown.

[0066] Figure 2A schematic diagram of the architectural concept of large language model enhanced visual analysis provided by one embodiment of the present invention is shown.

[0067] Figure 3 An overview diagram of the implementation requirements required for integrating the present invention into a visual analysis system is shown.

[0068] Figure 4 A schematic diagram of a system interface provided by an embodiment of the present invention is shown.

[0069] Figure 5 A schematic diagram of an interactive tutorial provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0070] After extensive and in-depth research, the inventors have developed a visual analysis system based on the LEVA architecture (hereinafter referred to as the LEVA system). The LEVA system integrates large language models into the various stages of human visual analysis processes. Through interactive means, it intelligently assists human exploration, significantly improving the efficiency and quality of VA. Therefore, it is particularly suitable for more effective understanding, reasoning, and decision-making of large and complex data sets. This is the basis for the present invention.

[0071] Specifically, the LEVA system of the present invention uses a large language model (LLM) to enhance multiple stages of the visual analysis workflow, including:

[0072] In the learning visual analysis system phase, the system of the present invention provides a solution that enables LLM to interpret the visualization of each view and the relationship between these views according to the specifications of the VA system. This makes it possible to flexibly create tutorials for various VA systems.

[0073] During the exploration phase, the system of the present invention uses an insight recommendation strategy to guide the LLM to recommend insights based on its understanding of the system, analysis tasks, user interactions, and data, thereby promoting hybrid proactive exploration. During the exploration phase, the system of the present invention typically adopts a two-step process, including selecting an insight type and performing insight evaluation.

[0074] During the results summary phase, the system presents insights integrated into the VA system as annotations rather than textual descriptions, making communication between LLM and end users more intuitive. During the summary phase, LEVA helps users trace their analysis history through interactive flow visualizations, enabling them to select the analytical path for report generation. The system also includes report generation, where LLM synthesizes visualizations and interpretations to produce a report using LaTeX code.

[0075] the term

[0076] It should be noted that in this patent application, relational terms such as first and second, etc., are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Furthermore, the terms "comprise," "include," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element specified by the phrase "comprising a" does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element. In this patent application, reference to performing an action in accordance with an element means performing the action in accordance with at least that element, including two situations: performing the action in accordance with that element alone, and performing the action in accordance with that element and other elements. Expressions such as "plurality," "multiple times," and "many" include "two," "twice," "two kinds," and "more than two," "more than two times," and "more than two kinds."

[0077] The specification of this application records a large number of technical features, which are distributed in various technical solutions. If all possible combinations of technical features of this application (i.e., technical solutions) are to be listed, the specification will be too lengthy. In order to avoid this problem, the various technical features disclosed in the above-mentioned invention content of this application, the various technical features disclosed in the various embodiments and examples below, and the various technical features disclosed in the accompanying drawings can be freely combined with each other to form various new technical solutions (these technical solutions are all deemed to have been recorded in this specification), unless such a combination of technical features is technically infeasible. For example, in one example, feature A+B+C is disclosed, and in another example, feature A+B+D+E is disclosed. Features C and D are equivalent technical means that play the same role. Technically, only one of them can be used, and it is impossible to use them at the same time. Feature E can be technically combined with feature C. Then, the solution of A+B+C+D should not be considered as having been recorded because it is technically infeasible, while the solution of A+B+C+E should be considered as having been recorded.

[0078] Large Language Model Enhanced Interactive Visual Analysis System (LEVA System)

[0079] The present invention provides an interactive visual analysis system (LEVA system) based on large language model enhancement.

[0080] As used herein, the terms "visual analysis system of the present invention," "interactive visual analysis system of the present invention," "LEVA system of the present invention," "interactive visual analysis system enhanced by a large language model of the present invention," or "system of the present invention" are used interchangeably to refer to the interactive visual analysis system enhanced by a large language model (LEVA system) of the present invention. It should be understood that the above terms include both LEVA-based analysis methods and LEVA-based analysis devices or apparatuses.

[0081] like Figure 1 As shown, the LEVA system of the present invention has a large language model enhanced visual analysis architecture (LEVA architecture). The LEVA system of the present invention utilizes the power of large language models to assist users in learning and using visual analysis systems more effectively and generate in-depth data insights.

[0082] The LEVA architecture of the present invention can effectively connect analysts and LLMs to achieve hybrid proactive exploration. Compared with the original visual analysis system, the LEVA-enhanced system of the present invention adds natural language dialogue, interactive system learning tutorials, recommended insights, analysis history visualization and generated reports.

[0083] The architecture of the present invention has gone through three stages: (1) System Learning (or Guided Learning): Using LLM to generate a system learning tutorial for users, the generated tutorial is integrated into the visual analysis system through an interactive interface, guiding users to learn system functions and operations. (2) Exploration: Using LLM to perform data analysis and insight mining to promote user exploration. (3) Summary: Visually review and select analysis history, and use LLM to generate insight reports.

[0084] In a preferred embodiment, the LEVA system of the present invention adopts the following technical solutions:

[0085] S1: Build an interactive tutorial for a visual analysis system, which includes (but is not limited to): creating a grammar, generating tutorial prompts, and integrating these tutorials into the original system to help users learn how to use the system.

[0086] S2: After the user has mastered the system's usage, the system will automatically generate data insights to assist the user in their exploration. This includes (but is not limited to): defining insight types, providing insight recommendations, or a combination thereof. Furthermore, the system preferably integrates insights into the visual analysis system in the form of annotations and highlights.

[0087] S3: Organizing the user's exploration process, including (but not limited to) converting automatically generated insights into a report with images and text. Representative steps for organizing the user's exploration process include (but not limited to) visualizing historical exploration records, automatically summarizing the user's exploration history, converting these summaries into a code-generated report, or a combination thereof.

[0088] Preferably, for the stage of learning the visual analysis system, step S1 may include one or more of the following sub-steps:

[0089] s11: Based on the LEVA architecture, develop a set of grammatical rules for the visual analysis system to facilitate LLM to interpret various views and their relationships according to the specifications of the VA system, and then generate systematic learning tutorials for different VA systems.

[0090] s12: Design interactive prompts to specifically guide LLM to dynamically generate customized tutorial content based on user needs and VA system characteristics.

[0091] s13: Integrate the generated tutorial into the VA system with an interactive interface, and greatly improve the user's understanding of system functions and operations through visual explanations and relationship mapping.

[0092] Preferably, for the exploration and insight stage, step S2 may include one or more of the following sub-steps:

[0093] s21: Analyze VA systems, analytical tasks, user interactions, and data through LLM, define different types of insights, and provide directional guidance for user exploration.

[0094] s22: Implement an understanding-based insight recommendation strategy to provide personalized insight recommendations based on user behavior and analysis needs.

[0095] s23: The integrated insights are presented in the form of visual annotations rather than just text descriptions, making the LLM-generated insights and user interactions more intuitive and attractive.

[0096] Preferably, for the result summary stage, step S3 may include one or more of the following sub-steps:

[0097] s31: Leverage LEVA's interactive flow visualization capabilities to help users effectively trace and understand the analysis process and decision paths.

[0098] s32: Automatically sort out and summarize the user's exploration history through LLM technology, accurately identify and highlight key insights and decision points in the analysis process.

[0099] s33: Uses LLM to synthesize visualization images and explanations, and automatically generates detailed analysis reports (for example, detailed analysis reports containing LaTeX code), providing users with a comprehensive and professional analysis summary document.

[0100] Equipment for performing LEVA analysis

[0101] The present invention also provides an apparatus for performing the LEVA assay of the present invention.

[0102] Typically, the apparatus of the present invention comprises:

[0103] (Z1) a data module, the data module being configured to store a data set to be analyzed;

[0104] (Z2) a LEVA analysis module, wherein the LEVA analysis module is configured to execute the large language model enhanced visual analysis method of the present invention, wherein the large language model enhanced visual analysis method comprises:

[0105] (Y1) Provide insights and recommendations to users based on data insights automatically generated by LLM;

[0106] (Y2) performing corresponding visual analysis on the dataset based on the insight type in the insight recommendation selected by the user; and

[0107] (Y3) Organize the user's exploration process and generate a summary report;

[0108] (Z3) an input module configured to input information and / or data related to the visualization analysis;

[0109] (Z4) An output module, configured to output: insight recommendations, a summary report of the visual analysis, or a combination thereof.

[0110] In another preferred embodiment, the insight recommendation includes: one or more insight recommendation types automatically generated based on the analysis results of the LLM.

[0111] In another preferred embodiment, the insight recommendation includes: after implementing an understanding-based insight recommendation strategy, personalized insight recommendations are provided by the system based on user behavior and analysis needs.

[0112] In another preferred embodiment, the summary report is synthesized by LLM.

[0113] In another preferred embodiment, the summary report includes visual images and explanations.

[0114] In another preferred embodiment, the summary report includes (preferably highlights) key insights and decision points in the analysis process.

[0115] Preferably, the device of the present invention further comprises:

[0116] (Z5) An interactive tutorial module configured to generate a system learning tutorial for the VA system and present the system learning tutorial to the user, thereby improving the user's understanding of the system's functions and operations.

[0117] Preferably, the interactive tutorial module is further configured to guide the LLM to dynamically generate customized tutorial content according to user needs and VA system characteristics.

[0118] In addition, the device of the present invention is also equipped with corresponding operation interfaces for performing LEVA operations. Representative interfaces include: chat view, original system view, interaction flow view and report view.

[0119] The following is combined with Figure 1-5 The system and various stages of the present invention are further described in the following figures and some examples. It should be understood that the drawings are only for illustrative purposes and are only schematic diagrams.

[0120] 1. Generate an interactive system learning tutorial

[0121] When faced with a complex visual analysis system containing multiple views, many users experience a steep learning curve. Therefore, the inventors propose a visual analysis system with a systematic learning tutorial generation method, which aims to support users in understanding the three types of knowledge in visual analysis systems: data, style, and coordination.

[0122] System learning can follow two types of approaches: one is bottom-up, that is, the details are introduced first; the other is top-down, that is, an overview is provided first. Top-down is more effective in introducing visualizations, and the inventors adopt a top-down approach, that is, introducing from the system level to the view level and the relationships in multiple views. To this end, the inventors use the ability of LLM to understand the information of the system and generate tutorials. Previous work has shown that the declarative syntax of visualizations is easily understood by language models. Therefore, for automatic tutorial generation, a unified specification is needed to describe the visual analysis system. To this end, the inventors propose a visual analysis system specification designed in accordance with the top-down approach as the input of LLMS.

[0123] The specification starts with a high-level system information SystemSpec. This invention provides the systeminfo attribute to specify the name and total number of views of the system. It includes view information that describes the style and coordination of each view.

[0124] As a complement to the system metadata, the present invention introduces viewStyleInfo information to further delve into the design of each individual view. This information captures the essence of the visual analysis view through the viewName attribute, representing its identity, while the layers attribute describes its visual components. Each layer details the mark type (e.g., area) and encoding properties, providing insight into how data fields are intuitively mapped to the axis, color scale, and size scale. In addition, an optional tooltip specification can be used to enrich the user's interactive experience by displaying supplementary information during interaction.

[0125] Understanding the coordination of different views for a consistent user experience is crucial, and this invention provides the conceptualization of ViewCoordinationInfo. This information connects the interactions between the source and target views through the sourceViewName and targetViewName properties, respectively. In addition, the coordinationType property describes the nature of the interaction (e.g., filter, brush), and the interactions array provides a granular breakdown of user-triggered events and their subsequent branches on the target view.

[0126] In the present invention, a key-value pair approach is used to determine the selection of fields and their formats. This approach provides flexibility and is applicable to a variety of visualization types, not just common visualizations. For example, users can define custom cards using title and context instead of using x and y. Therefore, for these special cases, it is sufficient to use a key-value pair format to describe the view so that LLM can understand it. In addition, this specification not only helps generate intuitive system learning tutorials, but also helps LLM understand data types and find suitable tasks.

[0127] 2. Insight Recommendation Strategy

[0128] The insight recommendation strategy constructed by the present invention mainly undertakes the following functions: providing suggestions for enhanced insights through LLM, including three key steps: insight type selection, insight calculation, and scoring before final output.

[0129] 2.1 Insight Definition

[0130] In the framework of the present invention, insight is considered the basic unit of visual analytical exploration. Typically, the present invention uses four attributes to define insights, including:

[0131] Insight: = <type, parameter, theme, score>

[0132] In other words, the four representative attributes of an insight are insight type, insight parameter, insight topic, and insight score.

[0133] Insight type: In the present invention, there is no particular limitation on the insight type. Representative insight types may include a variety of different insight types, for example, including extreme values, outliers, change point trends, etc. In order to cover more complex data types, such as text and graphs, the present invention further adds three insight types, namely text summaries and key nodes or edges. Since the insights in the visual analysis work are linked to the analysis of related views within the visual analysis system, the insights can be consistent with a single view (for example, sales trends in the sales view) or across multiple views (for example, a positive correlation between sales and profits in the sales view and profit view, or events with time and space information). Therefore, if the user (or analyst) wants to obtain richer insights at once, a cross-view insight type can be defined instead of analyzing only a single view.

[0134] Insight Parameters: For each insight type, you can use parameters to describe the characteristics of the insight, such as the direction of the correlation, the title of the event summary, the location, and the time.

[0135] Insight topic: defines the data scope of the insight and contains four attributes

[0136] Insight topic: = <subspace, dimension, measurement, context>

[0137] Each insight theme corresponds to a subset of data in a visual analytics system view. Dimension values ​​can be mapped to the x-axis, and measures can be mapped to the y-axis. Typically, the subspace is user-selected; the system transforms the data, eliminating the need for adjustments. For example, in a line chart showing sales for two products, the dimension is time, the measure could be sales or profit, and the subspace could be the time range. For text data analysis, measures can be used to reference columns of text data in a dataset.

[0138] In addition to the datasets in the original view, additional data can also serve as an important source. First, during the ongoing interactive exploration process, insights can be retained to discern similar or different patterns, which can inform subsequent analysis stages. Second, complex tasks, such as finding important nodes related to a specific event, are beyond the scope of conventional topics due to the need for event summaries. Based on the requirements presented by these scenarios, this additional data is categorized as context in this invention.

[0139] Insight score: Different insights have different appeal to users. In order to implement insight recommendations in a visual analysis system, it is necessary to evaluate which insight types can be used to solve tasks and determine the importance of insights. The importance insight score includes influence and significance scores. Therefore, in the present invention, the insight score can be calculated by combining factors such as influence score, significance score and relevance score. Preferably, the insight score is a comprehensive score based on the above-mentioned insight evaluation, and representative examples include weighted scores and non-weighted scores, with weighted scores being preferred.

[0140] In the present invention, a significance score can be used to assess the importance of data insights. Specifically, this score is calculated based on a hypothesis test. The hypothesis test takes a range of values, for example, finding the first item that stands out in a group. It is assumed that the data follows the null hypothesis, that is, a long-tailed distribution. The p-value is used to indicate whether the data excludes the maximum value that would violate the null hypothesis. Therefore, a significance score is 1-p (p is the p-value). Although the analysis of the present invention can use p-values ​​to assess the significance of insights, the use of p-values ​​also has certain limitations. First, p-values ​​are unstable and cannot convey the size of an effect. Second, p-values ​​are not suitable for all types of insights. To address these issues, the preferred insight scoring mechanism of the present invention adopts a highly adaptable mechanism. Therefore, in preferred embodiments, the present invention adopts alternative parameters such as effect size, Cohen's d, OddsRatio, or coefficient of determination to obtain wider applicability. For text analysis, BLEU and ROUGE or LLM themselves can serve as more suitable alternative parameters. It should be understood that in the present invention, one or more different methods or combinations thereof can also be used to comprehensively assess the importance of data insights to obtain a significance score.

[0141] There are two aspects to evaluating the influence score. One is the coverage of the data subspace on the entire dataset, and the other is the semantics of the data subspace. In this invention, influence can be defined as the data subspace that covers the entire dataset.

[0142] For example, in social media event analysis, "Weibo" and "Call Center" are two types of messages that can represent two different data subspaces. If the data from "Weibo" exceeds the data from "Call Center", then the insights generated by the "Weibo" data are considered to have greater influence due to its wider coverage. However, from a semantic perspective, the message from the "Call Center" is more impactful because it is more reliable and timely. Emphasizing data coverage may obscure the insights from smaller data. To alleviate this problem, in the present invention, the impact of the LLM can be evaluated based on the language model's understanding of the dataset and its extensive knowledge in data analysis.

[0143] Since users usually use visual analysis systems with analysis tasks, the present invention can use the correlation score to quantitatively measure the consistency between the emergent insights and the overall analysis tasks. In the system of the present invention, the LLM is used to evaluate the insights, considering how closely the insights are related to the tasks.

[0144] For example, in Figure 4 and Figure 5 In the event analysis shown, the task is to detect risky events and areas in a city. Therefore, analyzing information to summarize events will score high on relevance because it is directly related to the task.

[0145] Taking the influence score, significance score, and relevance score as an example, a preferred approach is to calculate the insight score based on the following formula by comprehensively considering these three scores:

[0146] Score=∑ k∈{significance ,impact,relevance}w k ×score k

[0147] in,

[0148] Score indicates the insight score;

[0149] k∈{significance,impact,relevance} indicates that the evaluation factors include significance, impact, and relevance;

[0150] score k Represent the scores of significance score, influence score and relevance score respectively;

[0151] w k Represent the weights of significance score, influence score and relevance score respectively.

[0152] For example, in Figure 4 and Figure 5 In the event analysis shown, since the purpose of the present invention to obtain data insights is to reveal its patterns, the importance score should be more important than other scores. Therefore, the weights can be set to 0.5, 0.2 and 0.3 respectively based on experience.

[0153] It should be understood that the weights are adjustable to suit different application scenarios or data sets.

[0154] 2.2 Recommendation Strategy

[0155] In the present invention, the insight recommendation strategy is intuitive, adaptive, and relevant to the user's task.

[0156] In the present invention, insight recommendation is usually performed through the following two-step strategy:

[0157] Step 1 (Insight Type Selection):

[0158] Considering the numerous possible insight types, in the present invention, the insight type that is most relevant to the task may be preferentially selected.

[0159] When a user selects a specific view, LLM can be used to find relevant insight types based on the user's selection and task, and assign a relevance score. Insight types with high relevance scores will be used for further calculations.

[0160] To automatically execute these insight types, LLM needs to schedule the data required to calculate the insight (such as data tables and column names). The output includes the selected insight type, relevance score, and data information for the insight calculation.

[0161] Step 2 (Insight Assessment):

[0162] This step aims to comprehensively evaluate insights. First, the insight type selected in the last step will be executed to generate insights. The system of the present invention uses LLM to convert the calculation results into complete sentences. Secondly, the system of the present invention calculates the comprehensive score of the insight from three aspects. As mentioned above, in the present invention, preferably, the significance score is calculated by statistical methods, and the influence and relevance scores are evaluated by LLM. The final output of this step is insights sorted according to the comprehensive score and the corresponding data points in the annotation view.

[0163] After these two steps, the insights should be annotated to the relevant views in the visual analytics system. Users can continuously interact with the views to gain new insights from LLM.

[0164] 3. Generate summary report

[0165] After an exploratory interaction with the LLM, users may want to start a new round of exploration to continue acquiring new knowledge.

[0166] Given that not every interaction stage will yield the desired insights, the system of the present invention preferably uses interactive strategies to filter exploration results and create comprehensive reports, and leverages the text summarization capabilities of LLM.

[0167] 3.1 Record Retention:

[0168] Throughout the analytical process, a detailed record of the user's exploration journey, including interactions and insights, is essential. The system stores these elements together, as insights and interactions provide key findings and valuable context for report creation. Specifically, user interactions act as filters, modifying the state of the analysis and defining the subspace of insights.

[0169] For example, consider a scenario where a user selects a specific time period. This action prompts the LLM to summarize relevant events within this time period. Providing the time period is crucial when presenting an event summary. This approach ensures comprehensive and consistent insights and the relevance of the resulting insights.

[0170] Each recorded dataset is defined as spanning m analysis rounds. In each round, there are n distinct steps, representing the insights users adopt from LLM or their self-motivated interactions. In this invention, the exploration journey can be represented as a matrix M of dimension m×n, where each element M_ij represents the jth step in round i. For each step M_ij, the following details are retained:

[0171] FocusedView: A descriptor that describes which visualization or data view is in focus.

[0172] Insight: A specific insight generated by the LLM. If the user performs the interaction themselves, the interaction object is recorded.

[0173] Screenshots: Saved screenshots of the focused view, essential for subsequent report illustrations.

[0174] 3.2 Selective Reporting:

[0175] To provide users with a coherent understanding of their analytical process, the system of the present invention provides an interactive mechanism. This interactive mechanism graphically illustrates the matrix M, allowing users to retrace the insights of the report. Once the user selects a run of data, the organized sequence from the matrix M serves as the input to the LLM. Using this structured input, the LLM can generate a report summarizing the user's exploration process.

[0176] 4. Implementation of LEVA Architecture

[0177] For some existing visual analysis systems, corresponding LEVA systems can be developed based on the LEVA architecture and method of the present invention.

[0178] The implementation of the architecture of the present invention includes two parts: the original visual analysis system needs to be supplemented with extensions, and the LLM-driven components ( Figure 3 ).

[0179] The original visual analysis system requires configuration files and data handlers to process user selections and LLM output. The LLM-driven component consists of a prompt handler and a presentation module for displaying output across three new views.

[0180] 4.1 Expansion of the original visual analysis system

[0181] Typically, integrating the architecture of the present invention into a given visual analysis system requires preparing four configuration files and four data processing programs.

[0182] Syntax: This specification serves as the foundation for the LLM to understand the visual analytics system, enabling it to gain a deep understanding and understanding of its design and usage. Understanding this specification allows the LLM to tailor detailed and appropriate tutorials for the visual analytics system. Furthermore, because the specification contains the data and coordination information for each view, it helps the LLM explore the data by understanding the system state and determining which view and data columns to use to compute insights.

[0183] Analysis Task: Before recommending insights, it is necessary to propose a user task described in a natural language sentence. If not specified, LLM can also be used to propose a task for selecting relevant insight types.

[0184] For example, in Figure 4 and Figure 5 In the visualization shown, when using social media data for public risk analysis, one task might be to discover events in a city that pose a high risk to the public.

[0185] Insight Function: In the present invention, different types of insights are assigned corresponding insight functions to perform calculations. For example, for a selected insight type, representative examples include: using the insight function to obtain the most significant value (such as an extreme value) in a set of data, or calculating the correlation between two or more sets of data.

[0186] Given common types of insights, they may not be sufficient for certain specific cases. Therefore, developers can add functions based on their domain-specific analytical needs. Previous taxonomies have summarized complex tasks that may be conducive to the development of computational functions, such as taxonomies for graph analysis, spatial and temporal analysis, and social media analysis.

[0187] Insight Function API: The system allows the LLM to select a function from the Insight Function API list and then use its system understanding to supplement the function parameters, thereby invoking the function to calculate insights. To achieve this, a function definition is required, including two attributes: name and description, which explain the function's purpose and the expected results after execution.

[0188] For example, to get the maximum value in a set of data, the name can be "get_outstanding_top1" and the description can be "calculate the maximum value that is significantly higher than all remaining values". The API that provides insight functionality enables LLM to select appropriate analysis methods and execute them by completing the parameters.

[0189] Data Handlers: To support the connection between LLM and the original system, these handlers need to be implemented to capture changes in the system and add new features from LLM.

[0190] Add Tutorials: Receive tutorials from LLM and use the tour guide tool to display the tutorials.

[0191] Get selection data: When the user interacts with a specific view, the system needs to get the selection data, including filters and updated data on each view.

[0192] Add annotations: The original visual analysis system needs to highlight the insights generated by LLM on the target view. This function should allow selecting visual elements, changing their style, and adding annotations around them.

[0193] Apply Annotation Filters: Allows selection of annotations across views, such as filtering and deletion.

[0194] 4.2 System Learning Tutorial Generation

[0195] During the system learning phase, the system uses the LLM to generate a tutorial by inputting the visual analysis system's specifications. When designing the output of the learning and memory module, considering that text-based instruction increases user reading time, the present invention designed the tutorial as an interactive tour. Therefore, the output of the LLM is a list of steps in the interactive tutorial.

[0196] Each step includes two properties, including title and description. The title is the view name v ie wName. Description may include (but is not limited to): visualization type, encoding, and coordination.

[0197] Preferably, in the system of the present invention, LLM output can be described in HTML format, long text can be translated into fewer lines, and font styles can be set for clear observation. When the user chooses to start the tour, the tutorial will be triggered and added to the original system to introduce each view (for example, see Figure 5 ).

[0198] 4.3 Insight Recommendation

[0199] The implementation of this strategy involves a two-step dialogue with the LLM. The first step is to select the appropriate insight type based on the selected data and analytical task, and the second step is to execute and evaluate the insights to select the final results.

[0200] In the first round, the input consists of four components. It includes a specification, the current interaction analysis task, and a list of insight functions. In the system of the present invention, specifications including view style information and view coordinate information are added to the input. View coordination information allows LLM to know the target view after the user selects it. viewStyleInfo contains data information in the view and will be used as parameters to compose the data for insight calculation. The current interaction is represented as a triple: (viewName, dimName, value). If the user selects non-contiguous elements, such as two locations on a map, California and New York, the current selection will be two triplets in the array. In addition, if the selection from the previous analysis step is not canceled, it will be kept together to provide context for further analysis. The LLMS then determines the type of insight that can be analyzed based on the insight function and the analysis task, and evaluates it using the relevant score. To perform these insight functions, the output of the LLM can be defined as four parts: functionName, viewName, variableName, dimName.

[0201] In the second round, the insight function is executed to calculate the significance score as the insight calculation result, which is used as input for further evaluation. LLM then organizes the calculation results into natural language sentences to describe the insights. After generating insights and obtaining significance scores, the next step is to evaluate the impact score. Let LLM assign an impact score based on the nature of each insight, such as potential consequences, urgency and timeliness, and impact on decision-making. Developers can further modify the definition of impact scores to suit specific analysis scenarios. Insights will correspond to a triple: viewName, dimName, valuei, just like the current selection mentioned above in the present invention, positioning insights as annotations on their views.

[0202] After obtaining the generated insights in a structured format, the Add Annotation function in the original visual analysis system is executed to link the data objects from the insights to the elements in the view. Each insight type can correspond to a unique view or multiple views. Therefore, if an insight is cross-view, the annotation will also be added to multiple views. Considering that not all insight types may be what the user wants to see, in the present invention, it is preferred to use single-view insights to perform analysis step by step.

[0203] When using LLM directly for data analysis, it is crucial to consider the advantages and limitations of language models. The inventors tested the data analysis performance of LLM on tabular datasets. The results showed that their accuracy for basic tasks was relatively low. Therefore, for tabular data analysis, in the system of the present invention, a rule-based approach is used to ensure accurate results, and LLM is used to call these functions based on an understanding of the underlying data and user tasks. However, recent studies have shown that LLM exhibits extraordinary capabilities when analyzing text data. Therefore, the system of the present invention can use LLM to directly analyze these text data tasks. In addition, if the recommended insights are incomplete or inaccurate, the system of the present invention provides an open-ended question answering function. Users can type follow-up questions to learn more about the insights. LLM will analyze the underlying data in its current state and return highlights of the insights and source data, thereby improving the explainability of how the insights were generated.

[0204] In a preferred embodiment of the present invention, two strategies can be adopted to solve the response speed problem of LLM. First, control output to improve efficiency. Considering that some computing tasks (such as text summarization) will produce lengthy output, it is very necessary to define the output length in the prompt template to ensure that it remains concise. Second, interactive Q&A. Before executing the insight function, the inventors adopted an interactive approach in which LLM suggested questions based on the evaluation of relevant insight types from the insight selection stage. The user can select a desired question to obtain insight. This alternative strategy can reduce the waiting time for calculating all insights.

[0205] In summary, the limitations of input and output have an impact on both computational methods and interaction paradigms. Depending on the specific analysis scenario, the above strategies can be fine-tuned to achieve the right balance.

[0206] 4.4 Report Generation

[0207] In the present invention, a typical report generation method is implemented in two steps.

[0208] First, the insight annotation is saved. The data is a 5-tuple: (Insight, type, value, viewName, imageName). Typically, each step M_ij obtains a corresponding screenshot based on viewName. Then, for each round of historical analysis data, the system of the present invention allows LLM to generate a report in text form.

[0209] Secondly, given LLM's superior performance in code generation, text reports can be converted to LATE format presentation reports. The system can add requirements to the output format to set the report style. The resulting report is intuitive to read and presented to the user with interactive visualizations of markers.

[0210] 4.5 System Interface

[0211] In order to support the entire architecture, the present invention designs a system interface to connect users, LLM and underlying data. The interface consists of four main components: chat view, original system view, interaction flow view and report view, such as Figure 4 shown.

[0212] Chat view: In order to receive feedback and control the entire workflow in the architecture of the present invention, this view serves as an interactive interface where users can receive feedback and control other views ( Figure 4 A in the figure). During the login phase, the chat view launches a wizard to introduce the original system. When the user starts exploring, this view displays the questions that the LLM recommends exploring. The user can make selections, and the selected questions and insights will be recorded and visualized in the interaction flow view ( Figure 4 C in ), and supports generating reports in the report view ( Figure 4 D in the chat view). In addition, the chat view allows for open-ended questions, allowing users to engage in a fluid conversation with the LLM to clarify doubts or gain new insights. Since the LLM may generate incorrect formats, the system of the present invention allows for a feedback mechanism to display error messages in the chat view and make users aware of failed issues.

[0213] Original System View: The original visual analysis system is merged into this interface. To enhance the system, LEVA introduces annotations as parameters. These annotations serve as guide markers to help users identify interesting data patterns. To effectively manage these annotations, a dedicated control panel ( Figure 4 b2 in the previous section). It provides filtering functionality and the option to clear all annotations on a view. Users can interact with the view, prompting LLM to ask questions based on their selections. After recommending cross-view insights, if the user selects an annotated area in the source view, the system will display the remaining insights in the target view.

[0214] Interaction Flow View & Report View: The Interaction Flow View visualizes the analytical insights users gain from LLM ( Figure 4 C in the figure). Users can hover over nodes to review the detailed information at each step. The analytical insights of each step are automatically saved. Considering that users may want to stop the current round of exploration and start a new round, the system of the present invention provides an end button. The system of the present invention sets the interaction view to be hidden by default. After selecting an interaction path, the report view ( Figure 4 D) in the figure is triggered to display a summary report of this round of exploration.

[0215] The main advantages of the present invention include:

[0216] (a) The LEVA system of the present invention utilizes the power of large language models to assist users in learning and using visual analysis systems more effectively and generating in-depth data insights therefrom.

[0217] (b) The LEVA system of the present invention can automatically generate data insights to assist user exploration, thereby significantly improving the efficiency and quality of visual analysis.

[0218] (c) The LEVA system of the present invention helps users trace the analysis history through interactive flow visualization, thereby selecting the analysis path for generating reports, and can automatically generate reports containing visualization images and explanations.

[0219] (d) The LEVA system of the present invention enables LLM to interpret the visualization of each view and the relationship between these views based on the specifications of the visual analysis system, thereby flexibly creating tutorials for various visual analysis systems.

[0220] In summary, the method of the present invention enhances the user's visual analysis workflow at multiple stages: system learning, data exploration, and result summarization. For system learning, the present invention uses a large language model to generate a system learning tutorial based on a unified system specification grammar. For exploration, a large language model is used to provide insight recommendations based on the understanding of system status and data analysis to promote hybrid active exploration. In terms of summarization, the present invention combines flow visualization to trace the analysis history and generates insight reports with the help of a large language model.

[0221] discuss

[0222] Visual analytics often requires significant effort. On the user side, data interpretation and analysis often lead to inefficiencies. Key challenges contributing to these inefficiencies include a steep learning curve for visual analytics systems, the ease of getting lost in data exploration, and the difficulty in generating insights.

[0223] Specifically, the challenge of system learning stems from unfamiliarity with visual analytics systems, resulting in reduced efficiency in mastering visual mapping and interaction. Second, during data exploration, there are numerous ways to analyze data, and a lack of structured guidance and direction makes it difficult to effectively discover insights. Finally, tracking and summarizing key findings is often difficult, and summarizing insights is a time-consuming task that requires considerable effort and attention to detail.

[0224] To enhance the visual analysis workflow, the inventors carefully identified the important steps users take when using visual analysis and proposed LEVA to improve analysis efficiency and enrich insight exploration.

[0225] Large language models (LLMs) demonstrate a broader range of knowledge and problem-solving capabilities, making it possible to address the aforementioned challenges across the three stages of visual analysis. Through research and development, the inventors have developed a novel and unique visual analysis method and system enhanced by large language models. The LEVA system of the present invention is the first to embed LLMs within complex visual analysis workflows, providing powerful support to users at different stages of analysis.

[0226] First, since LLM can interpret the declarative syntax of visualization, in the present invention, a unified syntax of a visual analysis system is provided and used as the input of LLM to generate a learning tutorial for the visual analysis system.

[0227] Second, because LLM can handle various data types, it can recommend insights and support user exploration by analyzing system status and underlying data.

[0228] Finally, LLM displays strong summarization capabilities, which can help automate tasks such as report generation during the summarization phase.

[0229] The visual analysis method and system based on large language model enhancement of the present invention provides a unique and intelligent guiding architecture, thereby integrating the large language model into the process of various stages of human visual analysis, and intelligently assisting human exploration through interaction, thereby greatly improving the efficiency and quality of visual analysis.

[0230] All documents mentioned in this application are incorporated herein by reference, just as if each document were incorporated herein by reference individually. It should also be understood that after reading the above teachings of the present invention, those skilled in the art may make various changes or modifications to the present invention, and that such equivalents also fall within the scope of the claims appended hereto.

Claims

1. A visual analysis method enhanced by a large language model, characterized in that: Including steps: (1) Provide a data set to be visually analyzed; (2) analyzing the data of the dataset using a large language model to generate one or more insight recommendations, and providing the insight recommendations to the user for selection, thereby determining a first insight type, wherein the first insight type is selected by the user from the insight recommendations, or is not selected from the insight recommendations but is input by the user; (3) for the selected first insight type, using a large language model to analyze the data of the dataset and perform insight exploration, thereby obtaining an insight result for the first insight type; (4) Optionally, based on the insight result in the previous step, repeat steps (2) and (3) to obtain an insight result of the Nth insight type, where N is a positive integer ≥ 2; and (5) Generate a summary report of the visual analysis based on the analyzed insight types and insight results.

2. The method according to claim 1, wherein The visual analysis is performed in a visual analysis system based on the LEVA architecture.

3. The method according to claim 1, wherein In step (2), through large language model analysis, insight recommendations are provided based on the understanding of the system, analysis tasks, user interactions, and data.

4. The method according to claim 1, wherein In step (5), it includes: automatically sorting and summarizing the user's exploration history through LLM technology, accurately identifying and highlighting the key insights and decision points in the analysis process.

5. The method according to claim 1, wherein The insights are defined by the following four attributes: insight type, insight parameter, insight topic, and insight score.

6. The method according to claim 5, wherein The insight topics are defined using four attributes: subspace, dimension, measurement, and context.

7. The method according to claim 5, wherein The insight score is obtained based on the influence score, significance score and relevance score; Preferably, the insight score is calculated using the following formula: Score=∑ k∈{significance ,impact,relevance}w k ×score k in, Score indicates the insight score; k∈{significance,impact,relevance} indicates that the evaluation factors include significance, impact, and relevance; score k Represent the scores of significance score, influence score and relevance score respectively; w k Represent the weights of significance score, influence score and relevance score respectively.

8. The method according to claim 1, wherein The method further comprises: (i) Provide users with interactive teaching of the Large Language Model Enhancement Visual Analysis method (LEVA method).

9. A device for performing visual analysis enhanced by a large language model, characterized in that include: (Z1) a data module, the data module being configured to store a data set to be analyzed; (Z2) a LEVA analysis module, wherein the LEVA analysis module is configured to perform a large language model-enhanced visual analysis method, wherein the large language model-enhanced visual analysis method comprises the steps of: (Y1) Provide insights and recommendations to users based on data insights automatically generated by LLM; (Y2) performing corresponding visual analysis on the dataset based on the insight type in the insight recommendation selected by the user; and (Y3) Organize the user's exploration process and generate a summary report; (Z3) an input module configured to input information and / or data related to the visualization analysis; and (Z4) An output module, configured to output: insight recommendations, a summary report of the visual analysis, or a combination thereof.

10. A visual analysis method, characterized in that: Including steps: During the visual analysis of the data set, insights and recommendations are provided based on large language model analysis and interactively presented to users; as well as Based on the selected insight, the recommended insight type is analyzed visually to obtain insight results.

Images