Semantic search interface for data repository

Abstract

A method is disclosed that provides visual analysis of a data set. A system receives a natural language search query directed to a data store that includes a data source and a data visualization. The system parses search terms to determine whether the natural language search query contains an analysis intent. The system also uses a semantic search to determine whether the search term matches a field in one or more data sources. When (i) the search term matches a field in the one or more data sources and (ii) the natural language search query contains an analysis intent, the system generates and displays a visual response. When (i) the search term does not match a field in the data source or (ii) the natural language search query does not contain the analysis intent, the system displays pre-written content from the data visualization.

Description

[0001] priority

[0002] This application is a continuation-to-file of U.S. Patent Application No. 18 / 427,799, entitled "Semantic Search Interface for Data Repositories," filed January 30, 2024. U.S. Patent Application No. 18 / 427,799 claims priority to U.S. Provisional Application Serial No. 63 / 457,367, entitled "Semantic Search Interface for Data Repositories," filed April 5, 2023. U.S. Patent Application No. 18 / 427,799 also claims priority to U.S. Provisional Application Serial No. 63 / 461,237, entitled "Semantic Search Interface for Data Repositories," filed April 21, 2023. Each of the foregoing applications is hereby incorporated herein by reference in its entirety. Technical Field

[0003] The disclosed implementations generally involve data visualization, and more specifically, systems, methods, and user interfaces for semantic search of data repositories. Background Technology

[0004] User expectations of search interfaces are evolving. There is a growing expectation that search engines can answer questions while providing contextually relevant content to help achieve the searcher's goals. Current keyword-based search methods are largely designed for content retrieval. Their main potential drawback is limited support for structured query types that typically expect focused and specific responses. On the other hand, natural language (NL) question-answering (Q&A) interfaces support more fact-finding but do not support content or document discovery and retrieval.

[0005] With the increasing number of data repositories online, including structured data in the form of relational databases, documents, and knowledge graphs, a wealth of information supports combining the generation of responses to fact-finding questions with document retrieval. Similarly, data repositories and visualization tools host hundreds or thousands of visualizations representing a wide range of datasets, making them rich platforms for knowledge sharing and consumption. Search plays a crucial role in these repositories, enabling users to filter content of interest (e.g., charts on specific topics, charts showing data trends, and customized visualizations such as Sankey diagrams or charts created by specific individuals). Current search systems tend to rely on document retrieval techniques to provide relevant search results for a given query. However, a challenge with data repositories lies in the sparsity of searchable text within them; data sources and charts often have limited textual information, such as titles, explanatory words, and textual data values. Alternative methods for indexing and searching content based on this limited availability of textual information need to be explored.

[0006] Another challenge is that the current search features of data repositories offer limited expressiveness in specifying search queries, restricting users to keyword searches for content primarily based on visual titles and authors. In contrast, other contemporary search interfaces, such as general web search, image and video search, and social networking sites, enable users to find and discover content through a rich combination of textual content (e.g., keywords or topics covered in the website), visual features within the content (e.g., finding images with a specific background color), date (e.g., watching videos from the most recent week), geographic location (e.g., limiting the search to postal codes or cities), and even different types of media (e.g., searching for similar images via reverse image search).

[0007] Given the current limitations of these systems, designing expressive search interfaces for data repositories requires a deeper empirical understanding of people's search needs. For example, what goals do people consider when using search within the context of a data repository? How do people formulate their search queries? Is text alone sufficient as a search modality? If not, what are complementary / alternative modalities? What supporting metadata do people want to query or use to filter search results? Summary of the Invention

[0008] Therefore, systems, methods, and interfaces for semantic search in data repositories are needed. Some implementations are based on a hybrid approach called semantic search that bridges the gap between two very different search paradigms: keyword-based search and natural language (NL) question-and-answer (Q&A) interfaces. Semantic search applies user intent and the meaning of words and phrases (e.g., semantics) to determine the correct content that may not be immediately apparent in the text (the keywords themselves) but is closely related to what the searcher wants. By using information such as entity recognition, word disambiguation, and relation extraction to interpret the searcher's intent in the query, information retrieval techniques go beyond simple keyword matching. For example, keyword search might find documents with the query "French press," while queries such as "How do I quickly make strong coffee?" or "manual coffee brewing methods" are better suited for semantic search to generate targeted responses.

[0009] Some implementations offer hybrid search capabilities for data repositories containing curated data sources and visualizations. Others provide a search interface that offers a blend of results, including both automatically generated visualization responses and pre-written charts, to combine analytical question-and-answer with content discovery search objectives. Some implementations support three search scenarios for the data repository: question-and-answer, exploratory search, and design search. The interface also provides faceted search support to refine and filter general best-priority search results based on parameters such as author name, time, and chart type. Preliminary user evaluations of the system indicate that the interface and hybrid search described herein collectively provide a more expressive way for users to discover insights and visualizations within the data repository.

[0010] According to some implementations, a method for visual analysis of a dataset is provided. The method is executed at a computing system having one or more processors and memory configured for executing one or more programs by the one or more processors. The method includes: receiving a natural language search query directed to multiple data repositories including multiple data sources and one or more data visualizations. The method further includes: parsing search terms corresponding to the natural language search query to determine whether the natural language search query contains one or more analytical intents. The method further includes: using semantic search to determine whether the search terms match fields in one or more of the multiple data sources. The method further includes: displaying and generating one or more visualization responses when (i) the search terms match fields in one or more data sources or (ii) the natural language search query contains one or more analytical intents. The method further includes: displaying pre-written content from one or more data visualizations when (i) the search terms do not match fields in the multiple data sources or (ii) the natural language search query does not contain one or more analytical intents.

[0011] In some implementations, the method further includes using a federated query search to obtain search terms, wherein the federated query search distributes the query to multiple search repositories and combines the results into a single merged search result.

[0012] In some implementations, one or more analytical intents are one or more of the following: grouping, aggregation, association, filtering and limiting, and time and geospatial.

[0013] In some implementations, parsing search terms further includes identifying data attributes and values ​​along with one or more analytical intents based on multiple data sources and metadata from those data sources.

[0014] In some implementations, identifying data attributes and values ​​includes comparing N-grams corresponding to search terms with available data attributes for syntactic and semantic similarity.

[0015] In some implementations, syntactic similarity is identified using Levenshtein distance and semantic similarity is identified using Wu-Palmer similarity scores.

[0016] In some implementations, semantic search includes: indexing each of the multiple data repositories and their corresponding metadata to obtain an index; and performing a federated search to determine, based on the index, whether a search term matches a field in one or more of the multiple data sources.

[0017] In some implementations, indexing includes: representing each file as a corresponding document vector for each data repository and visualization context and associated metadata; and storing N-gram strings from the document vectors to support partial and exact matching.

[0018] In some implementations, performing a federated search includes: obtaining query vectors corresponding to search terms; encoding the query vectors into query string terms using an encoder used to generate the index; and selecting a predetermined number of candidate document vectors from the document vectors for each data repository and visualization context along with associated metadata, based on the amount of overlap between the query string terms and the document string terms of the document vectors.

[0019] In some implementations, the method further includes ranking a predetermined number of candidate document vectors using a scoring function that scores documents based on the search terms appearing in each document, regardless of the proximity of the search terms within the document.

[0020] In some implementations, the method further includes generating and displaying one or more visual responses by interpreting one or more analytical intents based on a list of a predetermined number of data sources returned by semantic search.

[0021] In some implementations, the method further includes generating and displaying one or more visual responses based on attributes, values, and analytical intent in natural language search queries.

[0022] In some implementations, the method further includes: determining multiple data sources based on semantic search, providing a dropdown list of data source alternatives along with the corresponding percentage match score for each data source.

[0023] In some implementations, the method further includes: displaying suggested queries for the data source when (i) the semantic search returns a matching data source for a natural language query and (ii) the search terms are not resolved to valid attributes and values ​​within that data source.

[0024] In some implementations, the method further includes using a template-based approach to generate suggested queries based on a combination of attributes from the data source and data interest metrics.

[0025] In some implementations, the method further includes using three encoding channels (x, y, color) and four marker types (bars, lines, dots, and geographic shapes) to generate and display one or more visualization responses, thereby supporting the dynamic generation of bar charts, line charts, scatter plots, and maps that cover a range of analytical intents.

[0026] In some implementations, the method further includes determining the tag type of one or more visual responses based on the mapping between visual encoding and attribute data types.

[0027] In some implementations, the method further includes using one or more statistical computations and large language models to generate and display dynamic text summaries describing one or more visual responses.

[0028] In some implementations, the method further includes: providing a hint containing a statistical description to a large language model, the statistical description being extracted from the generated visualization using a predefined set of heuristics; and receiving a dynamic text summary from the large language model in response to providing the hint.

[0029] In some implementations, a predefined set of heuristics is selected based on the type of visualization being generated.

[0030] In some implementations, the hints correspond to (i) minimum / maximum and average values ​​for bar charts, and (ii) Pearson correlation coefficients for scatter plots.

[0031] On the other hand, a method for visual analytics of datasets is provided. The method is executed at a computing system having one or more processors and memory configured for executing one or more programs by the one or more processors. The method includes: receiving a user's selection of a data source; presenting a graphical user interface for analyzing data in the selected data source; and providing three search options. The search options include: a question-and-answer search for interpreting analytical intent within a selected set of data sources; an exploratory search for document-based information retrieval of indexed visualizations; and a design search using visualization metadata from the data source.

[0032] In some implementations, the method further includes providing a faceted browser for trimming search results by author name, time range, and / or visualization type.

[0033] In some implementations, the method further includes: providing a login screen that displays samples of the data source as thumbnail images; and displaying metadata information corresponding to the data source in response to hovering the mouse over the thumbnail image of the data source.

[0034] In some implementations, the method further includes responding to a user typing a search query into an input text box: using the terms of the search query as keywords to match any pre-written visualization, and displaying a grid of thumbnails as a preview for browsing and exploration. Each thumbnail is hyperlinked to its corresponding visualization file, which can be selected for more detailed viewing or download.

[0035] In some implementations, the visual title, author's name, and creation date are displayed below each thumbnail to provide additional context.

[0036] In some implementations, the method further includes generating a map as a response to the search query in response to detecting a term representing a geographic location in the search query.

[0037] In some implementations, the method further includes displaying a scent-themed widget on a portion of the exploratory search panel to support faceted browsing of a pre-written visualization.

[0038] In some implementations, the method further includes augmenting the data source with metadata and semantics before receiving a search query, including linking attributes and values ​​in the data source to ontology concepts, including synonyms and related terms.

[0039] In some implementations, metadata includes data types and attribute semantics, including identifiers for attributes as measures and dimensions.

[0040] In another aspect, an electronic device includes one or more processors, a memory, a display, and one or more programs stored in the memory. The programs are configured to be executed by the one or more processors and are configured to perform any of the methods described herein.

[0041] In another aspect, a non-transitory computer-readable storage medium stores one or more programs configured for execution by a computing device having one or more processors, memory, and a display. The one or more programs are configured to perform any of the methods described herein.

[0042] Therefore, methods, systems, and graphical user interfaces that allow users to perform visual analysis on datasets are disclosed.

[0043] The foregoing general description and the following detailed description are both exemplary and illustrative, and are intended to provide further explanation of the claimed invention. Attached Figure Description

[0044] To better understand the systems, methods, and graphical user interfaces described above, as well as other systems, methods, and graphical user interfaces that provide data visualization analysis, reference should be made to the detailed embodiments described below in conjunction with the accompanying drawings, in which the same reference numerals refer to corresponding parts in the various figures.

[0045] Figure 1A This illustrates an example search scenario for question-and-answer (Q&A) based on an input query, according to some implementation methods.

[0046] Figure 1B An example search scenario is shown, based on some implementation methods of exploratory search.

[0047] Figure 1C An example search scenario is shown, based on a design search method that is implemented in some ways.

[0048] Figure 2 An example login screen is shown, based on some implementations.

[0049] Figure 3 An example search interface is shown, based on some implementation methods.

[0050] Figure 4 It is a schematic diagram of an example system based on some implementation methods.

[0051] Figure 5 This section presents a list of example JSONs showing visualization types and their concepts stored as metadata to support design searches, based on some implementation methods.

[0052] Figure 6 A schematic diagram of an example pipeline of a semantic search framework based on some implementations is shown.

[0053] Figure 7 Example interfaces for supporting Q&A search are shown, based on some implementation methods.

[0054] Figure 8 Examples of exploratory search results based on some implementation methods are shown.

[0055] Figure 9 The example design of search results is shown based on some implementation methods.

[0056] Figure 10 Example interfaces are shown, displaying available data sources indexed from a database and pre-written visualizations, according to some implementation methods.

[0057] Figure 11 The example search performed using the search interface is shown, based on some implementation methods.

[0058] Figure 12A An example search interface is shown, based on some implementation methods.

[0059] Figure 12B A preview with metadata is shown, based on some implementation methods. Figure 12A Example search interface.

[0060] Figure 13A This shows a sample search interface after a user enters a search query, based on some implementation methods.

[0061] Figure 13B This shows the implementation methods. Figure 13A Another view of the example search interface shown.

[0062] Figure 13C This illustrates how, according to some implementations, after a user enters a search query... Figure 13A Another view of the example search interface shown.

[0063] Figure 13D This illustrates how, depending on the implementation, different data sets are selected by the user. Figure 13A Another view of the example search interface shown.

[0064] Figure 13E This illustrates the process following a user-input query ("election results"), depending on the implementation. Figure 13A Another view of the example search interface shown.

[0065] Figure 14A An example search interface is shown, based on some implementation methods.

[0066] Figure 14B illustrates some implementation methods. Figure 14A Another view of the example search interface shown.

[0067] Figure 15 Sample user interfaces for question answering, exploratory search, and design search are shown, based on some implementation methods.

[0068] Figure 16 It is a block diagram illustrating a computing device according to some implementation methods, the computing device being able to display a graphical user interface and support visual analysis of datasets.

[0069] Figure 17 A flowchart is shown for example methods of visual analysis of datasets, based on some implementations.

[0070] Figure 18 A flowchart is shown for another example method for visual analysis of a dataset, based on some implementations.

[0071] Reference will now be made to implementations, examples of which are shown in the accompanying drawings. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring these specific details. Detailed Implementation

[0072] Depending on the implementation, the interface supports semantic search behavior by dynamically generating visual responses and pre-written visualizations of the data repository. Specifically, the interface implements three search scenarios within the semantic search framework: Q&A search by interpreting analytical intent within a curated set of data sources; exploratory search using document-based information retrieval methods on existing indexed visualizations; and design search by leveraging the visualization metadata of the content. The interface also supports faceted browsing that trims search results by author name, time frame, and / or visualization type.

[0073] To obtain feedback on the implemented metadata and query features, a qualitative study was conducted to identify system design and implementation challenges and to better understand user behavior. This study confirmed that the semantic search paradigm supports search objectives across different data repositories. The ability to conduct Q&A and obtain pre-written content facilitates a smooth analytical search experience.

[0074] As used in this article, "data repository" generally refers to a repository for data visualization, including metadata and parameters used for data visualization.

[0075] Example search scenario

[0076] Figures 1A to 1C An example semantic search scenario is shown based on some implementation methods. Figure 1A Example search scenario 100 is shown for the question and answer (Q&A) response to input query 102, “How has the trend of movie budgets changed over time for different generations?” Some implementations detect that this is a Q&A search with the analytical intent of 'trend'. The selected data source 104 ('movies') is the highest-scoring match for this query, and the system generates a multivariate line graph response 106. The generated text summary 108 describes the visualization, as shown in the figure. Pre-written visualization content 110 is also displayed as additional information in thumbnail form below the generated response.

[0077] Figure 1B An example search scenario 112 for exploratory search is shown. Some implementations identify the input query 114 (“elections”) as a keyword search query and show a pre-written visualization 116 with text content 118 related to 'elections'.

[0078] Figure 1CAn example search scenario 120 is shown. Query 122 (“treemap stocks”) is identified as a search for all content containing treemap visualizations related to 'stocks'. Some implementations return a set of relevant, pre-written visualizations 124 for that query and display them as thumbnails. If the user expects to continue with the analysis workflow, the thumbnails are linked to the actual visualizations.

[0079] Typically, there are three search scenarios in the context of a data repository:

[0080] • Question and Answer (Q&A). People using an organization-specific repository that hosts several data sources typically search for answers to analytical questions. This goal is similar to information lookup in the broader context of web search, where user queries map to short, discrete pieces of information (e.g., entities, dates, or calculated values). However, for data repositories, participants ask analytical questions (e.g., “What are sales trends across regions?”, “highest covid cases by country”) and receive appropriate responses containing visualizations and / or text generated from available data sources.

[0081] • Exploratory Search. Similar to the concept of exploratory search in web search, users leverage a data repository to learn about a topic through available charts and data. Examples of exploratory search queries include “NFL drafts,” “USA covid trends,” or “FIFA World Cup.” These queries are generally open-ended and do not provide refined filtering criteria beyond the topic itself. For example, a user (e.g., a visualization consultant) might use exploratory search as one of their primary goals during the initial phase of a client interaction. This is analogous to searching visualizations on an investment firm's private equity dashboard. As another example, a visualization designer might use exploratory search to find examples created by other users and data sources related to a particular topic.

[0082] • Design Search. The ability to find visualizations based on design features (such as chart type or color) is another use case for search, especially among designers / consultants or novice visualization writers. Examples of design search queries include “sunburst chart,” “bar and line combination chart,” or “map with icons.” This type of search is typically performed when users are looking for learning resources. For example, a developer might look for examples of force-oriented layouts. A user might try to create a custom visualization like a Sankey diagram, or a user might try to understand design practices and find inspiration for their own work (e.g., using a search like “maps with a dark background” to find examples of charts with specific color constraints).

[0083] It's important to note that these scenarios are neither exhaustive nor mutually exclusive. For example, queries like "sales by state and segment as a heatmap" or "maps showing covid trends" combine Q&A and design search, as well as design and exploratory search, respectively.

[0084] The three scenarios described above are not explicit and ordered sets, but rather examples of a broad range of user tasks designed for search systems used in data repositories.

[0085] In addition to defining search categories based on when and why people use searches in the data repository (i.e., the scenarios described above), the search system described in this paper can also identify the most relevant information that people look for when searching and browsing visualizations. To this end, some implementations provide a list of curated metadata fields. These fields may include the visualization title and description, chart type (e.g., 'bar chart', 'map', 'heatmap'), graphic encoding (such as tag type), visualization author, and / or the chart's creation date.

[0086] Example Design

[0087] Some implementations offer a unified experience supporting all three search scenarios. Others provide the same interface and modalities to perform different tasks. Still others provide a seamless experience supporting a common input modal (NL) and mixed Q&A (tasks typically performed on a collection of data sources) with exploratory and design search (tasks typically performed using pre-written visualization repositories).

[0088] Some implementations support language variation in queries. Previous work on NLI in visualization and web search has shown that people use multiple phrases in search queries to achieve the same goal. Users often use linguistically different examples when discussing the same goal (e.g., "What are sales trends across regions?" vs. "sales by region over time"). Some implementations support multiple query formats, from concise keywords to queries phrased as questions or sentence fragments, and in either case, they understand the analytical intent related to the data repository.

[0089] Some implementations display text responses and provide guidance for Q&A queries. Other implementations offer different types of visualizations for different queries. For example, in addition to charts, some implementations also provide text responses to text queries, including supplementary text along with the generated charts. Given a Q&A query, some implementations, besides generating text content that uses charts to help answer the input query, also select an appropriate data source and generate a chart. Furthermore, because Q&A queries can map to multiple data sources, and users may not be aware of the available data sources and fields, some implementations guide users in asking questions (e.g., via query suggestions) and provide metadata information about the relevant data sources (e.g., available data fields and the data values ​​to be queried).

[0090] Some implementations offer visual summaries and filtering options for search results. Conventional visual search systems don't provide an easy way to understand and filter results beyond manual inspection. To overcome this limitation, some implementations offer visual summaries and support dynamic filtering to help people overview, organize, and create meaningful facets for visual search results.

[0091] Example interface

[0092] Some implementations provide interfaces that support semantic search behavior by dynamically generating visual responses and pre-written visualizations from data repositories. Example interfaces are described through brief use cases. Details of example system components and implementations follow.

[0093] Figure 2Example login screen 200 is shown according to some implementations. The login page shows a search input box (labeled 'A') with placeholder query suggestions generated based on one of the available data sources. Thumbnail previews of some available data sources are shown (labeled 'B'). In some implementations, hovering the mouse over a data source (e.g., the 'Housing' data source) displays a tooltip showing metadata about the data source's properties and values. According to some implementations, samples of pre-written visualizations that can be used for searching are also shown (labeled 'C').

[0094] In some implementations, the interface initially displays a login screen showing a sample of data sets available for Q&A searches. Users can hover their mouse over the data source thumbnails to view their corresponding metadata information. Figure 3 Example search interfaces 300 are shown according to several implementations. Some implementations show a search input box (labeled 'A'). Some implementations show dynamically generated content (labeled 'B'), including a chart on the right and text highlighting key points in the chart on the left. Users can hover the mouse cursor over the icon to display [the relevant text]. Figure 2 The dataset summary tooltip shown is similar to other dataset summary tooltips. Some implementations show a pre-defined set (e.g., the top 50) of pre-written visualizations mapped to the input query (labeled 'C'). Some implementations show a scented widget that supports dynamically filtering the pre-written results (labeled 'D'). Suppose a user types the search query "housing prices usa (US housing prices)" into the input text box. The system detects that the term 'usa' is a geographic location and searches relevant data sources in its data repository. The system finds a housing data source that matches and dynamically generates a map as a Q&A response to the query (…). Figure 3 (Marked as 'B'). Additionally, as part of the exploratory search, query terms are used to match any pre-written visual keywords. A thumbnail grid is displayed to serve as a preview for the user to browse and explore. Figure 3 (Marked as 'C' in Chinese). Each thumbnail hyperlinks to its corresponding visualization file, which users can select for more detailed viewing or download to their local machine. The visualization's title, author's name, and creation date are displayed below each thumbnail to provide additional context. A smell-enabled widget appears on the right side of the exploratory search panel to support access to pre-written visualizations (…). Figure 3 Use the 'D' option to browse faceted search results. Users can narrow down their search results by applying one or more filters simultaneously, such as author name, visualization type, and creation date.

[0095] Example System

[0096] Figure 4 This is a schematic diagram of an example system 400 based on some implementation methods. The example illustrates various components: a query classifier 402, a parser 404, a semantic search framework 406, a Q&A module 408, and a general search module 410. The query classifier 402 checks for the presence of terms 412 in the query 416, which refer to fields from the data source and analytical intents (e.g., intent, attributes, and values) in the parsed query 414 from the parser 404. If present (418), a dynamically generated visualization 420 from the Q&A search component 408 is rendered, along with pre-written visualizations 422 from the general search component (exploratory and design) 410. The semantic search framework 406 includes both a data source search index 424 and a visualization search index 426.

[0097] In some implementations, the interface is implemented using Python and a Flask backend connected to a Node.js frontend, creating a web-based application. Other implementations use Elasticsearch, an open-source full-text Java search engine designed to be distributed, scalable, and offer near real-time query execution performance.

[0098] In some implementations, the system includes a repository of curated data sources for Q&A searches. These data sources can be stored in various formats, such as relational databases, JSON, XML, flat files, or tabular CSV files. Some implementations also include data sources spanning a variety of familiar topics, such as sales, sports, world events, entertainment, and civic issues.

[0099] Example data repository and metadata

[0100] Unlike traditional document search, data sources and visualizations are often text-sparse, with limited searchable text content. Therefore, some implementations augment the data repository by helping the system understand and interpret additional metadata and semantics of search queries. Specifically, in some implementations, data fields (attributes) and data values ​​in the data source are linked to ontology concepts, including synonyms (e.g., 'film' and 'movie') and related terms (e.g., 'theft', 'burglary', and 'crime'). In some implementations, the system includes a small hierarchical structure of hypernyms and hyponyms from Wordnet, typically at a depth of two hierarchical levels (e.g., ['beverage', 'drink'] → ['espresso', 'cappuccino']). In some implementations, metadata also includes data types (e.g., 'text', 'date', 'Boolean', 'geospatial', 'temporal', and 'numeric') and attribute semantics, such as currency type (e.g., US dollar). This information can also be inferred using data pattern matching techniques. Metadata also identifies attributes as measures (e.g., attributes that can be measured, aggregated, or used in mathematical operations) and dimensions (e.g., fields that are essentially categorical and cannot be summed except for counting). This set of metadata information is then added to a semantic search framework.

[0101] In some implementations, the pre-written content is derived from a set of 75,000 or more visualizations from Tableau Public (a community-based, free platform). In other implementations, the visualizations reflect the user's demographics and / or include themes such as natural disasters, health, world events, financial news, entertainment, and sports.

[0102] In some implementations, given the Tableau workbook's XML visualization specification, the system traverses the Document Object Model (DOM) structure and indexes any textual metadata that can be extracted from the visualization. The extracted metadata may include the visualization title, captions, logo, description, author's name, profile, visualization tags encoded in the visualization, and / or visualization type. To support designs for identifying visualization types mentioned in search queries, some implementations include a general list of visualization types and language variants of visualization types within the semantic search framework. Figure 5 Example JSON list 500 shows visualization types and their concepts stored as metadata to support design search, based on some implementation methods.

[0103] In addition to CSV data sources and Tableau visualizations, the architecture can also include additional data repositories, including Vega-lite charts and knowledge base articles.

[0104] Example Query Classifier

[0105] In some implementations, natural language search queries are input into a query classifier. In other implementations, the classifier supports federated query search, which distributes queries across multiple search repositories and combines the results into a single merged search result. Therefore, from the user's perspective, it appears as if they are interacting with a single search instance. In this context, users can search heterogeneous data repositories (i.e., both data sources and visualizations) without changing or modifying how they formulate their query input. The query classifier passes search terms to the parser and the data source search index (which is part of the semantic search framework) and determines whether the system needs to generate Q&A searches to dynamically generate visualization responses, or simply perform a generic search that supports both exploratory and design searches. Algorithm 1, shown below, is an example query classification process based on some implementations. In summary, the query classifier passes query terms to the parser (line 7) to determine whether the query contains any analytical intent, such as aggregation, association, temporal, or geospatial representations. The query classifier also passes query terms to the semantic search framework to determine whether the query terms match any field in any data source (e.g., matching 'prices' with 'price' in the housing data source) and whether the normalized match score is greater than a pre-determined threshold (line 10). In practice, fieldMatch = 2 and normMatch = .3 provide reasonable thresholds for relevant data source matches. If both conditions (i.e., analytical intent exists and the match score meets the threshold criteria) are met, a Q&A search is first invoked to dynamically generate a visual response to the given query (line 13). Otherwise, a general search is invoked to return pre-written content from the data repository (line 16).

[0106] An example algorithm (Algorithm 1 cited above) is shown below. The algorithm categorizes search behavior based on whether the query contains analytical intent and whether there is a match for one or more selected data sources.

[0107] 1: Function QUERYCLASSIFIER(query)

[0108] ⊳ A Boolean used to check if there is analytical intent in the query.

[0109] 2:hasAnalyticalIntent ←False

[0110] ⊳ A boolean used to check if a data source match exists.

[0111] 3:hasDSMatch ←False

[0112] ⊳ Contains matching scores for the query and each data source ds.

[0113] 4:dsScores ← getDSScores (query, ds)

[0114] ⊳ Contains normalized match scores for the query and each data source ds.

[0115] 5:normScores ← norm(dsScores)

[0116] ⊳Predetermined thresholds set for field matching in ds and normScores.

[0117] 6: fieldMatch, normMatch

[0118] ⊳ Check whether the parsed query contains analytical intent.

[0119] 7: If (parseForAnalyticalIntent(query), then

[0120] 8: hasAnalyticalIntent ← True

[0121] 9: End if

[0122] ⊳ Check whether the query terms match the fields in ds and whether the normalized match score of ds is greater than a predetermined threshold.

[0123] 10: If (dsScores['fields'] > fieldMatch) and (normScores > normMatch)), then

[0124] 11:hasDSMatch ←True

[0125] 12: End if

[0126] If the query has analytical intent and contains terms that match ds, the Q&A search is invoked before the general search; otherwise, only the general search is invoked.

[0127] 13: If (hasAnalyticalIntent and hasDSMatch), then

[0128] 14: invokeQ&ASearch(query, ds)

[0129] 15: End if

[0130] 16: invokeGeneralSearch(query)

[0131] 17: Termination Function

[0132] Example parser

[0133] In some implementations, the parser removes stop words (e.g., 'a' and 'the') and connectives / disjunctions (e.g., 'and' and 'or') from the search query and extracts a list of N-grams (e.g., 'Seattle houseprices' → [Seattle], [house], [prices], [houseprices], [Seattle house prices]). In some implementations, the parser employs the Cocke-Kasami-Younger (CKY) parsing algorithm and generates dependency trees to understand the relationships between words in the query. The input to the underlying CKY parser is context-free grammar, whose generation rules are enhanced with both syntactic and semantic predicates to detect the following analytical intents in the search query:

[0134] Grouping. Divide the data into categories (e.g., by data attributes).

[0135] • Aggregation. Grouping values ​​from multiple rows of data together to form a single value based on mathematical operations (such as 'average', 'median', 'count', or 'distinct count').

[0136] • Correlation. A statistical measure of the strength of the relationship between two data attributes (measures) (e.g., 'correlate' or 'relate').

[0137] • Filters and Limits. A finite set of operators that return a subset of data attribute fields (e.g., 'filter to', 'at least', 'between', or 'at most'). Limits are also a finite set of operators, similar to filters that return a subset of attribute fields, but limit to at most n rows (e.g., 'top' or 'bottom').

[0138] • Time. Time and date expressions containing time motifs and phrases (e.g., 'over time', 'year', 'in 2020', or 'when').

[0139] • Geospatial. Geospatial expressions that refer to locations and places (e.g., 'in Canada', 'by location', or 'where').

[0140] To aid in detecting the association graph of data attributes and values, the parser gains access to the set of selected data sources and their metadata. The parser then compares the N-gram with available data attributes for both syntactic similarity (e.g., spelling errors) and semantic similarity (e.g., synonyms) using Levenshtein distance and Wu-Palmer similarity scores, respectively. If the parser detects one or more of the aforementioned analytical intents, it returns the intent along with its corresponding data attributes and values ​​to the query classifier.

[0141] Semantic search framework

[0142] Some implementations include semantic search frameworks that consist of two phases: indexing content and metadata, and searching for content and metadata within a data repository. This two-phase process applies to content in the data repository, including both curated data sources and visual content. Figure 6 A schematic diagram of an example pipeline 600 of a semantic search framework according to some implementations is shown. Document vectors X602 from pre-written visualizations 604 and data source 606, along with their corresponding encoded string terms S 608, are indexed (624) in the semantic search framework 406 into two data repository indexes (data source search index 424 and visualization search index 426). During a search, a query vector from the input search query 610 (e.g., "Seattle house prices") is used. 612 is encoded into string terms 614 and used to retrieve a set of relevant results from both the visualization content and the data source 616. A ranking function 618 T is used. i ∈{i1,i2,...,i r} = min(| − x i |), perform a final ranking of the result set to return the highest-scoring result (related visualization 620 and related data source 622).

[0143] Indexing phase 624 creates an index for each data repository (data source and visualization content) in the data repository, along with the data repository's metadata, to support federated search. Given a data source and visualization content and associated metadata (i.e., attributes, data values, chart type, and author name), each file is represented as a document vector x. i ,in:

[0144] (1)

[0145] Some implementations also store N-gram strings from these document vectors to support partial and exact matching in the system:

[0146] (2)

[0147] For a given encoder, The encoder transforms the document vector into a set of string tokens of cardinality n. The original vector X and the encoded tokens S are stored in the index by mapping the specified content (i.e., defining the type and format of fields in the index). Some implementations store text as keywords in the index, thus supporting exact value searches, fuzzy matching to handle typos and spelling variations, and N-grams for phrase matching. Scoring algorithms, tokenizers, and filters are specified as part of the search index setup to determine how to score matching documents relative to the input query and token processing (such as adding synonyms from the thesaurus, removing stop words (e.g., "a", "the", and "for") and duplicate tokens, and converting tokens to lowercase).

[0148] In some implementations, the search phase 626 has the following two steps: retrieval 616 and ranking 618. Given an input query q, it is represented as a query vector. The query term is Some implementations use the same encoder from the indexing stage to encode vectors into string tokens. The search process is based on the query string term set. and The most relevant r document vectors are retrieved based on the overlap between document string terms. As candidates. More specifically, the scoring function maximizes search relevance by calculating the following:

[0149] (3)

[0150] Some implementations were subsequently based on the query vector. BM25 score for candidate search results set The vectors in the document are ranked. BM25 is essentially a bag-of-words scoring function that ranks documents based on the occurrence of query terms in each document, regardless of the proximity of the query terms within the document. It is the preferred metric for calculating the similarity between vectors because the method corrects for variations in vector magnitude caused by documents of uneven length. Given Document vector x i The BM25 score is:

[0151] (4)

[0152] in yes In document vectors The number of times it appears in the index, and avgdl is the average document vector length in the search index. b and b are constants used to further optimize the scoring function.

[0153] Furthermore, b = 0.75 tends to provide reasonable ranking behavior. Inverse Document Frequency (IDF) measures the frequency of a term across all documents and boosts the ranking of the only term in a document. It is calculated as follows:

[0154] (5)

[0155] Where docCnt is a string containing the given query term. The total number of documents with the value of , and It is the number of documents containing the i-th query term.

[0156] The BM25 scoring function sorts the vector in descending order of its normalized BM25 scores, where That is, the higher the score, the higher the ranking, thus producing a final ranked search results set based on the minimum difference between the query and each document vector. :

[0157] (6)

[0158] In some implementations, the search request is then passed to the Elastic Search server. The system calculates Equations 3 and 4 and returns a ranked set of results from the data source (for Q&A) or visualization content used for both exploratory and design search scenarios.

[0159] Example Q&A module

[0160] The Q&A module interprets the analytical intent expressed in the input search query and dynamically generates a visual response based on a list of best-matching data sources returned from the semantic search framework, as described above. In some implementations, the module accepts a tabular CSV dataset of the best-matching data sources as input. In some implementations, the visualizations in the tool are created using Vega-Lite.

[0161] Figure 7 Example interface 700 for supporting Q&A search is shown, based on some implementations. The interface provides interactive and scaffolded support (see label 'A'). When multiple data sources are identified as the best match for a given search query, a dropdown list displays a ranking list of data sources along with their corresponding percentage match scores. When a match exists with a data source but a valid visualization cannot be generated for a given search query, some implementations display a list of query suggestions (see label 'B') for selection to generate a visual response.

[0162] In some implementations, the interface and functionality for Q&A search are similar to those for Natural Language Interfaces (NLI) used for visual analytics, but with some extensions inherent to Q&A behavior within the context of semantic search. For example, the interface displays text indicating matches with one or more data sources (if any), along with dropdown menus of the matched data sources. Visualizations are based on analytical intent in attributes, values, and queries, along with a text summary describing the visualization (e.g., ...). Figure 1A The text in the dropdown menu (108) is used for rendering. Users can view the dropdown menu for other data source alternatives and the corresponding percentage match scores for other data source alternatives (calculated as described above), and choose to switch to another data source in the dropdown menu, such as... Figure 7 As shown (see label 'A'). In cases where a query matches a data source, but the terms in the query do not resolve to valid attributes and values ​​within that data source, some implementations display suggested queries for that data source. Examples are shown in... Figure 7 As shown in the diagram (see label 'B'). These query suggestions are generated using a template-based approach, which is based on a combination of attributes from the data source and data interest metrics.

[0163] In some implementations, the visualization generation process for Q&A search supports three encoding channels (x, y, and color) and four marker types (bars, lines, dots, and geographic shapes). These markers and encodings support the dynamic generation of bar charts, line graphs, scatter plots, and maps covering a range of analytical intents described in Section 4.5. Some implementations select a default visualization that uses rules to determine the marker type based on a mapping between visual encoding and attribute data types (e.g., a scatter plot is shown if two quantitative attributes are mapped to x and y channels, and a line graph is shown if a time attribute is visualized on the x-axis and a quantitative attribute is visualized on the y-axis).

[0164] Some implementations display dynamic text summaries of the generated visualizations. While template-based approaches are a viable option for the summarization process, some implementations employ a Large Language Model (LLM)-based approach. Passing chart data as-is to an LLM application to generate descriptions can produce erroneous statistics, or even illusions, depending on the data domain context. To overcome these challenges while still providing convincing descriptions, some implementations follow a combined approach using both basic statistical computation and LLM. Specifically, input to an LLM-based chat application is a prompt containing statistical descriptions extracted from the generated visualizations using a set of heuristics defined in earlier data insight recommendation tools. For example, for bar charts, some implementations identify minimum / maximum and average values; for scatter plots, some calculate Pearson correlation coefficients, and so on. Consider a search query “sales by region”, which produces a bar chart showing sales across all four regions. An example of the statistical descriptions (keyStats) in this bar chart would be:

[0165] Region: Central has a minimum sales value of $220.

[0166] The maximum sales value for the South region is $240.

[0167] Average sales across the region is $230.

[0168] The corresponding prompt from the LLM application then becomes a more persuasive rewrite of the following input: '${keyStats}\n', ultimately generating a text summary: "The Sales in Central Region had the lowest value of $220, while South Region had the highest value of $240. The average Sales across all Regions was $230."

[0169] Example General Search Module

[0170] In some implementations, the general search module 410 displays thumbnails of pre-written visualizations along with information such as titles and dates. If the user chooses to download or analyze the visualization in more detail, the thumbnail image is hyperlinked to the corresponding Tableau Public workbook URL. The module implements two types of searches: exploratory search and design search.

[0171] Figure 8 Example exploratory search results 800 are shown, based on two superimposed parts according to some implementations. Some implementations display pre-written thumbnail visualizations 802 and 804 based on keywords found in the input search query (e.g., "world population" 806 and "crime in USA" 808). Exploratory search returns visualizations based on keyword matching in the input search query (e.g., "world population" 806).

[0172] Figure 9 Example design search results 900 are shown, based on the overlay of two parts according to some implementations. Some implementations display thumbnails of pre-written visualizations for combinations of chart types (e.g., bar charts and line charts) and analytical concepts that imply a specific visualization type (e.g., 'correlation' for scatter plots) 902 and 904. Design search is a special form of exploratory search that specifically returns visualization results for keywords containing terms that mention the visualization type, synonyms of terms, and related concepts (e.g., "covid correlations" 906).

[0173] Example search interface

[0174] Figure 10 A sample interface 1000 is shown, based on some implementation, displaying a sample of available data sources 1002 indexed from a database (e.g., Tableau public) and pre-written visualizations 1004. Users of the search interface may ask numerous questions while interacting with it. For example, a user might want to find examples of tree diagrams to build upon. Another example is a user wanting to understand the differences in COVID-19 cases across countries. Some users might ask other users what kind of visualizations they are creating (e.g., to present election results).

[0175] Figure 11 Example searches 1100 using a search interface are shown according to some implementations. According to some implementations, the search interface enables users to perform multiple searches 1100. A first example 1102 shows a user asking “How has the trend of movie budgets changed over time for different genres?” A second example 1104 shows a user searching for “election results.” And a third example 1106 shows a search for “treemapstocks.”

[0176] Figure 12A A sample search interface 1200 is shown, based on some implementation methods. This example shows a sample dataset 1202 and a sample graph 1204 that are searchable in the repository.

[0177] Figure 12B A sample search interface 1200 with a preview of metadata 1208 is shown, according to some implementations. Users can hover the mouse over 1206 on (or point to) the data source to preview the metadata and use the search box to issue queries based on the metadata.

[0178] Figure 13A A sample search interface 1300 is shown after a user enters a search query 1302, according to some implementation methods. In this case, after receiving the query "what are average budgets by genre?", the system examines available data sources to find any relevant data sources. Determining a match with the movie data set, the system then dynamically generates a sorted bar chart 1304 to answer the given query and produces a text summary 1306 describing the key points in the chart. In addition to finding relevant data sources and generating a dynamic response, the system also returns fifty relevant visualizations 1308.

[0179] Figure 13BAnother view of an example search interface 1300, based on some implementations, is shown. A different set of related visualizations 1310 are shown. Users can browse and explore these visualizations, or use dynamic filtering widgets (e.g., writer filter 1312 or chart type filter 1314) to identify visualizations of interest. In some implementations, if the query is ambiguous or unclear, the system also provides the user with options to override their default selection.

[0180] Figure 13C Another view of a sample search interface 1300, implemented in some manner, is shown after a user enters search query 1316 (“show sales across the US”). See above for reference. Figure 4 Similar to the preview of the previous query description, the system looks for relevant data sources on sales figures for large supermarkets and generates a map 1320 because the query includes a geographic intent ("across the US"). The example also shows the system indicating the existence of another sales data source 1318 relevant to the query, allowing the user to switch between the two data sources.

[0181] Figure 13D Another view of an example search interface 1300, shown according to some implementations, is presented after the user selects different data groups 1322 (sometimes referred to as data sources). (Refer to the above) Figure 13C For the same query description, similar previews are generated, and because the query includes a geographic intent (“across the US”), the system generates a map 1328, a text summary 1324 describing the key points in the chart, and a set of pre-written visualizations 1326.

[0182] Figure 13E Another view of the sample search interface 1300 is shown after the user enters query 1330 (“election results”). In the case of exploratory searches (such as election results), if the entered query is too vague or the system cannot match the query with the data source, the system may not generate a visualization. The system only displays a pre-written visualization 1332, allowing the user to explore the content using filters 1334.

[0183] In some implementations, the system automatically populates the user interface with appropriate pre-written visuals as the user begins typing. Figure 14AA sample search interface 1400 is shown according to some implementations. As the user begins typing query 1402 (“map with overlaid pie”), the system begins filling a pre-written visualization area 1404 with a map and a pie chart. Figure 14B shows another view of the sample search interface 1400 according to some implementations. In this example, the user types query 1406 (“square chart”), and the system begins filling a pre-written visualization area 1408 with a chart containing squares (or rectangles). In this way, in addition to helping users answer questions or explore topics broadly, some implementations also allow users to search for specific visualization types by including the visualization's graphical metadata attributes (e.g., square) as part of its search engine index. Combined with filtering options, the interface allows users to search for visualizations in a more expressive way by explicitly specifying the chart type used to combine different visualization types or even specifying lower-level graphical attributes (such as shape) that make up the visualization.

[0184] Figure 15 Example user interfaces 1500 for question answering 1502, exploratory search 1504, and design search 1506, based on some implementation methods, are shown. In this way, the example semantic search techniques described herein not only support search targets for different data repositories but also facilitate a smooth analytical search experience.

[0185] Example computing device for visual analytics of datasets

[0186] Figure 16This is a block diagram illustrating a computing device 1600 according to some implementations, which can display a graphical user interface and support visual analysis of datasets. Various examples of the computing device 1600 include desktop computers, laptop computers, tablet computers, and other computing devices having a display and a processor capable of running data visualization applications 1630. The computing device 1600 generally includes one or more processing units (processors or cores) 1602, one or more network or other communication interfaces 1604, memory 1606, and one or more communication buses 1608 for interconnecting these components. The communication bus 1608 optionally includes circuitry (sometimes referred to as a chipset) that interconnects and controls communication between system components. The computing device 1600 includes a user interface 1610. The user interface 1610 generally includes a display device 1612. In some implementations, the computing device 1600 includes input devices such as a keyboard, mouse, and / or other input buttons 1616. Alternatively, or additionally, in some implementations, the display device 1612 includes a touch-sensitive surface 1614, in which case the display device 1612 is a touch-sensitive display. In some implementations, the touch-sensitive surface 1614 is configured to detect various swipe gestures (e.g., continuous gestures in the vertical and / or horizontal directions) and / or other gestures (e.g., single-click / double-click). In computing devices having a touch-sensitive display 1614, a physical keyboard is optional (e.g., a soft keyboard may be displayed when keyboard input is required). In some implementations, the user interface 1610 also includes an audio output device 1618, such as a speaker or an audio output connection connected to a speaker, headphones, or headset. Furthermore, some computing devices 1600 use a microphone and speech recognition to supplement or replace the keyboard. In some implementations, the computing device 1600 includes an audio input device 1620 (e.g., a microphone) to capture audio (e.g., speech from the user).

[0187] Memory 1606 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices, and may include non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some implementations, memory 1606 includes one or more storage devices located remotely from processor 1602. The non-volatile memory devices within memory 1606, or alternatively memory 1606, include non-transitory computer-readable storage media. In some implementations, memory 1606 or the computer-readable storage media of memory 1606 stores programs, modules, and data structures, or subsets or supersets thereof:

[0188] • Operating system 1622, which includes procedures for handling various basic system services and for performing hardware-related tasks;

[0189] • Communication module 1624, which is used to connect computing device 1600 to other computers and devices via one or more communication network interfaces 1604 (wired or wireless) (such as the Internet, other wide area networks, local area networks, metropolitan area networks, etc.);

[0190] • An optional web browser 1626 (or other application capable of displaying web pages) that enables users to communicate with remote computers or devices over a network;

[0191] • An optional audio input module 1628 (e.g., a microphone module) is used to process audio captured by the audio input device 1620. The captured audio may be sent to a remote server and / or processed by an application (e.g., a data visualization application 1630) running on the computing device 1600;

[0192] • Data visualization application 1630 generates data visualizations and related features. Application 1630 includes a graphical user interface 1632 (e.g., the graphical user interface 100 shown in Figure 1) for users to construct visualizations. For example, a user selects one or more data sources 1640 (which may be stored on computing device 1600 or in remote storage), selects data fields from the data sources, and uses the selected fields to define a visualization; and

[0193] One or more databases or data sources 1640 (e.g., first data source 1640-1 and second data source 1640-2) are used by the data visualization application 1630. In some implementations, the data sources are stored as spreadsheet files, CSV files, text files, JSON files, XML files, or flat files, or stored in a relational database. (See above for reference.) Figures 1A to 1C , Figure 2 , Figure 4 and Figure 6 Examples describing data sources (sometimes called curated data sources).

[0194] In some implementations, the data visualization application 1630 includes a data visualization generation module 1634, which receives user input (e.g., visualization specification 1636) and generates a corresponding visualization graphic. The data visualization application 1630 then displays the generated visualization graphic in a user interface 1632. In some implementations, the data visualization application 1630 executes as a standalone application (e.g., a desktop application). In some implementations, the data visualization application 1630 executes within a web browser 1626 or another application using web pages provided by a web server (e.g., a server-based application).

[0195] In some implementations, user-provided information (e.g., user input) is stored as visual specification 1636. In some implementations, visual specification 1636 includes attributes received from previous natural language commands from the user or specified by the user via natural language commands.

[0196] In some implementations, the data visualization application 1630 includes a language processing module 1638 for processing (e.g., interpreting) commands provided by a user of the computing device. In some implementations, the commands are natural language commands (e.g., captured by an audio input device 1620). In some implementations, the language processing module 1638 includes submodules such as a parser 1642 (e.g., parser 404), a query classifier 1644 (e.g., query classifier 402), a Q&A module 1646 (e.g., Q&A module 408), a general search module 1648 (e.g., general search module 410), and / or a semantic search module 1650 (e.g., semantic search framework 406), examples of which are referenced above. Figure 4 and Figure 6 The description is provided. Depending on some implementations, the data visualization application 1630 also includes pre-written visualizations and / or results 1652, examples of which are shown above with reference to Figure 1 and... Figure 2 It has been described.

[0197] In some implementations, memory 1606 stores intermediate data (e.g., query terms, query vectors, intents, attributes, values, Levenshtein distances, syntactic and / or semantic similarities identified using Wu-Palmer similarity scores, metrics, and / or scores) determined or computed by language processing module 1638. In some implementations, memory 1606 stores Pearson correlation coefficients, minimum / maximum values, average values, and / or cues for large language models. Furthermore, memory 1606 may store thresholds and other criteria compared to metrics and / or scores determined by language processing module 1638.

[0198] Each of the executable modules, applications, or procedures identified above may be stored in one or more of the previously mentioned memory devices and corresponds to an instruction set for performing the functions described above. The modules or programs identified above (i.e., instruction sets) do not need to be implemented as separate software programs, procedures, or modules, and therefore subsets of these modules may be combined or otherwise rearranged in various implementations. In some implementations, memory 206 stores a subset of the modules and data structures identified above. Furthermore, memory 206 may store additional modules or data structures not described above.

[0199] although Figure 16The computing device 1600 is shown, but Figure 16 This is intended more as a functional description of the various possible features than as a structural diagram of the implementation described herein. In practice, and as those skilled in the art will recognize, items shown individually can be combined and some items can be separated.

[0200] Example methods for visual analysis of datasets

[0201] Figure 17 A flowchart of an example method 1700 for visual analysis of a dataset, according to some implementation, is shown. The method is executed at a computing system 1600 having one or more processors 1602 and memory 1606 configured for execution of one or more programs by the one or more processors.

[0202] The method includes receiving (1702) (e.g., by query classifier 402) a natural language search query (e.g., query 416) pointing to multiple data repositories including multiple data sources 1640 and one or more data visualizations (e.g., pre-written visualizations 1652).

[0203] The method further includes parsing (1704) (e.g., by parser 404) search terms (e.g., term 412) corresponding to a natural language search query to determine whether the natural language search query contains one or more analytical intents (e.g., intent 414). In some implementations, parsing search terms further includes identifying data attributes and values ​​along with one or more analytical intents based on multiple data sources and metadata from multiple data sources. In some implementations, identifying data attributes and values ​​includes comparing N-grams corresponding to search terms with available data attributes for syntactic and semantic similarity. In some implementations, syntactic similarity is identified using Levenshtein distance and semantic similarity is identified using Wu-Palmer similarity scores.

[0204] The method also includes: using semantic search (e.g., using semantic search framework 406) to determine whether (1706) search terms match data fields (i.e., attributes) in one or more of a plurality of data sources. In some implementations, the method further includes: using a federated query search to obtain search terms, the federated query search distributing the query to multiple search repositories and combining the results into a single merged search result. In some implementations, semantic search includes: indexing each of the plurality of data repositories and the metadata of the plurality of data repositories to obtain an index; and performing a federated search to determine, based on the index, whether search terms match fields in one or more of the plurality of data sources. In some implementations, indexing includes: representing each document as a corresponding document vector for each data repository and visualization context and associated metadata; and storing N-gram string terms from the document vectors to support partial and exact matching. In some implementations, performing a federated search includes: obtaining query vectors corresponding to search terms; encoding the query vectors into query string terms using an encoder for generating an index; and selecting a predetermined number of candidate document vectors from the document vectors for each data repository and visualization context with associated metadata, based on the amount of overlap between the query string terms and the document string terms of the document vectors. In some implementations, the method further includes: ranking the predetermined number of candidate document vectors using a scoring function that scores documents based on the search terms appearing in each document, regardless of the proximity of the search terms within the document. In some implementations, the method further includes: when the semantic search returns multiple data sources, providing a dropdown list of alternative data sources along with a corresponding percentage match score for each data source. In some implementations, the method further includes: when (i) the semantic search returns a matching data source for a natural language query and (ii) the search terms do not resolve to valid attributes and values ​​within that data source, displaying suggested queries for that data source. In some implementations, the method further includes generating suggested queries using a template-based approach based on a combination of attributes (i.e., data fields) from the data source and data interest metrics. Examples of these operations, depending on the implementation, are referenced above. Figure 6 It has been described.

[0205] The method further includes generating and displaying (1708) (e.g., by the data visualization generation module 1634) one or more visualization responses when (i) a search term matches a field in one or more data sources and (ii) a natural language search query contains one or more analytical intents. In some implementations, the one or more analytical intents are one or more of the following: grouping, aggregation, association, filtering and limiting, time and geospatial. In some implementations, the method further includes generating and displaying one or more visualization responses by interpreting one or more analytical intents based on a predetermined list of data sources returned by a semantic search. In some implementations, the method further includes generating and displaying one or more visualization responses based on attributes, values, and analytical intents in the natural language search query. In some implementations, the method further includes generating and displaying one or more visualization responses using three encoded channels (x, y, and color) and four marker types (bars, lines, dots, and geographic shapes), thereby supporting the dynamic generation of bar charts, line charts, scatter plots, and maps covering a range of analytical intents. In some implementations, the method further includes determining the tag type of one or more visual responses based on the mapping between visual encoding and attribute data types.

[0206] In some implementations, the method further includes: using one or more statistical calculations and a large language model to generate and display (e.g., by language processing module 1638) a dynamic text summary 108 describing the responses of one or more visualizations. In some implementations, the method further includes: providing a hint containing a statistical description to the large language model, the statistical description being extracted from the generated visualization using a predefined set of heuristics. In response to providing the hint, the method receives the dynamic text summary from the large language model. In some implementations, the predefined set of heuristics is selected based on the type of visualization generated. In some implementations, the hint corresponds to (i) minimum / maximum and average values ​​for bar charts, and (ii) Pearson correlation coefficients for scatter plots.

[0207] The method also includes: when (i) search terms do not match fields in multiple data sources or (ii) a natural language search query does not contain one or more analytical intents, displaying (1710) (e.g., displayed by data visualization generation module 1634) one or more pre-written visualizations from one or more data visualizations (e.g., Figure 1B Visualization 116 in the middle, and pre-written visualization 1652).

[0208] Figure 18A flowchart of another example method 1800 for visual analysis of a dataset, according to some implementation, is shown. The method is executed at a computing system 1600 having one or more processors 1602 and memory 1606 configured for execution of one or more programs by the one or more processors.

[0209] The method includes: receiving (1802) (e.g., by a data visualization application 1630) a user's selection of a data source (e.g., data source 1640-1), presenting (1804) a graphical user interface 1632 for analyzing data in the selected data source, and providing (1806) three search options. The search options include: a question-and-answer search (1808) for interpreting analytical intent within a selected set of data sources; an exploratory search (1810) for document-based information retrieval of indexed visualization content; and a design search (1812) using visualization metadata from the data source.

[0210] In some implementations, the method further includes providing a faceted browser for trimming search results by author name, time range, and / or visualization type.

[0211] In some implementations, the method further includes: providing a login screen 200 that displays samples of the data source as thumbnail images; and in response to a user hovering the mouse over the thumbnail image of the data source, the method displays metadata information corresponding to the data source.

[0212] In some implementations, the method further includes: responding to the user typing a search query into an input text box (e.g., when the user enters...). Figure 8 After entering the query "world population" in text box 806: Use search query terms (e.g., Figure 4 The term 412 in the search query is used as a keyword to match any pre-written visualization (e.g., pre-written result 422), and a grid of thumbnails is displayed as a preview for browsing and exploration. Each thumbnail is hyperlinked to its corresponding visualization file, which can be selected for more detailed reading or download. In some implementations, the title of the visualization, the author's name, and / or the creation date are displayed below each thumbnail to provide additional context. In some implementations, the method further includes generating a map in response to detecting a term as a geographic location in the search query. In some implementations, the method further includes displaying a smell-enabled widget on a portion of the exploratory search panel to support faceted browsing of the pre-written visualizations.

[0213] In some implementations, the method further includes augmenting the data source with metadata and semantics before receiving search queries, including linking attributes and values ​​in the data source to ontology concepts, including synonyms and related terms. In some implementations, the metadata includes data types and attribute semantics, including identifiers for attributes as measures and dimensions.

[0214] The terminology used in describing this invention is for the purpose of describing particular implementations only and is not intended to limit the invention. As used in the description of the invention and the appended claims, the singular forms “a,” “an,” and “described” are intended to equally include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or,” as used herein, refers to and covers any and all possible combinations of one or more of the associated listed items. It should be further understood that, when used in this specification, the term “comprising” specifies the presence of the stated features, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0215] For purposes of explanation, the foregoing description has been given with reference to specific implementations. However, the above illustrative discussion is not intended to be exhaustive or to limit the invention to the precise forms disclosed. In view of the above teachings, many modifications and variations are possible. The implementations have been chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to best utilize the invention and various embodiments with various modifications suitable for the specific uses covered.

[0216] The following clauses describe some embodiments or implementations:

[0217] Clause A1. A method for visual analysis of a dataset, comprising:

[0218] In a computing system having one or more processors and memory configured for executing one or more programs by said one or more processors:

[0219] Receive natural language search queries that point to multiple data repositories, including multiple data sources and one or more data visualizations;

[0220] Parse the search terms corresponding to the natural language search query to determine whether the natural language search query contains one or more analytical intents;

[0221] Use semantic search to determine whether the search terms match data fields in one or more of the plurality of data sources;

[0222] Based on (i) matching the search terms with fields in the one or more data sources and (ii) the determination that the natural language search query contains one or more analytical intents, generate and display one or more visual responses; and

[0223] Based on the determination that (i) the search terms do not match the fields in the multiple data sources or (ii) the natural language search query does not contain the one or more analytical intents, pre-written content from the one or more data visualizations is displayed.

[0224] Clause A2. The method described in Clause A1 further includes:

[0225] The search terms are obtained using a federated query search, which distributes the query to multiple search repositories and combines the results into a single merged search result.

[0226] Clause A3. The method according to any one of Clauses A1 to A2, wherein the one or more analytical intents are selected from the group consisting of: grouping, aggregation, association, filtering and limiting, time and geospatial.

[0227] Clause A4. The method according to any one of Clauses A1 to A3, wherein parsing the search terms further comprises: identifying data fields and data values ​​together with the one or more analytical intents based on the plurality of data sources and the metadata of the plurality of data sources.

[0228] Clause A5. The method according to Clause A4, wherein identifying data fields and data values ​​includes: comparing an N-gram corresponding to the search term with available data fields for syntactic similarity and semantic similarity.

[0229] Clause A6. The method described in Clause A5, wherein the syntactic similarity is identified using Levenshtein distance and the semantic similarity is identified using Wu-Palmer similarity score.

[0230] Clause A7. The method according to any one of Clauses A1 to A6, wherein the semantic search includes:

[0231] Indexing the metadata of each of the plurality of data repositories to obtain an index; and

[0232] A federated search is performed based on the index to determine whether the search terms match fields in one or more of the plurality of data sources.

[0233] Clause A8. The method described in Clause A7, wherein the indexing comprises:

[0234] For each data repository and visualization context, along with associated metadata, each file is represented as a corresponding document vector; and

[0235] N-gram strings from the document vectors are stored to support partial and exact matching.

[0236] Clause A9. The method described in Clause A7, wherein performing the federal search comprises:

[0237] Obtain the query vector corresponding to the search term;

[0238] The query vector is encoded into query string terms using an encoder used to generate the index; and

[0239] Based on the amount of overlap between the query string terms and the document string terms of the document vector, a predetermined number of candidate document vectors are selected from the document vector for each data repository and visualization context with associated metadata.

[0240] Clause A10. The method described in Clause A9 further includes:

[0241] The predetermined number of candidate document vectors are ranked using a scoring function that scores each document based on the search terms appearing in each document, regardless of the proximity of the search terms within the document.

[0242] Clause A11. The method according to any one of Clauses A1 to A10 further comprises:

[0243] The one or more visual responses are generated and displayed based on the data fields, data values, and the analytical intent in the natural language search query.

[0244] Clause A12. The method according to any one of Clauses A1 to A11 further comprises:

[0245] Based on (i) the semantic search returns a matching data source for the natural language query and (ii) the search terms are not parsed into valid data fields and data values ​​within the data source, a suggested query is displayed for the data source.

[0246] Clause A13. The method described in Clause A12 further includes:

[0247] The suggested query is generated using a template-based approach, which combines data fields from the data source with data interest metrics.

[0248] Clause A14. The method according to any one of Clauses A1 to A13 further comprises:

[0249] The system uses three encoded channels (x, y, and color) and four marker types (bars, lines, dots, and geographic shapes) to generate and display one or more visualization responses, thereby supporting the dynamic generation of bar charts, line charts, scatter plots, and maps that cover a range of analytical intents.

[0250] Clause A15. The method according to any one of Clauses A1 to A14 further comprises:

[0251] The tag type of the one or more visual responses is determined based on the mapping between visual encoding and the data type of the data field.

[0252] Clause A16. The method according to any one of Clauses A1 to A15 further comprises:

[0253] Use one or more statistical computations and large language models to generate and display dynamic text summaries describing the one or more visual responses;

[0254] Provide the large language model with hints containing statistical descriptions, which are extracted from the one or more visual responses using a predefined set of heuristics; and

[0255] In response to providing the prompt, the dynamic text summary is received from the large language model.

[0256] Clause A17. The method described in Clause A16, wherein the hint corresponds to (i) the minimum / maximum and average values ​​for the bar chart, and (ii) the Pearson correlation coefficient for the scatter plot.

[0257] Clause A18. The method according to any one of Clauses A1 to A17 further includes, before receiving the natural language search query:

[0258] Receive the user's selection of a data source;

[0259] Presents a graphical user interface for analyzing data from the selected data source; and

[0260] Three search options are provided, including:

[0261] (i) Question-and-answer search used to interpret analytical intent within the selected data source;

[0262] (ii) Exploratory search for document-based information retrieval of indexed, visualized content from the selected data source; and

[0263] (iii) Design search using the visual metadata of the selected data source.

[0264] Clause B1. A computer system for visual analysis of datasets, comprising:

[0265] One or more processors; and

[0266] Memory;

[0267] The memory stores one or more programs configured to be executed by the one or more processors, and the one or more programs include instructions for any of the terms A1 to A18.

[0268] Clause C1. A non-transitory computer-readable storage medium storing one or more programs configured for execution by a computer system having a display, one or more processors, and memory, said one or more programs including instructions for any of Clauses A1 to A18.

Claims

1. A method for visual analysis of a dataset, comprising: In a computing system having one or more processors and memory configured for executing one or more programs by said one or more processors: Receive natural language search queries that point to multiple data repositories, including multiple data sources and one or more data visualizations; Parse the search terms corresponding to the natural language search query to determine whether the natural language search query contains one or more analytical intents; Use semantic search to determine whether the search terms match data fields in one or more of the plurality of data sources; Based on (i) matching the search terms with fields in the one or more data sources and (ii) the determination that the natural language search query contains one or more analytical intents, generate and display one or more visual responses; as well as Based on the determination that (i) the search terms do not match the fields in the multiple data sources or (ii) the natural language search query does not contain the one or more analytical intents, pre-written content from the one or more data visualizations is displayed.

2. The method according to claim 1, further comprising: The search terms are obtained using a federated query search, which distributes the query to multiple search repositories and combines the results into a single merged search result.

3. The method according to any one of claims 1 to 2, wherein the one or more analytical intentions are selected from the group consisting of: grouping, aggregation, association, filtering and limiting, time and geospatial.

4. The method according to any one of claims 1 to 2, wherein parsing the search terms further comprises: Data fields and data values, along with one or more analytical intents, are identified based on the plurality of data sources and their metadata.

5. The method of claim 4, wherein identifying the data field and the data value comprises: The N-gram corresponding to the search term is compared with the available data fields for syntactic and semantic similarity.

6. The method of claim 5, wherein the syntactic similarity is identified using Levenshtein distance and the semantic similarity is identified using Wu-Palmer similarity score.

7. The method according to any one of claims 1 to 2, wherein the semantic search comprises: Index the metadata of each of the plurality of data repositories to obtain an index; as well as A federated search is performed based on the index to determine whether the search terms match fields in one or more of the plurality of data sources.

8. The method of claim 7, wherein the indexing comprises: For each data repository and visualization context and associated metadata, each file is represented as a corresponding document vector; as well as N-gram strings from the document vectors are stored to support partial and exact matching.

9. The method of claim 7, wherein performing the federated search comprises: Obtain the query vector corresponding to the search term; The query vector is encoded into query string tokens using an encoder used to generate the index; as well as Based on the amount of overlap between the query string terms and the document string terms of the document vector, a predetermined number of candidate document vectors are selected from the document vector for each data repository and visualization context with associated metadata.

10. The method of claim 9, further comprising: The predetermined number of candidate document vectors are ranked using a scoring function that scores each document based on the search terms appearing in each document, regardless of the proximity of the search terms within the document.

11. The method according to any one of claims 1 to 2, further comprising: The one or more visual responses are generated and displayed based on the data fields, data values, and the analytical intent in the natural language search query.

12. The method according to any one of claims 1 to 2, further comprising: Based on (i) the semantic search returns a matching data source for the natural language query and (ii) the search terms are not parsed into valid data fields and data values ​​within the data source, a suggested query is displayed for the data source.

13. The method of claim 12, further comprising: The suggested query is generated using a template-based approach, which combines data fields from the data source with data interest metrics.

14. The method according to any one of claims 1 to 2, further comprising: The system uses three encoded channels (x, y, and color) and four marker types (bars, lines, dots, and geographic shapes) to generate and display one or more visualization responses, thereby supporting the dynamic generation of bar charts, line charts, scatter plots, and maps that cover a range of analytical intents.

15. The method according to any one of claims 1 to 2, further comprising: The tag type of the one or more visual responses is determined based on the mapping between visual encoding and the data type of the data field.

16. The method according to any one of claims 1 to 2, further comprising: Use one or more statistical computations and large language models to generate and display dynamic text summaries describing the one or more visual responses; The large language model is provided with hints containing statistical descriptions, which are extracted from the one or more visual responses using a predefined set of heuristics; as well as In response to providing the prompt, the dynamic text summary is received from the large language model.

17. The method of claim 16, wherein the hints correspond to (i) minimum / maximum and average values ​​for a bar chart, and (ii) Pearson correlation coefficients for a scatter plot.

18. The method according to any one of claims 1 to 2, further comprising, before receiving the natural language search query: Receive the user's selection of a data source; Presents a graphical user interface for analyzing data from the selected data source; as well as Three search options are provided, including: (i) Question-and-answer search used to interpret analytical intent within the selected data source; (ii) Exploratory search for document-based information retrieval of indexed, visualized content from the selected data source; and (iii) Design search using the visual metadata of the selected data source.

19. A computer system for visual analysis of datasets, comprising: One or more processors; as well as Memory; The memory stores one or more programs configured to be executed by the one or more processors, and the one or more programs include instructions for: Receive natural language search queries that point to multiple data repositories, including multiple data sources and one or more data visualizations; Parse the search terms corresponding to the natural language search query to determine whether the natural language search query contains one or more analytical intents; Use semantic search to determine whether the search terms match data fields in one or more of the plurality of data sources; Based on (i) matching the search terms with fields in the one or more data sources and (ii) the determination that the natural language search query contains one or more analytical intents, generate and display one or more visual responses; as well as Based on the determination that (i) the search terms do not match the fields in the multiple data sources or (ii) the natural language search query does not contain the one or more analytical intents, pre-written content from the one or more data visualizations is displayed.

20. A non-transitory computer-readable storage medium storing one or more programs configured for execution by a computer system having a display, one or more processors, and a memory, said one or more programs comprising instructions for: Receive natural language search queries that point to multiple data repositories, including multiple data sources and one or more data visualizations; Parse the search terms corresponding to the natural language search query to determine whether the natural language search query contains one or more analytical intents; Use semantic search to determine whether the search terms match data fields in one or more of the plurality of data sources; Based on (i) matching the search terms with fields in the one or more data sources and (ii) the determination that the natural language search query contains one or more analytical intents, generate and display one or more visual responses; as well as Based on the determination that (i) the search terms do not match the fields in the multiple data sources or (ii) the natural language search query does not contain the one or more analytical intents, pre-written content from the one or more data visualizations is displayed.

Images