Systems and methods of visualizing data associated with search queries and computing devices

Abstract

Systems and methods of visualizing data associated with a search query and computing devices. A method and system of visualizing data associated with a search query is provided. The method includes the steps of receiving a data set comprising a plurality of data items, each data item having associated rank information and similarity information indicating a similarity between the data item and one or more other data items; ordering the data set into a ranked list based on the rank information; masking successive portions of the ranked list based on a boundary parameter; iteratively clustering portions of the ranked list other than the masked successive portions based on the similarity information to produce a clustered ranked data list; and generating and displaying a visualization representative of the clustered ranked data list, the visualization comprising shaped objects representative of each cluster in the clustered ranked data list.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to automatically ranking hierarchical items and generating a visual representation of the ranked results, and more specifically, to systems and methods of hierarchically clustering hierarchical items for effective data exploration. BACKGROUND

[0002] In related art systems such as data search result displays and data visualization systems, hierarchical clustering is a method used to organize data items that are sometimes used. The hierarchical clustering in these related art systems can produce a tree structure that provides a summary that can be beneficial for users who need to understand and navigate large data sets. More specifically, in related art systems, a tree structure can be generated by progressively grouping similar items together, where leaf nodes represent original items and internal nodes (i.e., clusters) represent aggregated items. Related art methods are either agglomerative (bottom-up) or divisive (top-down). With this hierarchy, users of the related art systems can be able to navigate and summarize an entire data set, which is often large, in a multi-scale manner.

[0003] However, the hierarchical clustering algorithms of the related art do not differentiate data items, which creates a problem when the hierarchical clustering is used for sequential or hierarchical data such as search results. For example, in many real-world scenarios, data items can be presented in a certain order or hierarchy. For example, real-world scenarios involving data presented in a certain order or hierarchy can include retrieved search results, a recommended list of media to play, and algorithmic outputs with weights or probabilities (e.g., topic keywords in a corpus of text). In contrast, in the hierarchical clustering algorithms of the related art, all data items are typically treated equally and out of order (e.g., hierarchy or order is ignored).

[0004] While the hierarchical clustering algorithms of the related art can help users view a data set, ignoring hierarchy information can lead to inefficient exploration when the data is hierarchical and large. For example, due to the grouping criteria, a high-weighted item can be in a very deep layer of the tree, such that a user navigating the data set can miss this important (e.g., high-level) item either because the user stops expanding the tree before that layer or because it is effectively buried within the clustered tree. Some related art systems can attempt to address this problem by selecting the highest-weighted item as a representative in a visual interface, but this related art system cannot address the problem because when the high-level item is in a very deep layer, many internal nodes can be identical or similar. SUMMARY

[0005] Some aspects of the present application can include a method of visualizing data associated with a search query. The method includes the steps of receiving a data set comprising a plurality of data items, each data item having associated rank information and similarity information indicative of a similarity between the data item and one or more other data items; ordering the data set into a ranked list based on the rank information; masking a contiguous portion of the ranked list based on a boundary parameter; iteratively clustering portions of the ranked list other than the masked contiguous portion based on the similarity information to produce a clustered ranked data list; and generating and displaying a visualization representative of the clustered ranked data list, the visualization comprising a shaped object representative of each cluster in the clustered ranked data list.

[0006] Further aspects of the present application can include a non-transitory computer readable medium encoded with instructions for causing a computing device to perform a method of visualizing data associated with a search query. The method includes the steps of receiving a data set comprising a plurality of data items, each data item having associated rank information and similarity information indicative of a similarity between the data item and one or more other data items; ordering the data set into a ranked list based on the rank information; masking a contiguous portion of the ranked list based on a boundary parameter; iteratively clustering portions of the ranked list other than the masked contiguous portion based on the similarity information to produce a clustered ranked data list; and generating and displaying a visualization representative of the clustered ranked data list, the visualization comprising a shaped object representative of each cluster in the clustered ranked data list.

[0007] Still further aspects of the present application can include a computing device comprising a storage device configured to store a database of data items; and a processor communicatively coupled to the storage device. The processor can be configured to receive a search query and perform a method of visualizing data associated with the search query. The method includes the steps of receiving a data set comprising a plurality of data items, each data item having associated rank information and similarity information indicative of a similarity between the data item and one or more other data items; ordering the data set into a ranked list based on the rank information; masking a contiguous portion of the ranked list based on a boundary parameter; iteratively clustering portions of the ranked list other than the masked contiguous portion based on the similarity information to produce a clustered ranked data list; and generating and displaying a visualization representative of the clustered ranked data list, the visualization comprising a shaped object representative of each cluster in the clustered ranked data list.

[0008] Yet other aspects of the present application can include a computing device comprising: a database for storing data items; a means for receiving a search query; a means for receiving a data set comprising a plurality of data items, each data item having associated rank information and similarity information indicating a similarity between the data item and one or more other data items; a means for ordering the data set into a ranked list based on the rank information; a means for masking a contiguous portion of the ranked list based on a boundary parameter; a means for iteratively clustering portions of the ranked list other than the masked contiguous portion based on the similarity information to produce a clustered ranked data list; and a means for generating and displaying a visualization representative of the clustered ranked data list, the visualization comprising shaped objects representative of each cluster in the clustered ranked data list. BRIEF DESCRIPTION OF DRAWINGS

[0009] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0010] Figure 1 A flowchart illustrating a process of hierarchical clustering of ranked items and exporting a visualization of ranked items in accordance with an example implementation of the present application is illustrated.

[0011] Figure 2 A data tree produced by a hierarchical agglomerative clustering (HAC) algorithm in accordance with a comparative example is illustrated.

[0012] Figure 3 A data tree produced by a ranked hierarchical agglomerative clustering (RHAC) algorithm in accordance with an example implementation of the present application is illustrated.

[0013] Figure 4 A graphical plot illustrating an average path length for accessing the top N documents in a data tree in accordance with a comparative example showing related art is illustrated. Figure 2

[0014] Figure 5 A graphical plot illustrating an average path length for accessing the top N documents in a data tree in accordance with an example implementation of the present application is illustrated. Figure 3

[0015] Figure 6 is a rendering of a user interface produced using a process in accordance with an example implementation.

[0016] Figure 7 A flowchart illustrating a process of hierarchical clustering of ranked items and exporting a visualization of ranked items in accordance with other example implementations of the present application is illustrated.

[0017] Figure 8 ​​An example computing environment having an example computer device suitable for use with some example implementations of the present application is illustrated. DETAILED DESCRIPTION

[0018] The following detailed description provides more detail on the drawings and example implementations of the present application. Reference designators are included in the figures to facilitate the description of the drawings. The terms used throughout the specification are provided as examples only and are not intended to be limiting. For example, use of the term "automatic" can refer to fully automatic or semi-automatic implementations that involve user or operator control of some aspects of the implementation in accordance with the desired implementation of one of ordinary skill in the art practicing the implementations of the present application. Additionally, ordinal terms such as "first," "second," "third," etc. can be used in the description and claims merely for purposes of clarity, and are not to be construed as indicating any order or sequence unless expressly so stated. The actions or items can be reordered or performed concurrently or dynamically, without departing from the scope of the present application.

[0019] In the present application, the term computer readable medium can include local storage, cloud-based storage, remotely located servers, or any other storage that can be apparent to one of ordinary skill in the art.

[0020] As described above, the hierarchical clustering algorithms of the related art do not differentiate between data items, which can create problems when the hierarchical clustering is used for sequential or hierarchical data such as search results. Specifically, in the hierarchical clustering algorithms of the related art, all data items are typically treated equally and out of order (e.g., hierarchical or sequential information is ignored). While such hierarchical clustering algorithms of the related art can facilitate a user's review of a data set, ignoring hierarchical information can result in inefficient exploration when data is hierarchical and large. For example, due to grouping criteria, a high-weighted item can be located at a very deep level of the tree such that a user reviewing the data set can miss this important (e.g., high-level) item either because the user stopped expanding the tree before that level or because it was effectively buried within the clustered tree.

[0021] To address this situation, example implementations of the present application can include a novel method, referred to as a hierarchical hierarchical agglomerative clustering or RHAC, for hierarchically clustering data items while considering both similarity and rank when organizing structured data. These example implementations can enable high rank or high weight items to be placed higher in the hierarchy while preserving item clusters through similarity. For example, the example implementations can facilitate high rank or high weight items in the hierarchy without letting many similar items fall apart (e.g., can preserve the integrity of clusters associated with weighted items). If a user explores a data set according to a tree structure (e.g., with tree visualization techniques), this process can enable more important items to be seen earlier.

[0022] Example implementations of the present application can be used to analyze and rank search results, arrange media to be displayed to a user, organize documents in an electronic library of documents.

[0023] Figure 1 A flowchart of a process 100 to hierarchically cluster ranked items and export a visualization of the ranked items according to example implementations of the present application is illustrated. As described below, an aspect of some embodiments of the present application can involve first excluding the highest ranked items from the clustering, and then iteratively clustering the most relevant items that are not excluded. Each time through the iteration, a certain percentage of the excluded items are re-included in the clustering. By doing this, high ranked items will be included for merging consideration during the hierarchical clustering, and thus they can be placed higher in the hierarchy. This process can be performed by a computing device such as the computing device 805 in the computing environment 800 illustrated in Figure 8

[0024] As illustrated in Figure 1 The process 100 begins at 105 with the system receiving or identifying a data set containing ranked data information. The ranked data information can be search results produced by an Internet or web search engine, media files (e.g., songs, images, or videos) produced with or associated with a media viewing platform (e.g., a streaming media service, an image gallery, or other media viewing platform), documents or papers stored in an electronic library or document storage platform, and predictive identifications or probabilities associated with the results of a predictive algorithm such as an identification algorithm.

[0025] ​After receiving the dataset, at 110, the data items are ordered into a list based on the ranking information associated with each item in the dataset. For example, all of the data items in the dataset can be ranked based on their relevance to a search request provided by a user. The items determined to be most relevant to the search request have the highest rank, and the items determined to be least relevant to the search request have the lowest rank. For purposes of explanation herein, the list can be denoted as L, where |L| = N.

[0026] After ordering the data items into a list based on the ranking of the data items, at 115, the system can mask or exclude the ranked items from the beginning of the list (e.g., the top of the list or the items with the highest ranks) by an initial further analysis. The ranked items that are masked or excluded can fall within a specified boundary index at the top of the list. In other words, the system can mask the items from the beginning (top) of the list L to a boundary index b determined by a threshold t, where b = |t-N|; 0 < t < 1.

[0027] In some example implementations, the boundary index can be pre-defined by a system designer, system administrator, or user, or can be automatically determined by the system based on the received dataset and an expected or desired tree depth. Additionally, in some example implementations, the boundary index can be automatically dynamically determined by the system, or can be dynamically adjusted by a user as feedback or control over the resulting visualization. In such implementations, when the boundary index value is adjusted, the visualization associated with the results of the clustering can be updated. For example, a user can specify a new boundary index, and the resulting visualization will be updated by re-clustering using the new boundary index.

[0028] For example, a user can adjust the boundary index downward to decrease the impact of the ranking on the clustering, or adjust the boundary index upward to increase the impact of the ranking on the clustering. In the experimental implementation described herein, the boundary index was set to 40% (e.g., b = 40% or the top 40% of the ranked data was excluded or masked in the initial iteration). As discussed below with respect to 130, b will be gradually decreased from bo = 40 to b Final = 0 as all excluded items are gradually added back into the clustering process.

[0029] After the start or top items in the list have been shielded or excluded, at 120, the remaining unshielded items undergo an iteration of the hierarchical agglomerative clustering process. From the items in the list designated by L|b...N|, based on a similarity (or correlation) measure, the two closest items (or group of previously merged items) are merged. In some example implementations, the similarity can be based on one or more of author, topic, volume or section as a larger whole, or any other clustering similarity that can be apparent to one of ordinary skill in the art.

[0030] At 125, it is determined whether only a single item remains in the list L (e.g., the root of the tree). As can be apparent to one of ordinary skill in the art, when all items in the list have been included (unshielded) in the clustering process (e.g., b Final = 0), there can be only one item in the list (e.g., the root of the tree).

[0031] If there is more than one item in the list (NO at 125), the process 100 continues to 130 and the boundary index b is decreased or lowered by a specified amount or step s, such that the value b for any subsequent iteration t can be defined as b t = b t-1 - |s · N|, 0 < s < 1, where s is a step parameter. This lowers the boundary index b, so that more data items in the original list under consideration are merged during the clustering process.

[0032] In some example implementations, the step parameter can be pre-defined by a system designer, system administrator, or user, or can be determined automatically by the system based on the received data set and the expected or desired tree depth. Additionally, in some example implementations, the step parameter can be determined automatically by the system dynamically, or can be adjusted dynamically by a user as feedback or control over the resulting visualization. In such implementations, when the step parameter value is adjusted, the visualization associated with the results of the clustering can be updated. For example, a user can specify a new step parameter, and the resulting visualization will be updated by re-clustering using the new step parameter.

[0033] For example, a user can adjust the step parameter up to reduce the impact of the hierarchical ranking on the clustering, or adjust the step parameter down to increase the impact of the hierarchical ranking on the clustering. In the experimental implementation described herein, the step parameter was set to 1% (e.g., the lowest 1% of the ranked data, s = 1%, was included in the clustering or excluded or shielded during subsequent iterations).

[0034] Once the boundary index has been decreased, Figure 1The process 100 then returns to 115 and the items in the ranked list are masked based on the newly reduced boundary index b computed at 130. The process 100 repeats 115, 120, and 125 until only a single item remains in the ranked list L (e.g., the root of the tree) and b = 0 (e.g., the list L includes all the items).

[0035] Once the ranked list L has been reduced to a single item and all the items have been included in the list L (e.g., "Yes" at 125), the process 100 moves to 135 where a visualization of the ranked and clustered list L is exported or displayed to the user to allow for interactive exploration of the data. Example implementations of the visualization of the ranked and clustered list L are provided below in Figure 4

[0036] In some example implementations, the process 100 can also be configured to exit at 125 before reducing the ranked list L to a single item. For example, the early stop at 125 can be triggered by another or additional stopping criteria. For example, 125 can be stopped by reaching a desired number of clusters, so the cluster count determination can also be tested as part of the stopping condition for 125 (e.g., an "or" of the two conditions can be used). Other stopping conditions will be apparent to those of ordinary skill in the art.

[0037] After the visualization has been exported or displayed, in some example implementations, the process 100 can end. In other example implementations, the process 100 can be repeated if the user updates or changes the boundary index value, the step parameter, or both.

[0038] Figure 2 A data tree 200 produced by a hierarchical agglomerative clustering (HAC) algorithm according to a comparative example is illustrated. In addition, Figure 3 A data tree 300 produced by a ranked hierarchical agglomerative clustering (RHAC) algorithm according to an example implementation of the application is illustrated. Figure 2 and Figure 3 Results of a HAC algorithm similar to the related art and a RHAC algorithm according to an example implementation using a data set containing 100 documents are illustrated. The distance metric between the documents used for clustering to generate these graphs is cosine similarity based on a bags-of-words model. A single link was employed to compute cluster distances.

[0039] Figure 2 A comparison between the results of Figure 3 can illustrate the benefits of using a Figure 3 ​The RHAC algorithm of data tree 300 produces a more balanced tree. In addition, according to example implementations of the present application, items with higher ranks (e.g., more relevant to a query) are placed higher in the tree in data tree 300 produced by the RHAC algorithm. For example, nodes of ranks #0 and #1 appear on the sixth level in the RHAC, as highlighted by the ellipse 305 illustrated in Figure 3 Conversely, nodes of ranks #0 and #1 are annotated on the 18th level in the traditional HAC, as highlighted by the ellipse 205 illustrated in Figure 2

[0040] Because the two high-rank data items to be viewed are important, but users have to zoom in (18 levels) in reality to locate them using the hierarchy in Figure 2 compared to the much shallower level (6 levels) illustrated in Figure 3 , significantly reducing the chance of accessing them for visual exploration of data in real-world applications. This problem can become more severe for larger data sets.

[0041] Method Mean of item distance Standard of item distance Diameter HAC 0.960(0.04) 0.040(0.02) 0.979(0.05) RHAC 0.941(0.05) 0.030(0.02) 0.955(0.06)

[0042] Table 1: Comparison of coherency of clusters in HAC and RHAC. The number of each metric is shown with the mean (standard value).

[0043] In addition, the Applicant further investigated the quality of the clusters. For each cluster at each level of the tree containing a leaf node of an internal node (e.g., represented by the internal node), a number of metrics were computed about the cluster, including the mean of the pairwise distances between the member documents, the standard deviation of these distances, and the diameter of the cluster (i.e., the maximum distance). Table 1 (above) illustrates the statistical information for all clusters for these metrics.

[0044] One of ordinary skill in the art can assume that, with the introduction of the shielded highest rank elements, the effectiveness of the clustering is hampered, which naturally can be considered to reduce the overall coherency of the clustering process. However, as can be observed from Table 1, the RHAC algorithm of example implementations of the present application does not only produce comparable coherency values to the related HAC algorithm, but actually produces better coherency values (e.g., smaller values for the RHAC algorithm). This indicates that the RHAC does not degrade the quality of the clusters compared to the original HAC.

[0045] Figure 4 A graphical plot 400 is illustrated for the average path length for accessing the top N documents in the data tree of Figure 2 according to a comparative example that demonstrates related technology. Figure 5 A graphical plot 400 is illustrated for the average path length for accessing the top N documents in the data tree of Figure 3 ​The graph shows the average path length of the first N documents in the data tree (Figure 500).

[0046] Additionally, the number of accesses to the two resulting levels is also calculated. Figure 2 and Figure 3 The average path length of documents with a ranking less than N. For example... Figure 4 and Figure 5 For example, for N=25, the average path length for HAC (related technology algorithm) is 37.16, while the average path length for RHAC (example implementation of this application) is 23.68. This means that users can spend less effort to access the highest-level documents in the dataset.

[0047] in addition, Figure 4 and Figure 5 A comparison of this metric for each N is shown for both methods. Figure 4 and Figure 5 The comparison between the two shows that RHAC produces a much smaller average path length, making it more effective for visual exploration of hierarchical datasets.

[0048] Figure 6 This is a presentation of a user interface 600 generated using the processing described in the example implementation. The user interface 600 illustrates a multi-scale data tree diagram generated using a clustering hierarchy of hierarchical data generated by the RHAC algorithm, according to the example implementation of this application. It can be transmitted by a communication device or in conjunction with, for example, the following... Figure 8 The computing environment 800 illustrated herein is displayed on a display device associated with a computing device such as computing device 805. For example, user interface 600 may be displayed on a display such as a computer screen of a portable computer device or a touch screen.

[0049] As mentioned above Figure 1 The processing described in section 135 of the data, based on the results of hierarchical clustering, can produce, export, or display visualizations. For example, scalable treeplots can be used to create multi-scale visualizations.

[0050] In user interface 600, each rectangle or square (e.g., 605a, 605b, 605c) may correspond to a separate cluster generated by the RHAC algorithm. Additionally, in some example implementations, the size of each square or rectangle (e.g., 605a, 605b, 605c) may represent the size of the associated cluster. For example, a taller square or rectangle 605a may correspond to a cluster with more inner layers than the cluster associated with a shorter square or rectangle 605b. Similarly, a wider square or rectangle 605b may correspond to a cluster with wider layers (e.g., more parallel sub-clusters) than the cluster associated with a narrower square or rectangle 605c.

[0051] Additionally, in some example implementations, another aspect of the squares or rectangles of the UI 600 can correspond to a priority in the hierarchy (e.g., a cluster containing data items of a higher level). For example, the color of the squares or rectangles of the UI 600 can correspond to the priority in the hierarchy.

[0052] Figure 6 A tree map visualization of a collection of documents is illustrated. In example implementations, a summary of the documents at different levels in each cluster can be generated using text analysis techniques. For example, a user can want to explore these documents retrieved from a query of a database. The user can then utilize the text summaries and hierarchy to navigate through this large result set. Since the RHAC promotes the highest level documents in the search results, the user can get an overview of the data without losing the benefits brought by the search. Thus, the user is able to more easily visualize the highest ranked (e.g., most relevant to the search) items while also visualizing the largest clusters associated with a topic or document. Initially, only the documents up to a certain level in the hierarchy are shown in the visualization, and the user can gain a basic understanding of the data. Subsequently, the user can further explore one or more regions of interest through the data tree map with an interface such as 600 to zoom in on that portion of the hierarchy. Figure 6

[0053] Figure 7 A flowchart of a process 700 to hierarchically cluster and export a visualization of hierarchical items in accordance with other example implementations of the present application is illustrated. As described below, an aspect of some embodiments of the present application can involve first excluding some of the lowest level items from the clustering, and then iteratively clustering the most relevant items that are not excluded. Each time through the iteration, a certain percentage of the excluded items are re-included in the clustering. By doing so, the lower level items will be included for consideration during the hierarchical clustering, and thus they can be placed at a lower level in the hierarchy. The process can be performed by a computing device such as the computing device 805 in the computing environment 800 illustrated in Figure 8

[0054] As illustrated in Figure 7 The process 700 begins at 705 with the system receiving or identifying a data set containing hierarchical data information. The hierarchical data information can be search results produced by an Internet or web search engine, media files (e.g., songs, images, or videos) produced or associated with a media viewing platform (e.g., a streaming media service, an image gallery, or other media viewing platform), documents or papers stored in an electronic library or document storage platform, and predictive identifications or probabilities associated with the results of a predictive algorithm such as an identification algorithm. ​​

[0055] After receiving the dataset, at 710, the data items are ordered into a list based on the ranking information associated with each item in the dataset. For example, all of the data items in the dataset can be ranked based on their relevance to a search request provided by a user. The items determined to be most relevant to the search request have the highest rank, and the items determined to be least relevant to the search request have the lowest rank. For explanatory purposes herein, this list can be denoted as L, where |L| = N.

[0056] After ordering the data items into a list based on their ranking, at 715, the system can mask or exclude ranked items from the end of the list (e.g., the bottom of the list or the items with the lowest rank) by an initial further analysis. The ranked items that are masked or excluded can fall within a specified boundary index at the bottom of the list. In other words, the system can mask the items from the end (bottom) of the list L to a boundary index b determined by a threshold t, where b = |t-N|; 0 < t < 1.

[0057] In some example implementations, the boundary index can be pre-defined by a system designer, system administrator, or user, or can be automatically determined by the system based on the received dataset and an expected or desired tree depth. Additionally, in some example implementations, the boundary index can be automatically dynamically determined by the system, or can be dynamically adjusted by a user as feedback or control over the resulting visualization. In such implementations, when the boundary index value is adjusted, the visualization associated with the results of the clustering can be updated. For example, a user can specify a new boundary index, and the resulting visualization will be updated by re-clustering using the new boundary index.

[0058] For example, a user can adjust the boundary index downward to decrease the impact of ranking on clustering, or adjust the boundary index upward to increase the impact of ranking on clustering. In the experimental implementation described herein, the boundary index was set to 40% (e.g., b = 40% or the bottom 40% of ranked data was excluded or masked in the initial iteration). As discussed below with respect to 730, b will be gradually decreased from bo = 40 to b Final = 0 as all excluded items are gradually added back into the clustering process.

[0059] At 740, if any items have been newly exposed in the iteration process 700, the newly exposed items are clustered around any existing clusters based on similarity to one or more separate items.

[0060] At 720, one iteration of the hierarchical deterministic clustering process is performed on the exposed items. Among the items in the list L |1...b| as specified, the largest cluster (or group of previously merged items) is split or separated into two clusters of approximately equal maximum similarity (or relevance) based on a similarity (or relevance) metric. In some example implementations, the similarity can be based on one or more of author, topic, volume or section as a larger whole, or any other clustering similarity that can be apparent to one of ordinary skill in the art. Thus, the least relevant items are separated or split to define separate branches.

[0061] At 725, a determination is made as to whether there are any remaining clusters that can be split in the list L (e.g., whether there are other branches of the possible tree or leaves of the tree). As can be apparent to one of ordinary skill in the art, when all items in the list have been included (unmasked) in the clustering process (e.g., b Final = 0), there can be only one item in the list (e.g., the root of the tree).

[0062] If there are remaining clusters in the list that are suitable for splitting ("yes" at 725), the process 700 continues to 730 and the boundary index b is increased by a specified amount or step s, such that the value b for any subsequent iteration t can be defined as bt= bt-1+ |s · N|, 0 < s < 1, where s is a step parameter. This raises the boundary index b, so that more data items in the original list that were merged are considered during the clustering process.

[0063] In some example implementations, the step parameter can be pre-defined by a system designer, system administrator, or user, or can be determined automatically by the system based on the received data set and an expected or desired tree depth. Additionally, in some example implementations, the step parameter can be determined automatically by the system dynamically, or can be adjusted dynamically by a user as feedback or control over the resulting visualization. In such implementations, the visualization associated with the results of the clustering can be updated when the step parameter value is adjusted. For example, a user can specify a new step parameter, and the resulting visualization will be updated by re-clustering using the new step parameter.

[0064] For example, a user can adjust the step parameter up to reduce the impact of the hierarchical ranking on the clustering, or adjust the step parameter down to increase the impact of the hierarchical ranking on the clustering. In the experimental implementation described herein, the step parameter was set to 1% (e.g., the lowest 1% of the ranked data was included s = 1% or excluded or masked out during subsequent iterations of the clustering).

[0065] Once the boundary index has been increased, Figure 7processing 700 returns to 715 and the items in the hierarchical list are masked based on the newly incremented boundary index b computed at 730. The iteration of 715, 740, 720, and 725 continues until there are no other clusters eligible for partitioning in the hierarchical list L (e.g., other branches possible in the tree) and b = 0 (e.g., the list L includes all items).

[0066] Once the hierarchical list L has identified individual clusters that are not eligible for further partitioning and all items have been included in the list L (e.g., "No" at 725), the processing 700 moves to 735 where a visualization of the hierarchical and clustered list L is exported or displayed to the user to allow for interactive exploration of the data. An example implementation of the visualization of the hierarchical and clustered list L is provided in the above Figure 4

[0067] In some example implementations, the processing 700 can also be configured to exit at 725 before the hierarchical list L has identified individual clusters that are not eligible for further partitioning. For example, the early stopping in 725 can be triggered by another or additional stopping criteria. For example, 725 can be stopped by reaching a desired number of clusters, so the cluster count determination can also be tested as part of the stopping condition for 725 (e.g., an "or" of the two conditions can be used). Other stopping conditions will be apparent to those of ordinary skill in the art.

[0068] After the visualization has been exported or displayed, in some example implementations, the processing 700 can end. In other example implementations, the processing 700 can be repeated if the user updates or changes the boundary index value, the step parameter, or both.

[0069] Example Computing Environment

[0070] Figure 8 An example computing environment 800 is illustrated with an example computing device 805 suitable for use in some example implementations. The computing device 805 in the computing environment 800 can include one or more processing units, cores, or processors 810, memory 815 (e.g., RAM, ROM, etc.), internal storage 820 (e.g., magnetic, optical, solid-state, and / or organic), and / or I / O interfaces 825, either of which can be coupled to a communication mechanism or bus 830 for communicating information, or built into the computing device 805.

[0071] ​The computing device 805 can be communicatively coupled to the input / user interface 835 and the output device / interface 840. Either or both of the input / user interface 835 and the output device / interface 840 can be wired or wireless interfaces and can be detachable. The input / user interface 835 can include any device, component, sensor, or physical or virtual interface that can be used to provide input (e.g., buttons, touch screen interfaces, keyboards, pointing / cursor controls, microphones, cameras, Braille, motion sensors, optical readers, etc.).

[0072] The output device / interface 840 can include a display, television, monitor, printer, speaker, Braille, etc. In some example implementations, the input / user interface 835 (e.g., a user interface) and the output device / interface 840 can be built into the computing device 805 or physically coupled to the computing device 805. In other example implementations, other computing devices can be used as or provide the functionality of the input / user interface 835 and the output device / interface 840 for the computing device 805. These elements can include, but are not limited to, well-known AR hardware inputs to permit a user to interact with an AR environment.

[0073] Examples of the computing device 805 can include, but are not limited to, highly mobile devices (e.g., smartphones, devices in vehicles and other machines, devices carried by people and animals, etc.), mobile devices (e.g., tablet computers, notebook computers, portable computers, personal computers, portable televisions, radios, etc.), and devices that are not designed for mobility (e.g., desktop computers, server devices, other computers, kiosks, televisions, radios, etc. that have one or more processors built in and / or coupled to them).

[0074] The computing device 805 can be communicatively coupled (e.g., via the I / O interface 825) to external memory 845 and a network 850 for communication with any number of networked components, devices, and systems, including one or more computing devices of the same or different configurations. The computing device 805 or any connected computing device can function as, provide services of, or be referred to as a server, a client, a thin server, a general-purpose machine, a special-purpose machine, or another label.

[0075] The I / O interface 825 can include, without limitation, wired and / or wireless interfaces that use any communication or I / O protocol or standard (e.g., Ethernet, 802.11 xs, universal system bus, WiMAX, modem, cellular network protocols, etc.) to communicate information with at least all connected components, devices, and networks in the computing environment 800. The network 850 can be any network or combination of networks (e.g., the Internet, local area networks, wide area networks, telephone networks, cellular networks, satellite networks, etc.).

[0076] The computing device 805 can use and / or communicate using computer-usable or computer-readable media, including transitory media and non-transitory media. Transitory media includes transmission media (e.g., metal cables, fiber optics), signals, carrier waves, and the like. Non-transitory media includes magnetic media (e.g., disks and tapes), optical media (e.g., CD ROM, digital video disks, Blu-ray disks), solid state media (e.g., RAM, ROM, flash, solid state storage), and other non-volatile storage or memory.

[0077] The computing device 805 can be used to implement techniques, methods, applications, processes, or computer-executable instructions in some example computing environments. Computer-executable instructions can be retrieved from transitory media and stored on and retrieved from non-transitory media. The executable instructions can be in one or more programming, scripting, and machine languages (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0078] The processor 810 can execute under any operating system (OS) (not shown), in native or virtual environments. One or more applications can be deployed, including the logic unit 855, the application programming interface (API) unit 860, the input unit 865, the output unit 870, the hierarchical unit 875, the masking unit 880, the clustering unit 885, and the visualization unit 890, and an inter-unit communication mechanism 895 to facilitate communication of the different units with each other, with the OS, and with other applications (not shown).

[0079] For example, the hierarchical unit 875, the masking unit 880, the clustering unit 885, and the visualization unit 890 can implement one or more processes illustrated in Figure 1 and Figure 7 The described units and elements can vary in design, function, configuration, or implementation, and are not limited to the provided descriptions.

[0080] In some example implementations, when the API unit 860 receives information or executes instructions, the information or executed instructions can be communicated to one or more other units (e.g., the ranking unit 875, the masking unit 880, the clustering unit 885, and the visualization unit 890). For example, the ranking unit 875 can rank data items received via the network or the input unit to generate a portion of a ranked list. Additionally, the masking unit 880 can mask or hide a portion of the ranked list, and the clustering unit 885 can iteratively cluster the exposed portion of the ranked list. Additionally, the visualization unit 890 can generate a visualization representative of the clustering of the ranked list, and export the visualization via the output unit 870.

[0081] In some cases, in some example implementations described above, the logic unit 855 can be configured to control the flow of information between units and direct the services provided by the API unit 860, the input unit 865, the ranking unit 875, the masking unit 880, the clustering unit 885, and the visualization unit 890. For example, the flow of one or more processes or implementations can be controlled by the logic unit 855, alone or in combination with the API unit 860.

[0082] While some example implementations have been illustrated and described, these example implementations are provided as examples to convey the subject matter described herein to those familiarized with the art. It should be understood that the subject matter described herein can be implemented in various forms without being limited to the example implementations described. The subject matter described herein can be practiced without those specifically limited or described elements or without other or different elements or subject matter described. Those familiar with the art should appreciate that changes can be made to these example implementations without departing from the subject matter described herein as defined by the appended claims and their equivalents.

Claims

1. A method of visualizing data associated with a search query, the method comprising the steps of: receiving, by a computing device, a data set comprising a plurality of data items, each data item having relevance information indicative of a relevance of the data item to the search query and similarity information indicative of a similarity between the data item and one or more other data items from the plurality of data items; ordering, by the computing device, the data set into a hierarchical list based on the relevance information associated with each of the plurality of data items; masking, by the computing device, a contiguous portion of the hierarchical list based on a boundary parameter; iteratively clustering, by the computing device, a portion of the hierarchical list other than the masked contiguous portion based on the similarity information associated with the plurality of data items to produce a clustered hierarchical data list; and generating and displaying a visualization representative of the clustered hierarchical data list, the visualization comprising shaped objects representative of each cluster in the clustered hierarchical data list, wherein the boundary parameter is a user-adjustable control variable, and the step of masking, by the computing device, a contiguous portion of the hierarchical list based on a boundary parameter comprises masking a contiguous portion extending from a highest ranked data item of the hierarchical list to a lower ranked data item calculated based on the boundary parameter or a contiguous portion extending from a lowest ranked data item of the hierarchical list to a higher ranked data item calculated based on the boundary parameter. The step of iteratively clustering a portion of the hierarchical list other than the masked contiguous portion comprises, for each successive iteration:

2. The method of claim 1, wherein, changing a value of the boundary parameter by a step parameter; exposing a portion of the previously masked contiguous portion of the hierarchical list based on the changed value of the boundary parameter; adding the exposed portion to the portion of the hierarchical list other than the masked contiguous portion; and clustering the combination of the exposed portion and the portion of the hierarchical list other than the masked contiguous portion. The step parameter is a user-adjustable control variable.

4. The method of claim 1, further comprising the steps of:

3. The method of claim 2, wherein, receiving user instructions to define new values for one or more of the boundary parameter and step parameter as the user-adjustable control variables, wherein the step parameter is used to change the value of the boundary parameter; exposing the entire hierarchical list; masking, by the computing device, a new contiguous portion of the hierarchical list; iteratively clustering, by the computing device, a new portion of the hierarchical list other than the masked new contiguous portion based on the similarity information associated with the plurality of data items according to the new values defined for one or more of the boundary parameter and the step parameter to produce a clustered hierarchical data list; and generating and displaying a new visualization representative of the clustered hierarchical data list, the new visualization comprising shaped objects representative of each cluster in the clustered hierarchical data list.

5. The method of claim 1, wherein the step of iteratively clustering a portion of the hierarchical list other than the masked contiguous portion comprises, for each successive iteration: changing a value of the boundary parameter by a step parameter; exposing a portion of the previously masked contiguous portion of the hierarchical list based on the changed value of the boundary parameter; adding the exposed portion to the portion of the hierarchical list other than the masked contiguous portion; and clustering the combination of the exposed portion and the portion of the hierarchical list other than the masked contiguous portion. The step parameter is a user-adjustable control variable.

6. The method of claim 1, wherein the step of iteratively clustering a portion of the hierarchical list other than the masked contiguous portion comprises, for each successive iteration: changing a value of the boundary parameter by a step parameter; exposing a portion of the previously masked contiguous portion of the hierarchical list based on the changed value of the boundary parameter; adding the exposed portion to the portion of the hierarchical list other than the masked contiguous portion; and clustering the combination of the exposed portion and the portion of the hierarchical list other than the masked contiguous portion. The step parameter is a user-adjustable control variable.

5. The method of claim 2, wherein, The step of generating the visualization includes the steps of: defining a size of each shaped object to represent a size of a cluster of the hierarchical data list of clusters associated with each shaped object of the clusters; and defining a color of each shaped object to represent a highest ranked item in the cluster of the hierarchical data list of clusters associated with each shaped object.

6. A non-transitory computer readable medium encoded with instructions for causing a computing device to perform a method of visualizing data associated with a search query, the method comprising the steps of: receiving, by a computing device, a data set comprising a plurality of data items, each data item having associated rank information indicative of a rank of the data item relative to the search query and similarity information indicative of a similarity of the data item to one or more other data items from the plurality of data items; ordering, by the computing device, the data set into a hierarchical list based on the rank information associated with each of the plurality of data items; masking, by the computing device, a contiguous portion of the hierarchical list based on a boundary parameter; iteratively clustering, by the computing device, a portion of the hierarchical list other than the masked contiguous portion based on the similarity information associated with the plurality of data items to produce a clustered hierarchical data list; and 7. The non-transitory computer readable medium of claim 6, wherein, generating and displaying a visualization representative of the clustered hierarchical data list, the visualization comprising a shaped object representative of each cluster in the clustered hierarchical data list, wherein the boundary parameter is a user-adjustable control variable and the step of masking, by the computing device, a contiguous portion of the hierarchical list based on a boundary parameter comprises masking a contiguous portion extending from a highest ranked data item of the hierarchical list to a lower ranked data item calculated based on the boundary parameter or a contiguous portion extending from a lowest ranked data item of the hierarchical list to a higher ranked data item calculated based on the boundary parameter. The iteratively clustering a portion of the hierarchical list other than the masked contiguous portion comprises, for each successive iteration: changing a value of the boundary parameter by a step parameter; exposing a portion of the previously masked contiguous portion of the hierarchical list based on the changed value of the boundary parameter; adding the exposed portion to the portion of the hierarchical list other than the masked contiguous portion; and 8. The non-transitory computer readable medium of claim 7, wherein, clustering the combination of the exposed portion and the portion of the hierarchical list other than the masked contiguous portion.

9. The non-transitory computer readable medium of claim 6, wherein, The step parameter is a user-adjustable control variable. The method further comprises the steps of: receiving user instructions to define new values for one or more of the boundary parameter and step parameter as the user-adjustable control variables, wherein the step parameter is used to change the value of the boundary parameter; exposing the entire hierarchical list; masking, by the computing device, a new contiguous portion of the hierarchical list; iteratively clustering, by the computing device, a new portion of the ranked list other than the new contiguous portion that is masked based on the similarity information associated with the plurality of data items for new values defined for one or more of the boundary parameter and the step parameter to produce a clustered ranked data list; and generating and displaying a new visualization representative of the clustered ranked data list, the new visualization including shaped objects representative of each cluster in the clustered ranked data list.

10. The non-transitory computer readable medium of claim 7, wherein, generating the visualization includes defining a size of each shaped object to be representative of a size of the cluster of the clustered ranked data list associated with each shaped object; and defining a color of each shaped object to be representative of a highest ranked item in the cluster of the clustered ranked data list associated with each shaped object.

11. A computing device, comprising: a storage device configured to store a database of data items; and a processor communicatively coupled to the storage device, the processor configured to receive a search query and perform a method of visualizing data associated with the search query, the method comprising the steps of: receiving a data set including a plurality of data items from the database, each data item having associated rank information indicative of a data item's relevance to the search query and similarity information indicative of a similarity between the data item and one or more other data items from the plurality of data items; ordering, by the computing device, the data set into a ranked list based on the rank information associated with each of the plurality of data items; masking, by the computing device, a contiguous portion of the ranked list based on a boundary parameter; iteratively clustering, by the computing device, a portion of the ranked list other than the masked contiguous portion based on the similarity information associated with the plurality of data items to produce a clustered ranked data list; and generating and displaying a visualization representative of the clustered ranked data list, the visualization including shaped objects representative of each cluster in the clustered ranked data list, wherein the boundary parameter is a user-adjustable control variable, and the step of masking, by the computing device, a contiguous portion of the ranked list based on a boundary parameter includes masking a contiguous portion extending from a highest ranked data item of the ranked list to a lower ranked data item calculated based on the boundary parameter or a contiguous portion extending from a lowest ranked data item of the ranked list to a higher ranked data item calculated based on the boundary parameter.

12. The computing device of claim 11, wherein, iteratively clustering a portion of the ranked list other than the masked contiguous portion includes for each successive iteration: changing a value of the boundary parameter by a step parameter; exposing a portion of the previously masked contiguous portion of the ranked list based on the changed value of the boundary parameter; adding the exposed portion to the portion of the ranked list other than the masked contiguous portion; and clustering a combination of the exposed portion and the portion of the ranked list other than the masked contiguous portion. ​ 13. The computing device of claim 12, wherein, the step parameter is a user-adjustable control variable; and wherein the method further comprises the steps of: receiving user instructions to define new values for one or more of the boundary parameter and the step parameter as the user-adjustable control variable; exposing the entire hierarchical list; masking, by the computing device, a new contiguous portion of the hierarchical list; iteratively clustering, by the computing device, new portions of the hierarchical list, other than the masked new contiguous portion, based on the similarity information associated with the plurality of data items, according to the new values defined for one or more of the boundary parameter and the step parameter, to produce a clustered hierarchical data list; and generating and displaying a new visualization representative of the clustered hierarchical data list, the new visualization including shaped objects representative of each cluster in the clustered hierarchical data list.

14. The computing device of claim 12, wherein, generating the visualization includes defining a size of each shaped object to be representative of a size of a cluster associated with each shaped object in the clustered hierarchical data list; and defining a color of each shaped object to be representative of a highest ranked item in the cluster associated with each shaped object in the clustered hierarchical data list.

Images