Document rediscovery system

Abstract

The document rediscovery system generates embeddings of concept clips provided by a user, the concept clips defining concepts for searching for content of interest to the user within a plurality of documents previously viewed by the user. The rediscovery system determines semantic relationships between concepts and document clips related to respective documents of the plurality of documents based on the concept embeddings and embeddings generated for the document clips. A graphical user interface depicting the semantic relationships is rendered to the user and is operable to enable the user to rediscover documents of the plurality of documents having content of interest to the user.

Description

[Background technology]

[0001] background Searching for content within various electronic documents, such as web pages, electronic files, emails, etc., is a routine task performed by users of various applications. Sometimes searching for content includes rediscovering a document that the user encountered in the past. For example, a user may want to rediscover a web page that he or she previously searched for. As another example, a user may want to rediscover an email or document that he or she previously viewed. A typical search system allows a user to search for such documents by using keywords, with search results retrieved based on the keywords. However, rediscovering a document using keywords can sometimes be difficult because users often do not remember the content of the document well enough to provide accurate keywords to search for the document. Summary of the Invention

[0002] It is with respect to these and other general considerations that the aspects disclosed herein have been made. It should also be understood that, although relatively specific problems may be discussed, the examples should not be limited to solving any particular problem identified in the context of this disclosure or elsewhere.

[0003] overview According to examples of the present disclosure, a refiner system may be configured to assist a user in rediscovering documents having content of interest to the user. In an aspect, the refiner system may allow a user to define concepts that may include clips of documents, images, chunks of text paragraphs, etc. that may relate to the content of interest to the user. The refiner system may determine semantic relationships between the user-defined concepts and clips of documents previously viewed by the user, and may visually depict the semantic relationships to the user. For example, the refiner system may display a multi-dimensional canvas showing representations of document clips and representations of concepts, where the representations of document clips are positioned relative to the positions of the representations of concepts according to the semantic relationships between the corresponding document clips and the corresponding concepts. Thus, a user may look through the document clips based on the semantic relationships between the document clips and the concepts, for example, to find document clips that are semantically similar and / or semantically dissimilar to each of the concepts. In some aspects, the refiner system may provide an interactive graphical user interface that allows a user to interactively reposition the representations of concepts within the graphical user interface, and may update the positions of the representations of document clips based on changes in the positions of the representations of concepts within the graphical user interface. Thus, when a user interactively rearranges a representation of a concept within a graphical user interface, representations of document clips that are relatively semantically similar to the concept may be visually moved closer to the representation of the concept within the graphical user interface, and representations of document clips that are relatively semantically dissimilar to the concept may be visually moved farther away from the representation of the concept. Thus, a user may rearrange the representation of a concept within a graphical user interface to organize the document clips according to document content that may be conceptually interesting to the user. These and other techniques described herein may enable a user to rediscover documents that are more difficult or impossible to rediscover using traditional rediscovery techniques, such as using traditional keyword searches, to search for documents having content that interests the user.

[0004] In an aspect, a method is provided for assisting a user in rediscovering a document. The method includes generating embeddings of document clips related to each document of a plurality of documents. The method also includes receiving a first set of concept clips defining a first concept for searching for content of interest to a user within the plurality of documents. In addition, the method includes generating embeddings of each concept clip in the first set of concept clips, and generating a first concept embedding based on a combination of the embeddings generated for each concept clip in the first set of concept clips. The method further includes determining a semantic relationship between the first concept and the document clip based on i) the embeddings generated for the document clips, and ii) the concept embeddings, and generating a graphical user interface depicting the semantic relationship between the first concept and the document clip. In addition, the method includes causing a display of the graphical user interface to be rendered at a client device, the graphical user interface being operable to enable the user to rediscover a document having content of interest to the plurality of documents.

[0005] In a further aspect, a system is provided. A system for assisting a user in rediscovering documents is provided. The system includes one or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media that, when executed by at least one processor, cause the at least one processor to perform operations. The operations include using a machine learning model to generate embeddings of document clips related to each document of the plurality of documents. The operations also include receiving a first set of concept clips that define a first concept for searching for content of interest to a user within the plurality of documents. In addition, the operations include using a machine learning model to generate embeddings of each concept clip in the first set of concept clips, and generating a first concept embedding based on a combination of the embeddings generated for each concept clip in the first set of concept clips. The operations further include determining a semantic relationship between the first concept and the document clip based on i) the embeddings generated for the document clips, and ii) the concept embeddings, and generating a graphical user interface that depicts the semantic relationship between the first concept and the document clip. Additionally, the operations include causing a display of a graphical user interface to be rendered at the client device, the graphical user interface operable to enable a user to rediscover documents among the plurality of documents that have content of interest to the user.

[0006] In yet a further aspect, a computer storage medium is provided. The computer storage medium includes computer-executable instructions that, when executed by at least one processor, perform a method. The method includes generating embeddings of document clips related to each document of the plurality of documents. The method also includes receiving a first set of concept clips defining a first concept for searching for content of interest to a user within the plurality of documents. The method further includes generating embeddings of each concept clip in the first set of concept clips and generating a first concept embedding based on a combination of the embeddings generated for each concept clip in the first set of concept clips. The method still further includes determining a semantic relationship between the first concept and the document clip based on i) the embeddings generated for the document clips and ii) the concept embeddings, and generating a graphical user interface depicting the semantic relationship between the first concept and the document clip. In addition, the method includes causing a display of the graphical user interface to be rendered at the client device, the graphical user interface being operable to enable a user to rediscover a document of the plurality of documents having content of interest to the user.

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features and / or advantages of examples will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practice of the disclosure.

[0008] BRIEF DESCRIPTION OF THE DRAWINGS Non-limiting and non-exhaustive examples are described with respect to the following figures. [Brief description of the drawings]

[0009] [Figure 1]1 illustrates an overview of an example system in which rediscovery techniques may be utilized, according to aspects of the present disclosure. [Diagram 2] 1 illustrates an example of a graphical user interface for providing user annotations to a web page, according to an aspect of the present disclosure. [Diagram 3] 1 illustrates an example of a graphical user interface of a refiner application, according to an aspect of the present disclosure. [Figure 4] 1 illustrates an example of a Concepts panel of a graphical user interface for creating concepts, according to an aspect of the present disclosure. [Diagram 5] 1 illustrates an example of a Concepts panel of a graphical user interface for viewing and / or manipulating concepts, according to an aspect of the present disclosure. [Figure 6] 1 illustrates an example of a finder panel that may be provided as a detail panel of a graphical user interface in accordance with an aspect of the present disclosure. [Figure 7] 1 illustrates an example of a selected clip panel that may be provided as a details panel of a graphical user interface in accordance with aspects of the present disclosure. [Figure 8] 1 illustrates an example of a finder panel that may be provided as a detail panel of a graphical user interface in accordance with an aspect of the present disclosure. [Figure 9] 10 details a method for assisting a user in rediscovering a document according to an aspect of the present disclosure. [Figure 10] FIG. 1 is a block diagram illustrating the physical components (e.g., hardware) of a computing device in which aspects of the present disclosure may be practiced. [Figure 11A] 1 illustrates a mobile computing device in which aspects of the present disclosure may be practiced. [Figure 11B] 1 illustrates a mobile computing device in which aspects of the present disclosure may be practiced. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Detailed Description In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and which show by way of illustration specific aspects or examples. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the disclosure. The aspects disclosed herein may be embodied as a method, system, or device. Thus, the aspects disclosed herein may take the form of a hardware implementation, an entirely software implementation, or an implementation combining software and hardware aspects. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the disclosure is defined by the appended claims and their equivalents.

[0011] According to examples of the present disclosure, a refiner system may be configured to assist a user in rediscovering documents having content of interest to the user. For example, a refiner system may be configured to assist a user in rediscovering web pages having particular content that the user viewed in the past. For ease of explanation, the refiner system is generally described herein in the context of rediscovering web pages that the user viewed in the past, however, the refiner system may additionally or alternatively be configured to assist a user in rediscovering documents other than web pages that the user viewed in the past. For example, a refiner system may be configured to assist a user in rediscovering emails, chat messages, documents stored in file repositories (e.g., documents stored in a computer's file system and / or on a cloud) that the user encountered in the past, and the like.

[0012] In an aspect, the refiner system may allow a user to define a concept, which may include clips of portions of documents, images, text paragraphs, etc., that may relate to content of interest to the user. The refiner system may determine a semantic relationship between the user-defined concept and clips of documents previously viewed by the user, such as web pages in the user's browsing history. In an aspect, the refiner system may generate semantic embeddings of the user-defined concept and clips of documents previously viewed by the user (e.g., using a machine learning model such as a Transformer model, although one skilled in the art would understand that other types of models may be used), and may determine a semantic relationship between the concept and the document clips based on a calculated distance between the semantic embedding generated for the concept and the semantic embedding generated for the clips of documents previously viewed by the user. The refiner system may then display the relationship between the concept and the document clip to the user. For example, the refiner system may provide a graphical user interface that displays a visual representation of the document clip and a visual representation of the concept, where the visual representation of the document clip is positioned relative to the visual representation of the concept based on the semantic relationship between the document clip and the concept. Thus, a user may visually inspect documents based on semantic similarities and / or dissimilarities between document clips and concepts to potentially rediscover documents of interest to the user. In some aspects, the refiner system may enable a user to interactively reposition a representation of a concept within the graphical user interface and may dynamically update the position of the representation of the document clip based on a change in the position of the representation of the concept within the graphical user interface. Thus, when a user interactively repositions a representation of a concept within the graphical user interface, a representation of a document clip that is relatively semantically similar to the concept may be visually moved closer to the representation of the concept within the graphical user interface, and a representation of a document clip that is relatively semantically dissimilar to the concept may be visually moved away from the representation of the concept within the graphical user interface.Thus, a user can rearrange the representation of concepts within a graphical user interface to organize document clips according to document content that may be conceptually interesting to the user. These and other techniques described herein may enable a user to rediscover documents that are more difficult or impossible to rediscover using traditional rediscovery techniques, such as using traditional keyword searches, to search for documents having content that interests the user.

[0013] FIG. 1 illustrates an overview of an example system 100 in which the rediscovery techniques described herein may be utilized, according to aspects of the present disclosure. The system 100 may include a user device 102 that may be configured to run or otherwise execute a refiner application 104. Examples of the user device 102 include, but are not limited to, a laptop, a tablet, a smartphone, and the like. The refiner application 104 may be a client application that may be configured to assist a user 110 of the user device 102 in rediscovering electronic documents that the user 110 may have viewed in the past. As an example, the refiner application 104 may assist the user 110 in rediscovering web pages that the user 110 viewed in the past. In other examples, the refiner application 104 may additionally or instead assist the user 110 in rediscovering other types of documents, such as electronic file documents, emails, chats, and the like. The refiner application 104 may be a client application that may be provided as an extension to a browsing application, such as, for example, a web browsing application or a file browsing application. In some aspects, the refiner application 104 may be a client application provided as an extension to another type of application for which searching for specific content may be necessary or desirable. For example, the refiner application 104 may be provided as part of or as an extension to an email application, a chat application, a document repository application, and the like. In other aspects, the refiner application 104 may be a stand-alone application. In some examples, the refiner application 104 may be a web application, such as an application 104 that runs within a web browser or is otherwise capable of executing instructions. In some examples, the application 104 may additionally or alternatively include a native client application residing on the user device 102.

[0014] The user device 102 may be communicatively coupled to the application server 106 via a communications network 108. The communications network 108 may be a wide area network (WAN) such as the Internet, a local area network (LAN), or any other suitable type of network. The communications network 108 may be a single network or may be comprised of multiple different networks, in some examples. The system 100 may also include a database 112, which may be communicatively coupled to the application server 106 and / or the user device 102 via the communications network 108. In other aspects, the database 112 may be coupled to the application server 106 and / or the user device 102 in other suitable manners. For example, the database 112 may be directly connected to the application server 106 or may be included as part of the application server 106, in some examples. The database 112 may be a single database or may include multiple different databases.

[0015] In aspects, the refiner application 104 may enable the user 110 to rediscover documents (e.g., web pages) that the user 110 has previously encountered, for example by accessing the documents on the user device 102 or another device, based on semantic relationships between document clips related to the content of the documents that the user 110 has previously encountered and concepts that the user 110 may define or otherwise provide. In this manner, the refiner application 104 may enable the user 110 to rediscover documents without having to remember the exact content of the documents and without having to provide exact keywords to search for the documents. For ease of explanation, the refiner application 104 is generally described herein in the context of rediscovering web pages that the user has previously encountered, although the refiner application 104 may additionally or alternatively be configured to assist the user in rediscovering documents other than web pages that the user has previously encountered. For example, the refiner application may be configured to assist the user in rediscovering emails, chat messages, documents stored in file repositories (e.g., documents stored in a computer's file system and / or on a cloud) that the user has previously encountered, and the like.

[0016] In an aspect, to enable user 110 to later rediscover web pages that user 110 has encountered, refiner application 104 may be configured to generate document clips 114 related to the content of web pages as user 110 encounters the web pages. In an aspect, refiner application 104 may include a document chunking module 116 configured to generate document clips 114 for web pages encountered by user 110. The document clips 114 generated for web pages may include, for example, document chunks that include text (e.g., text paragraphs) that document chunking module 116 can parse from the web pages. In some aspects, document chunking module 116 may discard some document chunks extracted from the web pages. For example, document chunking module 116 may discard duplicate chunks that include identical content extracted from the web pages. As another example, the document chunking module can discard document chunks that are too small, such as document chunks having a number of characters or words below a certain (e.g., predetermined) low threshold, and / or document chunks that are too large, such as document chunks having a number of characters or words above a certain (e.g., predetermined) high threshold. In addition to or instead of the text parsed from the web page, the document clip 114 generated for the web page can include information other than the text parsed from the web page. For example, the document clip 114 can include images extracted from the web page. As another example, the document clip 114 can include information related to the structure or feel of the web page. For example, the document clip 114 can include metadata information, such as information indicating the length of the web page, information indicating colors (e.g., background colors) used on the web page, information indicating font types, colors, sizes, etc. used on the web page, information indicating the number of sections or portions (e.g., tabs, etc.) on the web page, etc.

[0017] In some aspects, the document clip 114 may include additional information not obtained from the web page. For example, the refiner application 104 may enable the user 110 to provide (e.g., enter into the graphical user interface 118 of the refiner application 104) additional information, such as notes or other annotations, about the web page when the user 110 encounters the web page. Thus, the user 110 may add notes or other annotations that may help the user 110 to rediscover the web page at a later time, for example. In an example, the refiner application 104 may enable the user 110 to provide global annotations that describe the content and / or feel of the entire web page, and / or may enable the user 110 to provide local annotations that describe the content and / or feel of a particular section, paragraph, image, etc. on the web page. In an aspect, the document chunking module 116 may include such annotations provided by the user 110 in the document clip 114 generated for the web page. The document chunking module 116 may additionally or alternatively use annotations provided by the user 110 to prioritize portions (e.g., chunks) of the web page for inclusion in the document clip 114. For example, the user 110 may explicitly indicate the importance of a particular portion of the web page when providing annotations for the particular portion of the web page, and / or the document chunking module 116 may implicitly determine the importance of a particular portion of the web page based on annotations provided by the user 110 for the particular portion of the web page. The document chunking module 116 may then generate the document clip 114 to include relatively important portions of the web page and exclude relatively unimportant portions of the web page, such as portions that may not provide meaningful semantic information about the web page.

[0018] The refiner application 104 can provide the document clips 114 generated by the document chunking module 116 to the server 106 over the network 108. The server 106 can receive the document clips 114 and provide the document clips 114 to a data service application 124 that may be running on the server 106 or elsewhere. The data service application 124 can include a document embedding module 126, a concept embedding module 128, and a concept search module 130. The document embedding module 126 can be configured to generate embeddings of the document clips 114 to capture semantics within the document clips 114. In an aspect, the document embedding module 126 can utilize an embedding model 132 to generate embeddings of the document clips 114. In an aspect, the embedding model 132 can include a machine learning model, such as a transformer model, trained to generate embeddings based on content, where the embeddings capture the semantics of the content. In some aspects, a multimodal transformer model can be utilized. A multimodal Transformer model is trained based on multiple modalities (e.g., web pages, file documents, email chats, etc.) to generate embeddings of the various modalities in the same embedding space. In other aspects, a Transformer model may be trained based on a single modality, such as, for example, a web page. In some aspects, a suitable machine learning model other than the Transformer model may be utilized.

[0019] Data service application 124 can store the embeddings generated for document clips 114 in database 112 such that the embeddings can later be used to determine semantic relationships between corresponding document clips and concepts that may later be provided by user 110. In an aspect, data service application 124 can store the embeddings generated for document clips 114 in database 112 as part of document metadata 125 that may be included, for example, with a browsing history associated with user 110. If user 110 subsequently wishes to rediscover a web page that user 110 encountered in the past, user 110 can use finder application 104 to define or otherwise provide one or more concepts 134 for searching for web pages that may, for example, be in a browsing history associated with user 110. Such concepts may assist user 110 in searching for a particular web page that may, for example, be in document repository 127 (e.g., in database 112 or in a repository separate from database 112). In one aspect, the user 110 may provide a definition of the concept 134 through a concept panel 136 that may be displayed within the graphical user interface 118. In an example, the concept 134 may include one or more concept clips that are conceptually related to the content of the web page that the user 110 wishes to rediscover. The one or more concept clips may include text that the user 110 may obtain from other documents, such as one or more text paragraphs that the user 110 may copy and paste into the concept panel 136 from one or more other web pages that the user 110 may encounter. The one or more concept clips may additionally or alternatively include one or more custom text paragraphs that the user 110 may enter into the concept panel 136, one or more images that the user 110 may provide via the concept panel 136, and the like. In some aspects, the user 110 may obtain the concept definition from a concept repository, such as a library that may store previously defined concepts, such as standard concepts and / or saved concepts previously defined by the user 110 and / or users other than the user 110.In some aspects, user 110 can customize concepts, for example using concept panel 136, to make such concepts more suitable for the search that user 110 is currently conducting. As described in more detail below, in various aspects, user 110 can provide a definition of only a single concept for conducting a search, or can provide individual definitions of multiple concepts for conducting a search. Also, as described in more detail below, user 110 can change or modify concepts and / or provide additional concepts to refine the search based on search results obtained based on the concepts that user 110 initially provided.

[0020] The refiner application 104 can provide the concept 134 to the data service application 106 over the network 108. The data service application 106 can receive the concept 134 and provide the concept 134 to the concept embedding module 128. The concept embedding module 128 can generate an embedding of the concept 134, where the embedding can capture the semantics of the concept clip contained within the concept 134. In an aspect, the concept embedding module 128 can generate an embedding of the concept 134 based on a combination of embeddings that the concept embedding module 128 can generate for each concept clip contained within the concept 134. The embedding of the concept clip can be generated by the concept embedding module 128 using a machine learning model, such as a transformer model, that can generate an embedding that captures the semantics in the concept clip. In one aspect, the concept embedding module 128 can utilize the embedding model 132 to generate an embedding of the concept clip contained within the concept 134. In other aspects, another embedding model can be utilized to generate an embedding of the concept clip contained within the concept 134.

[0021] In aspects, the concept embedding module 128 may generate an embedding of the concept 134 as a linear combination of embeddings generated for the concept clips contained within the concept 134. In some aspects, the concept embedding module 128 may generate a concept embedding as a weighted linear combination of embeddings generated for the concept clips contained within the concept, where embeddings generated for some concept clips may be weighted higher than embeddings generated for other concept clips. The weights used for the embedding may be provided along with the concept definition, for example, by the user 110. In other aspects, the concept embedding module 128 may be configured to determine the weights used for the embedding in other suitable manners. In some examples, the concept embedding module 128 may generate an embedding of the concept 134 as an unweighted or equal weighted linear combination of embeddings generated for the concept clips contained within the concept 134, and / or may generate an embedding of the concept 134 based on a non-linear combination of embeddings generated for the concept clips contained within the concept 134.

[0022] The concept embeddings generated by the concept embedding module 128 for the concept 134 may be provided to the concept search module 130. The concept search module 130 may be configured to obtain (e.g., retrieve from the database 112) embeddings generated by the document embedding module 126 for the document clips 114 related to web pages previously encountered by the user 110, and determine a semantic relationship between the concept 134 and the document clips 114 based on i) the respective embeddings generated by the document embedding module 126 for the document clips 114, and ii) the concept embeddings generated by the concept embedding module 128 for the concept 134. For example, the concept search module 130 may be configured to determine a semantic relationship between the concept 134 and the document clips 114 based on a calculated distance between i) the respective embeddings generated by the document embedding module 126 for the document clips 114, and ii) the concept embeddings generated by the concept embedding module 128 for the concept 134.

[0023] The concept search module 130 can provide an indication of the determined semantic relationship between the concept 134 and the document clip 114 to the refiner application 104. In some aspects, the concept search module 130 can generate a concept canvas for displaying the semantic relationship between the concept 134 and the document clip 114 and can provide a concept canvas display to the refiner application 104 for display to the user 110. The concept canvas display can include a multi-dimensional (e.g., two-dimensional) canvas or map on which a representation of the concept 134 and a representation of the document clip 114 can be laid out to display the semantic relationship between the concept 134 and the document clip 114. As described in more detail below, the concept search module 130 can generate a graph having nodes corresponding to the document clip 114 and the concept 134 and edges representing the semantic relationship between the document clip 114 and the concept 134. As also described in more detail below, the concept search module 130 may then use a graph-based, force-directed graph drawing algorithm to determine a layout for displaying the semantic relationships between the document clips 114 and the concepts 134, where nodes that have relatively stronger connections to each other in the graph are brought closer to each other with relatively greater force in the layout of the graph. In other aspects, the layout for displaying the semantic relationships between the document clips 114 and the concepts 134 may be determined in other suitable manners.

[0024] The refiner application 104 can render the concept canvas to the user 110 in a canvas panel 154 of the graphical user interface 118. In an aspect, the concept panel 154 can thus display representations of the document clips 114 and representations of the concepts 134, where the representations of the document clips 114 can be positioned with respect to the representations of the concepts 134 based on a semantic relationship between the document clips 114 and the concepts 134. Thus, for example, a representation of a document clip 114 that is semantically similar to the concept 134 can be positioned relatively closer in the canvas panel 154 to the representation of the concept 134 in the canvas panel 154 compared to a representation of a document clip 114 that is not semantically similar to the concept 134. The user 110 can then look through the document clips 114 according to their semantic similarity to the concept 134 to search for one or more document clips, e.g., a subset of the document clips 114, that indicate that the one or more document clips may be relevant to the web page the user 110 is attempting to rediscover. In some aspects, the graphical user interface 118 may enable the user 110 to click, hover over, or otherwise engage with a representation of the document clip 114 in the canvas panel 154 to view the contents of the document clip 114 and browse through the document clip 114. In other aspects, the graphical user interface 118 may enable the user 110 to browse through the document clip 114 in other suitable manners. In one aspect, to enable the user 110 to browse through a web page associated with the document clip 114, the graphical user interface 118 may display a web page (e.g., retrieved from the document repository 127) associated with the document clip 114 selected by the user 110.

[0025] In some aspects, the graphical user interface 118 may be operable to enable the user 110 to interactively rearrange the representations of the concepts 134 within the canvas panel 154. For example, the graphical user interface 118 may be operable to enable the user 110 to drag the representations of the concepts 134 to a new position within the canvas panel 154. As the user 110 rearranges the representations of the concepts 134 within the canvas panel 154, the refiner application 114 may dynamically update the positions of the representations of the document clips 114 within the canvas panel 154 based on the changing positions of the representations of the concepts 134. Thus, when the user 110 rearranges the representations of the concepts 134 within the concept panel 154, the representations of the document clips 114 that are relatively semantically similar to the concept 134 may be visually moved closer to the representation of the concept 134 within the concept panel 154, and the representations of the document clips that are relatively semantically dissimilar to the concept 134 may be visually moved farther away from the representation of the concept 134 within the concept panel 154. Thus, the user may rearrange the representation of the concepts 134 in the graphical user interface 118 to organize the document clips 114 according to content that may be conceptually interesting to the user. In some aspects, the refiner application 104 may enable the user 110 to modify the concepts 134 based on the search results displayed in the concept canvas 154. For example, the user 110 may modify the content of one or more concept clips included in the concept 134, delete such concept clips, and / or adjust their importance (e.g., weight), and / or provide one or more additional concept clips for inclusion in the concept 134. As another example, the user 110 may delete one or more concepts 134 entirely and / or add one or more additional concepts 134. The refiner application 104 may update the display in the canvas panel 154 based on the revised concept definitions provided by the user 110.These and other techniques described herein may enable a user 110 to rediscover documents that are more difficult or impossible to rediscover using traditional rediscovery techniques, such as using traditional keyword searches, to search for documents having content that is of interest to the user.

[0026] Continuing to refer to FIG. 1, while the document chunking module 116 is illustrated in FIG. 1 as being part of a refiner application running or otherwise running on the user device 102, it should be understood that in some aspects the document chunking module 116 may be at least partially executed on the server 106. Similarly, while the document embedding module 126, the concept embedding module 128, and the concept search module 130 are illustrated in FIG. 1 as being part of a data services application running or otherwise running on the server 106, it should be understood that in some aspects the document embedding module 126, the concept embedding module 128, and / or the concept search module 130 may be at least partially executed on the refiner application 104 of the user device 102. For example, the document embedding module 126 may operate at least in part to generate embeddings locally on the refiner application 104 running or otherwise running on the user device 102. Additionally, the concept embedding module 128 and / or the concept search module 130 may be at least partially resident on the refiner application running or otherwise running on the user device 102.

[0027] Figure 2 illustrates an example of a graphical user interface 200 configured to enable a user to provide document (e.g., web page) annotations in accordance with aspects of the present disclosure. Graphical user interface 200 may be utilized with system 100 of Figure 1, and graphical user interface 200 is described for illustrative purposes with respect to system 100 of Figure 1. In other aspects, graphical user interface 200 may be utilized with systems different than system 100 of Figure 1.

[0028] The graphical user interface 200 may be provided, for example, as an extension to an Internet search application, such as an Internet browser. In one aspect, the graphical user interface 200 may be displayed on the user device 102 of FIG. 1 when the user 110 visits a web page using a browser running or otherwise operational on the user device 102. The graphical user interface 200 may include a bar (e.g., a sidebar) 202 for providing notes or other annotations for a web page encountered by the user 110. The bar 202 may be displayed to the user 110 when the user 110 clicks or otherwise engages with an icon 204, which may be provided, for example, as an extension to a browser utilized by the user 102. The bar 202 may include an area (e.g., a box) 206 for entering custom information, such as notes or other annotations, that the user 110 desires to add regarding the web page to facilitate subsequent rediscovery of the web page. In one aspect, such annotations may be included within a document clip 114 associated with the web page. For example, the document chunking module 116 may include notes or other annotations provided by the user 110 as one or more document clips 114 generated for the web page along with one or more document clips 114 generated based on the content of the web page. In this manner, the custom information entered by the user 110 may later be used in a semantic concept search to assist the user 110 in rediscovering the web page.

[0029] 3 illustrates an example of a refiner application graphical user interface 300 according to an aspect of the present disclosure. In one aspect, the graphical user interface 300 corresponds to the graphical user interface 118 of the refiner application 104 of FIG. 1. In another aspect, the graphical user interface 300 may be provided by a different refiner application than the refiner application 104 of FIG. 1.

[0030] The graphical user interface 300 may include a concept panel 302, a canvas panel 304, and a details panel 306. In an aspect, the concept panel 302 may enable a user to create and / or manipulate one or more concepts 310 for searching the user's browsing history. With brief reference to FIG. 4, the concept panel 302 may include a concept title field 412, a drop box 414, and a custom text box 416 that may be utilized to create the concept 310. A user may enter a title for the concept 310 in the concept field 410 and define one or more concept clips for the concept 310 using the drop box 414 and / or the custom text box 416. For example, a user may use the drag and drop box 414 to copy text or an image from another document, such as another web page, that the user may wish to include as a concept clip in the concept 310. A user may additionally or alternatively enter custom text, such as a written paragraph, into the custom text box 414 of the concept 310 to be included as a concept clip in the concept 310. In some aspects, the concept panel 302 may additionally allow a user to specify a weight to use for each of one or more concept clips. The weight may indicate the relative importance of each concept clip to the concept 310. In aspects, the weight may be utilized when generating concept embeddings based on embeddings generated for corresponding concept clips.

[0031] Referring now briefly to FIG. 5, a concept 310 created by a user may be displayed as an expandable menu 510 in the concept panel 302. A user may click on a plus sign associated with a concept 310 to expand a view of the concept 310 in the concept panel 302. The user may then manipulate one or more concept clips within the expanded view of the concept 310. For example, the user may modify the content of the concept 310 and / or remove one or more concept clips from the concept 310. As another example, the user may adjust the weight of one or more concept clips included within the concept 310. As yet another example, the user may provide one or more additional concept clips to the concept 310. In some aspects, the user may additionally or alternatively completely remove one or more concepts 310 from the search and / or add one or more additional concepts 310 to the search. In aspects, as described in more detail below, such manipulation of concepts 310 may be performed by a user to refine an initial search performed based on one or more concepts 310 initially defined by the user.

[0032] Referring again to FIG. 3, the canvas panel 304 may display representations of concepts 310 provided for search by the user and representations of clips 312 that may be semantically similar to each of the concepts 310 provided by the user. The representations of clips 312 may include representations of document clips (e.g., document clips 114 of FIG. 1) generated for web pages in the browsing history associated with the user. The clips 312 may further include concept clips included within the concept 310. In the example shown in FIG. 3, the concept 310 is represented by a large square in the canvas panel 304, and the clips 312 are presented by a small square in the canvas panel 304. In some aspects, the representations of the concept clips included within the concept 310 may be visually displayed as related to the representation of the concept 310. For example, the representations of the concept clips included within the concept 310 may be of the same color as the representation of the concept 310.

[0033] In one aspect, the representations of the clips 312 may be positioned relative to one another within the canvas panel 304 based on the degree of semantic similarity between the clips 312 and may be further configured based on the degree of semantic similarity between the clips 312 and the concepts 310 within the canvas panel 304. For example, the position of the representations of the clips 312 within the canvas panel 304 may be determined based on the distance between the embeddings generated for the individual clips 312 (e.g., by the document embedding module 126 and / or the concept embedding module 128 of FIG. 1 ) and further based on the distance between the embeddings generated for the clips 312 and the embeddings generated for the concepts 310 (e.g., by the concept embedding module 128 of FIG. 1 ). Thus, a user may view clips 312 that are semantically similar to one another and may be more semantically similar to multiple concepts 310 and / or may be more semantically similar to a particular concept 310 than others of the concepts 310. Thus, a user may attempt to explore a web page based on the semantic similarity and / or dissimilarity of the clips 312 associated with the web page to respective ones of the concepts 310. In the example shown in FIG. 3, the user has provided four concepts 310 for search, including the concept 310-1 of "food and beverages," the concept 310-2 of "nutrition," the concept 310-3 of "cooking," and the concept 310-4 of "vegetarian cooking." If the user is attempting to rediscover web pages related to both nutrition and food and beverages, the user may wish to view clips 312 positioned in an area within the canvas panel 304 that may be at least approximately equidistant between the location of the concept 310-1 of "nutrition" and the concept 310-2 of "food and beverages" within the canvas panel 304. On the other hand, if the user is attempting to find web pages related to food and beverages but not nutrition, the user may wish to view clips 312 positioned in an area within the canvas panel 304 that is closer to the concept 310-2 of "food and beverages" and farther from the concept 310-2 of "nutrition."

[0034] In some aspects, a user may wish to refine a search by manipulating (e.g., modifying or deleting) one or more concepts 310 originally provided by the user and / or by providing one or more additional concepts 310. As an example, the concept 310-2 of "nutrition" may be provided by the user after viewing the initial search results in the canvas panel 304 to "pull away" or separate clips 312 that are more closely semantically related to nutrition from clips 312 that are more semantically related to the concept of "food and drink" rather than "nutrition". In response to the user modifying, deleting and / or adding concepts 310, the refiner application 104 may update the canvas panel 304 based on the semantic relationships determined based on the new definition of the clips 312 and the concept 310. For example, if a representation of a first concept (e.g., the "food and drink" concept 310-1) is placed in a first position within the canvas panel 304, and then a representation of a second concept (e.g., the "nutrition" concept 310-2) is placed in a second position within the canvas panel 304, an updated position of the representation of the clip 312 within the canvas panel 304 may be determined based on i) the first position of the representation of the first concept (e.g., the "food and drink" concept 310-1) within the canvas panel 304, ii) the calculated distance between the embedding generated for the corresponding clip 312 and the first concept embedding, iii) the second position of the representation of the second concept (e.g., the "nutrition" concept 310-2) within the canvas panel 304, and iv) the calculated distance between the embedding 312 generated for the corresponding clip and the second concept embedding. The refiner application 104 may then move the representation of the clip 312 to the updated position. Thus, if a web page that a user is attempting to rediscover relates to a first concept (e.g., food and beverages) but not to a second concept (e.g., nutrition), the user can more easily explore the web page.

[0035] In one aspect, the positions of the representations of the clips 312 relative to one another within the canvas panel 304 may be determined (e.g., by the concept search module 130 of FIG. 1) based on an undirected graph of the clips 312. In some aspects, types of graphs other than undirected graphs may be utilized. For example, in one aspect, a directed graph may be utilized. In generating the graph of the clips 312, the concept search module 130 may place the clips 312 as nodes in the graph and may add edges based on the distance between embeddings generated for the clips 312. In one aspect, the concept search module 130 may add edges between nodes corresponding to clips 312 that have a similarity (e.g., cosine similarity) above a certain (e.g., predetermined) threshold. For example, the concept search module 130 may add edges between nodes corresponding to clips 312 that have a similarity (e.g., cosine similarity) above zero. In another aspect, a suitable threshold value above zero may be utilized. In some aspects, the concept search module 130 may perform a sampling of edges in the graph to reduce the number of edges and accordingly reduce the complexity of the graph. For example, for each node in the graph, the concept search module 130 can reduce the number of edges connecting to the node such that the total number of edges connecting to the node is k% of all edges connecting to all of the nodes in the graph. In this manner, the concept search module 130 can proportionally sample k% of all edges in the graph. Thus, in one aspect, the number of edges in the graph can be reduced to O(N 2 ), then the number of edges sampled is k / 100 * O(N 2) In one example, the value of k is 1. In this example, the concept search module 130 may proportionally sample 1% of all edges in the graph. In at least some aspects, this sampling may provide a balanced tradeoff between the computation time and power required to determine the edges and an accurate representation of the relationships between the nodes, so that the position of the representation of the clips 312 may be updated at a responsive, interactive rate as the user 110 is, for example, repositioning the concepts 310 on the canvas panel 304. In other aspects, other suitable values ​​of k may be utilized, and other suitable percentages of all edges may be sampled accordingly.

[0036] In one aspect, the concept search module 130 can use a graph-based force-directed graph drawing algorithm to calculate a layout for displaying the representations of the clips 312 in the canvas panel 304, where nodes that have relatively stronger connections to each other in the graph are drawn closer to each other with a relatively greater force in the layout of the graph. In this manner, representations of clips 312 that are relatively similar to each other are positioned near each other in the canvas panel 304, and representations of clips 312 that have a sufficiently large degree of similarity can form a cluster in the canvas panel 304. To provide a visualization of the similarity between the clips 312 and the concept 310 in the canvas panel 304, the concept search module 130 can further add the concept 310 as a node to the underlying graph used to calculate the layout of the display in the canvas panel 304. Upon adding the concept 310 as a node to the graph, the concept search module 130 can connect the node corresponding to the concept 310 to other nodes in the graph based on the semantic similarity between the concept 310 and the clips 312 that correspond to the other nodes in the graph. In one embodiment, the concept search module 130 connects the node corresponding to the concept 310 to m other nodes in the graph that have a relatively high semantic similarity between the concept 310 and the clips 312 that correspond to other nodes in the graph. In one example, the value of m is 25. In this example, the concept search module 130 connects the node corresponding to the concept 310 to 25 other nodes in the graph that have a relatively high semantic similarity between the concept 310 and the clips 312 that correspond to other nodes in the graph. Connecting each node corresponding to each concept 312 to 25 other nodes in the graph that have a relatively high semantic similarity between the concept 310 and the clips 312 that correspond to other nodes in the graph can provide a balanced tradeoff between computation time and usefulness of the layout of the graph.

[0037] In an aspect, the concept search module 130 may generally treat the location of the representation of the concept 310 as a fixed location on the canvas panel 304, and may use the fixed location of the representation of the concept 310 as a constraint for the force-directed algorithm when determining the layout of the graph in the canvas panel 304. However, as described above, the refiner application 104 may allow the user to interactively move (e.g., drag) the representation of the concept 310 across the canvas panel 304. As the user interactively moves the representation of the concept 310 to a new location in the canvas panel 304, the refiner application 104 may update the locations of the representations of the clips 312 that are directly and indirectly connected to the nodes corresponding to the concept 310 based on the changing location of the representation of the concept 310. In an aspect, the connections between the nodes corresponding to the concept 310 may be dynamically updated such that m other nodes with relatively high semantic similarity to the concept 310 are closer to the new location of the representation of the concept 310. Thus, in one aspect, as a user moves the representation of the concept 310 across the canvas panel 304, representations of clips 312 that have a relatively high semantic similarity to the concept 310 may be displayed closer to the representation of the concept 310 in the canvas panel 304.

[0038] In one aspect, the refiner application 104 may allow a user to view the contents of the clip 312 in the canvas panel 304, for example, by hovering over or otherwise engaging with a representation of the clip 312 in the canvas panel 304. By looking through the clips 312 based on their location relative to the concept 310, the user may recognize that a particular clip 312 belongs to a web page that the user is attempting to rediscover. In some aspects, the user may click on or otherwise engage with the clip 312 to be directed to that web page.

[0039] In some aspects, the refinder application 104 may display details of one or more clips 312 that may be of interest to the user in the details panel 306. For example, the user may use the finder tool 344 to select an area of ​​interest to the user, and the details panel 306 may display a list of one or more clips 312 located in the canvas panel 304 within the area selected by the user. In some aspects, the user may click or otherwise engage with a representation of a particular clip 312 in the canvas panel 304 to cause a view to appear in the details panel 306 that displays the content of the particular clip 312. In some aspects, the details panel 306 may allow the user to perform a keyword query search of the content of clips 312 associated with web pages in the user's browsing history. In this view, the details panel 306 may display all clips 312 that contain the query provided by the user.

[0040] In some aspects, one or more keywords in the clip 312 may be highlighted to a user when the content of the clip 312 is displayed in the details panel 304. In one aspect, the one or more keywords highlighted in the clip 312 may be determined using a machine learning model, such as a transformer-based machine learning model, that may identify keywords that have corresponding embeddings that are sufficiently close to the embeddings of the clip 312. For example, the model may identify one or more keywords in the clip 312 that have embeddings that have the highest cosine similarity to the embeddings of the clip 312. Highlighting the identified keywords in the clip 312 in the details panel 306 may help a user more quickly assess the importance of the clip 312 and the relevance of the content of the clip 312 to the one or more concepts 310 provided by the user.

[0041] FIG. 6 illustrates an example of a finder panel 600 that may be provided as a details panel 306 according to an aspect of the disclosure. The refinder application 104 may display the finder panel in response to detecting that a user has positioned the finder tool 344 over a particular area in the canvas panel 304. The finder panel 600 may display the content of one or more clips 312 located within the particular area in the canvas panel 304 selected by the user. In one aspect, the content of the clips 312 in the finder panel 600 may be displayed in association with a web page of which the clips 312 may be a part. For example, the content of each clip 312 in the finder panel 600 may be displayed on an individual page card 602 that corresponds to the web page of which the clip 312 is a part. In some aspects, the individual page card 602 in the finder panel 600 may include additional information about the corresponding web page, such as the title of the web page, the date and time the user last visited the web page, etc. In one aspect, if a user wants to visit a web page related to a clip 312, the user can click on a portion of the page card 602 that corresponds to the web page (e.g., displays the title of the web page), and the refiner application 104 can direct the user to the web page.

[0042] FIG. 7 illustrates an example of a selected clip panel 700 that may be provided as a details panel 306 according to an aspect of the disclosure. The refiner application 104 may display the selected clip panel 700 in response to detecting that a user has clicked or otherwise engaged with a particular clip 312 in the canvas panel 304. In one aspect, the selected clip panel 700 may display the contents of the particular clip 312 selected by the user. In some aspects, the selected clip panel 700 may additionally display the contents of clips 312 that are closely related to the particular clip 312 selected by the user. For example, the selected clip panel 700 may display a predetermined number of clips 312 that are closest members of the selected clip 312. In one aspect, the contents of each clip 312 displayed in the selected clip panel 700 may be displayed in association with a web page associated with the clip 312. For example, similar to the finder user interface 600 of FIG. 6, the contents of each clip 312 in the selected clip panel 700 may be displayed on an individual page card 702. In some aspects, each page card 702 in the selected clip panel 700 may include additional information about the corresponding web page, such as the title of the web page, the date and time the user last visited the web page, etc. In one aspect, if a user wishes to visit a web page associated with a clip 312, the user may click on a portion of the page card 702 that corresponds to the web page (e.g., that displays the title of the web page), and the refiner application 104 may direct the user to the web page.

[0043] FIG. 8 illustrates an example of a keyword search panel 800 that may be provided as a details panel 306 according to aspects of the disclosure. In one aspect, the refiner application 104 may display the keyword search panel 800 in response to detecting a user clicking or otherwise engaging with a search icon (not shown) that may be displayed within the details panel 306. The keyword search panel 800 may include a search box 804 that may enable a user to enter a keyword search query for searching the user's browsing history. In response to a user entering a search query in the search box 804, the keyword search panel 800 may display clips 312 associated with web pages in the user's browsing history that contain text similar to the query and / or the exact wording of the query. In one aspect, the clips 312 associated with a particular web page may be displayed within the keyword search panel 800 on a page card 806 that corresponds to the particular web page. The page card 806 may include additional information about the web page, such as the title of the web page, the date and time the user last visited the web page, etc. In one aspect, if a user wants to visit a web page, the user can click on a portion of the page card 806 (e.g., that displays the title of the web page) and the refiner application 104 can direct the user to the corresponding web page.

[0044] FIG. 9 details a method 900 for assisting a user in rediscovering a document according to an aspect of the present disclosure. The general sequence of steps of the method 700 is shown in FIG. 9. The method 900 may be executed by a computer system and implemented as a set of computer-executable instructions encoded or stored on a computer-readable medium. Furthermore, the method 900 may be performed by gates or circuits associated with a processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SOC), or other hardware device. The method 900 will be described below with respect to the systems, components, modules, software, data structures, user interfaces, etc. described in conjunction with FIGS. 1-8.

[0045] At block 902, embeddings of document clips related to each document of the plurality of documents are generated. In one aspect, a document clip related to a particular document of the plurality of documents includes one or more of i) one or more text paragraphs parsed from the particular document, and ii) one or more images from the particular document. In some aspects, a document clip related to a particular document additionally or instead includes information related to metadata information of the document, such as information about the structure and / or semantics of the particular document (e.g., the length of the document, the number and / or length of individual sections (e.g., paragraphs), fonts used within the particular document (e.g., font type, size, color, etc.), background color of the document, etc.). In some aspects, the document clips related to the particular document include at least one user-defined clip that includes information not extracted from the document. In one aspect, embeddings of document clips are generated using a machine learning model, such as a Transformer model configured to capture semantics.

[0046] At block 904, a first set of concept clips defining a first concept for searching for content of interest to a user within a plurality of documents. In one aspect, the first set of clips defining the first concept includes one or more of: i) one or more custom text paragraphs provided by a user, ii) one or more text paragraphs pasted by a user from one or more documents, and iii) one or more images provided by a user. Although a particular type of content is described herein, one skilled in the art will appreciate that the set of clips may include other types of content, such as, for example, video clips or audio clips.

[0047] At block 906, an embedding of each concept clip in the first set of concept clips is generated. In one embodiment, the embedding of the concept clip is generated using a machine learning model, such as a Transformer model, configured to capture the semantics of the content. Although the embodiment is described as using a Transformer model, one skilled in the art will understand that other types of machine learning models may be employed in the disclosed embodiment. In one example, the embedding of the concept clip at block 906 is generated using the same machine learning model used to generate the embedding of the document clip at block 902. In another example, a different machine learning model is utilized.

[0048] At block 908, a first concept embedding is generated based on a combination of embeddings generated for each concept clip in the first set of concept clips. In one example, the first concept embedding is generated as a weighted linear combination of the embeddings of the concept clips generated at block 906. In another example, the first concept embedding is based on another suitable combination of the embeddings of the concept clips generated at block 906.

[0049] At block 910, a semantic relationship between the first concept and the document clip is determined based on i) each embedding generated for the document clip and ii) the concept embedding. For example, the semantic relationship is determined based on individual distances (e.g., cosine similarity distances) between i) each embedding generated for the document clip and ii) the concept embedding.

[0050] At block 912, a graphical user interface is generated depicting the semantic relationship between the first concept and the document clip. In an embodiment, generating the graphical user interface at block 912 includes arranging representations of the document clips in a multidimensional canvas, the representations of the document clips being positioned relative to one another based on the calculated distance between embeddings generated for the corresponding document clips. In an embodiment, generating the graphical user interface further includes arranging the representation of the first concept in a first position in the multidimensional canvas, determining a position of the representation of the document clip relative to the first position of the representation of the first concept based on the calculated distance between the embedding generated for the corresponding document clip and the first concept embedding, and positioning the representation of the document clip relative to the representation of the first concept in the multidimensional canvas according to the determined distance. In other embodiments, the graphical user interface depicting the semantic relationship between the first concept and the document clip is generated in other suitable manners. At block 914, the graphical user interface is rendered at the client device. The graphical user interface is operable to enable a user to rediscover a document having content of interest among a plurality of documents.

[0051] 10-11 and the associated discussion provide an illustration of various operating environments in which aspects of the disclosure may be practiced. However, the devices and systems shown and discussed with respect to FIGURES 10-11 are for purposes of example and illustration, and are not intended to limit the numerous computing device configurations that may be utilized to practice aspects of the disclosure described herein.

[0052] 10 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 1000 in which aspects of the disclosure may be practiced. The computing device components described below may be suitable for such a computing device. In a basic configuration, the computing device 1000 may include at least one processing unit 1002 and a system memory 1004. Depending on the configuration and type of computing device, the system memory 1004 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memory.

[0053] The system memory 1004 may include one or more program modules 1006 suitable for executing software applications 1020, such as an operating system 1005 and one or more components supported by the system described herein. By way of example, the system memory 1004 may store a document chunking module 1016 (e.g., corresponding to the document chunking module 126 of FIG. 1), a document embedding module 1026 (e.g., corresponding to the document embedding module 126 of FIG. 1), a concept embedding module 1028 (e.g., corresponding to the concept embedding module 128 of FIG. 1), and / or a concept search module 1030 (e.g., corresponding to the concept search module 130 of FIG. 1). The operating system 1005 may be suitable for controlling the operation of the computing device 1000, for example.

[0054] Additionally, aspects of the disclosure may be practiced with graphics libraries, other operating systems, or any other application programs and are not limited to any particular application or system. This basic configuration is illustrated in FIG. 10 by those components within dashed line 1008. Computing device 1000 may have additional features or functionality. For example, computing device 1000 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 10 by removable storage 1009 and non-removable storage 1010.

[0055] As mentioned above, a number of program modules and data files can be stored in the system memory 1004. While executing on at least one processing unit 1002, the program modules 1006 (e.g., applications 1020) can perform processes including, but not limited to, aspects described herein. Other program modules that can be used in accordance with aspects of the present disclosure can include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, and the like.

[0056] Additionally, aspects of the disclosure may be practiced in electrical circuits including discrete electronic elements, packaged or integrated electronic chips including logic gates, circuits utilizing microprocessors, or on a single chip including electronic elements or a microprocessor. For example, aspects of the disclosure may be practiced by a system-on-chip (SOC) in which each or many of the components shown in FIG. 10 may be integrated on a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communication units, system virtualization units, and various application functions, all of which are integrated (or "burned") onto the chip substrate as a single integrated circuit. When operated by a SOC, the functionality described herein with respect to the client's ability to switch protocols may be operated by application-specific logic integrated with other components of the computing device 1000 on a single integrated circuit (chip). Aspects of the disclosure may also be practiced using other technologies capable of performing logical operations, such as, for example, AND, OR, and NOT, including, but not limited to, mechanical, optical, fluidic, and quantum technologies. Additionally, aspects of the disclosure may be practiced in a general-purpose computer or any other circuit or system.

[0057] The computing device 1000 may also have one or more input devices 1012, such as a keyboard, mouse, pen, audio or voice input device, touch or swipe input device, etc. Output devices 1014, such as a display, speakers, printer, etc. may also be included. The aforementioned devices are examples and other devices may be used. The computing device 1000 may include one or more communications connections 1017 that enable communication with other computing devices 1050. Examples of suitable communications connections 1017 include, but are not limited to, radio frequency (RF) transmitter, receiver and / or transceiver circuitry, universal serial bus (USB), parallel and / or serial ports.

[0058] As used herein, the term computer readable medium may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information, such as computer readable instructions, data structures, or program modules. System memory 1004, removable storage 809, and non-removable storage 1010 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, Electrically Erasable Read Only Memory (EEPROM), Flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other article of manufacture that can be used to store information and that can be accessed by computing device 1000. Any such computer storage media may be part of computing device 1000. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0059] Communication media may be embodied by computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" may refer to a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared and other wireless media.

[0060] 11A-11B illustrate a mobile computing device 1100, such as a mobile phone, a smartphone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, and the like, on which aspects of the disclosure may be practiced. In some aspects, a client (e.g., computing systems 104A-E) may be a mobile computing device. Referring to FIG. 11A, one aspect of a mobile computing device 1100 for implementing aspects is shown. In a basic configuration, the mobile computing device 1100 is a handheld computer having both input and output elements. The mobile computing device 1100 typically includes a display 1105 and one or more input buttons 1110 that allow a user to input information into the mobile computing device 1100. The display 1105 of the mobile computing device 1100 may also function as an input device (e.g., a touch screen display). If included, an optional side input element 1115 allows for further user input. The side input element 1115 may be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the mobile computing device 1100 may incorporate more or fewer input elements. For example, the display 1105 may not be a touch screen in some aspects. In yet another alternative aspect, the mobile computing device 1100 is a mobile phone system, such as a mobile phone. The mobile computing device 1100 may also include an optional keypad 1135. The optional keypad 1135 may be a physical keypad or a "soft" keypad generated on a touch screen display. In various aspects, output elements include the display 1105 for displaying a graphical user interface (GUI), visual indicators 1120 (e.g., light emitting diodes), and / or audio transducers 1125 (e.g., speakers). In some aspects, the mobile computing device 1100 incorporates a vibration transducer for providing tactile feedback to the user.In yet another aspect, the mobile computing device 1100 incorporates input and / or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port) for transmitting signals to or receiving signals from an external source.

[0061] 11B is a block diagram illustrating the architecture of one aspect of a computing device, server, or mobile computing device. That is, computing device 1100 may incorporate a system (e.g., architecture) 1102 for implementing some aspects. System 1102 may be implemented as a "smartphone" capable of running one or more applications (e.g., browser, email, calendar, contact manager, messaging client, games, and media client / player). In some aspects, system 1102 is integrated as a computing device such as an integrated personal digital assistant (PDA) and wireless phone.

[0062] One or more application programs 1166 may be loaded into memory 1162 and executed on or in association with operating system 1164. Examples of application programs include a phone dialer program, an email program, a personal information manager (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. System 1102 includes a non-volatile storage area 1168 in memory 1162. Non-volatile storage area 1168 may be used to store persistent information that should not be lost even if system 1102 is powered down. Application programs 1166 may use and store information in non-volatile storage area 1168, such as emails or other messages used by an email application. A synchronization application (not shown) is also on system 1102 and is programmed to interact with a corresponding synchronization application on the host computer to keep information stored in non-volatile storage area 1168 synchronized with corresponding information stored on the host computer. As should be understood, other applications may be loaded into memory 1162 and executed on the mobile computing device 1100 described herein (e.g., a document chunking module, a document embedding module, a concept embedding module, a concept search module, etc.).

[0063] The system 1102 has a power source 1170, which may be implemented as one or more batteries. The power source 1170 may further include an external power source, such as an AC adapter or a powered base, that supplements or recharges the batteries.

[0064] The system 1102 may include a radio interface layer 1172 that performs the functions of transmitting and receiving radio frequency communications. The radio interface layer 1172 facilitates a wireless connection between the system 1102 and the "outside world" via a communications carrier or service provider. Transmissions to and from the radio interface layer 1172 are performed under the control of the operating system 1164. In other words, communications received by the radio interface layer 1172 may be disseminated by the operating system 1164 to the application programs 1166, and vice versa.

[0065] The visual indicator 1120 may be used to provide visual notifications and / or the audio interface 1174 may be used to produce audible notifications by the audio converter 1125. In the illustrated configuration, the visual indicator 1120 is a light emitting diode (LED) and the audio converter 1125 is a speaker. These devices may be directly coupled to the power source 1170 to remain on for a period of time dictated by the notification mechanism upon startup, even though the processor 1160 and other components may shut down to conserve battery power. The LED may be programmed to remain on indefinitely to indicate a powered-on state of the device until the user takes action. The audio interface 1174 is used to provide audible signals to and receive audible signals from the user. For example, the audio interface 1174, in addition to being coupled to the audio converter 1125, may also be coupled to a microphone to receive audible input to facilitate a telephone conversation. According to aspects of the present disclosure, the microphone may also function as an audio sensor to facilitate control of notifications, as described below. The system 1102 may further include a video interface 1176 that enables operation of the on-board camera 1130 for recording still images, video streams, and the like.

[0066] A mobile computing device 1100 implementing system 1102 may have additional features or functionality. For example, mobile computing device 1100 may also include additional data storage devices (removable and / or non-removable) such as magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 11B by non-volatile storage area 1168.

[0067] The data / information generated or captured by the mobile computing device 1100 and stored by the system 1102 may be stored locally on the mobile computing device 1100 as described above, or the data may be stored on any number of storage media that may be accessed by the device via the wireless interface layer 1172 or via a wired connection between the mobile computing device 1100 and another computing device associated with the mobile computing device 1100, such as a server computer in a distributed computing network such as the Internet. It should be understood that such data / information may be accessed by the mobile computing device 1100 via the wireless interface layer 1172 or a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use in accordance with known data / information transfer and storage means, including electronic mail and collaborative data / information sharing systems.

[0068] Aspects of the present disclosure have been described above with reference to block diagrams and / or operational illustrations of, for example, methods, systems, and computer program products according to aspects of the present disclosure. The functions / acts noted in the blocks may occur out of the order shown in any flowchart. For example, two blocks shown in succession may in fact be executed substantially simultaneously or the blocks may sometimes be executed in the reverse order depending on the functions / acts involved.

[0069] The description and illustrations of one or more aspects shown in this application are not intended to limit or restrict the scope of the claimed disclosure in any way. The claimed disclosure should not be construed as being limited to any aspect, example, or detail provided in this application. It is intended that various features (structural as well as methodological) be selectively included or omitted to create an embodiment having a particular set of features, whether illustrated and described in combination or separately. Having provided the description and illustrations of this application, those skilled in the art may devise modifications, alterations, and alternatives that fall within the spirit of the broader aspects of the general inventive concept embodied in this application without departing from the broader scope of the claimed disclosure.

Claims

1. 1. A method for assisting a user in rediscovering a document, comprising: generating an embedding of a document clip relating to each document of the plurality of documents; receiving a first set of concept clips defining first concepts for searching for content of interest to a user within the plurality of documents; generating an embedding of each concept clip in the first set of concept clips; generating a first concept embedding based on a combination of the embeddings generated for each of the concept clips in the first set of concept clips; determining a semantic relationship between the first concept and the document clip based on i) the embedding generated for the document clip, and ii) the concept embedding; generating a graphical user interface depicting the semantic relationship between the first concept and the document clip; causing a representation of the graphical user interface to be rendered on a client device, the graphical user interface being operable to enable the user to rediscover documents of the plurality of documents having the content of interest to the user; The method includes:

2. The method of claim 1 , wherein the plurality of documents comprises a plurality of web pages in the user's browsing history.

3. The document clip, which relates to content of a particular document of the plurality of documents, includes one or more of: i) one or more text paragraphs parsed from the particular document; and ii) one or more images from the particular document; and 2. The method of claim 1, wherein the first set of clips defining the first concept includes one or more of: i) one or more custom text paragraphs provided by the user, ii) one or more text paragraphs pasted by the user from one or more documents, and iii) one or more images provided by the user.

4. The method of claim 1 , wherein the document clips relating to content of a particular document include at least one user-defined clip that contains information not extracted from the document.

5. 2. The method of claim 1 , wherein generating an embedding for each of the document clips comprises generating the embedding for the document clip using a machine learning model configured to generate embeddings based on content, the embeddings capturing semantics of the content, and generating an embedding for each of the concept clips comprises generating the embedding for each of the concept clips using the machine learning model.

6. 2. The method of claim 1 , wherein generating the first concept embedding comprises generating the first concept embedding based on a weighted combination of the embeddings generated for the respective concept clips in the first set of concept clips.

7. Generating the graphical user interface includes: arranging representations of the document clips within a multidimensional canvas, the representations of the document clips being positioned relative to one another based on the calculated distances between the embeddings generated for corresponding document clips; placing a representation of the first concept at a first location within the multidimensional canvas; determining a position of the representation of the document clip relative to the first position of the representation of the first concept based on the calculated distance between the embedding generated for the corresponding document clip and the first concept embedding; positioning the representation of the document clip relative to the representation of the first concept within the multidimensional canvas according to the determined distance; The method of claim 1 , comprising:

8. receiving a second set of concept clips defining second concepts for searching for content of interest to the user within the plurality of documents; placing a representation of the second concept at a second location within the multidimensional canvas; i) updating positions of the representations of the document clips such that representations of document clips that are semantically similar to the first concept than to the second concept are positioned closer to the representation of the first concept in the multidimensional canvas, and ii) updating positions of the representations of the document clips such that representations of document clips that are semantically similar to the second concept than to the first concept are positioned closer to the representation of the second concept in the multidimensional canvas. The method of claim 7 further comprising:

9. Updating the position of the representation of the document clip comprises: generating an embedding of each concept clip in the second set of concept clips; generating a second concept embedding based on a combination of the embeddings generated for each of the concept clips in the second set of concept clips; determining an updated position of the representation of the document clip within a multidimensional canvas based on i) a first position of a representation of the first concept within the multidimensional canvas, ii) a calculated distance between the embedding generated for the corresponding document clip and the first concept embedding, iii) a second position of a representation of a second concept within the multidimensional canvas, and iv) a calculated distance between the embedding generated for the corresponding document clip and the second concept embedding; moving the representation of the document clip to the updated position within the multidimensional canvas; The method of claim 1 , comprising:

10. Detecting that the representation of the first concept has been repositioned by the user to a new location within the multidimensional canvas; in response to detecting that the representation of the first concept has been repositioned, repositioning the representation of the document clip within the multidimensional canvas such that representations of document clips that are semantically similar to the first concept than to the second concept are moved closer to the new location of the representation of the first concept. The method of claim 9 further comprising:

11. 1. A system for assisting a user in rediscovering a document, comprising: one or more computer readable storage media; When executed by at least one processor, generating embeddings of document clips relating to each document of the plurality of documents using a machine learning model; receiving a first set of concept clips defining first concepts for searching for content of interest to a user within the plurality of documents; generating an embedding of each concept clip in the first set of concept clips using the machine learning model; generating a first concept embedding based on a combination of the embeddings generated for each of the concept clips in the first set of concept clips; determining a semantic relationship between the first concept and the document clip based on i) the embedding generated for the document clip, and ii) the concept embedding; generating a graphical user interface depicting the semantic relationship between the first concept and the document clip; causing a representation of the graphical user interface to be rendered on a client device, the graphical user interface being operable to enable the user to rediscover documents of the plurality of documents having the content of interest to the user; program instructions stored on the one or more computer readable storage media that cause the at least one processor to A system including:

12. The system of claim 11 , wherein the plurality of documents comprises a plurality of web pages in the user's browsing history.

13. The document clips relating to content of a particular document of the plurality of documents include one or more of: i) one or more text paragraphs parsed from the particular document; ii) one or more images from the particular document; and iii) at least one user-defined clip that includes information not extracted from the document; and 12. The system of claim 11, wherein the first set of clips defining the first concept includes one or more of: i) one or more custom text paragraphs provided by the user, ii) one or more text paragraphs pasted by the user from one or more documents, and iii) one or more images provided by the user.

14. When executed by at least one processor, generating an embedding of a document clip relating to each document of the plurality of documents; receiving a first set of concept clips defining first concepts for searching for content of interest to a user within the plurality of documents; generating an embedding of each concept clip in the first set of concept clips; generating a first concept embedding based on a combination of the embeddings generated for each of the concept clips in the first set of concept clips; determining a semantic relationship between the first concept and the document clip based on i) the embedding generated for the document clip, and ii) the concept embedding; generating a graphical user interface depicting the semantic relationship between the first concept and the document clip; causing a representation of the graphical user interface to be rendered on a client device, the graphical user interface being operable to enable the user to rediscover documents of the plurality of documents having the content of interest to the user; A computer storage medium comprising computer-executable instructions for performing a method including:

15. The method comprises: arranging, in the graphical user interface, representations of the document clips within a multidimensional canvas, the representations of the document clips being positioned relative to one another based on the calculated distances between the embeddings generated for corresponding document clips; placing a representation of the first concept at a first location within the multidimensional canvas; determining a position of the representation of the document clip relative to the first position of the representation of the first concept based on the calculated distance between the embedding generated for the corresponding document clip and the first concept embedding; positioning the representation of the document clip relative to the representation of the first concept within the multidimensional canvas according to the determined distance; The computer storage medium of claim 14 further comprising:

Images