Temporary copy using an embedded content database

Abstract

Systems and methods are provided for generating design difference captions. In particular, a computing device may receive a capture request to capture a content item, capture the content item along with metadata associated with the content item, and generate one or more semantic embeddings of the content data using one or more semantic models. The content data may include the content item and the metadata. The computing device may further store the one or more semantic embeddings in a content database, provide a captured content history including a list of captured content items stored in the content database, receive a user query to search the captured content history, and provide search results based on the user query.

Description

[Background technology]

[0001] background Computing devices include various productivity tools and information that facilitate the accomplishment of various tasks, including copying and pasting content items between different devices and applications. For example, a clipboard tool allows a user to copy and store content items (e.g., images and text) from an original location and paste the copied content items into a new location. The clipboard may also provide a clipboard history, allowing a user to scroll back through the stored content items and select any of the previously stored content items. In addition, the clipboard may allow a user to pin frequently used content items so that they remain protected from deletion from the clipboard history and appear at the top of the clipboard history for easy access. However, as the clipboard history grows over time, it can be difficult for a user to find a particular content item from a long list of previously stored content items. It is with respect to these and other general considerations that the aspects disclosed herein are addressed. Also, while relatively specific problems may be discussed, it should be understood that embodiments should not be limited to solving the specific problems identified in the background or elsewhere in this disclosure. Summary of the Invention [Means for solving the problem]

[0002] overview According to an embodiment of the present disclosure, a content management tool provides for capturing, storing, and retrieving content items across different devices and applications. Content items are captured along with corresponding metadata associated with the content items. One or more semantic embeddings associated with the content items are generated using one or more semantic embedding models based on the content items and associated metadata and stored in a content database. The content management tool allows users to retrieve content items from previously captured content items using natural language queries.

[0003] According to at least one embodiment of the present disclosure, a method for capturing content items to a content database is provided. The method may include receiving a capture request to capture a content item, capturing the content item along with metadata associated with the content item, and generating one or more semantic embeddings of the content data using one or more semantic models. The content data may include the content item and the metadata. The method may further include storing the one or more semantic embeddings in a content database and providing a captured content history stored in the content database. The captured content history may include a list of captured content items. The method may further include receiving a user query to search the captured content history and providing search results based on the user query.

[0004] According to at least one embodiment of the present disclosure, a computing device for generating design difference captions is provided. The computing device may include a processor and memory storing a plurality of instructions that, when executed by the processor, cause the computing device to receive a capture request to capture a content item, capture the content item along with metadata associated with the content item, generate one or more semantic embeddings of the content data using one or more semantic models, the content data including the content item and the metadata, store the one or more semantic embeddings in a content database, provide a captured content history stored in the content database, the captured content history including a list of captured content items, receive a user query to search the captured content history, and provide search results based on the user query.

[0005] In accordance with at least one embodiment of the present disclosure, a non-transitory computer-readable medium is provided that stores instructions for generating design difference captions, which, when executed by one or more processors of a computing device, cause the computing device to receive a capture request to capture a content item, capture the content item along with metadata associated with the content item, generate one or more semantic embeddings of the content data using one or more semantic models, the content data including the content item and the metadata, store the one or more semantic embeddings in a content database, provide a captured content history that is stored in the content database, the captured content history including a list of captured content items, receive a user query to search the captured content history, and provide search results based on the user query.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of embodiments will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practice of the disclosure.

[0007] BRIEF DESCRIPTION OF THE DRAWINGS Non-limiting and non-exhaustive examples are described with reference to the following figures: [Brief explanation of the drawings]

[0008] [Figure 1] 1 illustrates a block diagram of an example operating environment in which a content management tool may be implemented, according to an example embodiment of the present disclosure. [Figure 2] 1 illustrates a flow diagram of an example method for capturing content items into a semantic content database using a clipboard, according to an embodiment of the present disclosure. [Figure 3] 1 illustrates a flow diagram of an example method for retrieving a content item from a semantic content database using a clipboard, according to an embodiment of the present disclosure. [Figure 4A] 1 illustrates an overview of an example generative machine learning model that may be used by embodiments of the present disclosure. [Figure 4B] 1 illustrates an overview of an example generative machine learning model that may be used by embodiments of the present disclosure. [Figure 5] FIG. 1 is a block diagram illustrating physical components of an example computing device in which aspects of the present disclosure may be implemented. [Figure 6] 1 is a schematic block diagram of a computing device in which aspects of the present disclosure may be implemented. [Figure 7] 1 is a schematic block diagram of a distributed computing system in which aspects of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE INVENTION

[0009] Detailed Description In the following detailed description, reference is made to the accompanying drawings which form a part hereof, and which show, by way of illustration, specific aspects or examples. These aspects may be combined, other aspects may be utilized, or structural changes may be made without departing from the disclosure. The aspects may be embodied as methods, systems, or devices. Thus, the aspects may take the form of a hardware implementation, an entirely software implementation, or an implementation combining software and hardware aspects. The following detailed description is therefore not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0010] Computing devices include various productivity tools and information that facilitate the accomplishment of various tasks, including copying and pasting content items between different devices and applications. For example, a clipboard tool allows a user to copy and temporarily store a content item from an original location and paste the copied content item in a new location. The clipboard may also provide a clipboard history that allows a user to scroll back through the stored content items and select any of the previously captured content items. In addition, the clipboard may allow a user to pin frequently used content items to prevent them from being deleted from the clipboard history and to appear at the top of the clipboard history for easy access. However, as the clipboard history grows over time, it can be difficult for a user to find a particular content item from a long list of previously captured content items. According to embodiments of the present disclosure, a content management tool provides for capturing, storing, and retrieving content items between different devices and applications. When a content item is captured, metadata associated with the content item is also identified and stored in a content database. For example, a content item may include text, documents, photos, video, and audio, and metadata may include the date and time the content item was captured, the type of content item captured, the filename and path of a file associated with the captured content item, the computing device used to capture the content item, the location on the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item. It should be appreciated that the term "content data" is used throughout this application to collectively refer to content items and metadata associated with content items.

[0011] The content management tool may also allow a user to view the captured content history of previously captured content items. The content management tool may also allow a user to filter and / or search the captured content history based on content data. To do so, a user may use a natural language query to search the list of previously captured content items and further filter the search results based on the content data. For example, a user may indicate that they are looking for information about dogs that may have been copied two months ago. The content management tool may provide search results including a list of captured content items related to dogs that were captured within a certain time range (e.g., ±3 days) from a date two months prior to the search date. Additionally, a user may further filter the search results to view content items captured on the user's mobile device.

[0012] In some aspects, a user may search a captured content history using a keyword, which requires iterating through each content item from a large amount of content history to find which content items are related to the keyword. However, storing and retrieving content items from a large amount of content information can be computationally inefficient, such as by requiring a relatively large amount of memory to store the content items to be searched. Additionally, keyword searches may not take into account the abstract meaning behind the content items, and search results may therefore be relatively inaccurate or incomplete.

[0013] According to embodiments of the present disclosure, a content management tool extracts abstract meaning from content data in the form of embeddings that can be mathematically compared against the abstract meaning of other content data in the form of embeddings, such that the content data from which the embedding was generated can be identified by quantitative similarity between the embeddings and the results. The similarity between the content data can provide semantic context to a computing device when identifying whether and / or what action should be performed.

[0014] The content data may further be provided to one or more semantic embedding models that generate one or more semantic embeddings, e.g., using a generative large-scale language machine learning model, a transformer model, other types of machine learning models, or a combination of models. The one or more semantic embeddings may then be inserted into an embedded object memory or a content database. The semantic embeddings may be associated with respective instructions that correspond to references to the content data associated with the semantic embeddings. Furthermore, the insertion may trigger a spatial storage operation to store vector representations of the one or more semantic embeddings. After the embeddings are stored, multiple collections of stored embeddings may be received from the content database, such as based on user input. Furthermore, an action may be identified based on a subset of the collection of stored embeddings and user input. A subset of the collection of stored embeddings may be retrieved based on similarity to the input embeddings.

[0015] 1 illustrates a block diagram of one example of an operating environment 100 in which a content management tool may be implemented, according to an embodiment of the present disclosure. To do so, operating environment 100 includes a computing device 120 associated with a user 110. Operating environment 100 may further include one or more remote devices, such as a productivity platform server 160, communicatively coupled to computing device 120 via a network 150. Network 150 may include any type of computing network, including, but not limited to, a wired or wireless local area network (LAN), a wired or wireless wide area network (WAN), and / or the Internet.

[0016] The computing device 120 includes a content management tool 130 executing on the computing device 120, which has a processor 122, a memory 124, and a communication interface 126. The content management tool 130 allows the user 110 to copy and paste or capture and retrieve content items. For example, the content management tool 130 may be a clipboard or any other productivity tool executing on the computing device 120 with copy and paste or capture and retrieve capabilities. The content items may be one or more of text, documents, images, pictures, photos, videos, or audio. Additionally, the computing device 120 may be, but is not limited to, a computer, notebook, laptop, mobile device, smartphone, tablet, portable device, wearable device, or any other suitable computing device capable of executing the content management tool 130. To do so, the content management tool 130 further includes a content capture manager 132 and a content acquisition manager 134.

[0017] The content capture manager 132 is configured to capture content items in the semantic content database 136. A user may capture any content item that the user may want to retrieve or recall in the future. To do so, the content capture manager 132 is configured to receive a capture request to capture the content item to be stored in the content database 136. A capture request is any indicator of a user's intent to capture or store a content item for future retrieval or access. A capture request may be a shortcut and / or gesture assigned by the operating system or by the user. For example, a keyboard shortcut for content capture (e.g., Ctrl+r or Windows logo key+r) may be predefined by the operating system of the user's computing device. Additionally or alternatively, a user may assign a gesture as a capture request. For example, a user may indicate that they want to store a screenshot content item in the content database 136 whenever they take a screenshot on their mobile device. In another example, a user may indicate that they want to store a photo content item in the content database 136 whenever they take a photo of their dog. In other words, a user may define one or more rules or action-based rules as capture requests to capture and store content items in content database 136. It should be appreciated that content database 136 may be synchronized across a user's multiple devices so that a user may capture content items from any of the user's computing devices. However, it should be appreciated that in some aspects the content database may be a cloud-based content database shared across a user's multiple devices.

[0018] The content capture manager 132 is further configured to capture content items with metadata associated with the content items (collectively referred to throughout this application as "content data"). The content data may include the original content item, the date and time the content item was captured, the type of content item captured, the file name and / or path of a folder or file associated with the captured content item, the computing device used to capture the content item, the location on the computing device where the content item was captured, and the application used to capture the content item. In addition, the content data may also include any information extracted from the content item.

[0019] The content capture manager 132 is configured to provide the content data to one or more semantic models to generate one or more semantic embeddings. In embodiments, the semantic models may be generative large-scale language machine learning models, transformer models, etc., or a combination of machine learning models. The one or more semantic models may include one or more semantic embedding models, one or more generative large-scale language models, and one or more machine learning models. The content capture manager 132 is configured to extract abstract meaning from the content data in the form of one or more semantic embeddings. As described further below in FIG. 3 , the semantic embeddings of the content items are used to find and retrieve one or more stored content items from all previously captured content items in response to a user query (e.g., a natural language query from a user).

[0020] Depending on the resources, capabilities, and capacity of the computing device used to capture the content item, one or more semantic embeddings may be generated from the computing device or from server 160. For example, if a user captures a content item on the user's laptop computer, content management tool 130 on the user's laptop computer generates a semantic embedding associated with the captured content item. However, if the user captures a content item on the user's mobile device, which has fewer resources to perform semantic analysis, content capture manager 132 sends the captured content data to server 160 to generate a semantic embedding associated with the captured content data. The semantic embedding is then sent back to the user's mobile device for insertion into a content database.

[0021] The content capture manager 132 is configured to store or insert one or more semantic embeddings into the content database 136. The semantic embeddings may be associated with respective instructions corresponding to references to content data associated with the semantic embeddings. Furthermore, the insertion may trigger a spatial storage operation to store vector representations of the one or more semantic embeddings. As described above, after the semantic embeddings are stored, multiple collections of stored embeddings may be received from the content database, such as based on a user query. Furthermore, an action may be identified based on a subset of the collection of stored embeddings and the user query. A subset of the collection of stored embeddings may be retrieved based on similarity to the user query embedding.

[0022] Additionally, content capture manager 132 is further configured to determine whether an action is associated with a content item based on the content data and / or the capture request used to capture the content item. For example, a user may set a rule indicating that whenever the user takes a screenshot of a recipe using the user's mobile device, the user wants the screenshot content item to be stored in a content database and an email sent to the user's email along with the screenshot content. In another example, a user may set a rule indicating that whenever the user takes a picture of a dog using the user's mobile device, the user wants a photo content item to be stored in a content database and a text message sent to a particular friend along with the photo of the dog.

[0023] The content acquisition manager 134 is configured to retrieve previously captured content items from the content item database 136. To do so, the content acquisition manager 134 is configured to receive a retrieve request. The retrieve request may be a shortcut and / or gesture assigned by the operating system or by the user. For example, a keyboard shortcut (e.g., Ctrl+v or Windows logo key+v) may be predefined by the operating system of the user's computing device. Additionally or alternatively, the user may assign a shortcut or gesture as a retrieve request. In response to receiving the retrieve request, the content acquisition manager 134 is configured to provide a captured content history (i.e., a list of previously captured content items) in reverse order. As described above, the content database is synchronized across a user's multiple devices so that the user can retrieve captured content items from any of the user's computing devices. However, it should be appreciated that in some aspects, the content database may be a cloud-based content database shared across a user's multiple devices.

[0024] The content acquisition manager 134 is further configured to receive a user query from a user and search the list of previously captured content items to retrieve one or more specific content items. The user query may be a natural language query. The content acquisition manager 134 is configured to generate input embeddings based on the user query using one or more semantic models (e.g., semantic embedding models, generative large-scale language models, machine learning models, etc.) and retrieve one or more stored semantic embeddings based on their similarity to the input embedding.

[0025] For example, the content acquisition manager 134 may be configured to, upon receiving a natural language query from a user, generate an input embedding corresponding to the user query. The abstract meaning of the natural language query in the form of the input embedding is compared to one or more semantic embeddings associated with stored content items in a content database. In doing so, the content acquisition manager 134 may be configured to identify the content data from which the embedding was generated by measuring the quantitative similarity between the stored semantic embeddings and the input embedding. The similarity between the content data and the natural language query can provide semantic context to the computing device when identifying whether and / or what action should be performed.

[0026] The content acquisition manager 134 is configured to provide search results to the user. In addition, the content acquisition manager 134 is further configured to allow the user to further filter the search results. For example, the user may further filter the search results based on content data, such as a particular time range (e.g., last hour, last 24 hours, last 7 days, this month, and this year), information associated with the content item, a particular computing device used to capture the content item, a particular type of content item, and / or a particular application used to capture the content item. The user can further search the filtered list of captured content history using natural language queries. The content acquisition manager 134 is further configured to allow the user to find and view a specific content item selected by the user from the search results and / or copy and paste it to a specific location indicated by the user.

[0027] Referring now to FIG. 2 , a method 200 for capturing content items into a semantic content database is provided, according to an embodiment of the present disclosure. A general sequence for the steps of method 200 is shown in FIG. 2 . Generally, method 200 begins at 202 and ends at 218. Method 200 may include more or fewer steps, or may arrange the order of steps differently than that shown in FIG. 2 . In an exemplary aspect, method 200 is performed by a computing device of user 110 (e.g., user device 120). However, it should be appreciated that one or more steps of method 200 may be performed by another device (e.g., server 160).

[0028] Notably, in some aspects, method 200 may be performed by a content management tool (e.g., 130) executing on user device 120. For example, content management tool 130 may be a clipboard or any other productivity tool executing on computing device 120 that has copy-and-paste or capture-and-retrieve functionality. For example, computing device 120 may be, but is not limited to, a computer, notebook, laptop, mobile device, smartphone, tablet, portable device, wearable device, or any other suitable computing device capable of executing content management tool (e.g., 130). For example, server 160 may be any suitable computing device capable of communicating with computing device 120. Method 200 may be executed by a computer system and may be implemented as a set of computer-executable instructions encoded or stored on a computer-readable medium. Furthermore, method 200 may be performed by gates or circuits associated with a processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system-on-chip (SOC), or other hardware device. The method 200 is described below with reference to the systems, components, modules, software, data structures, user interfaces, etc., described in connection with FIG. 1 and FIGS.

[0029] Method 200 begins at operation 202, where flow may proceed to 204. At operation 204, content management tool 130 receives a capture request to capture a content item to be stored in a content database (e.g., 144). A capture request is any indicator of a user's intent to capture or store a content item for future retrieval or access. A content item may be one or more of text, documents, images, pictures, photographs, videos, or audio. A capture request may be a shortcut and / or gesture assigned by an operating system or by a user. For example, a keyboard shortcut for content capture (e.g., Ctrl+r or Windows logo key+r) may be predefined by the operating system of a user's computing device. Additionally or alternatively, a user may assign a gesture as a capture request. For example, a user may indicate that they want the screenshot content item to be stored in a content database whenever they take a screenshot on their mobile device.

[0030] In operation 206, the content management tool 130 captures the content item with metadata associated with the content item (collectively referred to throughout this application as "content data"). The content data may include the original content item, the date and time the content item was captured, the type of content item captured, the file name and / or path of a folder or file associated with the captured content item, the computing device used to capture the content item, the location on the computing device where the content item was captured, and the application used to capture the content item. In addition, the content data may further include any information extracted from the content item.

[0031] For example, a user may have a screenshot function configured on their mobile device as a capture request, and may take a screenshot of a pizza recipe that the user wants to try later from a web browser application on their mobile device. Content management tool 130 may then store a pizza recipe content item, including text and photos, and identify content data associated with the pizza recipe content item. For example, content management tool 130 may identify the date and time of capture of the content item, the user's mobile device as the device used to capture the content item, and the web browser application as the application used to capture the content item. In addition, content management tool 130 may further process the photo to extract meaningful information, such as ingredients shown in the photo, which is stored in a content database.

[0032] In operation 208, the content management tool 130 generates one or more semantic embeddings using one or more semantic models (e.g., a semantic embedding model, a generative large-scale language model, a machine learning model, etc.). To do so, the content data is provided to the one or more semantic models, which generate one or more semantic embeddings. The content management tool 130 extracts abstract meaning from the content data in the form of one or more semantic embeddings. As described further below in FIG. 3, the semantic embeddings of the content items are used to find and retrieve one or more stored content items from all previously captured content items in response to a user query (e.g., a natural language query from a user).

[0033] As described above, the content database is synchronized across a user's multiple devices so that a user can capture content items from any of the user's computing devices. Depending on the resources, capabilities, and capacity of the computing devices, one or more semantic embeddings may be generated from the computing device and / or a server (e.g., 160) containing one or more semantic models. For example, if a user captures a content item on the user's laptop computer, the content management tool 130 on the user's laptop computer generates a semantic embedding associated with the captured content item. However, if a user captures a content item on the user's mobile device, which has less capability to perform semantic analysis, the captured content item is sent to a server (e.g., 160) to generate a semantic embedding associated with the captured content item. The semantic embedding is then sent back to the user's mobile device and inserted into the content database.

[0034] Thereafter, in operation 210, the content management tool 130 stores or inserts one or more semantic embeddings into a content database. The semantic embeddings may be associated with respective instructions corresponding to references to content data associated with the semantic embeddings. Furthermore, the insertion may trigger a spatial store operation to store vector representations of the one or more semantic embeddings. As described above, after the semantic embeddings are stored, multiple collections of stored embeddings may be received from the content database, such as based on a user query. Furthermore, an action may be identified based on a subset of the collection of stored embeddings and the user query. A subset of the collection of stored embeddings may be retrieved based on similarity to the user query embedding.

[0035] Subsequently or simultaneously, at operation 212, content management tool 130 determines whether actions are associated with the content item based on the content item and content data. In some aspects, content management tool 130 determines whether actions are associated with the content item based on a capture request used to capture the content item. If content management tool 130 determines that no actions are associated with the content item, method 200 may end at operation 218.

[0036] However, if the content management tool 130 determines that there is an action associated with the content item, then the method 200 proceeds to operation 216 and performs the associated action. For example, a user may set a rule indicating that whenever the user takes a screenshot of a recipe using the user's mobile device, the user wants the screenshot content item stored in a content database and an email sent to the user's email with the screenshot content.

[0037] In another embodiment, a user may set a rule indicating that whenever the user takes a picture of a dog using the user's mobile device, the user wants the photo content item stored in a content database and a text message sent to a particular friend along with the photo of the dog. Method 200 may then end at operation 218.

[0038] Referring now to FIG. 3 , a method 300 for retrieving content items from a content item database is provided, according to an embodiment of the present disclosure. A general sequence for the steps of method 300 is shown in FIG. 3 . Generally, method 300 begins at 302 and ends at 320. Method 300 may include more or fewer steps, or may arrange the order of steps differently than that shown in FIG. 3 . In an exemplary aspect, method 300 is performed by a computing device of user 110 (e.g., user device 120). However, it should be appreciated that one or more steps of method 300 may be performed by another device (e.g., server 160).

[0039] Notably, in some aspects, method 300 may be performed by a content management tool (e.g., 130) executing on user device 120. For example, content management tool 130 may be a clipboard or any other productivity tool executing on computing device 120 that has copy-and-paste or capture-and-retrieve functionality. For example, computing device 120 may be, but is not limited to, a computer, notebook, laptop, mobile device, smartphone, tablet, portable device, wearable device, or any other suitable computing device capable of executing content management tool (e.g., 130). For example, server 160 may be any suitable computing device capable of communicating with computing device 120. Method 300 may be executed by a computer system and may be implemented as a set of computer-executable instructions encoded or stored on a computer-readable medium. Furthermore, method 300 may be performed by gates or circuits associated with a processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system-on-chip (SOC), or other hardware device. The method 300 is described below with reference to the systems, components, modules, software, data structures, user interfaces, etc., described in connection with FIG. 1 and FIGS.

[0040] Method 300 may begin at operation 302, where flow may proceed to 304. At operation 304, content management tool 130 receives a retrieve request. The retrieve request may be a shortcut and / or gesture assigned by the operating system or by the user. For example, a keyboard shortcut (e.g., Ctrl+v or Windows logo key+v) may be predefined by the operating system of the user's computing device. Additionally or alternatively, the user may assign a shortcut or gesture as the retrieve request. In response to receiving the retrieve request, at operation 306, content management tool 130 provides a captured content history (i.e., a list of previously captured content items) in reverse order. As described above, the content database is synchronized across a user's different devices so that a user can retrieve captured content items from any of the user's computing devices. However, it should be appreciated that in some aspects, the content database may be a cloud-based content database shared across a user's multiple devices.

[0041] The user may further search the captured content history to retrieve one or more specific content items. To do so, at operation 308, the content management tool 130 receives a user query from the user to search the captured content history. The user query may be a natural language query. For example, the user may say, "I want to find a recipe that I think I saved two months ago." It should be appreciated that in some aspects, the method 300 may receive the retrieval request at operation 304 and skip to operation 308 to receive the user query (e.g., by providing a search bar for the user to enter the user query or by hearing an audible user query).

[0042] At operation 310, the content management tool 130 generates input embeddings based on the user query using one or more semantic models (e.g., a semantic embedding model, a generative large-scale language model, a machine learning model, etc.). At operation 312, the content management tool 130 retrieves one or more stored semantic embeddings based on their similarity to the input embeddings. For example, when a user submits a natural language query, the content management tool 130 may generate an input embedding corresponding to the user query. The abstract meaning of the natural language query in the form of the input embeddings is compared to one or more semantic embeddings associated with stored content items in a content database. In doing so, the content management tool 130 may identify the content data from which the embeddings were generated by measuring the quantitative similarity between the stored semantic embeddings and the input embeddings. The similarity between the content data and the natural language query can provide semantic context to the computing device when identifying whether and / or what action should be performed.

[0043] At operation 314, the content management tool 130 provides the search results. In addition, the user may further filter the search results at operation 316. For example, the search results may be filtered based on content data. For example, the user may further filter the search results by information associated with the content items, by the particular computing device used to capture the content items, by a particular type of content item, and / or by the particular application used to capture the content items.

[0044] In the example provided above, a user may query the content management tool 130 to find recipes that the user captured approximately two months ago. The content management tool 130 may provide search results including all recipes captured within ±3 days of a date two months prior to the query date. The user may further filter the search results to show one or more recipes containing chicken that were captured on the user's mobile device by taking a screenshot image. At operation 318, the user may select a particular content item from the search results. The content management tool 130 copies the selected content item and pastes it into the identified location indicated by the user. However, it should be appreciated that in some embodiments, a user may remove and view a particular content item from the search results without necessarily copying it to another location.

[0045] 4A and 4B illustrate an overview of an example generative machine learning model that may be used in accordance with aspects described herein. Referring initially to FIG. 4A, a conceptual diagram 400 illustrates an overview of a pre-trained generative model package 404 that processes input 402 to generate model outputs for storing entries in and / or retrieving information from generative model outputs 406 (e.g., suggestions and / or proposed modifications) in accordance with aspects described herein.

[0046] In embodiments, the generative model package 404 is pre-trained according to a variety of inputs (e.g., various human languages, various programming languages, and / or various content types) and therefore does not need to be fine-tuned or trained for a particular scenario. Rather, the generative model package 404 may be pre-trained more generally such that the input 402 includes prompts that are generated, selected, or otherwise manipulated to cause the generative model package 404 to generate a particular generative model output 406. It will be appreciated that the input 402 and the generative model output 406 may each include any of a variety of content types, including, but not limited to, text output, image output, audio output, video output, program output, and / or binary output, among other embodiments. In embodiments, the input 402 and the generative model output 406 may have different content types, such as may be the case when the generative model package 404 includes a generative multimodal machine learning model.

[0047] As such, the generative models package 404 may be used in any of a variety of scenarios, and further, a different generative models package may be substituted for the generative models package 404 without substantially modifying other associated aspects (e.g., similar to those described herein with respect to FIGS. 1-3). Thus, the generative models package 404 operates as a tool upon which machine learning processes are performed, where certain inputs 402 to the generative models package 404 are programmatically generated or otherwise identified, thereby causing the generative models package 404 to generate model outputs 406 that may then be used for further processing.

[0048] The generative models package 404 may be provided or otherwise used according to any of a variety of paradigms. For example, the generative models package 404 may be used locally to a computing device (e.g., computing device 140 in FIG. 1 ) or may be accessed remotely from a machine learning service (e.g., server 160 in FIG. 1 ). In other embodiments, aspects of the generative models package 404 are distributed across multiple computing devices. In some examples, the generative models package 404 is accessible via an application programming interface (API), such as may be provided by the operating system of a computing device and / or by a machine learning service, among other embodiments.

[0049] Referring now to the illustrated embodiment of generative model package 404, generative model package 404 includes input tokenizer 408, input embedding 410, model layer 412, output layer 414, and output decode 416. In an embodiment, input tokenizer 408 processes input 402 to generate input embedding 410, which includes a sequence of symbolic representations corresponding to input 402. Input embedding 410 is then processed by model layer 412, output layer 414, and output decode 416 to generate model output 406. An example architecture corresponding to generative model package 404 is shown in FIG. 4B and is described in further detail below. Even so, the architecture shown and described herein should not be construed in a limiting sense, and it will be appreciated that any of a variety of other architectures may be used in other embodiments.

[0050] 4B is a conceptual diagram illustrating an example architecture 450 of a pre-trained generative machine learning model that may be used in accordance with aspects described herein. As noted above, any of a variety of alternative architectures and corresponding ML models may be used in other embodiments without departing from aspects described herein.

[0051] As shown, architecture 450 processes input 402 to generate generative model output 406, aspects of which were discussed above with respect to FIG. 4A. Architecture 450 is shown as a transformer model including an encoder 452 and a decoder 454. Encoder 452 processes input embedding 458 (which may be similar in aspect to input embedding 410 of FIG. 4A) that includes a sequence of symbolic representations corresponding to input 456. In an embodiment, input 456 includes content data 402 corresponding to a content item.

[0052] Additionally, positional encoding 460 may introduce information regarding relative and / or absolute positions for the tokens of input embedding 458. Similarly, output embedding 474 includes a sequence of symbolic representations corresponding to output 472, while positional encoding 476 may similarly introduce information regarding relative and / or absolute positions for the tokens of output embedding 474.

[0053] As shown, the encoder 452 includes an example layer 470. It should be appreciated that any number of such layers may be used, and the depicted architecture is simplified for illustrative purposes. The example layer 470 includes two sublayers: a multi-head attention layer 462 and a feedforward layer 466. In the example, residual connections are included around each layer 462, 466, followed by normalization layers 464 and 468, respectively.

[0054] The decoder 454 includes an example layer 490. As with the encoder 452, any number of such layers may be used in other embodiments, and the depicted architecture of the decoder 454 is simplified for illustrative purposes. As shown, the example layer 490 includes three sublayers: a masked multi-head attention layer 478, a multi-head attention layer 482, and a feedforward layer 486. Aspects of the multi-head attention layer 482 and the feedforward layer 486 may be similar to those discussed above with respect to the multi-head attention layer 462 and the feedforward layer 466, respectively. Additionally, the masked multi-head attention layer 478 performs multi-head attention on the output of the encoder 452 (e.g., output 472). In an example, the masked multi-head attention layer 478 prevents a position from being attached to a subsequent position. Such masking may be combined with offsetting the embedding (e.g., by one position, as illustrated by multi-head attention layer 482) to ensure that predictions for a given position depend on known outputs for one or more positions smaller than the given position. As shown, residual connections are also included around layers 478, 482, and 486, followed by normalization layers 480, 484, and 488, respectively. Multi-head attention layers 462, 478, and 482 may each linearly project the query, key, and value using a set of linear projections onto the corresponding dimensions. Each linear projection may be processed using an attention function (e.g., dot product or sum attention), thereby resulting in an n-dimensional output value for each linear projection. The resulting values ​​may be concatenated and re-projected so that the values ​​are subsequently processed (e.g., by the corresponding normalization layer 464, 480, or 484) as illustrated in FIG. 4B.

[0055] Each feedforward layer 466 and 486 may be a fully connected feedforward network applied to each location. In an embodiment, each feedforward layer 466 and 486 includes multiple linear transformations with normalized linear unit activations between them. In an embodiment, each linear transformation may be the same across different locations, while using different parameters compared to other linear transformations in the feedforward network.

[0056] Additionally, aspects of the linear transform 492 may be similar to the linear transforms discussed above with respect to the multi-head attention layers 462, 478, and 482 and the feedforward layers 466 and 486. A softmax 494 may further convert the output of the linear transform 492 into a probability of a predicted next token, as indicated by output probability 496. The illustrated architecture is provided as one example, and it will be appreciated that in other examples, any of a variety of other model architectures may be used in accordance with the disclosed aspects. Accordingly, the output probability 496 may form the generative model output 406 in accordance with aspects described herein, such that the output of the generative ML model (which may include, for example, one or more semantic embeddings and one or more retrieved content items) is used as an input for identifying an action in accordance with aspects described herein. In other examples, the generative model output 406 is provided as a generated output for retrieving one or more previously captured content items.

[0057] 5-7 and the associated discussion provide a discussion of various operating environments in which aspects of the present disclosure may be implemented. However, the devices and systems shown and discussed with respect to FIGS. 5-7 are for purposes of example and explanation, and are not intended to limit the vast number of computing device configurations that may be utilized to implement aspects of the present disclosure described herein.

[0058] 5 is a block diagram illustrating the physical components (e.g., hardware) of a computing device 500 in which aspects of the present disclosure may be implemented. The computing device components described below may be suitable for the computing devices described above, including one or more devices associated with machine learning services (e.g., productivity platform server 160), as well as computing device 140 discussed above with respect to FIG. 1. In a basic configuration, computing device 500 may include at least one processing unit 502 and system memory 504. Depending on the configuration and type of computing device, system memory 504 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memory.

[0059] The system memory 504 may include an operating system 505 and one or more program modules 506 suitable for executing software applications 520, such as one or more components supported by the system described herein. As an example, the system memory 504 may store a content capture manager 521 and / or a content acquisition manager 522. The operating system 505 may be suitable for controlling the operation of the computing device 500, for example.

[0060] Additionally, aspects of the present disclosure may be implemented with graphics libraries, other operating systems, or any other application programs and are not limited to any particular application or system. This basic configuration is illustrated in FIG. 5 by the components within dashed line 508. Computing device 500 may have additional features or functionality. For example, computing device 500 may also include additional data storage devices (removable and / or non-removable), such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 5 by removable storage device 509 and non-removable storage device 510.

[0061] As mentioned above, a number of program modules and data files may be stored in the system memory 504. While executing on the processing unit 502, the program modules 506 (e.g., applications 520) may perform processes including, but not limited to, aspects described herein. Other program modules that may be used in accordance with aspects of the present invention may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, etc.

[0062] Furthermore, aspects of the present disclosure may be implemented in electrical circuits comprising discrete electronic elements, packaged or integrated electronic chips including logic gates, circuits utilizing a microprocessor, or on a single chip including electronic elements or a microprocessor. For example, aspects of the present disclosure may be implemented via a system-on-chip (SOC) in which each or many of the components shown in FIG. 5 may be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communications units, system virtualization units, and various application functions, all of which are integrated (or "burned") onto the chip substrate as a single integrated circuit. When operating via an SOC, the functionality described herein with respect to the client's ability to switch protocols may be operated via application-specific logic integrated with other components of the computing device 500 on a single integrated circuit (chip). Aspects of the present disclosure may also be implemented using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including, but not limited to, mechanical, optical, fluidic, and quantum technologies. Additionally, aspects of the present disclosure may be implemented within a general-purpose computer or any other circuit or system.

[0063] The computing device 500 may also have one or more input devices 512, such as a keyboard, mouse, pen, sound or voice input device, touch or swipe input device, etc. It may also include output devices 514, such as a display, speakers, printer, etc. The foregoing devices are examples, and others may be used. The computing device 500 may include one or more communications connections 516 that enable communication with other computing devices 550. Examples of suitable communications connections 516 include, but are not limited to, radio frequency (RF) transmitter, receiver, and / or transceiver circuitry, a universal serial bus (USB), a parallel port, and / or a serial port.

[0064] The term computer-readable medium as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer-readable instructions, data structures, program modules, etc. System memory 504, removable storage device 509, and non-removable storage device 510 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, Electrically Erasable Read-Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other articles that can be used to store information and that can be accessed by computing device 500. Any such computer storage media may be part of computing device 500. Computer storage media do not include carrier waves or other propagated or modulated data signals.

[0065] Communication media may be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" describes a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0066] 6 illustrates a system 600, which may be a mobile computing device such as a mobile phone, a smartphone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc., in which aspects of the present disclosure may be implemented. In one embodiment, system 600 is implemented as a "smartphone" capable of running one or more applications (e.g., a browser, email, calendaring, a contact manager, a messaging client, games, and a media client / player). In some aspects, system 600 is integrated as a computing device such as an integrated personal digital assistant (PDA) and wireless phone.

[0067] In its basic configuration, such a mobile computing device is a handheld computer having both input and output elements. System 600 typically includes a display 605 and one or more input buttons that allow a user to input information into system 600. Display 605 may also function as an input device (e.g., a touchscreen display).

[0068] If included, optional side input elements allow for further user input. For example, the side input elements may be rotary switches, buttons, or any other type of manual input element. In alternative embodiments, system 600 may incorporate more or fewer input elements. For example, display 605 may not be a touchscreen in some embodiments. In another embodiment, optional keypad 635 may also be included, which may be a physical keypad or a "soft" keypad generated on a touchscreen display.

[0069] In various embodiments, the output elements include a display 605 for showing a graphical user interface (GUI), a visual indicator (e.g., a light emitting diode 620), and / or an audio transducer 625 (e.g., a speaker). In some embodiments, a vibration transducer is included to provide tactile feedback to the user. In yet other embodiments, input and / or output ports are included, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port) for sending signals to or receiving signals from external devices.

[0070] One or more application programs 666 may be loaded into memory 662 and execute on or in association with operating system 664. Examples of application programs include a phone dialer program, an email program, a personal information manager (PIM) program, a word processing program, a spreadsheet program, an internet browser program, a messaging program, etc. System 600 also includes a non-volatile storage area 668 within memory 662. Non-volatile storage area 668 may be used to store persistent information that should not be lost if system 600 is powered off. Application programs 666 may use and store information in non-volatile storage area 668, such as emails or other messages used by an email application. A synchronization application (not shown) also resides on system 600 and is programmed to interact with a corresponding synchronization application resident on the host computer to keep information stored in non-volatile storage area 668 synchronized with corresponding information stored on the host computer. It should be appreciated that other applications may be loaded into memory 662 and run on the system 600 described herein (eg, a content capture manager, a content acquisition manager, etc.).

[0071] The system 600 includes a power supply 670, which may be implemented as one or more batteries. The power supply 670 may also include an external power supply, such as an AC adapter or a powered docking cradle, that replenishes or charges the batteries.

[0072] System 600 may also include a wireless interface layer 672 that performs the function of transmitting and receiving radio frequency communications. Wireless interface layer 672 facilitates wireless connectivity between system 600 and the "outside world" via a communications carrier or service provider. Transmissions to and from wireless interface layer 672 occur under the control of operating system 664. In other words, communications received by wireless interface layer 672 may be disseminated to application programs 666 via operating system 664, and vice versa.

[0073] The visual indicator 620 may be used to provide visual notifications, and / or the audio interface 674 may be used to generate audible notifications via the audio transducer 625. In the illustrated embodiment, the visual indicator 620 is a light-emitting diode (LED), and the audio transducer 625 is a speaker. These devices may be directly coupled to the power supply 670 so that, when activated, they remain on for a duration dictated by the notification mechanism, even though the processor 660 and other components may shut down to conserve battery power. The LED may be programmed to remain on indefinitely until the user takes an action indicating the device's power-on state. The audio interface 674 is used to provide audible signals to and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 674 may also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to aspects of the present disclosure, the microphone may also function as an audio sensor to facilitate control of notifications, as described below. The system 600 may further include a video interface 676 that enables operation of the on-board camera 630 to record still images, video streams, and the like.

[0074] It will be appreciated that system 600 may have additional features or functionality. For example, system 600 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 6 by non-volatile storage area 668.

[0075] Data / information generated or captured and stored via system 600 may be stored locally, as described above, or the data may be stored on any number of storage media that may be accessed by the device via wireless interface layer 672 or via a wired connection between system 600 and a separate computing device associated with system 600, for example, a server computer in a distributed computing network such as the Internet. As should be appreciated, such data / information may be accessed via wireless interface layer 672 or via a distributed computing network. Similarly, such data / information may be readily transferred between computing devices for storage and use according to any of a variety of data / information transfer and storage means, including email and collaborative data / information sharing systems.

[0076] 7 illustrates one aspect of a system architecture for processing data received at a computing system from a remote source, such as a personal computer 704, a tablet computing device 706, or a mobile computing device 708, as described above. Content displayed at the server device 702 may be stored on different communication channels or other storage types. For example, various documents may be stored using a directory service 724, a web portal 725, a mailbox service 726, an instant messaging store 728, or a social networking site 730.

[0077] An application 720 (e.g., similar to application 520) may be employed by a client in communication with server device 702. Additionally or alternatively, a content capture manager 791 and / or a content acquisition manager 792 may be employed by server device 702. Server device 702 may provide data to and from client computing devices, such as personal computer 704, tablet computing device 706, and / or mobile computing device 708 (e.g., smartphone) over network 715. As an example, the computer system described above may be embodied in personal computer 704, tablet computing device 706, and / or mobile computing device 708 (e.g., smartphone). Any of these examples of computing devices may retrieve content from store 716 in addition to receiving graphics data that can be used either pre-processed in a graphics originating system or post-processed in a receiving computing system.

[0078] It will be appreciated that aspects and functionality described herein may operate on a distributed system (e.g., a cloud-based computing system), where application functionality, memory, data storage and retrieval, and various processing functions may operate remotely from one another over a distributed computing network such as the Internet or an intranet. Various types of user interfaces and information may be displayed via on-board computing device displays or via remote display units associated with one or more computing devices. For example, various types of user interfaces and information may be displayed and interacted with on a wall surface onto which various types of user interfaces and information are projected. Interactions with numerous computing systems in which aspects of the present disclosure may be implemented include keystroke input, touchscreen input, voice or other audio input, gesture input where an associated computing device is equipped with detection (e.g., camera) capabilities to capture and interpret user gestures to control functions of the computing device, and the like.

[0079] Aspects of the present disclosure are described above with reference to, for example, block diagrams and / or operational illustrations of methods, systems, and computer program products according to aspects of the present disclosure. The functions / acts noted in the blocks may occur out of the order shown in any flow diagram. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality / acts involved.

[0080] The description and illustration of one or more embodiments provided herein in no way restricts or limits the scope of the disclosure as claimed. The embodiments, examples, and details provided herein are believed to be sufficient to convey ownership and enable others to make and use the claimed embodiments of the present disclosure. The claimed disclosure should not be construed as limited to any embodiment, example, or detail provided herein. Various features (both structural and methodological), whether shown and described in combination or separately, are intended to be selectively included or omitted to produce embodiments having a particular set of features. While description and illustration of the present application have been provided, those skilled in the art may envision variations, modifications, and alternative embodiments that fall within the spirit of the broader aspects of the general inventive concepts embodied herein without departing from the broader scope of the claimed disclosure.

[0081] Additionally, aspects and functionality described herein may operate on a distributed system (e.g., a cloud-based computing system), where application functionality, memory, data storage and retrieval, and various processing functions may operate remotely from one another over a distributed computing network such as the Internet or an intranet. Various types of user interfaces and information may be displayed via on-board computing device displays or via remote display units associated with one or more computing devices. For example, various types of user interfaces and information may be displayed and interacted with on a wall surface onto which various types of user interfaces and information are projected. Interactions with numerous computing systems in which aspects of the present disclosure may be implemented include keystroke input, touchscreen input, voice or other audio input, gesture input where an associated computing device is equipped with detection (e.g., camera) capabilities to capture and interpret user gestures to control functions of the computing device, and the like.

[0082] The phrases "at least one," "one or more," "or," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, the phrases "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," "A, B, and / or C," and "A, B, or C" each mean A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0083] The term "a" or "an" entity refers to one or more of that entity. Thus, the terms "a" (or "an"), "one or more," and "at least one" can be used interchangeably herein. It should also be noted that the terms "comprising," "including," and "having" can be used interchangeably.

[0084] The term "automatic" and its variations, as used herein, refers to any process or operation that is typically continuous or semi-continuous and occurs in the absence of substantial human input when the process or operation is performed. However, a process or operation can be automatic if input is received before the process or operation is performed, even if the performance of the process or operation uses substantial or intangible human input. Human input is considered substantial if such input affects the way the process or operation is performed. Human input consenting to the performance of a process or operation is not considered "substantial."

[0085] Any of the steps, functions, and operations discussed herein may be performed continuously and automatically.

[0086] The systems and methods of the present disclosure have been described with reference to computing devices. However, to avoid unnecessarily obscuring the present disclosure, the foregoing description omits some well-known structures and devices. This omission should not be construed as limiting. Specific details are described to provide an understanding of the present disclosure. However, it should be appreciated that the present disclosure may be practiced in a variety of ways beyond the specific details set forth herein.

[0087] Furthermore, while aspects of the illustrated embodiments herein show various components of the system being collocated, certain components of the system may be remotely located, such as in a remote portion of a distributed network, such as a LAN and / or the Internet, or within a dedicated system. Accordingly, it should be appreciated that components of the system may be combined into one or more devices, such as a server, a communication device, or may be collocated on a particular node of a distributed network, such as an analog and / or digital telecommunications network, a packet-switched network, or a circuit-switched network. From the foregoing discussion and for reasons of computational efficiency, it should be appreciated that components of the system may be located anywhere within the distributed network of components without affecting the operation of the system. Furthermore, it should be appreciated that the various links connecting the elements may be wired or wireless links, or any combination thereof, or any other known or later-developed elements capable of providing and / or communicating data between the connected elements. These wired or wireless links may also be secure links and may be capable of communicating encrypted information. The transmission medium used as the link may be any suitable carrier for electrical signals including, for example, coaxial cables, copper wire and optical fibers, and may also take the form of acoustic or light waves such as those generated during radio wave and infrared data communications.

[0088] Although the flow diagrams are discussed and illustrated with respect to a particular sequence of events, it will be appreciated that modifications, additions, and omissions to this sequence can be made without materially affecting the operation of the disclosed configurations and aspects.

[0089] Several variations and modifications of the present disclosure can be used: it is possible to provide some features of the present disclosure without providing other features.

[0090] In yet another configuration, the systems and methods of the present invention can be implemented with special purpose computers, programmed microprocessors or microcontrollers and peripheral integrated circuit elements, hardwired electronic or logic circuitry such as ASICs or other integrated circuits, digital signal processors, discrete element circuits, programmable logic devices or gate arrays such as PLDs, PLAs, FPGAs, PALs, special purpose computers, any equivalent means, etc. In general, any device or means capable of implementing the methods illustrated herein can be used to implement various aspects of the present disclosure. Example hardware that can be used for the present disclosure includes computers, handheld devices, telephones (e.g., cellular, Internet-enabled, digital, analog, hybrid, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage, input devices, and output devices. Furthermore, alternative software implementations, including, but not limited to, distributed processing or component / object distributed processing, parallel processing, or virtual machine processing, can also be constructed to implement the methods described herein.

[0091] In yet another configuration, the disclosed methods may be readily implemented with software using object or object-oriented software development environments that provide portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed systems may be implemented partially or fully in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement a system in accordance with the present disclosure depends on the speed and / or efficiency requirements of the system, the particular functionality, and the particular software or hardware system or microprocessor or microcomputer system being utilized.

[0092] In yet another configuration, the disclosed methods may be implemented in part in software that can be stored on a storage medium and executed on a programmed general-purpose computer, special-purpose computer, microprocessor, etc. in cooperation with a controller and memory. In these examples, the disclosed systems and methods can be implemented as programs embedded on a personal computer, such as applets, JAVA, or CGI scripts, as resources resident on a server or computer workstation, as routines embedded in dedicated measurement systems, system components, etc. The systems can also be implemented by physically incorporating the systems and / or methods into software and / or hardware systems.

[0093] The disclosure is not limited to the standards and protocols, if any, described. Other similar standards and protocols not mentioned herein exist and are included in this disclosure. Moreover, the standards and protocols mentioned herein, as well as other similar standards and protocols not mentioned herein, are periodically superseded by faster or more efficient equivalents having essentially the same functionality. Such replacement standards and protocols having the same functionality are considered equivalents included in this disclosure.

[0094] According to at least one embodiment of the present disclosure, a method for capturing content items to a content database is provided. The method may include receiving a capture request to capture a content item, capturing the content item along with metadata associated with the content item, and generating one or more semantic embeddings of the content data using one or more semantic models. The content data may include the content item and the metadata. The method may further include storing the one or more semantic embeddings in a content database and providing a captured content history stored in the content database. The captured content history may include a list of captured content items. The method may further include receiving a user query to search the captured content history and providing search results based on the user query.

[0095] According to at least one aspect of the above method, the method may include determining, based on the content data, whether an action associated with the content item exists; and, in response to determining that the action exists, performing the action associated with the content item.

[0096] According to at least one aspect of the above method, the method may include a case where the content item includes one or more of text, documents, images, pictures, photographs, video, or audio. According to at least one aspect of the above method, the method may include a case where the metadata includes a date and time the content item was captured, a type of the captured content item, a file name and path of a file associated with the captured content item, a computing device used to capture the content item, a location on the computing device where the content item was captured, an application used to capture the content item, and any information extracted from the content item.

[0097] According to at least one aspect of the above method, the method may include a case where the user query is a natural language query.

[0098] According to at least one aspect of the above method, the method may include where providing search results based on the user query includes generating input embeddings based on the user query and retrieving semantic embeddings associated with one or more previously captured content items from a content database based on similarity to the input embeddings. According to at least one aspect of the above method, the method may further include filtering the search results based on content data associated with the content items in the search results.

[0099] According to at least one aspect of the above method, the method may include where the one or more semantic models include a generative large-scale language model (LLM).

[0100] According to at least one embodiment of the present disclosure, a computing device for generating design difference captions is provided. The computing device may include a processor and memory storing a plurality of instructions that, when executed by the processor, cause the computing device to receive a capture request to capture a content item, capture the content item along with metadata associated with the content item, generate one or more semantic embeddings of the content data using one or more semantic models, the content data including the content item and the metadata, store the one or more semantic embeddings in a content database, provide a captured content history stored in the content database, the captured content history including a list of captured content items, receive a user query to search the captured content history, and provide search results based on the user query.

[0101] According to at least one aspect of the computing device described above, the computing device may further include a plurality of instructions that, when executed, cause the computing device to determine, based on the content data, whether an action associated with the content item exists, and, in response to the action being determined to exist, perform the action associated with the content item.

[0102] In accordance with at least one aspect of the computing device described above, the computing device may include a case in which the content item includes one or more of text, a document, an image, a picture, a photograph, a video, or an audio.

[0103] According to at least one aspect of the computing device described above, the computing device may include a case in which the metadata includes the date and time the content item was captured, the file name and path of a file associated with the captured content item, the computing device used to capture the content item, the location on the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item.

[0104] According to at least one aspect of the computing device described above, the computing device may include a case where causing the computing device to provide search results based on a user query includes causing the computing device to generate input embeddings based on the user query and retrieving semantic embeddings associated with one or more previously captured content items from a content database based on similarity to the input embeddings.

[0105] According to at least one aspect of the computing device described above, the computing device may include a plurality of instructions that, when executed, further cause the computing device to filter search results based on content data associated with content items in the search results.

[0106] In accordance with at least one aspect of the computing device described above, the computing device may include where the one or more semantic models include a generative large-scale language model (LLM).

[0107] In accordance with at least one embodiment of the present disclosure, a non-transitory computer-readable medium is provided that stores instructions for generating design difference captions, which, when executed by one or more processors of a computing device, cause the computing device to receive a capture request to capture a content item, capture the content item along with metadata associated with the content item, generate one or more semantic embeddings of the content data using one or more semantic models, the content data including the content item and the metadata, store the one or more semantic embeddings in a content database, provide a captured content history that is stored in the content database, the captured content history including a list of captured content items, receive a user query to search the captured content history, and provide search results based on the user query.

[0108] According to at least one aspect of the non-transitory computer-readable medium described above, the instructions, when executed by the one or more processors, may further cause the computing device to determine, based on the content data, whether an action associated with the content item exists, and, in response to determining that the action exists, perform the action associated with the content item.

[0109] According to at least one aspect of the non-transitory computer-readable medium described above, the instructions, when executed by one or more processors of a computing device, may include where the content item includes one or more of text, documents, images, pictures, photographs, video, or audio, and the metadata includes the date and time the content item was captured, the file name and path of a file associated with the captured content item, the computing device used to capture the content item, the location on the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item.

[0110] According to at least one aspect of the non-transitory computer-readable medium described above, the instructions, when executed by one or more processors of the computing device, may include causing the providing of search results based on a user query to include generating input embeddings based on the user query; and retrieving semantic embeddings associated with one or more previously captured content items from a content database based on similarity to the input embeddings.

[0111] According to at least one aspect of the non-transitory computer-readable medium described above, the instructions, when executed by one or more processors, may further cause the computing device to filter search results based on content data associated with content items in the search results.

[0112] The present disclosure includes, in various configurations and aspects, components, methods, processes, systems, and / or apparatus substantially as shown and described herein, including various combinations, subcombinations, and subsets thereof. After understanding the present disclosure, one of ordinary skill in the art will understand how to make and use the systems and methods disclosed herein. The present disclosure, in various configurations and aspects, includes providing devices and processes in the absence of items not shown and / or described herein or in various configurations or aspects herein, including the absence of items that may have been used in previous devices or processes, e.g., to improve performance, achieve ease, and / or reduce cost of implementation.

Claims

1. 1. A method for capturing content items into a content database, comprising: receiving a capture request to capture a content item; capturing the content item along with metadata associated with the content item; generating one or more semantic embeddings of content data using one or more semantic models, the content data including the content items and the metadata; storing the one or more semantic embeddings in a content database; providing a captured content history stored in the content database, the captured content history including a list of captured content items; receiving a user query to search the captured content history; providing search results based on the user query; A method comprising:

2. determining whether there is an action associated with the content item based on the content data; responsive to identifying the action as present, performing the action associated with the content item; and The method of claim 1 further comprising:

3. The method of claim 1 , wherein the content items include one or more of text, documents, images, pictures, photographs, video, or audio.

4. 2. The method of claim 1, wherein the metadata includes the date and time the content item was captured, the type of the captured content item, the file name and path of a file associated with the captured content item, the computing device used to capture the content item, the location on the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item.

5. The method of claim 1 , wherein the user query is a natural language query.

6. providing the search results based on the user query, generating an input embedding based on the user query; retrieving semantic embeddings associated with one or more previously captured content items from the content database based on similarity to the input embedding; The method of claim 1 , comprising:

7. The method of claim 1 , further comprising filtering the search results based on the content data associated with content items in the search results.

8. The method of claim 1 , wherein the one or more semantic models include a generative large-scale language model (LLM).

9. 1. A computing device for generating design difference captions, comprising: a processor; a memory storing a plurality of instructions that, when executed by the processor, cause the computing device to: receiving a capture request to capture a content item; capturing the content item along with metadata associated with the content item; generating one or more semantic embeddings of content data using one or more semantic models, said content data including said content items and said metadata; storing the one or more semantic embeddings in a content database; providing a captured content history stored in the content database, the captured content history including a list of captured content items; receiving a user query to search the captured content history; a memory for causing search results to be provided based on the user query; A computing device comprising:

10. The instructions, when executed, further cause the computing device to: determining whether there is an action associated with the content item based on the content data; responsive to the action being identified as present, causing the action associated with the content item to be performed; The computing device of claim 9.

11. The computing device of claim 9 , wherein the content items include one or more of text, documents, images, pictures, photographs, video, or audio.

12. 10. The computing device of claim 9, wherein the metadata includes the date and time the content item was captured, the file name and path of a file associated with the captured content item, the computing device used to capture the content item, the location on the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item.

13. Providing the search results based on the user query includes: generating an input embedding based on the user query; retrieving semantic embeddings associated with one or more previously captured content items from the content database based on similarity to the input embedding; The computing device of claim 9 , comprising:

14. The computing device of claim 9 , wherein the instructions, when executed, further cause the computing device to filter the search results based on the content data associated with content items in the search results.

15. The computing device of claim 9 , wherein the one or more semantic models include a generative large-scale language model (LLM).

16. 1. A computer-readable medium storing instructions for generating design difference captions, the instructions, when executed by one or more processors of a computing device, causing the computing device to: receiving a capture request to capture a content item; capturing the content item along with metadata associated with the content item; generating one or more semantic embeddings of content data using one or more semantic models, said content data including said content items and said metadata; storing the one or more semantic embeddings in a content database; providing a captured content history stored in the content database, the captured content history including a list of captured content items; receiving a user query to search the captured content history; providing search results based on the user query; Computer-readable medium.

17. The instructions, when executed by the one or more processors, further cause the computing device to: determining whether there is an action associated with the content item based on the content data; responsive to the action being identified as present, causing the action associated with the content item to be performed; 17. The computer-readable medium of claim 16.

18. 17. The computer-readable medium of claim 16, wherein the content item includes one or more of text, documents, images, pictures, photographs, video, or audio, and the metadata includes the date and time the content item was captured, the file name and path of a file associated with the captured content item, the computing device used to capture the content item, the location on the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item.

19. Providing the search results based on the user query includes: generating an input embedding based on the user query; retrieving semantic embeddings associated with one or more previously captured content items from the content database based on similarity to the input embedding; 17. The computer-readable medium of claim 16, comprising:

20. The computing device of claim 9 , wherein the instructions, when executed, further cause the computing device to filter the search results based on the content data associated with content items in the search results.

Images