Temporal replication using embedded content database

Abstract

Systems and methods for generating design difference descriptions are provided. Specifically, a computing device may receive a capture request to capture a content item, capture the content item and metadata associated with the content item, and generate one or more semantic embedding of the content data using one or more semantic models. The content data may include content items and metadata. The computing device may also embed and store the one or more semantics in a content database, provide a captured content history stored in the content database, the captured content history including a list of captured content items, receive a user query to search for the captured content history, and store the captured content history in the content database. And providing search results based on the user query.

Description

Background Art

[0001] Computing devices include various productivity tools and information that facilitate completing various tasks (including copying and pasting content items between different devices and applications). For example, a clipboard tool allows a user to copy and store content items (e.g., images and text) from an original location and paste the copied content items to a new location. The clipboard can also provide a clipboard history, wherein the user can scroll back through the stored content items to select any previously stored content items. Additionally, the clipboard can allow the user to pin frequently used content items to continue to keep them, prevent them from being removed from the clipboard history and appear at the top of the clipboard history for easy access. However, as the clipboard history grows over time, finding a specific content item from a long list of previously stored content items may be challenging for the user.

[0002] It is with respect to these and other general considerations that aspects of the present disclosure have been made.In addition, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving specific problems identified elsewhere in the background or this disclosure. Summary of the Invention

[0003] According to an example of the present disclosure, a content management tool provides for capturing, storing, and retrieving content items between different devices and applications. Content items and corresponding metadata associated with the content items are captured. One or more semantic embeddings associated with the content items are generated based on the content items and associated metadata using one or more semantic embedding models, and the one or more semantic embeddings are stored in a content database. The content management tool allows a user to retrieve content items from previously captured content items using natural language queries.

[0004] According to at least one example of the present disclosure, a method for capturing content items into a content database is provided. The method may include receiving a capture request to capture a content item, capturing the content item and metadata associated with the content item, and generating one or more semantic embeddings of the content data using one or more semantic models, where the content data may include the content item and the metadata. The method may also include storing the one or more semantic embeddings in the content database and providing a captured content history stored in the content database. The captured content history may include a list of captured content items. The method may also include receiving a user query to search the captured content history, and providing search results based on the user query.

[0005] According to at least one example of the present disclosure, a computing device for generating design difference description text is provided. The computing device may include a processor and a memory, the memory having a plurality of instructions stored thereon, which, when executed by the processor, cause the computing device to: receive a capture request to capture a content item; capture the content item and metadata associated with the content item; generate one or more semantic embeddings of the content data using one or more semantic models, the content data including the content item and the metadata; store the one or more semantic embeddings in a content database; provide a captured content history stored in the content database, the captured content history including a list of captured content items; receive a user query to search the captured content history; and provide search results based on the user query.

[0006] According to at least one example of the present disclosure, a non-transitory computer-readable medium storing instructions for generating design difference description text is provided. When executed by one or more processors of a computing device, the instructions cause the computing device to: receive a capture request to capture a content item; capture the content item and metadata associated with the content item; generate one or more semantic embeddings of the content data using one or more semantic models, the content data including the content item and the metadata; store the one or more semantic embeddings in a content database; provide a captured content history stored in the content database, the captured content history including a list of captured content items; receive a user query to search the captured content history; and provide search results based on the user query.

[0007] This summary is provided to introduce a selection of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Additional aspects, features, and / or advantages of the examples will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0009] Figure 1 depicts a block diagram of an example of an operating environment in which a content management tool may be implemented according to examples of the present disclosure;

[0010] Figure 2 A flowchart depicting an example method for capturing content items into a semantic content database using a clipboard according to examples of the present disclosure;

[0011] Figure 3 depicts a flow chart of an example method for retrieving content items from a semantic content database using a clipboard according to examples of the present disclosure;

[0012] Figure 4A and Figure 4B An overview of an example generative machine learning model that may be used in accordance with examples of the present disclosure is shown;

[0013] Figure 5 is a block diagram illustrating example physical components of a computing device in which aspects of the present disclosure may be practiced;

[0014] Figure 6 is a simplified block diagram of a computing device in which aspects of the present disclosure may be practiced; and

[0015] Figure 7 is a simplified block diagram of a distributed computing system in which aspects of the present disclosure may be practiced. DETAILED DESCRIPTION

[0016] In the following detailed description, reference is made to the accompanying drawings which form a part of this document, and in which specific aspects or examples are shown by way of illustration. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. Each aspect may be practiced as a method, system, or device. Therefore, each aspect may take the form of a hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Therefore, the following detailed description should not be regarded as limiting, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0017] Computing devices include various productivity tools and information that facilitate completing various tasks (including copying and pasting content items between different devices and applications). For example, a clipboard tool allows a user to copy and temporarily store content items from an original location and paste the copied content items to a new location. The clipboard can also provide a clipboard history, wherein the user can scroll back through the stored content items to select any previously captured content items. Additionally, the clipboard can allow the user to pin frequently used content items to continue to keep them, prevent them from being removed from the clipboard history and appear at the top of the clipboard history for easy access. However, as the clipboard history grows over time, finding a specific content item from a long list of previously captured content items may be challenging for the user.

[0018] According to an example of the present disclosure, a content management tool provides for capturing, storing, and retrieving content items between different devices and applications. When a content item is captured, metadata associated with the content item is also determined and stored in a content database. For example, a content item may include text, documents, photos, videos, and audio, and the metadata may include the date and time when the content item was captured, the type of the captured content item, the file name and path of the file associated with the captured content item, the computing device used to capture the content item, the location of the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item. It should be understood that the term "content data" used throughout this application collectively refers to content items and metadata associated with content items.

[0019] The content management tool also allows the user to view the captured content history of previously captured content items. The content management tool allows the user to filter and / or search the captured content history based on content data. For this reason, the user can use a natural language query to search a list of previously captured content items, and further filter the search results based on the content data. For example, the user can indicate that the user is looking for information about dogs, which may have been copied two months ago. The content management tool can provide search results that are included in a list of captured content items related to dogs that were captured within a specific time range (e.g., ±3 days) of a date two months before the search date. Additionally, the user can also filter the search results to view the content items captured on the user's mobile device.

[0020] In some aspects, users can search the captured content history using keywords, which requires iterating through each content item from a large amount of content history to find which content items are related to the keywords. However, storing and retrieving content items from a large amount of content information can be computationally inefficient, such as by requiring a relatively large amount of memory to store the content items to be searched. Additionally, keyword searches may not consider the abstract meaning behind the content items, and thus search results may be relatively inaccurate or incomplete.

[0021] According to an example of the present disclosure, a content management tool extracts abstract meaning from embedded content data, which can be mathematically compared to abstract meanings of other embedded content data, so that quantitative similarities between embeddings can be determined, and thus the content data from which the embeddings were generated can be determined. The similarities between the content data can provide semantic context to the computing device when determining whether an action should be performed and / or what action should be performed.

[0022] The content data is further provided to one or more semantic embedding models, which generate one or more semantic embeddings, for example, using a generative large language machine learning model, a transformer model, other types of machine learning models, or a combination of models. The one or more semantic embeddings can then be inserted into an embedding object memory or a content database. The semantic embedding can be associated with a corresponding indication corresponding to a reference to the content data associated with the semantic embedding. In addition, the insertion can trigger a spatial storage operation to store vector representations of one or more semantic embeddings. After the embeddings are stored, multiple sets of stored embeddings can be received from the content database, such as based on user input. In addition, actions can be determined based on a subset of the set of stored embeddings and user input. A subset of the set of stored embeddings can be retrieved based on similarity to the input embedding.

[0023] Figure 1 A block diagram depicts an example of an operating environment 100 in which a content management tool may be implemented according to an example of the present disclosure. To this end, the operating environment 100 includes a computing device 120 associated with a user 110. The operating environment 100 may also include one or more remote devices, such as a productivity platform server 160, communicatively coupled to the computing device 120 via a network 150. The network 150 may include any type of computing network, including, but not limited to, a wired or wireless local area network (LAN), a wired or wireless wide area network (WAN), and / or the Internet.

[0024] Computing device 120 includes a content management tool 130 executed on computing device 120 having a processor 122, a memory 124, and a communication interface 126. Content management tool 130 allows user 110 to copy and paste or capture and retrieve content items. For example, content management tool 130 can be a clipboard or any other productivity tool with copy and paste or capture and retrieve functionality executed on computing device 120. A content item can be one or more text, documents, images, pictures, photos, videos, or audio. Additionally, computing device 120 can be, but is not limited to, a computer, a notebook, a laptop computer, a mobile device, a smartphone, a tablet computer, a portable device, a wearable device, or any other suitable computing device capable of executing content management tool 130. To this end, content management tool 130 also includes a content capture manager 132 and a content retrieval manager 134.

[0025] The content capture manager 132 is configured to capture content items into the semantic content database 136. A user can capture any content item that the user may want to retrieve or remember in the future. To this end, the content capture manager 132 is configured to receive a capture request to capture a content item to be saved in the content database 136. A capture request is any indicator of the user's intent to capture or remember a content item for future retrieval or access. A capture request can be a shortcut and / or gesture assigned by the operating system or the user. For example, a keyboard shortcut for content capture (e.g., Ctrl+r or Window logo key+r) may be predefined by the operating system of the user's computing device. Additionally or alternatively, the user can assign a gesture as a capture request. For example, a user can indicate that whenever the user takes a screenshot on the user's mobile device, the user wants the screenshot content item to be stored in the content database 136. In other examples, the user can indicate that whenever the user takes a photo of their dog, the user wants the photo content item to be stored in the content database 136. In other words, the user can define one or more rules or action-based rules as capture requests for capturing and storing content items in the content database 136. It should be understood that the content database 136 is synchronized between the user's multiple devices so that the user can capture content items from any of the user's computing devices. However, it should be understood that in some aspects, the content database can be a cloud-based content database shared between the user's multiple devices.

[0026] The content capture manager 132 is also configured to capture content items and metadata associated with the content items (collectively referred to herein as "content data"). The content data may include the original content item, the date and time when the content item was captured, the type of content item captured, the file name and / or path of the folder or file associated with the captured content item, the computing device used to capture the content item, the location of the computing device where the content item was captured, and the application used to capture the content item. Additionally, the content data may also include any information extracted from the content item.

[0027] The content capture manager 132 is configured to provide the content data to one or more semantic models to generate one or more semantic embeddings. In an example, the semantic model can be a generative large language machine learning model, a transformer model, etc. or a combination of machine learning models. The one or more semantic models may include one or more semantic embedding models, one or more generative large language models, and one or more machine learning models. The content capture manager 132 is configured to extract abstract meaning from the content data in the form of one or more semantic embeddings. As shown below in Figure 3 As further described in , in response to a user query (eg, a natural language query from a user), semantic embeddings of content items are used to find and retrieve one or more stored content items from all previously captured content items.

[0028] Depending on the resources, capabilities, and capacity of the computing device used to capture the content item, one or more semantic embeddings may be generated from the computing device or the server 160. For example, if a user captures a content item on the user's laptop computer, the content manager tool 130 on the user's laptop computer generates a semantic embedding associated with the captured content item. However, if the user captures the content item on the user's mobile device, which has fewer resources to perform semantic analysis, the content capture manager 132 sends the captured content data to the server 160 to generate a semantic embedding associated with the captured content data. The semantic embedding is then sent back to the user's mobile device to be inserted into the content database.

[0029] The content capture manager 132 is configured to store or insert one or more semantic embeddings in the content database 136. The semantic embedding can be associated with a corresponding indication that corresponds to a reference to the content data associated with the semantic embedding. In addition, the insertion can trigger a spatial storage operation to store a vector representation of the one or more semantic embeddings. As described above, after the semantic embeddings are stored, multiple sets of stored embeddings can be received from the content database, such as based on a user query. In addition, an action can be determined based on a subset of the set of stored embeddings and a user query. A subset of the set of stored embeddings can be retrieved based on similarity to the user query embedding.

[0030] Additionally, the content capture manager 132 is further configured to determine whether an action is associated with a content item based on the content data and / or the capture request for capturing the content item. For example, a user may set a rule indicating that whenever a user uses the user's mobile device to take a screenshot of a recipe, the user wants the screenshot content item to be stored in the content database and an email with the screenshot content to be sent to the user's email address. In another example, a user may set a rule indicating that whenever a user uses the user's mobile device to take a photo of a dog, the user wants the photo content item to be stored in the content database and a text message with the dog photo to be sent to a specific friend.

[0031] The content retrieval manager 134 is configured to retrieve previously captured content items from the content item database 136. To this end, the content retrieval manager 134 is configured to receive a retrieval request. The retrieval request can be a shortcut and / or gesture assigned by the operating system or the user. For example, a keyboard shortcut (e.g., Ctrl+v or Window logo key+v) can be predefined by the operating system of the user's computing device. Additionally or alternatively, the user can assign a shortcut or gesture as a retrieval request. In response to receiving the retrieval request, the content retrieval manager 134 is configured to provide a captured content history (i.e., a list of previously captured content items) in reverse order. As described above, the content database is synchronized between the user's multiple devices so that the user can retrieve the captured content items from any of the user's computing devices. However, it should be understood that in some aspects, the content database can be a cloud-based content database shared between the user's multiple devices.

[0032] The content retrieval manager 134 is further configured to receive a user query from a user to search a list of previously captured content items to retrieve one or more specific content items. The user query can be a natural language query. The content retrieval manager 134 is configured to generate an input embedding based on the user query using one or more semantic models (e.g., a semantic embedding model, a generative large language model, a machine learning model, etc.), and retrieve one or more stored semantic embeddings based on similarity to the input embedding.

[0033] For example, the content retrieval manager 134 is configured to, upon receiving a natural language query from a user, generate an input embedding corresponding to the user's query. The abstract meaning of the natural language query in the form of the input embedding is compared with one or more semantic embeddings associated with the content items stored in the content database. By doing so, the content retrieval manager 134 can be configured to determine the content data from which the embedding is generated by measuring the quantitative similarity between the stored semantic embedding and the input embedding. The similarity between the content data and the natural language query can provide semantic context to the computing device when determining whether an action should be performed and / or what action should be performed.

[0034] The content retrieval manager 134 is configured to provide search results to the user. Additionally, the content retrieval manager 134 is also configured to allow the user to further filter the search results. For example, the user can also filter the search results based on content data, such as, a specific time range (such as, before the hour, before the 24 hours, before the 7 days, this month and this year), information associated with the content item, a specific computing device for capturing the content item, a specific type of the content item and / or a specific application for capturing the content item. The user can also use a natural language query to search for a filtered list of the captured content history. The content retrieval manager 134 is also configured to allow the user to find and view the specific content item selected by the user from the search results and / or copy and paste it in a specific location indicated by the user.

[0035] Now refer to Figure 2 According to an example of the present disclosure, a method 200 for capturing content items into a semantic content database is provided. The general order of steps of the method 200 is as follows: Figure 2 Generally, method 200 begins at 202 and ends at 218. Method 200 may include more or fewer steps, or may be combined with Figure 2 In the illustration, method 200 is performed by a computing device (e.g., user device 120) of user 110. However, it should be understood that one or more steps of method 200 may be performed by another device (e.g., server 160).

[0036] Specifically, in some aspects, method 200 can be performed by a content management tool (e.g., 130) executed on user device 120. For example, content management tool 130 can be a clipboard or any other productivity tool with copy and paste or capture and retrieve functionality executed on computing device 120. For example, computing device 120 can be, but is not limited to, a computer, a notebook, a laptop, a mobile device, a smart phone, a tablet, a portable device, a wearable device, or any other suitable computing device capable of executing a content management tool (e.g., 130). For example, server 160 can be any suitable computing device capable of communicating with computing device 120. Method 200 can be performed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer-readable medium. In addition, method 200 can be performed by gates or circuits associated with a processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system on a chip (SOC), or other hardware device. Hereinafter, method 200 will be referred to in conjunction with Figure 1 and Figures 4 to Figure 7 The described systems, components, modules, software, data structures, user interfaces, etc. are explained.

[0037] Method 200 begins at operation 202, where the process can proceed to 204. At operation 204, the content management tool 130 receives a capture request to capture a content item to be stored in a content database (e.g., 144). A capture request is any indicator of a user's intent to capture or remember a content item for future retrieval or access. A content item can be one or more text, documents, images, pictures, photos, videos, or audio. A capture request can be a shortcut and / or gesture assigned by an operating system or user. For example, a keyboard shortcut for content capture (e.g., Ctrl+r or Window logo key+r) can be predefined by the operating system of the user's computing device. Additionally or alternatively, a user can assign a gesture as a capture request. For example, a user can indicate that whenever a user takes a screenshot on the user's mobile device, the user wants the screenshot content item to be stored in the content database.

[0038] At operation 206, the content management tool 130 captures the content item and metadata associated with the content item (collectively referred to herein as "content data"). The content data may include the original content item, the date and time when the content item was captured, the type of the captured content item, the file name and / or path of the folder or file associated with the captured content item, the computing device used to capture the content item, the location of the computing device where the content item was captured, and the application used to capture the content item. Additionally, the content data may also include any information extracted from the content item.

[0039] For example, a user may have set a screenshot function on the user's mobile device to capture a request and may obtain a screenshot of a pizza recipe from a web browser application on the user's mobile device that the user wants to try later. The content management tool 130 may then store the content item for the pizza recipe, including text and photos, and determine content data associated with the pizza recipe content item. For example, the content management tool 130 may determine the date and time of capture of the content item, determine the user's mobile device as the device used to capture the content item, and determine the web browser application as the application used to capture the content item. Additionally, the content management tool 130 may also process the photo to extract meaningful information to be stored in the content database, such as the ingredients shown in the photo.

[0040] At operation 208, the content management tool 130 generates one or more semantic embeddings using one or more semantic models (e.g., semantic embedding models, generative large language models, machine learning models, etc.). To this end, the content data is provided to one or more semantic models that generate one or more semantic embeddings. The content management tool 130 extracts abstract meaning from the content data in the form of one or more semantic embeddings. As shown below in Figure 3As further described in , in response to a user query (eg, a natural language query from a user), semantic embeddings of content items are used to find and retrieve one or more stored content items from all previously captured content items.

[0041] As described above, the content database is synchronized between multiple devices of the user so that the user can capture content items from any of the user's computing devices. Depending on the resources, capabilities, and capacity of the computing device, one or more semantic embeddings may be generated from the computing device and / or a server (e.g., 160) comprising one or more semantic models. For example, if a user captures a content item on a user's laptop computer, the content manager tool 130 on the user's laptop computer generates a semantic embedding associated with the captured content item. However, if a user captures a content item on a user's mobile device that has less capability to perform semantic analysis, the captured content item is sent to a server (e.g., 160) to generate a semantic embedding associated with the captured content item. The semantic embedding is then sent back to the user's mobile device to be inserted into the content database.

[0042] Subsequently, at operation 210, the content management tool 130 stores or inserts one or more semantic embeddings in the content database. The semantic embedding can be associated with a corresponding indication that corresponds to a reference to the content data associated with the semantic embedding. In addition, the insertion can trigger a spatial storage operation to store a vector representation of the one or more semantic embeddings. As described above, after the semantic embeddings are stored, multiple sets of stored embeddings can be received from the content database, such as based on a user query. In addition, an action can be determined based on a subset of the set of stored embeddings and a user query. A subset of the set of stored embeddings can be retrieved based on similarity to the user query embedding.

[0043] Subsequently or simultaneously, at operation 212, the content management tool 130 determines whether an action is associated with the content item based on the content item and the content data. In some aspects, the content management tool 130 determines whether an action is associated with the content item based on the capture request for capturing the content item. If the content management tool 130 determines that there is no action associated with the content item, the method 200 can end at operation 218.

[0044] However, if the content management tool 130 determines that there is an action associated with the content item, the method 200 proceeds to operation 216 to perform the associated action. For example, the user may set a rule indicating that whenever the user takes a screenshot of a recipe using the user's mobile device, the user wants the screenshot content item to be stored in the content database and an email with the screenshot content to be sent to the user's email.

[0045] In other examples, the user may set a rule indicating that whenever the user takes a photo of a dog using the user's mobile device, the user wants the photo content item to be stored in the content database and a text message with the dog photo to be sent to a specific friend. Subsequently, method 200 may end at operation 218.

[0046] Now refer to Figure 3 , a method 300 for retrieving a content item from a content item database according to an example of the present disclosure is presented. The general order of steps of the method 300 is as follows: Figure 3 Generally, method 300 begins at 302 and ends at 320. Method 300 may include more or fewer steps, or may be combined with Figure 3 In the illustration, method 300 is performed by a computing device (e.g., user device 120) of user 110. However, it should be understood that one or more steps of method 300 may be performed by another device (e.g., server 160).

[0047] Specifically, in some aspects, method 300 can be performed by a content management tool (e.g., 130) running on user device 120. For example, content management tool 130 can be a clipboard or any other productivity tool with copy and paste or capture and retrieve functionality that is executed on computing device 120. For example, computing device 120 can be, but is not limited to, a computer, a notebook, a laptop, a mobile device, a smart phone, a tablet, a portable device, a wearable device, or any other suitable computing device capable of executing a content management tool (e.g., 130). For example, server 160 can be any suitable computing device capable of communicating with computing device 120. Method 300 can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer-readable medium. In addition, method 300 can be performed by gates or circuits associated with a processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a system on a chip (SOC), or other hardware device. Hereinafter, method 300 will be referred to in conjunction with Figure 1 and Figures 4 to Figure 7 The described systems, components, modules, software, data structures, user interfaces, etc. are explained.

[0048] Method 300 begins at operation 302, where the process can proceed to 304. At operation 304, content management tool 130 receives a retrieval request. The retrieval request can be a shortcut and / or gesture assigned by an operating system or a user. For example, a keyboard shortcut (e.g., Ctrl+v or Window logo key+v) can be predefined by the operating system of the user's computing device. Additionally or alternatively, the user can assign a shortcut or gesture as a retrieval request. In response to receiving the retrieval request, at operation 306, content management tool 130 provides the captured content history (i.e., a list of previously captured content items) in reverse order. As described above, the content database is synchronized between the different devices of the user so that the user can retrieve the captured content items from any of the user's computing devices. However, it should be understood that in some aspects, the content database can be a cloud-based content database shared between multiple devices of the user.

[0049] The user can further search the captured content history to retrieve one or more specific content items. To this end, at operation 308, the content management tool 130 receives a user query from the user to search the captured content history. The user query can be a natural language query. For example, the user can say "I want to find a recipe I saved two months ago." It should be understood that in some aspects, the method 300 can receive a retrieval request at operation 304 and jump forward to operation 308 to receive a user query (e.g., by providing a search bar for the user to enter the user query, or by listening to the user's voice query).

[0050] At operation 310, the content management tool 130 generates an input embedding based on the user query using one or more semantic models (e.g., a semantic embedding model, a generative large language model, a machine learning model, etc.). At operation 312, the content management tool 130 retrieves one or more stored semantic embeddings based on similarity to the input embedding. For example, when a user submits a natural language query, the content management tool 130 can generate an input embedding corresponding to the user query. The abstract meaning of the natural language query in the form of an input embedding is compared with one or more semantic embeddings associated with the content items stored in the content database. By doing so, the content management tool 130 can determine the content data from which the embedding is generated by measuring the quantitative similarity between the stored semantic embedding and the input embedding. The similarity between the content data and the natural language query can provide a semantic context to the computing device when determining whether an action should be performed and / or what action should be performed.

[0051] At operation 314, the content management tool 130 provides search results. Additionally, the user can filter the search results at operation 316. For example, the search results can be filtered based on content data. For example, the user can filter the search results by information associated with the content item, by a specific computing device used to capture the content item, by a specific type of content item, and / or by a specific application used to capture the content item.

[0052] In the example provided above, a user may query the content management tool 130 to find recipes that the user captured approximately two months ago. The content management tool 130 may provide search results for all recipes that were captured within ±3 days of a date two months prior to the query date. The user may also filter the search results to display one or more chicken recipes that were captured on the user's mobile device by taking screenshot images.

[0053] At operation 318, the user may select a specific content item from the search results. The content management tool 130 copies the selected content item and pastes it at the identified location indicated by the user. However, it should be understood that in some aspects, the user may retrieve and view a specific content item from the search results and not necessarily copy it to another location.

[0054] Figure 4A and Figure 4B An overview of an example generative machine learning model that can be used in accordance with aspects described herein is shown. Figure 4A , conceptual diagram 400 depicts an overview of a pre-trained generative model package 404 according to aspects described herein that processes input 402 to generate model output for storing entries in generative model output 406 and / or retrieving information (e.g., suggestions and / or suggested modifications) from generative model output 406.

[0055] In an example, the generative model package 404 is pre-trained based on a variety of inputs (e.g., various human languages, various programming languages, and / or various content types) and therefore does not need to be fine-tuned or trained for a specific scenario. Instead, the generative model package 404 can be pre-trained more generally such that the input 402 includes prompts that are generated, selected, or otherwise designed to induce the generative model package 404 to produce a specific generative model output 406. It should be understood that the input 402 and the generative model output 406 can each include any of a variety of content types, including but not limited to text output, image output, audio output, video output, programmatic output, and / or binary output, among other examples. In an example, the input 402 and the generative model output 406 can have different content types, as may be the case when the generative model package 404 includes a generative multimodal machine learning model.

[0056] Thus, generative model package 404 can be used in any of a variety of scenarios, and further, a different generative model package can be used in place of generative model package 404 without substantially modifying other associated aspects (e.g., similar to the examples described herein with respect to Figures 1 to 3 Thus, generative model package 404 operates as a tool for performing machine learning processing, wherein specific inputs 402 to generative model package 404 are programmatically generated or otherwise determined, causing generative model package 404 to produce model output 406, which can then be used for further processing.

[0057] The generative model package 404 can be provided or otherwise used according to any of a variety of paradigms. For example, the generative model package 404 can be provided or otherwise used in accordance with any of a variety of paradigms. Figure 1 140) locally, or can be used from a machine learning service (e.g., Figure 1 In some embodiments, the generative model package 404 may be accessed remotely from a server 160 in a computing device. In other examples, aspects of the generative model package 404 are distributed across multiple computing devices. In some instances, the generative model package 404 may be accessed via an application programming interface (API), such as may be provided by an operating system of a computing device and / or by a machine learning service, among other examples.

[0058] Referring now to the illustrated aspects of generative model package 404, generative model package 404 includes input tokenization 408, input embedding 410, model layer 412, output layer 414, and output decoding 416. In the example, input tokenization 408 processes input 402 to generate input embedding 410, which includes a sequence of symbol representations corresponding to input 402. Thus, input embedding 410 is processed by model layer 412, output layer 414, and output decoding 416 to produce model output 406. An example architecture corresponding to generative model package 404 is shown in FIG. Figure 4B , which example architecture is discussed in further detail below. Even so, it should be understood that the architecture shown and described herein should not be considered limiting, and in other examples, any of a variety of other architectures may be used.

[0059] Figure 4B 4 is a conceptual diagram depicting an example architecture 450 of a pre-trained generative machine learning model that can be used according to aspects described herein. As described above, any of a variety of alternative architectures and corresponding ML models can be used in other examples without departing from the aspects described herein.

[0060] As shown, the architecture 450 processes the input 402 to produce a generative model output 406, aspects of which are described above with respect to Figure 4A The architecture 450 is depicted as a transformer model comprising an encoder 452 and a decoder 454. The encoder 452 processes an input embedding 458 (which in various aspects may be similar to Figure 4A ), the input embedding includes a sequence of symbolic representations corresponding to the input 456. In an example, the input 456 includes content data 402 corresponding to a content item.

[0061] Furthermore, position encoding 460 can incorporate information about the relative and / or absolute positions of the tokens of input embedding 458. Similarly, output embedding 474 includes a sequence of symbol representations corresponding to output 472, and position encoding 476 can similarly incorporate information about the relative and / or absolute positions of the tokens of output embedding 474.

[0062] As shown, encoder 452 includes an example layer 470. It should be understood that any number of such layers can be used, and the architecture depicted is simplified for illustrative purposes. Example layer 470 includes two sublayers: a multi-head attention layer 462 and a feed-forward layer 466. In the example, residual connections are included around each layer 462, 466, followed by a normalization layer 464 and a normalization layer 468, respectively.

[0063] The decoder 454 includes an example layer 490. Similar to the encoder 452, any number of such layers can be used in other examples, and the architecture of the decoder 454 depicted is simplified for illustrative purposes. As shown, the example layer 490 includes three sublayers: a masked multi-headed attention layer 478, a multi-headed attention layer 482, and a feedforward layer 486. Various aspects of the multi-headed attention layer 482 and the feedforward layer 486 can be similar to those discussed above with respect to the multi-headed attention layer 462 and the feedforward layer 466, respectively. Additionally, the masked multi-headed attention layer 478 performs multi-headed attention on the output of the encoder 452 (e.g., output 472). In the example, the masked multi-headed attention layer 478 prevents a position from paying attention to subsequent positions. This mask combined with an offset embedding (e.g., offsetting one position, as shown in the multi-headed attention layer 482) can ensure that the prediction for a given position depends on the known output of one or more positions smaller than the given position. As shown, residual connections are also included around layers 478, 482, and 486, followed by normalization layers 480, 484, and 488, respectively.

[0064] The multi-head attention layer 462, the multi-head attention layer 478, and the multi-head attention layer 482 can each use a set of linear projections to linearly project the query, key, and value into corresponding dimensions. Each linear projection can be processed using an attention function (e.g., dot product or additive attention) to produce an n-dimensional output value for each linear projection. Figure 4B As shown (eg, by corresponding normalization layer 464, normalization layer 480, or normalization layer 484), the resulting values ​​may be concatenated and projected again so that the values ​​are subsequently processed.

[0065] Feed-forward layer 466 and feed-forward layer 486 can each be a fully connected feed-forward network applied to each position. In an example, feed-forward layer 466 and feed-forward layer 486 each include multiple linear transformations with rectified linear unit activations between them. In an example, each linear transformation is the same across different positions, but can use different parameters than other linear transformations in the feed-forward network.

[0066] Additionally, aspects of the linear transformation 492 can be similar to the linear transformations discussed above with respect to the multi-head attention layers 462, 478, and 482, as well as the feed-forward layers 466 and 486. A normalized exponential function (Softmax) 494 can also convert the output of the linear transformation 492 into a predicted next token probability, as shown in output probabilities 496. It should be understood that the illustrated architecture is provided as an example, and in other examples, any of a variety of other model architectures may be used in accordance with the disclosed aspects.

[0067] Thus, the output probability 496 can form a generative model output 406 according to aspects described herein, such that the output of the generative ML model (e.g., which can include one or more semantic embeddings and one or more retrieved content items) is used as input for determining an action according to aspects described herein. In other examples, the generative model output 406 is provided as a generated output for retrieving one or more previously captured content items.

[0068] Figures 5 to 7 and the associated descriptions provide a discussion of various operating environments in which aspects of the present disclosure may be practiced. Figures 5 to 7 The devices and systems shown and discussed are for purposes of example and explanation, and are not limiting of the wide variety of computing device configurations that can be used to practice the various aspects of the disclosure described herein.

[0069] Figure 5is a block diagram illustrating the physical components (e.g., hardware) of a computing device 500 that can practice various aspects of the present disclosure. The computing device components described below can be applicable to the computing devices described above, including one or more devices associated with the machine learning service (e.g., production platform server 160), as well as the above-described computing devices. Figure 1 The computing device 140 discussed. In a basic configuration, the computing device 500 may include at least one processing unit 502 and system memory 504. Depending on the configuration and type of the computing device, the system memory 504 may include, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of these memories.

[0070] The system memory 504 may include an operating system 505 and one or more program modules 506 suitable for running software applications 520, such as one or more components supported by the system described herein. As an example, the system memory 504 may store a content capture manager 521 and / or a content retrieval manager 522. The operating system 505 may, for example, be suitable for controlling the operation of the computing device 500.

[0071] Furthermore, aspects of the present disclosure may be practiced in conjunction with graphics libraries, other operating systems, or any other application programs and are not limited to any particular application or system. Figure 5 508. The computing device 500 may have additional features or functionality. For example, the computing device 500 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 5 , which is illustrated by a removable storage device 509 and a non-removable storage device 510.

[0072] As described above, a number of program modules and data files may be stored in system memory 504. When executed on processing unit 502, program modules 506 (e.g., applications 520) may perform processes including, but not limited to, the various aspects described herein. Other program modules that may be used in accordance with various aspects of the present disclosure may include email and contact applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-assisted applications, and the like.

[0073] Furthermore, aspects of the present disclosure may be practiced on circuits comprising discrete electronic elements, packaged or integrated electronic chips comprising logic gates, circuits utilizing a microprocessor, or a single chip comprising electronic elements or a microprocessor. For example, aspects of the present disclosure may be practiced via a system on a chip (SOC) wherein Figure 5Each or many components shown in can be integrated into a single integrated circuit. Such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit and various application functions, all of which are integrated (or "burned") onto a chip substrate as a single integrated circuit. When operated via the SOC, the functions described herein regarding the ability to switch protocols for the client can be operated via dedicated logic integrated on a single integrated circuit (chip) with other components of the computing device 500. Various aspects of the present disclosure may also be practiced using other technologies capable of performing logical operations (e.g., AND, OR, and NOT), including but not limited to mechanical, optical, fluidic, and quantum technologies. In addition, various aspects of the present disclosure may be practiced within a general-purpose computer or in any other circuit or system.

[0074] The computing device 500 may also have one or more input devices 512, such as a keyboard, a mouse, a pen, an audio or voice input device, a touch or slide input device, etc. It may also include output devices 514, such as a display, a speaker, a printer, etc. The above devices are examples, and other devices may be used. The computing device 500 may include one or more communication connections 516 that allow communication with other computing devices 550. Examples of suitable communication connections 516 include, but are not limited to, radio frequency (RF) transmitters, receivers, and / or transceiver circuit systems; universal serial bus (USB), parallel and / or serial ports.

[0075] The term computer-readable media as used herein may include computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures or program modules. System memory 504, removable storage device 509 and non-removable storage device 510 are all examples of computer storage media (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article that can be used to store information and can be accessed by computing device 500. Any such computer storage media may be part of computing device 500. Computer storage media does not include carrier waves or other propagated or modulated data signals.

[0076] Communication media may be embodied by computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0077] Figure 6 A system 600 is shown, which can be, for example, a mobile computing device, such as a mobile phone, a smart phone, a wearable computer (such as a smart watch), a tablet computer, a laptop computer, etc., with which aspects of the present disclosure can be practiced. In one example, the system 600 is implemented as a "smart phone" capable of running one or more applications (e.g., a browser, email, a calendar, a contact manager, a messaging client, a game, and a media client / player). In some aspects, the system 600 is integrated into a computing device, such as an integrated personal digital assistant (PDA) and a wireless phone.

[0078] In a basic configuration, such a mobile computing device is a handheld computer having both input and output elements. The system 600 typically includes a display 605 and one or more input buttons that allow a user to enter information into the system 600. The display 605 can also be used as an input device (e.g., a touch screen display).

[0079] If an optional bypass input element is included, it allows further user input. For example, the bypass input element can be a rotary switch, a button, or any other type of manual input element. In alternative aspects, the system 600 can include more or fewer input elements. For example, in some aspects, the display 605 can be other than a touch screen. In another example, an optional keypad 635 can also be included, which can be a physical keypad or a "soft" keypad generated on the touch screen display.

[0080] In various aspects, output elements include a display 605 for showing a graphical user interface (GUI), a visual indicator (e.g., a light emitting diode 620), and / or an audio transducer 625 (e.g., a speaker). In some aspects, a vibration transducer is included to provide tactile feedback to the user. In yet another aspect, input and / or output ports are included, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., an HDMI port) for sending or receiving signals to or from an external device.

[0081] One or more application programs 666 can be loaded into memory 662 and run on or in association with operating system 664. Examples of application programs include a phone dialer, an email program, a personal information management (PIM) program, a word processing program, a spreadsheet program, an Internet browser program, a messaging program, and the like. System 600 also includes a non-volatile storage area 668 within memory 662. Non-volatile storage area 668 can be used to store persistent information that should not be lost when system 600 loses power. Application programs 666 can use and store information in non-volatile storage area 668, such as emails or other messages used by an email application. A synchronization application (not shown) also resides on system 600 and is programmed to interact with a corresponding synchronization application residing on a host computer to synchronize information stored in non-volatile storage area 668 with corresponding information stored on the host computer. It should be understood that other applications can be loaded into memory 662 and run on system 600 as described herein (e.g., a content capture manager, a content retrieval manager, and the like).

[0082] The system 600 has a power source 670, which can be implemented as one or more batteries. The power source 670 can also include an external power source, such as an AC adapter or a powered docking station to replenish or recharge the batteries.

[0083] System 600 may also include a radio interface layer 672 that performs the functions of transmitting and receiving radio frequency communications. Radio interface layer 672 facilitates wireless connectivity between system 600 and the "outside world" via a communications carrier or service provider. Transmissions to and from radio interface layer 672 are controlled by operating system 664. In other words, communications received by radio interface layer 672 can be passed to application 666 via operating system 664, and vice versa.

[0084] The visual indicator 620 can be used to provide a visual notification, and / or the audio interface 674 can be used to generate an audible notification via the audio transducer 625. In the example shown, the visual indicator 620 is a light emitting diode (LED), and the audio transducer 625 is a speaker. These devices can be directly coupled to the power supply 670 so that when activated, they remain on for the duration specified by the notification mechanism, even if the processor 660 and other components may be turned off to save battery power. The LED can be programmed to remain on indefinitely until the user takes action to indicate the device's power-on status. The audio interface 674 is used to provide audible signals to the user and receive audible signals from the user. For example, in addition to being coupled to the audio transducer 625, the audio interface 674 can also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. According to aspects of the present disclosure, the microphone can also be used as an audio sensor to facilitate control of notifications, as described below. The system 600 can also include a video interface 676, which enables the operation of the onboard camera 630 to record still images, video streams, and the like.

[0085] It should be understood that the system 600 may have additional features or functionality. For example, the system 600 may also include additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 6 , shown by non-volatile storage area 668.

[0086] As described above, data / information generated or captured and stored via system 600 may be stored locally, or the data may be stored on any number of storage media that can be accessed by a device via radio interface layer 672 or via a wired connection between system 600 and a separate computing device associated with system 600 (e.g., a server computer in a distributed computing network such as the Internet). It should be understood that such data / information can be accessed via radio interface layer 672 or via a distributed computing network. Similarly, such data / information can be readily transferred between computing devices for storage and use according to any of a variety of data / information transmission and storage means, including email and collaborative data / information sharing systems.

[0087] As mentioned above, Figure 7 7. The diagram illustrates one aspect of the architecture of a system for processing data received at a computing system from a remote source, such as a personal computer 704, a tablet computing device 706, or a mobile computing device 708. The content displayed at the server device 702 may be stored in different communication channels or other storage types. For example, various documents may be stored using a directory service 724, a web portal 725, a mailbox service 726, an instant messaging store 728, or a social networking site 730.

[0088] Application 720 (e.g., similar to application 520) can be employed by a client communicating with server device 702. Additionally or alternatively, a content capture manager 791 and / or a content retrieval manager 792 can be employed by server device 702. Server device 702 can provide data to and from client computing devices, such as personal computers 704, tablet computing devices 706, and / or mobile computing devices 708 (e.g., smartphones), via network 715. As examples, the computer systems described above can be embodied in personal computers 704, tablet computing devices 706, and / or mobile computing devices 708 (e.g., smartphones). In addition to receiving graphics data that can be pre-processed at a graphics source system or post-processed at a receiving computing system, any of these examples of computing devices can retrieve content from storage 716.

[0089] It should be understood that the aspects and functions described herein can operate on a distributed system (e.g., a cloud-based computing system) where application functions, memory, data storage and retrieval, and various processing functions can operate remotely from each other on a distributed computing network (such as the Internet or an intranet). Various types of user interfaces and information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information can be displayed on and interacted with a wall surface on which various types of user interfaces and information are projected. Interactions with multiple computing systems that can practice various aspects of the present disclosure include: key input, touch screen input, voice or other audio input, gesture input, where the associated computing device is equipped with detection (e.g., camera) capabilities for capturing and interpreting user gestures to control the functions of the computing device, and the like.

[0090] For example, various aspects of the present disclosure are described above with reference to block diagrams and / or operational diagrams of methods, systems, and computer program products according to various aspects of the present disclosure. The functions / actions indicated in the blocks may not occur in the order shown in any flowchart. For example, depending on the functions / actions involved, two blocks shown in succession may actually be executed substantially simultaneously, or the blocks may sometimes be executed in the reverse order.

[0091] The description and explanation of one or more aspects provided in this application are not intended to limit or restrict the scope of this disclosure in any way. The aspects, examples and details provided in this application are considered to be sufficient to convey the claimed aspects of having and enabling others to make and use this disclosure. The disclosure claimed should not be interpreted as being limited to any aspect, example or details provided in this application. No matter whether it is shown and described in combination or individually, various features (both structure and method) are intended to be selectively included or omitted to produce aspects with a specific feature set. Having provided the description and explanation of this application, those skilled in the art can envision variations, modifications and alternative aspects within the spirit of the broader aspects of the overall inventive concept embodied in this application, which variations, modifications and alternative aspects do not depart from the broader scope of the disclosure claimed.

[0092] In addition, the aspects and functions described herein can be operated on a distributed system (e.g., a cloud-based computing system), where application functions, memory, data storage and retrieval, and various processing functions can operate remotely from each other on a distributed computing network (such as the Internet or an intranet). Various types of user interfaces and information can be displayed via an onboard computing device display or via a remote display unit associated with one or more computing devices. For example, various types of user interfaces and information can be displayed on and interacted with a wall surface on which various types of user interfaces and information are projected. Interactions with multiple computing systems that can practice various aspects of the present disclosure include: key input, touch screen input, voice or other audio input, gesture input, where the associated computing device is equipped with detection (e.g., camera) capabilities for capturing and interpreting user gestures to control the functions of the computing device, and the like.

[0093] The phrases "at least one," "one or more," "or," and "and / or" are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions "at least one of A, B, and C," "at least one of A, B, or C," "one or more of A, B, and C," "one or more of A, B, or C," "A, B and / or C," and "A, B, or C" means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B, and C together.

[0094] The term "a" or "an" entity refers to one or more of that entity. Thus, the terms "a" (or "an"), "one or more," and "at least one" can be used interchangeably herein. It should also be noted that the terms "including," "comprising," and "having" can be used interchangeably.

[0095] As used herein, the term "automatic" and its variations refer to any process or operation, typically continuous or semi-continuous, that is performed without substantial human input. However, if input is received prior to the execution of the process or operation, the process or operation may be automatic, even if the execution of the process or operation uses material or insignificant human input. Human input is considered significant if such input affects how the process or operation will be performed. Human input that consents to the execution of the process or operation is not considered "material."

[0096] Any of the steps, functions, and operations discussed herein may be performed continuously and automatically.

[0097] The example systems and methods of the present disclosure have been described with respect to computing devices. However, to avoid unnecessarily obscuring the present disclosure, the foregoing description omits several known structures and devices. This omission should not be construed as limiting. Specific details are set forth to provide an understanding of the present disclosure. However, it should be understood that the present disclosure can be practiced in a variety of ways beyond the specific details set forth herein.

[0098] Furthermore, while the example aspects illustrated herein show various components of the system being collocated, certain components of the system may be located in remote, distributed portions of a network (such as a LAN and / or the Internet) or within dedicated systems. Accordingly, it should be understood that the components of the system may be combined into one or more devices, such as servers, communications equipment, or collocated at specific nodes of a distributed network, such as an analog and / or digital telecommunications network, a packet-switched network, or a circuit-switched network. It will be understood from the foregoing description, and for reasons of computational efficiency, that the components of the system may be arranged at any location within the distributed network of components without affecting the operation of the system.

[0099] In addition, it should be understood that the various links connecting the elements can be wired or wireless links, or any combination thereof, or any other known or later developed element(s) capable of providing data to the connected elements and / or communicating data from the connected elements. These wired or wireless links can also be secure links and can communicate encrypted information. The transmission medium used as the link can be, for example, any suitable carrier for electrical signals, including coaxial cable, copper wire, and optical fiber, and can take the form of sound waves or light waves, such as those generated during radio wave and infrared data communications.

[0100] While the flow diagrams have been discussed and shown with respect to a particular sequence of events, it should be understood that changes, additions, and omissions to this sequence can occur without materially affecting the operation of the disclosed configurations and aspects.

[0101] Several variations and modifications of the present disclosure may be used. Some features of the present disclosure may be provided without providing others.

[0102] In another configuration, the systems and methods of the present disclosure can be implemented in conjunction with a dedicated computer, a programmed microprocessor or microcontroller and (multiple) peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, a hard-wired electronic or logic circuit such as a discrete element circuit, a programmable logic device or gate array such as a PLD, PLA, FPGA, PAL, a dedicated computer, any similar component, etc. In general, any device or component capable of implementing the methods shown herein can be used to implement the various aspects of the present invention. Example hardware that can be used for the present disclosure includes computers, handheld devices, phones (e.g., cellular, Internet-enabled, digital, analog, hybrid, etc.), and other hardware known in the art. Some of these devices include processors (e.g., single or multiple microprocessors), memory, non-volatile storage, input devices, and output devices. In addition, alternative software implementations including, but not limited to, distributed processing or component / object distributed processing, parallel processing, or virtual machine processing can also be constructed to implement the methods described herein.

[0103] In yet another configuration, the disclosed method can be readily implemented in conjunction with software using an object or object-oriented software development environment that provides portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed system can be implemented in partial or complete hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement a system according to the present disclosure depends on the speed and / or efficiency requirements of the system, the specific functionality, and the specific software or hardware system or microprocessor or microcomputer system being utilized.

[0104] In yet another configuration, the disclosed methods may be implemented in part in software that may be stored on a storage medium and executed on a programmed general-purpose computer using the cooperation of a controller and memory, a dedicated computer, a microprocessor, etc. In these cases, the disclosed systems and methods may be implemented as a program embedded on a personal computer, such as an applet, or CGI scripts, resources resident on a server or computer workstation, routines embedded in a dedicated measurement system, system components, etc. The system may also be implemented by physically incorporating the system and / or method into a software and / or hardware system.

[0105] The present disclosure is not limited to the standards and protocols described. Other similar standards and protocols not mentioned herein exist and are included in the present disclosure. In addition, the standards and protocols mentioned herein, as well as other similar standards and protocols not mentioned herein, are periodically replaced by faster or more efficient equivalents having substantially the same functionality. Such replacement standards and protocols having the same functionality are considered to be equivalents included in the present disclosure.

[0106] According to at least one example of the present disclosure, a method for capturing content items into a content database is provided. The method may include receiving a capture request to capture a content item, capturing the content item and metadata associated with the content item, and generating one or more semantic embeddings of the content data using one or more semantic models. The content data may include the content item and the metadata. The method may also include storing the one or more semantic embeddings in the content database and providing a captured content history stored in the content database. The captured content history may include a list of captured content items. The method may also include receiving a user query to search the captured content history and providing search results based on the user query.

[0107] According to at least one aspect of the above method, the method may include determining whether an action associated with the content item exists based on the content data, and performing the action associated with the content item in response to determining that the action exists.

[0108] According to at least one aspect of the above method, the method may include wherein the content item includes one or more text, documents, images, pictures, photos, videos, or audio.

[0109] According to at least one aspect of the above method, the method may include where the metadata includes the date and time when the content item was captured, the type of content item captured, the file name and path of the file associated with the captured content item, the computing device used to capture the content item, the location of the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item.

[0110] According to at least one aspect of the above method, the method may include wherein the user query is a natural language query.

[0111] According to at least one aspect of the above method, the method may include where providing search results based on the user query includes: generating an input embedding based on the user query, and retrieving semantic embeddings associated with one or more previously captured content items from a content database based on similarity to the input embedding.

[0112] According to at least one aspect of the above method, the method may further include filtering the search results based on content data associated with content items in the search results.

[0113] According to at least one aspect of the above method, the method may include wherein the one or more semantic models include a generative large language model (LLM).

[0114] According to at least one example of the present disclosure, a computing device for generating design difference description text is provided. The computing device may include a processor and a memory, the memory having a plurality of instructions stored thereon, the plurality of instructions, when executed by the processor, causing the computing device to: receive a capture request to capture a content item; capture the content item and metadata associated with the content item; generate one or more semantic embeddings of the content data using one or more semantic models, the content data including the content item and the metadata; store the one or more semantic embeddings in a content database; provide a captured content history stored in the content database, the captured content history including a list of captured content items; receive a user query to search the captured content history; and provide search results based on the user query.

[0115] According to at least one aspect of the above computing device, the computing device may include multiple instructions, which, when executed, further enable the computing device to: determine whether an action associated with the content item exists based on the content data; and perform the action associated with the content item in response to determining that the action exists.

[0116] According to at least one aspect of the above computing device, the computing device may include where the content item includes one or more text, documents, images, pictures, photos, videos, or audio.

[0117] According to at least one aspect of the above computing device, the computing device may include metadata including the date and time when the content item was captured, the file name and path of the file associated with the captured content item, the computing device used to capture the content item, the location of the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item.

[0118] According to at least one aspect of the above computing device, the computing device may include where providing search results based on a user query includes: generating an input embedding based on the user query, and retrieving semantic embeddings associated with one or more previously captured content items from a content database based on similarity to the input embedding.

[0119] According to at least one aspect of the above computing device, the computing device may include a plurality of instructions that, when executed, further cause the computing device to filter search results based on content data associated with content items in the search results.

[0120] According to at least one aspect of the computing device above, the computing device may include wherein the one or more semantic models include a generative large language model (LLM).

[0121] According to at least one example of the present disclosure, a non-transitory computer-readable medium storing instructions for generating design difference description text is provided. When executed by one or more processors of a computing device, the instructions cause the computing device to: receive a capture request to capture a content item; capture the content item and metadata associated with the content item; generate one or more semantic embeddings of the content data using one or more semantic models, the content data including the content item and the metadata; store the one or more semantic embeddings in a content database; provide a captured content history stored in the content database, the captured content history including a list of captured content items; receive a user query to search the captured content history; and provide search results based on the user query.

[0122] According to at least one aspect of the above non-transitory computer-readable medium, when the instructions are executed by one or more processors, the computing device may also: determine whether an action associated with the content item exists based on the content data, and in response to determining that the action exists, perform the action associated with the content item.

[0123] According to at least one aspect of the above non-transitory computer-readable medium, the instructions, when executed by one or more processors of a computing device, may include: wherein the content item includes one or more text, documents, images, pictures, photos, videos, or audio, and wherein the metadata includes the date and time when the content item was captured, the file name and path of the file associated with the captured content item, the computing device used to capture the content item, the location of the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item.

[0124] According to at least one aspect of the above non-transitory computer-readable medium, the instructions, when executed by one or more processors of a computing device, may include: wherein providing search results based on a user query includes generating an input embedding based on the user query, and retrieving semantic embeddings associated with one or more previously captured content items from a content database based on similarity to the input embedding.

[0125] According to at least one aspect of the above non-transitory computer-readable medium, the instructions, when executed by one or more processors, further cause the computing device to filter the search results based on content data associated with content items in the search results.

[0126] In various configurations and aspects, the present disclosure includes components, methods, processes, systems and / or devices substantially as depicted and described herein, including various combinations, subcombinations and subsets thereof. After understanding the present disclosure, those skilled in the art will understand how to make and use the systems and methods disclosed herein. In various configurations and aspects, the present disclosure includes providing devices and processes in the absence of items not depicted and / or described herein or in various configurations or aspects herein, including in the absence of such items that may have been used in previous devices or processes, for example, to improve performance, achieve convenience and / or reduce implementation costs.

Claims

1. A method for capturing a content item into a content database, the method comprising: receiving a capture request to capture a content item; capturing the content item and metadata associated with the content item; generating one or more semantic embeddings for content data, the content data comprising the content item and the metadata, using one or more semantic models; storing the one or more semantic embeddings in a content database; providing a captured content history stored in the content database, the captured content history comprising a list of captured content items; receiving a user query to search the captured content history; and Search results are provided based on the user query.

2. The method according to claim 1, further comprising: determining, based on the content data, whether an action associated with the content item exists; as well as In response to determining that the action exists, performing the action associated with the content item. 3 . The method of claim 1 , wherein the content item comprises one or more of text, document, image, picture, photo, video or audio.

4. The method of claim 1 , wherein the metadata includes a date and time when the content item was captured, the type of the content item captured, a file name and path of a file associated with the captured content item, a computing device used to capture the content item, a location of the computing device on which the content item was captured, an application used to capture the content item, and any information extracted from the content item. The method of claim 1 , wherein the user query is a natural language query.

6. The method of claim 1 , wherein providing the search results based on the user query comprises: generating an input embedding based on the user query; as well as Semantic embeddings associated with one or more previously captured content items are retrieved from the content database based on similarity to the input embedding.

7. The method of claim 1, further comprising filtering the search results based on the content data associated with content items in the search results.

8. The method of claim 1, wherein the one or more semantic models comprises a generative large language model (LLM).

9. A computing device for generating a description of a design difference, the computing device comprising: processor; as well as a memory having a plurality of instructions stored thereon that, when executed by the processor, cause the computing device to: receiving a capture request to capture a content item; capturing the content item and metadata associated with the content item; generating one or more semantic embeddings for content data, the content data comprising the content item and the metadata, using one or more semantic models; storing the one or more semantic embeddings in a content database; providing a captured content history stored in the content database, the captured content history comprising a list of captured content items; receiving a user query to search the captured content history; and Search results are provided based on the user query.

10. The computing device of claim 9, wherein the plurality of instructions, when executed, further cause the computing device to: determining, based on the content data, whether an action associated with the content item exists; and In response to determining that the action exists, performing the action associated with the content item.

11. The computing device of claim 9, wherein the content item comprises one or more of text, documents, images, pictures, photos, videos, or audio.

12. A computing device according to claim 9, wherein the metadata includes the date and time when the content item was captured, the file name and path of the file associated with the captured content item, the computing device used to capture the content item, the location of the computing device where the content item was captured, the application used to capture the content item, and any information extracted from the content item.

13. The computing device of claim 9, wherein providing the search results based on the user query comprises: generating an input embedding based on the user query; as well as Semantic embeddings associated with one or more previously captured content items are retrieved from the content database based on similarity to the input embedding.

14. The computing device of claim 9, wherein the plurality of instructions, when executed, further cause the computing device to filter the search results based on the content data associated with content items in the search results.

15. The computing device of claim 9, wherein the one or more semantic models comprise a generative large language model (LLM).

16. A computer-readable medium for generating a design difference description, the computer-readable medium storing instructions that, when executed by one or more processors of a computing device, cause the computing device to: receiving a capture request to capture a content item; capturing the content item and metadata associated with the content item; generating one or more semantic embeddings for content data, the content data comprising the content item and the metadata, using one or more semantic models; storing the one or more semantic embeddings in a content database; providing a captured content history stored in the content database, the captured content history comprising a list of captured content items; receiving a user query to search the captured content history; and Search results are provided based on the user query.

17. The computer-readable medium of claim 16, wherein the instructions, when executed by the one or more processors, further cause the computing device to: determining, based on the content data, whether an action associated with the content item exists; and In response to determining that the action exists, performing the action associated with the content item.

18. The computer-readable medium of claim 16, wherein the content item comprises one or more text, documents, images, pictures, photographs, videos, or audio, and wherein the metadata comprises a date and time when the content item was captured, a file name and path of a file associated with the captured content item, a computing device used to capture the content item, a location of the computing device at which the content item was captured, an application used to capture the content item, and any information extracted from the content item.

19. The computer-readable medium of claim 16, wherein providing the search results based on the user query comprises: generating an input embedding based on the user query; as well as Semantic embeddings associated with one or more previously captured content items are retrieved from the content database based on similarity to the input embedding.

20. The computing device of claim 9, wherein the plurality of instructions, when executed, further cause the computing device to filter the search results based on the content data associated with content items in the search results.

Images