Language model support provided by the operating system

Abstract

The method may receive a content selection. In response to receiving a selection, the method may provide a prompt based on the selection and the context associated with the selection. The method may display the prompt. In response to receiving a prompt selection, the method may generate an output by providing the content and prompt as input to a language model. The method may display the output.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of U.S. Provisional Application No. 63 / 501,100, filed on May 9, 2023, the entire disclosure of which is incorporated herein by reference.

Background Art

[0002] Currently, there are numerous applications that assist users in interacting with text and content, such as word processors, messaging, and email - sending applications. In some applications, at the time of user input, they propose corrections for misspellings and / or short phrases within long text or provide short snippets of predicted text. In other applications, they provide access to generative language models for users who need to provide prompts and context to the model.

Summary of the Invention

[0003] This disclosure describes a method for providing assistance in accessing and using generative language modeling tools within the operating system environment to help users create content, modify content, and obtain descriptions of content, without leaving the application window (i.e., without the application window losing focus), and regardless of whether the application itself supports generative language modeling tools. When content is selected within an application window, one or more prompts are determined and displayed. In one example, the prompts are generated based on the selected content and the context related to the selection. Other information available through the application or operating system may also be used to generate prompts. Upon receiving an input, which is a selection for a prompt, one or more outputs may be generated using the generative language model. When the user selects an output, the user may be presented with further prompts to modify the content that may be generated and presented. In another example, upon receiving a selection of content in an application, a prompt providing a description of the selected content may be generated and displayed.

[0004] In some embodiments, the techniques described herein relate to methods, the methods comprising: receiving a selection of content; providing a prompt in response to receiving the selection based on the selection and the context relating to the selection; displaying the prompt; generating an output by providing the content and the prompt as input to a language model in response to receiving a selection of the prompt; and displaying the output.

[0005] In some embodiments, the techniques described herein relate to a method, which includes receiving a selection of content, displaying options for generating a description of the selection in response to receiving the selection of content, generating the description by providing the selection of content and the options as input to a language model, and displaying the description.

[0006] In some embodiments, the techniques described herein relate to a system, the system including a processor and a memory containing instructions for receiving a selection of content, providing prompts based on the selection and context related to the selection, displaying prompts, and generating and displaying output by providing the content and prompts as input to a language model in response to receiving a selection of prompts.

[0007] In some embodiments, the techniques described herein relate to a system, the system including a processor and a memory containing instructions for receiving a selection of content, and in response to receiving the selection of content, for generating an explanation of the selection, and for generating and displaying the explanation by providing the selection of content and the options as input to a language model.

[0008] In some embodiments, the technology described herein relates to a computing device, which includes at least one processor and a non-temporary computer-readable medium storing executable instructions, which, when executed by at least one processor, causes the computing device to receive a selection of content,, in response to receiving the selection, provide a prompt based on the selection and the context associated with the selection, display the prompt, and, in response to receiving a selection of the prompt, generate an output by providing the content and the prompt as input to a language model, and display the output.

[0009] In some embodiments, the technology described herein relates to a computing device, which includes at least one processor and a non-temporary computer-readable medium for storing executable instructions, wherein, when executed by at least one processor, the computing device causes the computing device to receive a selection of content, to display options for generating a description of the selection in response to receiving the selection of content, and to generate and display a description by providing the selection of content and the options as input to a language model. [Brief explanation of the drawing]

[0010] [Figure 1A] An example window is shown. [Figure 1B] An example window is shown. [Figure 1C] An example window is shown. [Figure 1D] An example window is shown. [Figure 2] An example of a system is shown. [Figure 3A] An example window is shown. [Figure 3B] An example window is shown. [Figure 4A] An example window is shown. [Figure 4B] An example window is shown. [Figure 4C] An example window is shown. [Figure 4D] An example window is shown. [Figure 5A] An example menu is shown. [Figure 5B] An example menu is shown. [Figure 5C] An example menu is shown. [Figure 6A] An example of the method is shown. [Figure 6B] An example of the method is shown. [Modes for carrying out the invention]

[0011] Generating clear and meaningful content and communication requires significant user time and effort. Newly available language models allow users to input prompts and automatically generate content. However, to obtain useful output from these language models, users need to be familiar with the model's prompt design. Furthermore, new language model-based tools are often accessed as standalone applications, decoupled from the context of other tools or applications the user is using.

[0012] A language model (e.g., a generative language model) is a type of machine learning model that uses deep learning to generate human-like text or speech based on prompts. Language models are typically trained on vast amounts of data in text or speech form, and this data can be used to predict text output. Language models typically run on a server.

[0013] A language model's prompts can include commands, questions, or any other type of input, depending on the intended use of the model. A prompt might include a question that the user wants answered by running the language model, such as "What is the definition of this word?"

[0014] A prompt context may be provided to a language model along with a prompt to generate output. The prompt context can be any data (e.g., text, image files, audio files, embeddings, etc.) that helps the language model better interpret the prompt and / or generate a more accurate response. The prompt context may include data representing the results of past rounds or turns using the model (e.g., past prompts and past responses generated by the model). The prompt context provided with a prompt is typically not provided by the user.

[0015] For the purposes of the present disclosure, the window context includes any data that describes a setting where assistance from a large language model is required, including the content of the application window that currently has focus. This window context can represent the content within the application window that is displayed near the selection of the content. The window context can represent settings or attributes from the user profile, upon obtaining the user's permission. The window context can represent data obtained from the operating system environment, upon obtaining the user's permission. When the application window includes a browser window, the window context can include data that describes the source (domain) associated with the window. The window context can include data that describes editable user interface elements (e.g., the maximum number of characters in a text box). Although examples of text content are used herein, this is not intended to be limiting. In further examples, any type of content can be used as window content.

[0016] It is a technical problem that the quality of the language model output depends on the quality of the prompt used as input. However, few users are proficient in how to input a prompt into a language model to generate valuable output. Therefore, users may give up content generation using a language model tool after entering one or two prompts and may not be able to maximize the functionality of the language model to improve productivity.

[0017] A further technical challenge regarding the use of language models is that the output provided by the language model depends on the quality of the context provided to the model. However, the prompt context has conventionally not been provided by the user. Even when the user can provide a window context, providing the window context requires user-side knowledge, but the user may not understand which window context may be beneficial to the model or where the window context can be found.

[0018] The present disclosure provides a technical solution that includes using additional data available on the user device (context regarding the application and / or operating system environment, referred to as device context) to generate more targeted prompts. Further, the present disclosure describes using at least a portion of the additional data available on the user device, i.e., the device context, as a prompt context (the context provided to the language model) to improve the quality of the language model output. Finally, the present disclosure describes executing a language model on the user device such that the window context can be used while respecting privacy. Each of these technical solutions can improve the interaction between humans and machines by reducing the steps required to achieve a task and improving the quality of the output generated by the language model.

[0019] Figures 1A and 1B show an application window 100 containing content for an email application. In this example, the user is creating an email draft and highlighting a portion of it. As a result, the email draft includes both selected content 102 and unselected content 103. The method described herein may provide a prompt (e.g., a prompt suggestion section 112 in menu 108) that generates content using a language model. The prompt may be provided based on the selected content 102 and further window context. Further window context may include the unselected content 103. Further window context may be related to an application (e.g., an email application, the title or domain of the application). The method described herein may also further provide an option to generate a description of the selected content 102 via a language model.

[0020] Figure 2 shows a system 200 relating to an example. System 200 includes device 201. In this example, system 200 may further include server 250. Device 201 may be used by an end user who seeks to generate, modify, or learn more about content when using an application that displays and / or allows the user to interact with the content (e.g., text). In this example, device 201 may be a desktop or laptop computer, a handheld computer, a tablet, a smartphone, or any other type of user device.

[0021] In addition to the following description, the systems, programs, or functions described herein may provide the user with control over whether and when to enable the use of data from device 201, including user information (e.g., information about the user's computing activity, user preferences, and information about the user's data), in order to identify contexts that can be used to create prompts and generate output from language models. Some data may be processed in one or more ways before use so that personally identifiable information is removed. In this way, the user may have control over which information from device 201 is used, how that information is used, and, if any, which information is sent to the server.

[0022] Device 201 includes a processor 202, memory 204, communication interface 206, display 208, operating system 210, interactive content application 212, context selector 214, prompt generator 216, language model 218, rewrite module 220, and description module 222.

[0023] Server 250 may include a processor 252, memory 254, and a communication interface 256. Server 250 may further include any combination of a content context selector 214, a prompt generator 216, and / or a language model 218.

[0024] Device 201 includes a processor 202 and memory 204. The processor 202 may include multiple processors, and the memory 204 may include multiple memories. The processor 202 may consist of instructions that generate the start interface described in this disclosure. The instructions may include non-temporary, computer-readable instructions that are stored in and called from memory 204.

[0025] Device 201 includes memory 204, which may contain code to operate any combination of the operating system 210, interactive content application 212, content context selector 214, prompt generator 216, language model 218, rewrite module 220, and / or description module 222.

[0026] The communication interface 206 of device 201 may be capable of facilitating communication between device 201 and server 250 or any other computing device. For example, the communication interface 206 may utilize short-range wireless communication protocols such as BLUETOOTH®, Wi-Fi, Zigbee®, or any other wireless or wired communication method.

[0027] In the example, the processor 252, memory 254, and communication interface 256 of server 250 may include the same functions as processor 202, memory 204, and communication interface 206, respectively.

[0028] The display 208 of device 201 may include any type of display, either internal or external to the computing device 201. The processor 202 may render graphics for display on the display 208.

[0029] Operating system 210, running on processor 202, can provide a platform for running other applications. In this example, operating system 210 could be a browser-based operating system.

[0030] The processor 202 may include an interactive content application 212. The interactive content application 212 displays content. The interactive content application 212 may further allow the user to interact with the content in any combination of ways, such as creating, modifying, saving, or sending the content. In the example, the interactive content application 212 may be a native application or a browser application. In the example, the interactive content application 212 may be a browser application, a word processor application, a productivity application, a messaging application, an email application, or any other type of application or program. In the example where the interactive content application 212 is a native application, the interactive content application 212 may retrieve and store data from memory 204.

[0031] In an example where the interactive content application 212 is a browser application, the interactive content application 212 may retrieve web pages from a remote server hosting web pages and internet-based applications. The browser may be configured to display web pages and run web applications (e.g., applications that run in browser tabs). The browser may include additional functionality in the form of browser extensions, such as browser plugins. The browser may have access to its browser history. The browser may receive content from a URL that includes both displayable content and non-displayable content, including metadata. Metadata may include any data not displayed within the document displayed in the browser.

[0032] An interactive content application 212 may include a window in which content is displayed. For example, Figure 1A shows a window 100 in which content 102 is displayed. The exemplary application in Figure 1A is an email application, and in this example, content 102 is the text string "Have a nice weekend." However, this is not intended to be limiting. Any type of interactive content application and content is assumed in the example.

[0033] Processor 202 may further include a content context selector 214. In this example, the content context selector 214 may receive content from an application window 100, a URL rendered in the application window 100, the application rendering the application window 100, other applications running on the operating system 210, the operating system 210, or data files stored in memory 204.

[0034] The content context selector 214 may receive content and / or window context associated with the application window 100 and provide it to the language model 218. In the example, the window context may include content from windows excluded from selection. In the example, the window content may include any combination of text, images, audio data, associated metadata, or other data.

[0035] In the example, the content context selector 214 may include a set of user-defined settings that can be used to determine which window contexts can be used as input to the language model 218.

[0036] In the example, the content context selector 214 may receive selected content associated with the application window 100. Notification may be received that the user has selected content displayed in the window. In the example, the content selection may be editable and used to generate output that is replacement content for the selection. The replacement content is content generated by a language model intended to replace the selected content associated with the application window 100. In a further example, the content selection may be read-only. In other words, the content may not be modified or edited by the user of device 201 in window 100. In the example, the content may be text or an image. In the example, the notification may include an event notification that the user has highlighted a portion of the displayed text. In the example, the notification may include an event notification that the user has hovered the mouse pointer over a portion of the text (or image), or any other way of determining that the user has selected or is likely to select the text. In some embodiments, the selected content 402 may be selected by a smart or predictive selection process.

[0037] For example, in Figure 1A, the user has selected selected content 102, which is the body of an email, "Have a great weekend!". In the example, selected content 102 may be selected by hovering the cursor 118 over the text. In the example, selected content 102 may be selected by highlighting the text, selecting the text, or by any other method. In some embodiments, selected content 102 may be selected by a smart or predictive selection process.

[0038] In the example, with user consent, the content context selector 214 may receive content and window context. For example, Figure 1A shows unselected content 103 containing the text "Thank you, Dato," which is the closing of an email. In the example, unselected content 103 may include text, images, audio, semantic relationships, or metadata present in a document displayed in the application window 100. In the example, with user consent, window context may include any combination of past prompt selections made by the user, past outputs from the language model, information about the website where selected content 102 is displayed, search queries performed by the user, and links present in the email message.

[0039] In the example, with user consent, the context selector 214 may identify additional information as input to the language model 218, including data about user actions preceding the selection of background or prompt. This additional information may include information related to the interactive content application 212, the operating system 210, browser history, or other data available on device 201. In the example, the additional information may include one or more applications running on device 201. For example, the content context selector 214 may determine applications and / or processes that may be running via an application programming interface (API).

[0040] In the example, the context selector 214 may identify content that is displayed or open in another application. For example, if the selected content 102 is in an email application, the context selector 214 may identify a URL or photo displayed in a browser application as window content.

[0041] In this example, the processor 202 may include a prompt generator 216. The prompt generator 216 may identify prompts to provide as input to the language model 218.

[0042] Referring to Figure 1A, it can be seen that the prompt generator 216 may display a menu 106 to the user that contains one or more prompts. In one example, menu 106 may appear when the selected content 102 is highlighted and / or right-clicked. Menu 106 may contain several options, including rewrite options 109, which the user can choose to further interact with the selected content 102. Prompts available via the rewrite options 109 in menu 106 may provide output that is a modified version (e.g., rewritten) of the selected content 102.

[0043] In Figure 1B, the rewrite option 109 is selected, and a further menu 108 is displayed to allow the user to further refine the prompt. In one example, a prompt input box 110 may allow the user to enter a custom prompt. In one example, a prompt suggestion section 112 may display one or more suggested prompts for the user to select to modify the selected content 102. In this example, one or more suggested prompts in the prompt suggestion section 112 may include “Explain in detail,” “Shorten,” and “Add humor.” In another example, one or more suggested prompts may include prompts to interpret or further understand the selected content 102. In one example, suggested prompts may be derived from a model. For example, the system may invoke a model (e.g., a language model) with a prompt such as “Please provide additional suggestions to understand [selected content]” and context such as the currently running application, the location of a resource (e.g., a URL), and other screen content, and this model (e.g., a language model) may provide suggested prompts. In another example, suggested prompts may be based on a variety of factors. Factors may include popularity (for example, which prompts are frequently requested given the context). With user consent, factors may include personalized factors such as prompts previously selected by the user. Factors may include operating system factors such as which applications are running and which others are running. Factors may include what was selected or the window context related to what was selected. With user consent, factors may further include the applications the user uses, as well as patterns and history of operations performed by the user. Exemplary prompts not shown in Figure 1B include "Create an image for [Selected Content 102]" or "Describe what is shown in [Selected Image]."

[0044] In the example, the prompt suggestion section 112 may display prompts based on the popularity of the given prompts in a specific context. For example, the prompt suggestion section 112 may provide the user with options to shorten or elaborate on the selected content 102.

[0045] In one example, the prompt generator 216 may display a prompt based on the selected content 102 and / or window context. For example, if the body of a letter is selected, the prompt generator 216 may prompt the user to make the letter formal.

[0046] In one example, the prompt generator 216 may present a prompt that provides an explanation of the selected content 302, such as a definition of a term. For example, Figure 3A shows an exemplary word processor application window 300. The user selects the term "L / R" (selected content 302) from the content displayed in the word processor application window 300, right-clicks, and initiates the display of menu 106. Menu 106 includes a definition prompt 310. If the definition prompt 310 is selected, a definition output window 312 may appear on top of the word processor application window 300.

[0047] In one example, the prompt generator 216 may present prompts that provide further context, background, or explanation for the selected content. For example, Figure 4A shows a browser window 400 that displays a URL containing a list of riddle questions with corresponding answers. In this example, the user may right-click to select the selected content 402 while hovering the mouse pointer 404 over a riddle. In this example, the text within window 400 may be editable or read-only. In this example, the text may represent part of a larger document.

[0048] Figure 4B shows menu 106, which includes explanatory options 410 to aid understanding. Explanatory options 410 may be selected to provide further background or context to the content relating to the riddle. In some examples, explanatory options 410 may be used to provide interpretive explanations beyond definitional or encyclopedic background. In some examples, explanatory options 410 may provide further background on the relationships between ideas within the selected content 402, or how concepts such as irony or sarcasm relate to the selected content 402.

[0049] Figure 4C shows a further menu 413 that may appear when the explanatory option 410, which helps with understanding, is selected. The further menu 413 may include user interface elements for receiving prompts from the user, namely a text prompt field 415, and prompts related to the set of explanatory prompts 411. In this example, the section of the set of explanatory prompts 411 displays prompts specifically designed to provide context or background to the content, such as “Summary,” “Highlight Key Sentences,” and “What it Means.”

[0050] Referring to Figure 4B, it can be seen that menu 106 may provide web query options 412 to the selected content 402.

[0051] In one example, the prompt generator 216 may run on device 201 or server 250. With user consent, information (e.g., selected content 102) may be sent from device 201 to server 250 to generate one or more prompts. In one example, the prompt generator 216 may run a model with any of the above factors as input to generate prompts for display in menu 106 or a further menu 108, for example.

[0052] In the example, processor 202 may further include language model 218. The language model access module 218 may receive the window context determined by the content context selector 214 and the prompt determined using the prompt generator 216, and generate output. In these examples, language model 218 may run as part of a library accessible from multiple applications. In further examples, language model 218 may be integrated into an executable application.

[0053] The language model 218 can be trained on a wide range of datasets. In the example, the training data may include text and / or images extracted from web pages and other publications. In the example, the language model 218 can be optimized to run on device 201.

[0054] In this example, language model 218 may be executed on server 250. With user consent, prompts and window context may be sent to server 250, which may respond by sending output to device 201.

[0055] In one example, processor 202 may further include a rewrite module 220. In the example, the rewrite module 220 may initiate execution of the prompt generator 216, language model 218, and the rewrite module 220 itself. In the example, the rewrite module 220 may display one or more outputs from the rewrite module 220 itself. In one example, the rewrite module 220 may be accessible via the operating system from any application executed via one or more API calls.

[0056] Menu 106 in Figure 1A, menu 108 in Figure 1B, and menu 413 in Figure 4C are examples of menus that can be displayed by the rewrite module 220. In the examples, menus 106, 108, and 413 may be displayed when the selected content 402 is right-clicked. In the examples, menu 106 may be displayed automatically when the selected content 102 is long-pressed. Menu 106 may be displayed in response to any action that triggers a popup menu.

[0057] Referring to Figure 1B, the cursor 118 is positioned over the selected prompt 120, which is labeled "Formalize". Selecting the formalize prompt 120 from the further menu 108 may bring up the output window 130, as shown in Figure 1C.

[0058] The output window 130 may have a title 132. In this example, the title 132 may provide a background for any combination of the selected content 102, the window context, and the selected prompt. In this example, the title 132 is "Here are some more formal ways of saying 'Have a great weekend!'"

[0059] The output window 130 may display one or more outputs from the language model 218. In one example, the output window 130 may be a pop-up window displayed on top of the application window 100. In this example, the output window 130 displays three selectable outputs from the rewrite module 220, corresponding outputs 134A, 134B, and 134C. In one example, each of the corresponding outputs 134A, 134B, and 134C may be generated based on the same window context and prompt. The output window 130 may contain any number of corresponding outputs.

[0060] Upon receiving notification that one of the corresponding outputs 134A, 134B, and 134C has been selected, the system may copy the output to the clipboard. Upon receiving notification that an output has been selected, the system may also automatically replace the selected content 102 in the editable application window 100 with the selected output. For example, in Figure 1C, the cursor 118 selects output 134A. Referring to Figure 1D, the content of output 134A, "Have a great weekend filled with rest, relaxation, and fun!", is pasted into the application window 100, replacing the selected content 102.

[0061] In one example, a further prompt may be displayed in response to receiving notification that one of outputs 134A, 134B, and 134C has been selected. The further prompt may be operable to generate further outputs based on the first output and the second prompt as input to the language model. The steps of selecting content by the context selector 214, providing prompts by the prompt generator 216, generating outputs by the language model 218, and displaying the outputs by either the rewrite module 220 or the explanation module 222 may be repeated a desired number of times to assist the user in further iterating over or refining the output from the language model 218.

[0062] For example, upon receiving an output, such as a selection of one of the corresponding outputs 134A, 134B, and 134C, further prompts may be displayed that allow the system to generate further outputs based on the selected output and subsequent prompts.

[0063] In one example, output may be generated using selected content 102 associated with a first application and context associated with a second application as input. For example, a user might select text that may be related to a painting as selected content 102 in a first application (including a first tab in a web browser), which is a word processor. In the web browser, a second application (including another tab in the web browser) may display an image of the painting. The operating system may have access to additional context that applications running within the operating system do not have access to. The output generated by the language model may use the context of the image of the painting (e.g., in the second application / tab) when generating a textual description of the painting.

[0064] In one example, the processor 202 may further include an explanation module 222. The explanation module 222 may be used to provide a further explanation of the concepts expressed in the selected content 402. The explanation module 222 may generate explanations by providing the outputs from the context selector 214, the prompt generator 216, and the language model 218 as input to the language model 218. In the example, the explanation module 222 may be accessible via the operating system from any application executed via one or more API calls.

[0065] Referring to Figure 3B, it can be seen that when the selected content 302 and definition prompt 310 are selected, the explanation module 222 may display a definition output window 312 on top of the word processor application window 300. The definition output window 312 may further display a copy answer option 314. If the copy answer option 314 is selected, the user can copy the contents of the definition output window 312 to the clipboard and paste them elsewhere.

[0066] In the example, selected content 302 may contain a term with multiple meanings, and the explanation provided in the definition output window 312 may relate to one of those meanings based on the context. For example, selected content 302 may contain the term "L / R," which may have multiple meanings, such as an L / R polarizing filter, a living room in a real estate context, or the synthetic chemical element lawrencium. The window context surrounding selected content 302 in the word processor application window 300 includes the unselected text, "No light passes through. The first filter removes all [L / R] components of light, and the second filter removes..." By using at least a portion of the unselected text in the word processor application window 300 as input to the language model 218, a specific explanation for the L / R polarizing filter can be provided. Thus, the context (in this case, the unselected text) helps the model focus on generating a specific definition rather than covering all possible definitions. This is achieved with less input from the user.

[0067] Figure 4D shows that when selected content 402 and explanation option 410 (displayed as "Help me understand" in Figure 4D) are selected, the explanation module 222 may display an output window 416 that displays the output from the language model 218, based on the selected content 402 and explanation option (e.g., the prompt "Help me understand") being provided as input.

[0068] In one example, the description module 222 may receive selected content 402, which may initiate the display of options such as description option 410, which generate a description of the selected content 402. If description option 410 is selected, the description module 222 may provide and / or display prompts related to providing a description of the selected content 402. Exemplary prompts may include "Summary," "Highlight important sentences," or "What does it mean?". Upon receiving a selected prompt 414, the description module 222 may generate a description using the selected content 402 and the selected prompt 414 as input. This description may be displayed in the output window 416.

[0069] In one example, this explanation can describe the relationship between a first and second phrase in content selection. For instance, selected content 402 contains the phrases, "Why did the chicken cross the road?" and "Because it wanted to go to another slide." The explanation in output window 416 explains that the relationship between the two phrases is unexpected and amusing because the user expected the term "side" rather than "slide." In another example, unselected text in browser window 400 can be used as context for the selected text. Because this unselected content relates to a joke, this context helps the model provide a better explanation.

[0070] In one example, the description generated by the language model 218 may use a prompt 414 selected from a set of description prompts 411 as input. In one example, the description may be based on the selection of content 402, the selected prompt 414 in the set of description prompts 411, and the window context associated with the selection. In one example, the description module 222 may provide a user interface element, namely a text prompt field 415, configured to receive prompts from the user. The prompt received in the text prompt field 415 is one of the prompts provided by the description module 222.

[0071] In the example, the rewrite module 220 and the explanation module 222 may be able to communicate with multiple language models. For example, a particular prompt or a specific type of selected content may cause the system to send the prompt and window context to different language models (e.g., a translation model, a rewrite model, a definition model). Other factors that the rewrite module 220 uses to select a model may include the type of prompt selected, the window context, and operating system signals. In some embodiments, only a single language model may be used.

[0072] Figures 5A to 5C illustrate features that allow users to receive assistance in writing or creating content that is not based on selected content. For example, right-clicking on or hovering the mouse pointer over an editable space in an application window that does not have highlighted content may bring up menu 502.

[0073] Menu 502 may include a custom document creation prompt option 504. Selecting the custom document creation prompt option 504 may display the creation help menu 506. Returning to Figure 5B, an exemplary embodiment of the creation help menu 506 is shown. The creation help menu 506 may include a prompt field 508 where the user can enter a custom prompt to create something. For example, in Figure 5B, the user entered the prompt, "Please create a thank-you message for attending the meeting."

[0074] In one example, the rewrite module 220 sends the prompt and window context from the prompt field 508 to the language model 218 to generate output. With user consent, the window context for creation help may include the type of application on which the creation help menu 506 is displayed, other applications running on device 201, or content from other applications running on device 201.

[0075] Referring to Figure 5B, it can be seen that one or more outputs 510A, 510B from the language model 218 may be displayed. In one example, an approve or reject option 512 (shown in the figure as a thumbs-up hand and a hand pointing downwards) may be displayed along with each output 510A, 510B. If the approve option is selected, each output may be copied to an editable field. If the reject option is selected, each output may disappear from the display. In the example, a new output may appear in its place.

[0076] In one example, the Create Help menu 506 may include further prompts for modifying the output. In one example, further prompts for rewriting one or more outputs may be accessed by selecting the Enhancement option 514. Figure 5C shows the Enhancement option 514, which may be generated to display further prompts for modifying one or more outputs when selected. The prompts displayed in Enhancement option 514 may be selected in any of the ways described above with respect to the prompt generator 216. Alternatively, the output may be further modified by entering a prompt in the prompt field 508.

[0077] When an output is selected by clicking on one of outputs 510A or 510B, or by selecting one of the approve or reject options 512, the selected output may be pasted into the editable field where menu 502 was first accessed, for example, via the insert option 518.

[0078] For example, a user might compose an email in an email application. In the background, processor 202 might be running other applications, such as an image editing application with an image file open. The user might initiate a request for assistance in composing a message from the email application by right-clicking the email application window without selecting any text.

[0079] In one example, a prompt may be displayed to create a description of this image. This image may be used as input to the language model 218 along with the prompt to generate an output or a description of the image.

[0080] Figure 6A shows an example of method 600. Method 600 may be used within an application to generate output (e.g., information about new content, modified content, or content displayed in the window) based on the content within a window using a language model.

[0081] For example, according to one example, method 600 may be performed to provide an improved language model output. Method 600 may include any combination of steps 602 to 616.

[0082] Method 600 may begin with step 602. In step 602, selected content may be received from the content displayed in the window. For example, as described above, selected content may be received from the content context selector 214.

[0083] Method 600 may proceed to step 604. In step 604, a prompt may be generated based on the selection from the window and the context. For example, a prompt may be generated as described in the prompt generator 216 above.

[0084] Method 600 may proceed to step 606. In step 606, a prompt may be displayed. For example, as described above, the prompt suggestion section 112 is shown in Figure 1B.

[0085] Method 600 may proceed to step 607, in which a selected prompt may be received. For example, a selected prompt may be received as described in the prompt generator 216 above.

[0086] Method 600 may proceed to step 608, in which a first output may be generated by providing content and prompts as input to a language model. For example, as described above, the language model 218 may generate an output.

[0087] Method 600 may proceed to step 610, in which the first output may be displayed. For example, the output may be displayed as described in the rewrite module 220 above.

[0088] Method 600 may proceed to step 612, in which a second output may be generated by providing content and prompts as input to the language model. For example, as described above, the language model 218 may generate an output.

[0089] Method 600 may proceed to step 614. In step 614, a second output may be displayed. For example, the output may be displayed as described in the rewrite module 220 above.

[0090] Method 600 may proceed to step 616. In step 616, a notification may be received that the first output has been selected. For example, the notification may be received as described in the rewrite module 220 above.

[0091] In the example, steps 604 to 616 are repeated any number of times to iterate on the output. In this way, the user can continue to refine the content using method 600 and fully reap the productivity benefits provided by the language model.

[0092] Figure 6B shows an example of Method 650. Method 650 may be used in an application to generate output using a language model that includes a description of the selected content within a window (e.g., a definition, background description, and a description comparing the selected content to another concept).

[0093] Method 650 may begin from step 652, in which the content selection may be received from the window. For example, as described above, the selected content may be received from the content context selector 214.

[0094] Method 650 may proceed to step 654. In step 654, options for generating a description of the selection may be displayed. For example, a menu 106 containing the description option 410 may be displayed, as described above.

[0095] Method 650 may proceed to step 656, in which the user may receive an option selection. For example, as described above, the user may select the explanatory option 410.

[0096] Method 650 may proceed to step 658. In step 658, prompts related to the explanation may be provided. For example, as described above, a further menu 413 may be displayed, which includes a text prompt field 415 and a set of explanatory prompts 411.

[0097] Method 650 may proceed to step 660, in which a prompt selection may be received. For example, as described above, the user may select the selected prompt 414.

[0098] Method 650 may proceed to step 662, in which explanations and prompts may be generated using a content-based language model. For example, as described above, the interactive content application 212 may execute the language model 218.

[0099] Method 650 may proceed to step 664. In step 664, an explanation may be displayed. For example, as described above, the output window 416 may be displayed.

[0100] By providing access to the language model from within the application window 100, users can receive information more relevant to their needs because the information is generated considering the context of the application window 100. In the example, any combination of steps related to methods 600 and / or 650 can be performed in a browser application programming interface (API) via a browser plugin or operating system API. This allows any web or desktop application developer to integrate the capabilities of the language model into their application. This provides a more efficient and accessible experience, allowing users to access the language model's output information with less input or interaction to get the answers they are looking for.

[0101] This disclosure describes enhanced capabilities that can be implemented at the operating system level. This can provide improved functionality that can be consistent throughout the application / user interface, and context and ranking signals that are not available in the application / extension may be available in the operating system.

[0102] By enabling users to generate prompts related to selected content, generate language model output, and display that output, more relevant information can be provided to users more easily without having to switch between multiple applications.

[0103] Some of the exemplary embodiments described above are explained as processes or methods shown as flowcharts. While flowcharts describe operations as sequential, many operations may occur in parallel, concurrently, or simultaneously. The order of operations can also be changed. A process may terminate when its operations are complete, but it may also have additional steps not shown in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc.

[0104] The methods described above, some of which are illustrated by flowcharts, can be performed by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments for performing the required task can be stored in a machine- or computer-readable medium such as a storage medium. A processor(s) can perform the required task.

[0105] Furthermore, it should be noted that in some alternative embodiments, the functions / actions shown may occur in a different order than that shown in the diagrams. For example, two diagrams shown consecutively may actually be performed simultaneously, or sometimes in reverse order, depending on the functions / actions involved.

[0106] It should also be noted that the actions performed by the software in the exemplary embodiments are typically encoded in some form of non-temporary program storage medium or performed via some kind of transmission medium. The program storage medium may be magnetic (e.g., floppy disk or hard drive) or optical (e.g., compact disk read-only memory, i.e., CD-ROM), and may be read-only or random-access. Similarly, the transmission medium may be twisted wire pairs, coaxial cables, optical fibers, or any other suitable transmission medium known in the art. The exemplary embodiments are not limited to these aspects of any given embodiment.

[0107] Finally, while the attached claims describe specific combinations of the features described herein, it should be noted that the scope of this disclosure is not limited to the specific combinations claimed below, but rather extends to encompass any combination of the features or embodiments disclosed herein, regardless of whether or not such specific combinations are explicitly enumerated in the attached claims at this point.

[0108] In some embodiments, the techniques described herein relate to methods, and the context related to content includes data available to the operating system running on the user device.

[0109] In some embodiments, the techniques described herein relate to methods, the selections relate to a first application, and the context relating to the selections relates to a second application.

[0110] In some embodiments, the techniques described herein relate to methods, the selection of which is related to a first application and which further includes using content related to a second application as input to a language model to generate output.

[0111] In some embodiments, the techniques described herein relate to methods, where the selection is editable and the output is the replacement content of the selection.

[0112] In some embodiments, the techniques described herein relate to methods in which prompts are further generated based on the popularity of a given prompt.

[0113] In some embodiments, the techniques described herein relate to methods, and the context relates to the application associated with the selection.

[0114] In some embodiments, the techniques described herein relate to methods, and the output is a modified version of a selection generated based on the content and prompts.

[0115] In some embodiments, the techniques described herein relate to methods, wherein the output is a first output, and the language model further generates a second output by providing content and prompts as inputs to the language model, and the method further includes displaying the second output together with the first output, and the first and second outputs are selectable by the user, with the first output being selected by the user.

[0116] In some embodiments, the techniques described herein relate to methods, wherein a prompt is a first prompt, and an output is a first output, and the method further includes receiving a notification that a first output has been selected, and in response to receiving the notification, displaying a second prompt operable to generate a second output based on the first output and the second prompt as input to a language model.

[0117] In some embodiments, the techniques described herein relate to a method, which further includes receiving a selection of a second prompt and generating a second output by providing the first output and the second prompt as input to a language model.

[0118] In some embodiments, the techniques described herein relate to methods, where the language model is executed on the user device.

[0119] In some embodiments, the techniques described herein relate to methods, and the descriptions are further generated using the context related to the selection as input to a language model.

[0120] In some embodiments, the techniques described herein relate to methods, and the context includes data available to the operating system running on the user device.

[0121] In some embodiments, the techniques described herein relate to methods, and the context includes additional content displayed in the application that is outside the scope of content selection.

[0122] In some aspects, the techniques described herein relate to methods, the selection of content includes terms with multiple meanings, and the descriptions relate to one of the multiple meanings, based on the context.

[0123] In some embodiments, the techniques described herein relate to methods, and the descriptions relate to the definition of selection.

[0124] In some embodiments, the techniques described herein relate to methods, and the descriptions explain the relationship between a first phrase and a second phrase in content selection.

[0125] In some embodiments, the techniques described herein relate to methods, and the content selection is read-only content.

[0126] In some embodiments, the techniques described herein relate to methods, the methods further comprising receiving a selection of options, providing prompts relating to a description, and receiving a selection of one of the prompts, the description being based on one of the prompts and the selection.

[0127] In some aspects, the techniques described herein relate to methods, and the descriptions are based on content selection, one of the prompts, and the context relating to the selection.

[0128] In some embodiments, the techniques described herein relate to methods, and providing prompts relevant to the description includes providing a user interface element for receiving text prompts from a user, where the text prompt is one of the prompts.

[0129] In some embodiments, the technology described herein relates to a system, and the context related to the content includes data available to the operating system running on the user device.

[0130] In some embodiments, the technology described herein relates to a system in which the selection is editable and the output is the replacement content of the selection.

[0131] In some embodiments, the technology described herein relates to a system in which the language model is executed on a user device.

[0132] In some embodiments, the techniques described herein, with respect to a system, further include using context related to a selection as input to generate a description, where the context includes data available to the operating system running on the user device.

[0133] In some embodiments, the techniques described herein relate to systems, and the descriptions are further generated using context related to selection as input.

Claims

1. Receiving content selection and In response to receiving the aforementioned selection, a prompt is provided based on the selection and the context related to the selection. Displaying the aforementioned prompt, In response to receiving the selection of the prompt, the output is generated by providing the content and the prompt as input to the language model. A method including displaying the aforementioned output.

2. The method according to claim 1, wherein the context relating to the content includes data available to the operating system running on the user device.

3. The method according to claim 1 or 2, wherein the selection relates to a first application, and the context relating to the selection relates to a second application.

4. The method according to any one of claims 1 or 2, wherein the selection relates to a first application, and generating the output further comprises using content related to a second application as input to the language model.

5. The method according to any one of claims 1 or 4, wherein the selection is editable and the output is replacement content for the selection.

6. The method according to any one of claims 1 to 5, wherein the prompt is further generated based on the popularity of the given prompt in the context.

7. The method according to any one of claims 1 to 6, wherein the context relates to the application associated with the selection.

8. The method according to any one of claims 1 to 7, wherein the output is a modified version of the selection generated based on the content and the prompt.

9. The output is a first output, and the language model further generates a second output by providing the content and the prompt as input to the language model, and the method is The second output is further displayed together with the first output, The first output and the second output are selectable by the user. The method according to any one of claims 1 to 8, wherein the first output is selected by the user.

10. The prompt is a first prompt, the output is a first output, and the method is Receiving notification that the first output has been selected, The method according to any one of claims 1 to 9, further comprising: displaying a second prompt operable to generate a second output based on the first output and the second prompt as input to the language model, in response to receiving the notification;

11. Receiving the selection of the second prompt, The method according to claim 10, further comprising generating the second output by providing the first output and the second prompt as inputs to the language model.

12. The language model is executed on a user device, according to any one of claims 1 to 11.

13. Receiving content selection and In response to receiving the selection of the aforementioned content, the system displays options for generating a description of the selection. The description is generated by providing the content selection and options as input to the language model, A method including displaying the above description.

14. The method according to claim 13, wherein the above description is further generated by using the context related to the selection as input to the language model.

15. The method according to claim 14, wherein the context includes data available to the operating system running on the user device.

16. The method according to any one of claims 14 to 15, wherein the context includes additional content displayed in an application that is outside the scope of the selection of content.

17. The method according to any one of claims 14 to 16, wherein the selection of content includes terms having multiple meanings, and the description relates to one of the multiple meanings based on the context.

18. The foregoing description relates to the method according to any one of claims 13 to 17, relating to the definition of selection.

19. The method according to any one of claims 13 to 16, wherein the above description explains the relationship between the first phrase and the second phrase in the selection of content.

20. The method according to any one of claims 13 to 19, wherein the selected content is read-only content.

21. Receiving the selection of the aforementioned option, To provide prompts related to the explanation, The further includes receiving the selection of one of the aforementioned prompts, The method according to any one of claims 13 to 20, wherein the above description is based on one of the above prompts and the above selection.

22. The method according to claim 21, wherein the description is based on the selection of content, one of the prompts, and the context related to the selection.

23. The method according to claim 21, wherein providing a prompt related to the description includes providing a user interface element for receiving a text prompt from a user, the text prompt being one of the prompts.

24. Processor and A memory in which an instruction is set, and the instruction is Receiving content selection and In response to receiving the aforementioned selection, a prompt is provided based on the selection and the context related to the selection. Displaying the aforementioned prompt, In response to receiving the selection of the prompt, the output is generated by providing the content and the prompt as input to the language model. A system including the display of the aforementioned output.

25. The system according to claim 24, wherein the context relating to the content includes data available to the operating system running on the user device.

26. The system according to any one of claims 24 to 25, wherein the selection is editable and the output is replacement content for the selection.

27. The language model is executed on a user device, according to any one of claims 24 to 26.

28. Processor and A memory in which an instruction is set, and the instruction is Receiving content selection and In response to receiving the selection of the aforementioned content, the system displays options for generating a description of the selection. The description is generated by providing the content selection and options as input to the language model, A system that includes displaying the above-mentioned explanation.

29. The system according to claim 28, wherein generating the description further includes using a context related to the selection as input, the context including data available to an operating system running on a user device.

30. The above description is further generated using the context related to the selection as input, according to any one of claims 28 to 29.

31. A computing device, At least one processor, A non-temporary computer-readable medium for storing executable instructions, wherein the executable instructions, when executed by the at least one processor, are transmitted to the computing device. Receive content selection, In response to receiving the aforementioned selection, prompts are provided based on the selection and the context related to the selection. Display the aforementioned prompt, In response to receiving the selection of the prompt, the content and the prompt are provided as input to the language model to generate an output. A computing device that displays the aforementioned output.

32. A computing device, At least one processor, A non-temporary computer-readable medium for storing executable instructions, wherein the executable instructions, when executed by the at least one processor, are transmitted to the computing device. Receive content selection, In response to receiving the selection of the aforementioned content, display options for generating a description of the selection. By providing the aforementioned content selection and options as input to the language model, the description is generated. A computing device that displays the above description.

Images