Abstract summarization with user interface (UI) flow control and actionable information extraction

Abstract

The described techniques may be used to process transcriptional text of a transcriptional stream using a summary ML model to obtain a summary stream. In this manner, live conversations or other live voices may be provided to a user in real time using a display of a head-mounted device (HMD) or other suitable device. A change-over switch or other stream selector may be provided to the user interface for selecting either or both of a transcriptional stream or a digest stream at a given point in time. Actionable items may be identified within either or both of the transcriptional stream or summary stream, and a scroll bar or other action selector may be provided in the user interface to perform corresponding actions.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 364,478, filed May 10, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] This application also incorporates by reference the disclosures of the following related co-pending applications: U.S. Application No. 18 / 315,113, filed May 10, 2023; “Multi-Stage Summarization for Customized, Contextual Summaries” filed May 10, 2023 (Attorney Docket No. 0120-533WO1); “Dynamic Summary Adjustments for Live Summaries” filed May 10, 2023 (Attorney Docket No. 0120-534WO1); “Summary Generation for Live Summaries with User and Device Customization” filed May 10, 2023 (Attorney Docket No. 0120-535WO1); “Summarization with User Interface (UI) Stream Control and Actionable Information Extraction” filed May 10, 2023 (Attorney Docket No. 0120-541WO1); and “Incremental Streaming for Live Summaries” filed on May 10, 2023 (Attorney Docket No. 0120-589WO1). Technical Field

[0003] This specification relates to summarization using machine learning (ML) models. Background Art

[0004] A volume of text (such as a document or article) often includes useless or undesirable content for the consumer of the volume of text. Additionally or alternatively, the user may not be willing to spend time (or may not have enough time) to consume the entire volume of text.

[0005] Summarization generally refers to techniques that attempt to reduce a volume of text to obtain a reduced volume of text that retains most of the information of the volume of text within the summary. Thus, users can read the information in a more efficient and desirable manner. In order to achieve the necessary processing of the text, the latter can be represented by electronic data (text data). For example, an ML model can be trained to input text and output a summary of the text. Summary of the invention

[0006] The described technology processes input text data to reduce the data volume of the input text data and obtains output text data expressing a summary of the content of the input text data. The obtained output text data of reduced capacity can conform to the size of the display so as to optimize the size of the output text data with respect to the size of the display. In addition, compared with the prior art and method, the described technology can achieve such customized data volume reduction with reduced delay.

[0007] In a general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and includes instructions. When executed by at least one computing device, the instructions may be configured to cause the at least one computing device to: receive a transcription stream including a transcription text; input the transcription stream into a summary machine learning (ML) model to obtain a summary stream including the summary text; and render a stream selector icon on a user interface (UI). When executed by at least one computing device, the instructions may be configured to cause the at least one computing device to: receive at least one selected stream of a transcription stream and a summary stream via a stream selector icon; and cause display of the at least one selected stream on the UI.

[0008] According to another general aspect, an apparatus includes at least one processor, at least one display, and at least one memory storing instructions. When executed by the at least one processor, the instructions cause the apparatus to: receive a transcription stream including transcription text; input the transcription stream into a summary machine learning (ML) model to obtain a summary stream including summary summary text; render a stream selector icon on a graphical user interface (GUI); receive at least one selected stream of the transcription stream and the summary stream via the stream selector icon; and cause display of the at least one selected stream on the GUI.

[0009] According to another general aspect, a method includes: receiving a transcription stream including transcription text; inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream including summary summary text; and rendering a stream selector icon on a graphical user interface (GUI). The method further includes: receiving at least one selected stream of the transcription stream and the summary stream via the stream selector icon; and causing display of the at least one selected stream on the GUI.

[0010] According to another general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and includes instructions. When executed by at least one computing device, the instructions may be configured to cause the at least one computing device to: receive a transcription stream including a transcription text; input the transcription stream into a summary machine learning (ML) model to obtain a summary stream of summary summary text; and identify summary summary content associated with at least one action within the summary summary text. When executed by at least one computing device, the instructions may be configured to cause the at least one computing device to: render the summary summary text including the summary summary content therein on a graphical user interface (GUI) together with an action indicator that associates the summary summary content with at least one action; receive a selection of the summary summary content via the GUI; and perform at least one action in response to the selection.

[0011] According to another general aspect, an apparatus includes at least one processor, at least one display, and at least one memory storing instructions. When executed by the at least one processor, the instructions cause the apparatus to: receive a transcription stream including transcription text; input the transcription stream into a summary machine learning (ML) model to obtain a summary stream of summary summary text; and identify summary summary content associated with at least one action within the summary summary text. When executed by the at least one processor, the instructions cause the apparatus to: render the summary summary text including the summary summary content therein on a graphical user interface (GUI) together with an action indicator that associates the summary summary content with at least one action; receive a selection of the summary summary content via the GUI; and perform at least one action in response to the selection.

[0012] According to another general aspect, a method includes: receiving a transcription stream including transcription text; inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream of summary summary text; and identifying summary summary content associated with at least one action within the summary summary text. The method includes: rendering the summary summary text including the summary summary content on a graphical user interface (GUI) together with an action indicator that associates the summary summary content with the at least one action; receiving a selection of the summary summary content via the GUI; and performing at least one action in response to the selection.

[0013] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of a system for summarization with UI flow control and actionable information extraction.

[0015] Figure 2A It is an example Figure 1 Flowchart of a first example operation of a system.

[0016] Figure 2B It is an example Figure 1 Flowchart of a second example operation of the system.

[0017] Figure 3A It is an example Figure 1 A series of screenshots showing example operation of the system.

[0018] Figure 3B exemplify Figure 1 A pair of screenshots of the system at its first moment.

[0019] Figure 3C Example at the second time Figure 3B The pair of screenshots.

[0020] Figure 3D exemplify Figure 3C A first alternative embodiment of the example.

[0021] Figure 3E exemplify Figure 3C A second alternative embodiment of the example.

[0022] Figure 3F Examples of Figure 1 Screen shot of the settings menu in the system.

[0023] Figure 4A Screen shot illustrating an example summary of the first summary type.

[0024] Figure 4B exemplify Figure 4A An alternative implementation of the example.

[0025] Figure 4C Screen shot illustrating an example summary of the second summary type.

[0026] Figure 4D Screen shot illustrating an example summary of the third summary type.

[0027] Figure 5A exemplify Figure 1 An example display layout of a system.

[0028] Figure 5B Illustrate a series of screenshots showing action selection from the summary content.

[0029] Fig. 6A is an example screenshot of an alternative embodiment of action selection from a summary content.

[0030] Figure 6B Illustrate a series of screenshots demonstrating the use of a scroll bar to select actions from within a summary content.

[0031] Figure 7 It is a third-person view of the user in the surrounding computing environment.

[0032] Fig. 8A and Figure 8B Illustrated are front and back views of an example implementation of a pair of smart glasses. DETAILED DESCRIPTION

[0033] The described systems and techniques enable user interface (UI) or graphical user interface (GUI) display and control of one or both of the transcription stream of the transcribed text and / or the summary stream of the summary summary text that has been summarized from the transcribed text. For example, the described technology implements a customized context summary display and associated flow control during a live conversation between a speaker and a user. The input speech (audio data) received at the device during the live conversation can be transcribed to obtain a live transcription stream (data stream), and further processed using at least one trained summary summary model or summary summarizer (summarizer) to provide a summary of the speech. In this way, for example, a summary stream (data stream) that the above-mentioned commentary is updated as the speaker speaks can be provided. The described technology can provide users with the ability to easily view, interpret, modify and otherwise utilize either or both of the transcription stream and the summary stream, as well as various associated features. Therefore, even when the UI / GUI only includes a limited size for displaying the transcription stream and / or the summary stream, the user can also have a smooth experience of the live conversation, wherein the transcription stream and the summary stream help the user understand the live conversation.

[0034] Thus, for example, when a user is deaf or has hearing problems, the described techniques may be helpful because the transcription stream and / or summary stream may be visually provided to the user on a display. Similarly, when a user attempts to converse with a speaker of a foreign language, the summary stream may be provided to the user in the user's native language. For example, one or both of the transcription stream and the summary stream may be translated into a desired language.

[0035] The described techniques can be implemented for virtually any type of verbal input text (text data). For example, automatic speech recognition (ASR), speech-to-text (STT), and / or other transcription techniques can be used to provide live transcription of detected speech, which can then be provided or made available to the user as a transcription stream. The described UI techniques can then be used to provide the user with the ability to select and view the transcription stream, the summary stream, or both (e.g., in parallel or sequentially).

[0036] In addition, the described UI techniques may provide the user with various stream icons related to the transcription stream and the summary stream. For example, the described UI techniques may provide a stream status (e.g., currently being processed), a stream type (e.g., a transcription stream or a summary stream), or a stream context (e.g., the context or type of speech being transcribed, such as a lecture or conversation). The described UI techniques may further provide the user with the ability to scroll through (e.g., backward or forward in time) one or both of the transcription stream and the summary stream, or otherwise control aspects of the transcription stream and the summary stream.

[0037] Additionally, entities and other information may be identified within either or both of the transcript stream and the summary stream (e.g., using an entity extraction model), and the described UI techniques may enable a user to select implementation and execution of an associated action for each entity. An entity may be a portion of text included in a selected stream. For example, entity extraction may extract or identify an individual mentioned in the transcript stream or the summary stream, and an associated action may include adding the individual to a user's contact list. In another example, entity extraction may identify a phone number in the transcript stream or the summary stream, and an associated action may be calling the phone number. The action may be a predetermined processing of the data (e.g., based on the entity).

[0038] The rendering engine can be configured to render the transcribed stream and the summary stream and the associated stream icons, and execute the selections made by the user using the stream icons. Such a rendering engine can render the content of the transcribed stream and the summary stream in a manner that promotes or enables the use of the stream icons.

[0039] For example, the rendering engine may render the transcription stream and summary stream by color coding or otherwise visually highlighting the identified entities in a manner designed to enable easy identification and selection of the identified entities. Accordingly, any actions associated with such entities may be specified and performed.

[0040] For example, a user wearing smart glasses or a smart watch, or using a smart phone, may be provided with either or both of a transcription stream and a summary stream while listening to a speaker. In other examples, a user watching a video or participating in a video conference may be provided with either or both of a transcription stream and a summary stream.

[0041] The described techniques therefore overcome various shortcomings and deficiencies of existing summarization techniques while also enabling new implementations and use cases. For example, existing summarization techniques may over-reduced the input text, may not sufficiently reduce the input text, may include irrelevant text, or may include inaccurate information. In the above-mentioned scenario where it is desired to provide a transcription stream and a summary stream in parallel, existing summarization techniques (in addition to the shortcomings just mentioned) may not be able to generate the desired summary quickly enough, or may attempt to generate a summary at an inappropriate time (e.g., before the speaker finishes discussing the topic). Further, existing techniques may generate a summary that is too long (or otherwise inappropriate) to be effectively displayed on the available display area of ​​the device being used (e.g., smart glasses).

[0042] Furthermore, the prior art does not provide UI-based display and control techniques of the type described herein, so that the provided transcripts, even if available, are of limited practical use. For example, prior art that attempts to provide live transcriptions may not be able to keep up with the live speech, and does not provide any usable options for summary summarization.

[0043] In contrast, the described techniques address the above-mentioned and other issues by, for example, providing straightforward, intuitive UI-based use and control of either or both of the transcription stream and the summary stream. Thus, the described techniques are well suited to enabling dynamic, real-time summaries in conjunction with a live transcription that is also produced and available to the user, while providing the user with information and functionality that facilitates use of the available streams.

[0044] Figure 1 is a block diagram of a system for summarizing with UI flow control and actionable information extraction. Figure 1 In the example of , summary stream manager 102 processes speech 104 (audio data, also referred to as spoken input) of speaker 100 to obtain summary 106, which is provided to user 101 as part of live, dynamically adjusted summary stream 134. As mentioned above, speech 104 may include nearly any spoken speech or other verbal input. For example, speech 104 may be a lecture, a conversation, a session, an interview, a dialogue, or any other spoken verbal interaction of two or more participants. Such interaction may be primarily one-sided (monologue) (such as in the case of a lecture), or may be a conversation between speaker 100 (and possibly other speakers, not in the context of a lecture). Figure 1 ) and user 101.

[0045] For example, a conversation may be conducted between a speaker 100 and a user 101, and the conversation may be facilitated by a summary stream manager 102. As just noted, in other examples, the speaker 100 may represent a lecturer, and the user 101 may represent a lecture participant, such that the summary stream manager 102 facilitates the usefulness of the lecture to the user 101. The speaker 100 and the user 101 may be co-located and conduct a face-to-face conversation, or may be remote from one another and communicate via a web conference.

[0046] In other examples, speaker 100 may first record speech 104, and user 101 may view the recorded audio and / or video (and receive a summary 106 thereof) at a later time. In this sense, the term 'live conversation' should be understood primarily from the perspective of user 101. For example, as just noted, user 101 may hear live a previously recorded video of speaker 100, and be provided with a live, dynamically adjusted summary stream 134 of the type described herein.

[0047] therefore, Figure 1 It should be understood that this is an example of the ability of the summary stream manager 102 to provide summaries 106 in response to discrete instances of speech 104 in an independent or static manner (e.g., summarizing the audio of a single recorded video). Figure 1 Also illustrated is the ability of summary stream manager 102 to receive speech from speaker 100 during a first time interval and output summary 106 to user 101, and then repeat such speech-to-summary operations during second and subsequent time intervals to provide a dynamic summary summary of the type mentioned above, and described in detail below with reference to summary stream 134. In other words, as shown and described, summary 106 may be understood as a single discrete summary of corresponding discrete speech from speaker 100 during a single time interval representing a larger time period or time window of a conversation.

[0048] As also described in detail below, the summary stream manager 102 may be implemented in conjunction with any suitable device 138, such as a handheld computing device, smart glasses, earbuds, or a smart watch. For example, the summary stream manager 102 may be implemented in conjunction with one or more such devices in which a microphone or other input device is used to receive speech 104 and an audio output, a visual display (e.g., Figure 1 A display 140 in FIG. 1 and / or other output device is used to render or provide summary 106 and summary stream 134.

[0049] Summary Stream Manager 102 Figure 1 In the simplified example of FIG. 1 , the summary flow manager 102 is illustrated as a single component including multiple subcomponents. However, as also described below, the summary flow manager 102 may be implemented using multiple devices that communicate with each other.

[0050] like Figure 1 As shown, the summary flow manager 102 may include or utilize Figure 1 Device characteristics 108 of one or more devices represented by device 138 in FIG. For example, device characteristics may include the display size of display 140, available fonts or formats, or available scrolling rates of device 138 / display 140.

[0051] User preferences 110 may include any user preferences for receiving summary stream 134 (e.g., as reflected by a device setting selected by the user or other operation of the device by the user). For example, user preferences 110 may include a user preference for a slow, medium, or fast scroll rate of summary stream 134 on display 140. User preferences 110 may also specify a preferred font / format, or a preferred device among a plurality of available devices. User preferences 110 may also include preferences of user 101 with respect to available display options for displaying, controlling, or using one or both of transcription stream 130 and / or summary stream 134. User preferences 110 may be manually entered by user 101, and / or inferred by summary stream manager 102 based on actions of user 101.

[0052] The training data 112 generally represents any training data that can be processed by the training engine 114 to train one or more machine learning (ML) models as described herein. The training data 112 can represent one or more available repositories of labeled training data for training such ML models, and / or can represent training data compiled by a designer of the summary flow manager 102.

[0053] The speech analyzer 116 may be configured to receive the speech 104, for example, via a microphone or other input of the device 138, and process the speech 104 to determine relevant speech characteristics (as reflected by the audio data representing the speech). For example, the speech analyzer 116 may calculate or otherwise determine the rate, tonality, volume, pitch, emphasis, or any other characteristic of the speech 104. The speech analyzer 116 may also identify the speaker 100 individually or as a speaker category / type. For example, the speech analyzer 116 may identify the speaker 100 as a friend of the user 101, or a work colleague or teacher of the user 101. The speech analyzer 116 may also identify the language spoken by the speaker 100.

[0054] The entity extraction model 118 may be trained (e.g., using the training engine 114) and otherwise configured to extract or otherwise identify entities within the transcription stream 130 or summary stream 134. For example, such extracted content may include any type, category of information or information instances that may be structured in a known manner. Any type of fact, phrase, or other key information may be identified for extraction. Some specific but non-limiting examples of such content may include, for example, named entities such as persons, things, dates, times, events, locations (e.g., addresses), phone numbers, email addresses or other contact information, and the like.

[0055] Input handler 120 may be configured to receive input from or associated with display 140, which may be associated with controlling the display and / or operation of one or both of transcription stream 130 and summary stream 134. Rendering engine 122 may be configured to render one or more user interface elements using display 140, as described in more detail below.

[0056] Transcription generator 124 may be configured to convert the spoken words of speech 104 into transcribed text. Figure 1 1 is shown as transcription 126. For example, the transcription generator 124 may include an automatic speech recognition (ASR) engine or a speech-to-text (STT) engine.

[0057] The transcription generator 124 may include many different methods of generating text, including additional processing of the generated text. For example, the transcription generator 124 may provide a timestamp for the generated text, a confidence level for the generated text, and inferred punctuation for the generated text. For example, the transcription generator 124 may also utilize natural language understanding (NLU) and / or natural language processing (NLP) models or related techniques to identify semantic information (e.g., sentences or phrases), identify topics, or otherwise provide metadata for the generated text.

[0058] The transcription generator 124 may provide various other types of information in conjunction with the transcribed text, possibly using related hardware / software. For example, the transcription generator 124 may analyze the input audio stream to distinguish different speakers, or characterize the duration, pitch, speech rate or volume of the input audio, or other audio characteristics. For example, in some implementations, the transcription generator 124 may be understood to implement some or all of the speech analyzer 116.

[0059] exist Figure 1, the transcription generator 124 can utilize the transcription buffer 128 to output the transcription stream 130. That is, for example, the transcription generator 124 can process a live conversation, discussion, or other speech in real time as the speech is ongoing. Thus, the transcription 126 represents a transcription of a segment or instance of the transcribed text within a time interval that exists within a larger time period or time window of the conversation. For example, the summary 106 can represent a summary summary of the transcription 126, where the transcription 126 represents, for example, the transcription of the first 10 seconds of the speech 104.

[0060] For example, while the speaker 100 is speaking, the transcription generator 124 may output the transcribed text for storage in the transcription buffer 128. The transcribed text may be designated as intermediate text or final text within the transcription buffer 128 before being available as the transcription 126 / transcription stream 130. For example, the transcription generator 124 may detect the end of a sentence, a change of speaker, a pause of a predefined length, or other detected audio characteristics to designate a final transcription to be included in the transcription stream 130. In other examples, the transcription generator 124 may wait until a defined or detected time interval has expired to designate a final transcription of the audio.

[0061] Thus, the transcription stream 130 may be processed by a summary summarizer 136 to fill the summary buffer 132 and otherwise output the summary 106 / summary stream 134. The summary summarizer 136 may represent any trained model or algorithm designed to perform summary summarization. For example, the summary summarizer 136 may be implemented as a sequence-to-sequence generative large learning model (LLM).

[0062] In an example implementation, the entity extraction model 118, the summary summarizer 136, and various other ML models (some examples of which are provided herein) may be trained independently, or may be trained together in groups of two or more. The training of each model may be performed relative to, for example, input text representing examples of (transcribed) speech 104, relevant training data tags, the generated output of the model being trained, and the true value output of the model being trained (e.g., the true value summary output of the summary summarizer 136). Thus, the generated output may be compared to the true value output for back-propagation and error minimization to improve the accuracy of the trained model.

[0063] In an example implementation, summary stream manager 102 may be configured to manage various characteristics of summary stream 134 relative to or in conjunction with transcription stream 130. For example, summary stream manager 102 may utilize characteristics of transcription stream 130 to determine whether or when to invoke summary summarizer 136 to generate summary 106. For example, stream manager 102 may detect sentence endings, speech pauses, or rate (or other characteristics) of the audio to determine whether / when to invoke summary summarizer 136.

[0064] In other examples, the summary summarization operation of summary summarizer 136 may be manually invoked by user 101 at any desired time, for example, using input handler 120. For example, user 101 may manually invoke the summary summarization operation using an available touch screen, gesture recognition device, microphone, physical button, or other input device.

[0065] Display 140 may represent a hardware display of associated device 138, such as a touch screen or the lenses of a pair of smart glasses (e.g., Fig. 8A and Figure 8B 140 ). Additionally or alternatively, display 140 may represent a software display, such as a GUI (an example of which is also provided in more detail below). Rendering engine 122 may be implemented as part of or in communication with display 140 and may be configured to render transcription stream 130, summary stream 134, and plurality of stream icons 142.

[0066] For example, the rendering engine 122 may provide a stream icon 142 to, for example, notify the user 101 of the status or operation of the transcription stream 130 and the summary stream 134, or to provide the user 101 with various types of functionality of or related to the transcription stream 130 and the summary stream 134. For example, the stream icon 142 includes a stream selector 144 that is configured to provide the user 101 with the option and ability to view either or both of the transcription stream 130 and the summary stream 134 (e.g., to switch between them).

[0067] The rendering engine 122 may also be configured to provide a flow status indicator 146. For example, the rendering engine 122 renders the flow status indicator 146 on the GUI. For example, the flow status indicator 146 may be configured to notify the user 101 that the current portion of the summary flow 134 is being generated while the summary summarizer 136 is processing the corresponding portion of the transcription flow 130.

[0068] The flow type indicator 148 may be configured to display the type of one or more flows being displayed. For example, the rendering engine 122 renders the flow type indicator 148 on the GUI. For example, if the flow selector 144 is used to select display of the transcribed flow 130, the flow type indicator 148 may provide a written identifier or a designated icon indicating that the flow being displayed is the transcribed flow 130. For example, the provided / displayed flow may be identified as a transcribed flow or a summary flow, or in other examples, the flow may be identified as a translated flow, possibly in conjunction with the specific language used.

[0069] A context indicator 150 may be displayed that informs the user 101 of, for example, the type or other context of the summary 106. For example, as mentioned herein, different types or contexts of summaries may include lectures, conversations, ordered lists, unordered lists, directions (including spatial directions), or any other type of summaries. For example, the summary summarizer 136 may be configured (e.g., trained) to generate different types of summaries for different contexts in such various contexts, which may vary in content and / or layout depending on the relevant context. In a specific example, each summary context may be associated with a corresponding context template to which the summary 106 must conform.

[0070] In some scenarios, the summarizer 136 (or a separate ML model, such as a classifier model) may be trained using a variety of heuristics known to be associated with different summarization contexts. Such heuristics may include, for example, content heuristics related to the speech 104, such as the length of time the speaker 100 spoke, or the user of certain recognized speech (such as using first / second / third to identify ordered lists). External heuristics (such as the current location of the device 138) may also be used to train and infer context.

[0071] Then, at a given point in time, the current values ​​of the heuristics may be used together to determine the most likely current context. In a similar example, the heuristics may be used together to determine the top three (or other number of) contexts among the most likely contexts. The user 101 may then select the desired context using the context indicator 150. For example, the user 101 may select from the context options provided, or from a larger list of available contexts, or may provide a new context, for which the training engine 114 may then fine-tune the summary summarizer 136 to recognize new contexts in the future.

[0072] The stream scroll bar 152 may be provided in conjunction with (e.g., simultaneously and / or adjacently to) one or both of the transcription stream 130 and the summary stream 134. For example, the rendering engine 122 renders the scroll bar 152 on the GUI. By providing the stream scroll bar 152 in conjunction with the corresponding stream being displayed, the user 101 is provided with the ability to retrieve an earlier portion of the corresponding stream in response to movement of the stream scroll bar 152.

[0073] For example, as described above, the transcript buffer 128 and summary buffer 132 may be used to store the most recent "n" seconds of the transcript stream 130 and summary stream 134, respectively. For example, the display 140 may be limited in size so as to display a certain maximum number of words / lines of the transcript stream 130 and summary stream 134. When the amount of the transcript stream 130 and summary stream 134 exceeds this maximum, the remainder may be retained in its corresponding buffer. Thus, the user 101 is provided with the ability to scroll back through either or both of the transcript stream 130 and summary stream 134 (depending on which one / which ones are being displayed at the current point in time).

[0074] Additionally, as noted above, the entity extraction model 118 can identify entities within the transcription stream 130 and summary stream 134 that can be used to extract the entity information using Figure 1 14. For example, the entity extraction model 118 may identify actionable content within the summary flow 134, such as a calendar item, an email address, a physical location, or a phone number.

[0075] The rendering engine 122 may then be configured to render such entities within the transcript stream 130 and summary stream 134 in a recognizable predefined manner. For example, the rendering engine 122 may render such entities using a known coloring scheme or using other types of visual highlighting. For example, a word or phrase corresponding to the identified entity may be rendered using a different text color or font type than the rest of the transcript stream 130 or summary stream 134. In other examples, different text colors / fonts may be used to correspond to, visually identify, or otherwise provide visual distinction between various types of entities and / or different types of corresponding actions.

[0076] That is, in some implementations, the input handler 120 and the rendering engine 122 may be configured to facilitate or perform corresponding actions based on the content of the summary stream 134, such as generating a calendar entry, sending an email or text message, or placing a phone call. For example, the rendering engine 122 may visually identify a phone number within the summary stream 134, and the action selector 154 may identify a corresponding action type, such as saving the phone number to contacts, or placing a phone call. When implementing such an action, a corresponding service or application (e.g., a calendar application or a phone application) may be accessed and utilized.

[0077] Thus, for example, the input handler 120 may receive an action selection regarding, for example, summary content within the summary 106, as just mentioned, via the action selector 154. The input handler 120 may interact with or otherwise communicate with a separate application 156 to invoke performance of the action by the application 156. For example, as can be appreciated from the foregoing examples, the application 156 may include an email application, a phone application, a calendar application, or any application that may be configured to perform one or more types of actions described herein or similar actions.

[0078] exist Figure 1 In the simplified example of , various stream icons 142 are illustrated individually and separately from the transcribed stream 130 and the summary stream 134 within a single view of the display 140. However, in various implementations, the various stream icons 142 may be displayed in a separate display window, such as in a settings menu, or may be displayed as part of one or both of the transcribed stream 130 and the summary stream 134, such as within one or both of the transcribed stream and the summary stream.

[0079] In other examples, the functionality of two or more of the stream icons 142 may be combined. For example, as illustrated and described below, the stream scroll bar 152 may be used as an action selector 154. For example, the stream scroll bar 152 may be used to scroll to an extracted entity (e.g., a phone number). The scroll bar box or scroll bar slider may then be rendered selectable so that the user 101 may click or otherwise select the scroll bar box / slider to effectuate the corresponding action (e.g., placing a phone call using the phone number and the phone application).

[0080] Although transcription buffer 128 and summary buffer 132 are described herein as memories for providing short-term storage of transcription stream 130 and summary stream 134, respectively, it should be appreciated that the same or other suitable memories may be used for long-term storage of some or all of transcription stream 130 and summary stream 134. For example, user 101 may wish to capture a summary of a lecture attended by user 101 for later review. In these or similar scenarios, multiple instances or versions of summary 106 may be provided, and user 101 may be provided with the ability to select the most desirable summary for long-term storage.

[0081] exist Figure 1106 , the transcript stream 130 is shown separately from the summary stream 134 and the display 140. However, as noted above, the transcript stream 130 may be displayed on the display 140 simultaneously with, or in place of, the summary stream 134, for example using the stream selector 144. Thus, the transcript stream 130 and the summary stream 134 may be implemented as a single (e.g., interleaved) stream of commentary within a single window of the display 140. That is, for example, the transcript stream 130 may be displayed for a period of time, and then a summary request may be received via a suitable input device, and a corresponding summary (e.g., summary 106) may be generated and displayed. In other words, the output stream of the display 140 may alternate between displaying the transcript stream 130 and displaying the summary stream 134.

[0082] In the simplified example of the flow manager 102, each subcomponent 108-136 is illustrated in the singular, but it should be understood that it represents at least one instance of each subcomponent. For example, two or more training engines represented by the training engine 114 can be used to implement various types of training for training and deploying the entity extraction model 118 and / or the summary summarizer 136.

[0083] exist Figure 1 , the summary flow manager 102 is illustrated as being implemented and executed using a device 138. For example, as mentioned above, the device 138 may represent a handheld computing device, such as a smartphone, or a wearable computing device, such as smart glasses, smart earbuds, or a smart watch.

[0084] Device 138 may also represent a cloud or network resource in communication with a local device, such as one or more of the devices just mentioned. For example, various types of training data and training engine 114 may be implemented remotely relative to user 101 operating a local device, while the remainder of the instantiation components of summary flow manager 102 are implemented at one or more local devices.

[0085] Summary 106 and / or summary stream 134 are illustrated as being output to display 140. As noted herein, display 140 may be a display of device 138, or may represent a display of a separate device in communication with device 138. For example, device 138 may represent a smartphone, and display 140 may be the display of the smartphone itself, or a display of smart glasses or a smart watch worn by user 101 and in wireless communication with device 138.

[0086] For example, the following Figure 7 , Fig. 8A and Figure 8B More detailed examples of devices, displays, and network architectures are provided.In addition, summary 106 and summary stream 134 (as well as transcription 126 and transcription stream 130) can be output via audio, for example, using smart earbuds of the type mentioned above.

[0087] Figure 2A is an example Figure 1 Flowchart of a first example operation of a system. Figure 2B is an example Figure 1 A flowchart of a second example operation of the system. Figure 2A and Figure 2B In the examples, various operations are illustrated as separate sequential operations. However, in various example implementations, Figure 2A and / or Figure 2B The operations may be implemented in a different order than that illustrated, in an overlapping or parallel manner, and / or in a nested, iterative, cyclic or branching manner. In addition, various operations or sub-operations may be included, omitted or replaced. Figure 2A and / or Figure 2B The operation can be performed by Figure 1 The system is specifically executed by the device 138. The instructions stored in the memory of the device 138, when executed by at least one processor of the device 138, may cause the device 138 to perform operations.

[0088] exist Figure 2A In the example, a transcription stream including a transcribed text may be received (202). For example, a transcription stream 130 may be received from a transcription generator 124, thereby providing a transcription (text data, such as transcription 126) of a speaker 100's speech 104 (audio data).

[0089] The transcription stream 130 (data stream including text data) can be input into a summary machine learning (ML) model (e.g., the summarizer 136) to obtain a summary stream 134 (data stream) including summary text (such as the summary 106) (204). For example, the summarizer 136 can be executed in response to a user selection / initiation. In other examples, the summarizer 136 can be executed in response to characteristics of the speech 104 (e.g., as detected by the speech analyzer 116). For example, the speech analyzer 116 can initiate a summary operation in response to the volume (e.g., a certain number of words) or rate (words per minute) of the speech 104.

[0090] The stream selector icon (206) may be rendered on a graphical user interface (GUI). For example, the rendering engine 122 may render the stream selector 144 as one of the stream icons 142 described above. As described and illustrated below, the stream selector 144 may include a toggle switch or other selector icon for changing between a transcribed mode (in which the transcribed stream 130 is displayed) and a summary mode (in which the summary stream 134 is displayed). In other examples, the stream selector 144 may include other GUI techniques for selecting one or both of the transcribed stream 130 and the summary stream 134, such as a drop-down menu, a check box, a pop-up window, or other suitable techniques.

[0091] A selection of at least one of the transcribed stream and the summary stream may be received via the stream selector icon (e.g., by user interaction with the stream selector icon) (208). For example, user 101 may prefer to view summary stream 134 at a given time or in a given context, and may switch stream selector 144 accordingly. At any time thereafter, user 101 may prefer to view transcribed stream 130 instead, or to view both transcribed stream 130 and summary stream 134.

[0092] Thus, at least one selected stream may be displayed on the GUI (210). For example, when selected, summary stream 134 may be displayed on the GUI of display 140. In a more specific example, display 140 may be included as part of smart glasses or other HMD worn by user 101, as described herein.

[0093] Thus, a display of the most convenient and preferred conversion of speech 104 may be provided to user 101 at any given time. As described herein, in conjunction with such a display, a supplemental display may be provided to user 101 that further facilitates understanding and convenience of user 101. For example, a stream status indicator 146 may provide an operational status of the summary summarizer 136, particularly if there is a time delay between speech 104 and receipt of summary 106. For example, stream status indicator 146 may display "Summarizing . . . " or a predefined icon indicating that summary summarization is in progress. In addition, stream type indicator 148 may display to user 101 an indication of whether the stream currently being displayed is a transcription stream 130 or a summary stream 134.

[0094] exist Figure 2B In the example, similar to Figure 2A , a transcription stream (212) including transcribed text may be received. For example, transcription stream 130 may be received from transcription generator 124, thereby providing a transcription (e.g., transcription 126) of speech 104 of speaker 100. Figure 2A As also described, the transcription stream 130 can be input into a summary machine learning (ML) model (e.g., summary summarizer 136) to obtain a summary stream 134 (214) including summary summary text (such as summary 106).

[0095] Within the summary text, summary content associated with at least one action may be identified (216). For example, entity extraction model 118 may identify named entities within transcription 126 or summary 106 as summary content that is pre-designated as being associated with an action that is executable. For example, such summary content may include entities such as a person, a company, a phone number, a date, a location, or an event, to provide some non-limiting examples. A corresponding action refers to any automated action that may be associated with at least one of the entities (or other summary content or summary content types) and that may be pre-defined as executable (e.g., in response to a selection by user 101, or in response to some other triggering factor).

[0096] The summary summary text including the summary summary content is rendered on a graphical user interface (GUI) along with an action indicator that relates the summary summary content to at least one action (218). For example, the rendering engine 122 may be configured to render the summary stream 134, including the summary 106. The summary 106 may include specific summary content (e.g., specific words, phrases, names, or numbers) that is visually indicated as being associated with the corresponding action. For example, a visual action indicator, such as color, font size, font type, highlighting, or other visual indication or visual distinction, may be provided with respect to the summary content associated with the action that can be taken.

[0097] A selection of the summary content may be received via the GUI (220). For example, input handler 120 may be configured to receive a selection of action selector 154 by user 101. Action selector 154 may include simply clicking or otherwise selecting the summary content of the visual indication. Various additional example selection techniques are described below, for example, with respect to Fig. 8A and Figure 8B In more specific examples, such as regarding Figure 6B As illustrated and described, the stream scroll bar 152 may be combined with the action selector 154, such as by making a scroll button or other portion of the stream scroll bar 152 selectable. Then, when the scroll button is aligned with summary content rendered within the summary summary text, selection of the selectable scroll button may be received as an invocation of at least one action (and / or received as a selection of the summary content). In still other examples, the summary content may be identified and selected by the input handler 120 and / or the rendering engine 122. For example, the generation of a contact card for the speaker 100 (or other action) may be performed by default without requiring the user 101 to manually perform an action selection. Such default actions may be configured by the user 101 and / or a designer of the summary stream manager 102.

[0098] At least one action may be performed in response to the selection (222). For example, input handler 120 may be configured to interact with application 156 to invoke at least one action. Rendering engine 122 may render one of various indications on display 140 showing that the action has been invoked, is being executed, and / or has been completed.

[0099] Figure 3A It is an example Figure 1 3 , a first screenshot 302 at a first time illustrates an available screen area for displaying text (e.g., transcription or summary text), but in which no text is currently displayed. A transcription icon 316 indicates the potential availability of transcribed text, e.g., notifying the user 101 that speech 104 has been heard and is being processed / converted. For example, the transcription icon 316 may indicate that transcribed text is available (e.g., within the transcription buffer 128) but is not currently selected for viewing, or may indicate that speech 104 has been received and is currently in the process of being transcribed for display. Thus, the transcription icon 316 may be understood to indicate that Figure 1 An example implementation of the stream type indicator 148 is shown.

[0100] In response to Figure 1 In response to a tap 310 or other user selection / request for a summary received at the input handler 120 of the second screen shot 304, a summary icon 318 is shown, which indicates that Figure 1 304 further illustrates a processing indicator 319 that indicates to the user 101 that a summary request has been received (e.g., tap 310) and that summary summarization is in progress. In other words, a delay may be introduced while the user 101 waits for the transcription to eventually complete, and then waits for the summarizer 136 to complete the summary summarization operation. During this delay, the processing indicator 319 may be used to indicate to the user 101 that the speech 104 has been detected for transcription, the summarizer 136 has been invoked, and a summary is about to be provided. Figure 3A In the example of FIG. 3 , the processing indicator 319 is illustrated as an ellipse, but any suitable and predefined indicator may be used. More generally, the processing indicator 319 may be understood to represent Figure 1 An example implementation of the flow state indicator 146 is shown.

[0101] For example, if user 101 wears smart glasses (e.g., Fig. 8A and Figure 8B), the user 101 may tap the smart glasses to request a summary, or may use any other suitable and available input mechanism of the smart glasses. In other examples, summary summarization may be automatically initiated in response to external environmental conditions of the user 101 (including, for example, characteristics of the voice 104 as determined by the voice analyzer 116).

[0102] The relevant summary summarization process 312 may be performed by the summary stream manager 102 as described above. For example, the summary summarizer 136 may process the transcript 126 from within the transcript buffer 128 (not currently selected for display).

[0103] In screenshot 306, summary icon 318 is illustrated as having a title "Summary" 320, and a body 321 of the first portion of the summary generated by summary summarizer 136. Arrow 322 indicates the availability of a paging interface, i.e., summary body 321 may extend beyond the available screen area of ​​screenshot 306. Thus, user 101 may provide another tap 323 or other suitable input to advance to a subsequent summary portion, as shown in screenshot 308. Then, after another tap (or timeout) 314, Figure 3A The processing flow can return to the state of the screenshot 302. Therefore, Figure 3A It is illustrated that as speech 104 is received / available, the transcribed text may be continuously summarized and presented to the user 101 .

[0104] Figure 3B exemplify Figure 1 A pair of screenshots of the system at its first moment. Figure 3B , screenshot 324 shows a running transcription 330, while screenshot 326 shows a summary 334 of the running transcription 330. The transcription icon 328 in screenshot 324 thus identifies the transcription 330, while the summary icon 332 in screenshot 326 identifies the summary 334. As shown, the transcription "Hello, good morning everyone! Thank you very much for coming. Today" has been summarized as "Good morning everyone!".

[0105] Figure 3C Example at the second time Figure 3B The pair of screenshots of Figure 3C, screenshot 336 illustrates an updated / more current transcription "coming. Today we have a guest speaker who will betaking us through our morning exercises". Screenshot 338 illustrates the corresponding summary "Our guest speaker will lead exercises".

[0106] Thus, screenshots 324 / 336 illustrate real-time, verbatim rendering and layout of transcribed speech that prioritizes low-latency delivery of information in its complete form, with words presented as soon as they are available and punctuation added. Meanwhile, screenshots 326 / 338 illustrate summaries with additional delays imparted by the operation of the summary summarizer 136. In order to minimize disruption to such summaries and enhance the readability of such summaries, the summary summarizer 136 and / or rendering engine 122 may add additional delays beyond the delays caused by the operation of the summary summarizer so as to provide segments or blocks of summary summary text at a time (rather than when summary summary words become available). For example, a summary may be provided when a period is encountered in the summary, or when the entire transcribed segment is summarized.

[0107] therefore, Figure 3B and Figure 3C An example of displaying both a running transcript and a corresponding summary is illustrated. Such a dual display may be provided vertically or horizontally relative to each other, or in any suitable configuration. In addition to the use of a summary icon 332 and a transcript icon 328, the running transcripts of the screenshots 324, 336 may be distinguished from the running summaries 326, 338 by any suitable visual indication, including, for example, color, font, or size. In this way, the user 101 will always know clearly which text is being transcribed / summarized, in case, for example, a summary error occurs.

[0108] However, in many contexts, displaying both the running transcript and the running summary may be difficult, problematic, or undesirable for user 101. For example, in a context where glasses have a small field of view and limited display resolution, displaying both the running transcript and the running summary may be impractical. Figure 1 As described above, the stream selector 144 of can provide the user 101 with the ability to select to run a transcription or to run a summary at a given point in time.

[0109] For example, in Figure 3D (The example shows Figure 3C In the first alternative embodiment of the example of FIG. 3 , the screenshot 340 includes a title 342 and a Figure 1 144 of the example of the stream selector 144 of the stream switching switch 344. As shown in the figure, the switching switch 344 is a selectable two-position switch, which can be set to the transcription mode / transcription position (by Figure 3B ) or summary mode / summary position (indicated by the transcription icon 328 of Figure 3B In this example, the stream toggle switch 344 is set to summary mode so that a summary 346 (similar to Figure 3C 338), and the title 342 is marked as summary.

[0110] exist Figure 3E , screenshot 348 illustrates an example in which toggle switch 350 (which is similar or identical to toggle switch 344 but numbered differently for clarity) is set to transcription mode so that title 352 is labeled Transcribe and the same as Figure 3C The transcription of screenshot 336 corresponds to transcription 354 .

[0111] exist Figure 3D and Figure 3E In the example of , the toggle switch 344 / 350 is illustrated simultaneously with the provided text (e.g., summary 346 or transcription 354). However, in alternative implementations, the toggle switch 344 / 350 may be provided in a separate screen, such as in a settings menu.

[0112] For example, Figure 3F Examples of Figure 1 Screen shot 356 of a settings menu in the system. For example, user 101 may use settings icon 358 to access Figure 3F The settings menu, the settings icon may be displayed instead of Figure 3D Title 342 (or Figure 3E By selecting the settings icon 358 and accessing the settings menu of the screenshot 356, the user 101 can access the settings for managing Figure 1 The transcription stream 130 and / or summary stream 134 are associated with a plurality of tools of preference.

[0113] For example, as shown in the figure, a stream switch 360 may be provided in the context of a settings menu. At the same time, a translation switch 362 may be provided, with which the user 101 may select to receive, for example, Figure 1 one of a plurality of available translations of the transcript 126 and / or summary 106, or FIG. 3A to FIG. 3E Any previous transcription / abstract of Figure 3FIn the specific example of FIG. 3 , the translation toggle switch 362 has an option set to Spanish, but any available language may be provided.

[0114] Once a translation is selected, either or both of the transcript or summary may be translated. For example, a translation engine (of device 138 or another device communicatively connected to device 138) may be used to translate at least one selected stream into a selected language for display on the GUI. For example, in Figure 3F In the example of , the setting is for summary mode, where the summary is provided in English, but if the user 101 changes the translation toggle switch 362 to Spanish, the summary will be provided in Spanish. Similarly, the flow toggle switch 360 can be set to transcription, and the setting of the translation toggle switch 362 will determine whether the transcription is provided in English or Spanish.

[0115] Thus, real-time speech-to-text processing of the type described can be implemented to provide multiple types of conversions of received speech, including transcription, translation, summary summarization, or combinations thereof. The various conversions may inherently introduce variations in properties related to their presentation in the user interface, including, for example, latency, length / paging, formatting, and / or additional metadata. In addition, each of these conversions may have multiple types of conversions based on, for example, various defined contexts. The described techniques can be used to implement these variations so that all such conversions are presented within the available user interface in an efficient manner, particularly when screen real estate is limited (as is the case with smart glasses and other wearable devices).

[0116] For example, factors such as contextual signals from the surrounding environment, direct input as specified by the user 101, or context inferred from the speech itself may affect the type of transformation applied and the appropriate presentation strategy. For example, as mentioned above, the summary summarizer 136 can implement different types of summaries, such as summaries for lectures, conversations, directions, or ordered / unordered lists. These and various other contexts can be defined and, as just mentioned, can be determined by, for example, the speech analyzer 116, or specified by the user 101. For example, Figure 1 The context indicator 150 may be used by the user 101 to select a desired context via the input handler 120. In other examples, the speech analyzer 116 may determine the appropriate context, which may then be displayed explicitly or implicitly within the resulting summary.

[0117] FIG. 4A to FIG. 4D Examples of various strategies for layout and interaction mechanisms for various types of (context) summaries are illustrated. For example, Figure 4ASummary 402 is illustrated as an ordered list of instructions for cooking a meal, including step 1: preheat oven; step 2: season fish; and step 3: bake for 15 minutes. Figure 4B An alternative example of the same summary is illustrated, but the summary 404 is illustrated with an interactive checkbox instead of Figure 4A The number of the ordered list.

[0118] In contrast, Figure 4C An example summary 406 is presented, which is an unordered list of action items to be performed. Although not shown separately, Figure 4C An unordered list can also be presented with interactive checkboxes for the user to indicate completion of each item.

[0119] exist Figure 4D In the example, the summary title 408 explicitly lists the relevant context of the business name (Mike's Bistro), while the summary 410 includes the location, distance, and contact information (e.g., phone number). Figure 4D In the summary title 408 is used as Figure 1 An explicit implementation of the context indicator 150.

[0120] Therefore, in addition to FIG. 3A to FIG. 3F In addition to the examples provided previously, FIG. 4A to FIG. 4D Non-limiting examples of available summary types are also provided, which can be understood to be provided as default summary types, such as conversations or lectures. In some implementations, similar to Figure 3E A settings menu for setting summary preferences or contexts may be used. For example, a settings menu may be used to specify a work or social setting, a single person or a group setting, an indoor / outdoor setting, or any suitable context information for which the summary summarizer 136 may be trained. Thus, such a settings menu may also include settings for FIG. 4A to FIG. 4D Settings for various summary types / contexts, including ordered lists, unordered lists, how-to guides, or location / contact information.

[0121] Figure 5A exemplify Figure 1 An example display layout for a system. Figure 5A In the example of FIG. 5 , the layout template 502 includes a title portion 504 and a body portion 506. As shown, the title portion 504 can be used to display one or more icons and / or related meta information or metadata, and the body portion 506 can include a specified number of rows (e.g., rows 1 to 4), each row having a specified number of words (e.g., 4 words per row), and these rows are available, such as for scrolling through Figure 1 Summary of Stream 134.

[0122] Layout template 502 can be used Figure 1The display layout template 502 may be constrained or otherwise limited by one or more of the device characteristics 108 and / or user preferences 110 in the layout template 502. For example, the device characteristics 108 may specify, for example, a maximum number of lines and / or words per line, which may directly or indirectly affect other parameters such as font size. The device characteristics 108 may also specify a minimum or maximum scroll rate for the layout template 502, as well as any other display parameters and associated minimum, maximum, or optimal values.

[0123] Thus, user preferences 110 may specify the preferred values ​​of user 101 within the constraints of device characteristics 108. For example, user preferences 110 may specify less than four lines, or less than four words per line, in layout template 502 (e.g., so that the size of each word may be larger than in the example of FIG. 5 ). User preferences 110 may also specify the scrolling rate experienced by user 101, where the scrolling rate may be specified as slow / medium / fast (or between values ​​ranging between 0 and 1) defined relative to the minimum / maximum available scrolling rate of the associated device / display.

[0124] Title 504 may include nearly any information that may be useful to user 101 in interpreting, understanding, or otherwise using the summary flow provided in layout body 506. For example, as shown in example layout 508, title 510 indicates that body section 512 is being rendered in Spanish and conforms to body section 506 of layout template 502.

[0125] In further example layout 514, title 516 indicates that the summary summarization operation is being processed and / or has been processed. For example, as mentioned above, in addition to indicating that the summary summarization is being performed, there may be a delay associated with inputting the transcript 126 and outputting the summary 106, and title 516 may be used to convey the corresponding summary summarization status to user 101 until the summary is ready to be included in body portion 518.

[0126] Figure 5B A series of screenshots showing action selection from summary content are shown. That is, refer back to Figure 1 , Figure 5B Example operations of the entity extraction model 118 are illustrated in recognizing and identifying entities within either or both of the transcription 126 and the summary 106 , for which one or more actions may then be taken using the input handler 120 (eg, with respect to the application 156 ).

[0127] exist Figure 5B , screenshot 520 illustrates a body area 522 and a title 524 of the type of title described and illustrated above, with an icon, title text, and a status indicator (ellipse) used to indicate that a transcription is available and its summary is being processed.

[0128] In subsequent screenshot 526, title 528 indicates that a summary is being provided, and a corresponding summary 530 is rendered. Specifically, the rendered summary 530 states, as shown, "We'll have a meeting with the client next Thursday at 10am at the Woolworth building."

[0129] In subsequent screenshot 532, the title includes an action selector 534 and an action indicator 536. That is, the action selector 534 and the action indicator 536 indicate that Figure 1 Example implementation of the action selector 154. Figure 5B , the action indicator 536 is the word 'event', which indicates that the entity extraction model 118 has identified the event 538 within the summary 530 of "Thursday at 10am at the Woolworth building". Then, in the subsequent screenshot 540, the action selector 534 (numbered as action selector 542 for clarity) is visually indicated as having been selected by the user 101, and an action result 544 of "Added to Calendar" is displayed, indicating that the event 538 has been added to the calendar of the user 101 (as Figure 1 in the application 156).

[0130] Fig. 6A is an example screenshot of an alternative embodiment for selecting actions from the summary content. Fig. 6A In the example of , the screenshot 602 includes a transcription icon 604 indicating that the main body 606 of the text is live speech (e.g., Figure 1 The entity extraction model 118 has identified the phone number 608, which can be visually highlighted to indicate its status as an actionable item. Then, an action selector 612 can be provided separately from the phone number 608, thereby facilitating identification / selection by the user 101 in conjunction with the reproduced phone number 610.

[0131] Figure 6B Illustrate a series of screenshots showing the use of a scroll bar to select actions from within the summary content. Figure 6B, screenshot 614 includes a text body 616, which may represent a transcription or abstract summary text, wherein an address entity 618 "1244 Washington Ave." has been identified by the entity extraction model 118 and visually highlighted as representing an actionable item.

[0132] As Figure 1 The scroll bar 620 of the example of the flow scroll bar 152 includes a scroll button 622 implemented as an example of the action selector 154. In other words, as illustrated in the figure, the scroll button 622 can be implemented similar to Fig. 6A The action selector 612 is an action selector.

[0133] As about Figure 1 152, scroll bar 620 can be used to scroll through the text in the transcription buffer 128 or summary buffer 132, depending on whether the user uses the stream selector 144 ( Figure 6B 614 , the time stamp 624 indicates that the user 101 has scrolled back 15 seconds from the most recent available text.

[0134] To select the actionable item of address entity 618, user 101 may move / scroll scroll button 622 to align with address entity 618. In other words, in this example, the horizontal alignment of scroll button 622 with actionable address entity 618 indicates that the scroll button may be used as an action selector to select address entity 618 (as opposed to other action items / entities that may be included, such as Figure 6B ) and causes the associated action to be performed. For example, as shown in the figure, the scroll button 622 can be rendered as Fig. 6A An action selector 612 may include, for example, an arrow indicating the availability of an action to be taken.

[0135] Thus, after selecting the actionable scroll button 622, the screenshot 626 indicates that the actionable address entity 618 has been selected, and the action text 628 indicates that the corresponding action is being taken. For example, the action text 628 indicates that the map location of the address entity 618 is being opened, such as using Figure 1 156 is an example of a navigation application. The scroll button 622 and / or the scroll bar 620 may be visually modified (eg, changed to a different color) by the rendering engine 122 to indicate that a specified action is in progress.

[0136] In subsequent screenshot 630, the display has returned to the previously provided text, which has been scrolled forward in time within the associated buffer using scroll bar 620 to display text 632 including actionable item 634 "brunch" at timestamp 638, 7 seconds after the most recently available text. In the example of screenshot 630, scroll button 622 (shown as scroll button 636 for clarity) displays a different icon than the scroll button 622 of screenshots 614, 626, which is the same as the scroll button 622 of screenshots 614, 626. Fig. 6A 630. In contrast, since the scroll button 636 is not aligned with the actionable item 634 and is therefore not in an actionable or selectable state, the scroll button 636 may be rendered in a non-selectable state, for example, using any available / specified icon (such as the icon shown in the screenshot 630).

[0137] Further in Figure 6B In the example of , screenshot 640 illustrates further scrolling through the provided text to the most recently available text, so that a separate timestamp is not included. In the illustrated text 642, an actionable item 644 for a phone number is illustrated. Because scroll button 622 (labeled as scroll button 646 for clarity) is aligned with the identified phone number, scroll button 646 is rendered and designated as being in an actionable state similar to scroll button 622. Therefore, upon selection thereof, screenshot 648 displays text 650 indicating that the identified phone number is being called (e.g., using a phone application as a Figure 1 Applications 156).

[0138] Figure 7 is a user 702 in the surrounding environment 7000 (similar to Figure 1 101), in which one or more external computing systems are shown as additional resources 752 accessible to user 702 via network 7200. Figure 7 A plurality of different wearable devices operable by the user 702 are illustrated on one or more body parts of the user 702, including a first wearable device 750 in the form of glasses worn on the user's head, a second wearable device 754 in the form of earplugs worn on one or both ears of the user 702, a third wearable device 756 in the form of a watch worn on the user's wrist, and a computing device 706 held by the user 702. Figure 7 , computing device 706 is illustrated as a handheld computing device, but may also be understood to represent any personal computing device, such as a tablet computer or a personal computer.

[0139] In some examples, the first wearable device 750 is in the form of a pair of smart glasses that include, for example, a display, one or more image sensors that can capture images of the surrounding environment, audio input / output devices, user input capabilities, computing / processing capabilities, etc. Fig. 8A and Figure 8B Additional examples of the first wearable device 750 are provided.

[0140] In some examples, the second wearable device 754 is in the form of an ear-worn computing device such as headphones or earbuds, which may include audio input / output capabilities, image sensors that can capture images of the surrounding environment 7000, computing / processing capabilities, user input capabilities, etc. In some examples, the third wearable device 756 is in the form of a smart watch or smart bracelet, which includes, for example, a display, an image sensor that can capture images of the surrounding environment, audio input / output capabilities, computing / processing capabilities, user input capabilities, etc. In some examples, the handheld computing device 706 may include a display, one or more image sensors that can capture images of the surrounding environment, audio input / output capabilities, computing / processing capabilities, user input capabilities, etc. (such as in a smartphone). In some examples, the example wearable devices 750, 754, 756 and the example handheld computing device 706 can communicate with each other and / or communicate with the external computing system 752 to exchange information, receive and send input and / or output, etc. The principles described herein can be applied to Figure 7 Other types of wearable devices not specifically shown or described herein.

[0141] User 702 may choose to use any one or more of devices 706, 750, 754, or 756 (possibly in conjunction with external resource 752) to implement the above description of Figure 1 6C . For example, user 702 may use an application executed on device 706 and / or smart glasses 750 to receive, transcribe, and display Figure 1 Transcription flow 130 and / or Figure 1 Summary of Stream 134.

[0142] As mentioned above, device 706 may access additional resources 752 to facilitate various summarization techniques or related techniques described herein. In some examples, additional resources 752 may be partially or completely available locally on device 706. In some examples, some of additional resources 752 may be available locally on device 706, and some of additional resources 752 may be available to device 706 via network 7200. As shown, additional resources 752 may include, for example, server computer systems, processors, databases, memory storage, etc. In some examples, processors may include training engines, transcription engines, translation engines, rendering engines, and other such processors. In some examples, additional resources may include ML models, such as Figure 1 and / or various ML models of the architecture of FIG. 3 .

[0143] The device 706 may operate under the control of the control system 760. The device 706 may communicate with one or more external devices directly (via wired and / or wireless communication) or via the network 7200. In some examples, the one or more external devices may include various devices in the illustrated wearable computing devices 750, 754, 756, another mobile computing device similar to the device 706, etc. In some implementations, the device 706 includes a communication module 762 to facilitate external communication. In some implementations, the device 706 includes a sensing system 764, which includes various sensing system components. The sensing system components may include, for example, one or more image sensors 765, one or more position / orientation sensors 764 (including, for example, inertial measurement units, accelerometers, gyroscopes, magnetometers, and other such sensors), one or more audio sensors 766 that can detect audio input, one or more touch input sensors 768 that can detect touch input, and other such sensors. The computing device 706 may include more or fewer sensing devices and / or a combination of sensing devices.

[0144] The captured still and / or moving images may be displayed by a display device of the output system 772 and / or sent externally via the communication module 762 and the network 7200, and / or stored in the memory 770 of the device 706. The device 706 may include one or more processors 774. The processor 774 may include various modules or engines configured to perform various functions. In some examples, the processor 774 may include, for example, a training engine, a transcription engine, a translation engine, a rendering engine, and other such processors. The processor 774 may be formed in a substrate that is configured to execute one or more machine executable instructions or multiple pieces of software, firmware, or a combination thereof. The processor 774 may be semiconductor-based, including semiconductor materials that can execute digital logic. The memory 770 may include any type of storage device or non-transitory computer-readable storage medium that stores information in a format that can be read and / or executed by the processor 774. The memory 770 may store applications and modules that perform certain operations when executed by the processor 774. In some examples, the applications and modules may be stored in an external storage device and loaded into the memory 770.

[0145] Despite Figure 7 706, but it should be understood that the various resources of the computing device 706 may be implemented in whole or in part in one or more of the various wearable devices including the illustrated smart glasses 750, earbuds 754, and smart watches 756, which may communicate with each other to provide the various features and functions described herein. For example, the memory 770 may be used to implement the transcription buffer 128 and the summary buffer 132.

[0146] exist Figure 7 In the present invention, any audio and / or video output can be used to provide summaries of the type described herein and associated features. For example, the described technology can be implemented in any product where improved speech-to-text conversion would be helpful and high-quality summaries would be beneficial. In addition to head-mounted displays, wearable devices, and mobile devices, the described technology can also be used for teleconferencing and web apps (including, for example, providing transcripts / summaries within web conferencing software and / or pre-recorded videos).

[0147] The described techniques may also be useful in conjunction with, for example, the translation capabilities of additional resources 752. For example, user 702 may hear a translation from a separate speaker (with Figure 1 The user 702 may be provided with a conversation with a speaker 100 (corresponding to a speaker 100), who may be close to the user 702, or far from the user, where the speaker may speak in the first language. The translation engine of the processor of the additional resource 752 may provide an automated translation of the conversation into the native language of the user 702, and may also summarize the translated conversation using the techniques described herein.

[0148] Figure 7The architecture can be used to implement or access one or more large language models (LLMs) that can be used to implement the summary summarizer used in the previous examples. For example, the Pathways Language Model (PaLM) and / or the Language Model for Conversational Applications (LaMDA), both provided by Google, can be used.

[0149] For discussion and illustration purposes, Fig. 8A and Figure 8B An example head-mounted wearable device 800 in the form of a pair of smart glasses is shown in FIG. The example head-mounted wearable device 800 includes a frame 802 having an edge portion 803 surrounding a glass portion or lens 807, and an arm portion 830 coupled to the corresponding edge portion 803. In some examples, the lens 807 can be a corrective / prescription lens. In some examples, the lens 807 can be a glass portion that does not necessarily incorporate a corrective / prescription parameter. A bridge portion 809 can connect the edge portion 803 of the frame 802. In FIG. Fig. 8A and Figure 8B In the example shown, for purposes of discussion and illustration only, wearable device 800 is in the form of a pair of smart glasses, or augmented reality glasses.

[0150] In some examples, the wearable device 800 includes a display device 804 that can output visual content, such as at an output coupler that provides a visual display area 805, so that the visual content is visible to a user. Fig. 8A and Figure 8B In the example shown, for the purpose of discussion and illustration only, the display device 804 is provided in one of the two arm portions 830. The display device 804 may be provided in each of the two arm portions 830 to provide a binocular output of the content. In some examples, the display device 804 may be a perspective near-eye display. In some examples, the display device 804 may be configured to project light from a display source onto a portion of a teleprompter glass that acts as a beam splitter disposed at an angle (e.g., 30 to 45 degrees). The beam splitter may achieve a reflection value and a transmission value that allow light from the display source to be partially reflected and the remaining light to be transmitted. This optical design may allow a user to see physical objects in the world in two ways, such as through the lens 807, next to the content (e.g., digital images, user interface elements, virtual content, etc.) output by the display device 804. In some implementations, waveguide optics may be used to depict content on the display device 804.

[0151] like Fig. 8A and Figure 8BThe example wearable device 800 shown in the form of smart glasses includes one or more of the following: an audio output device 806 (such as, for example, one or more speakers), an illumination device 808, a sensing system 810, a control system 812, at least one processor 814, and an outward-facing image sensor 816 (e.g., a camera). In some examples, the sensing system 810 may include various sensing devices, and the control system 812 may include various control system devices, including, for example, at least one processor 814 that is operably coupled to components of the control system 812. In some examples, the control system 812 may include a communication module that provides communication and information exchange between the wearable device 800 and other external devices. In some examples, the head-mounted wearable device 800 includes a gaze tracking device 815 to detect and track eye gaze direction and movement. Data captured by the gaze tracking device 815 can be processed to detect and track gaze direction and movement as user input. In Fig. 8A and Figure 8B In the example shown, the gaze tracking device 815 is provided in one of the two arm portions 830 for purposes of discussion and illustration only. Fig. 8A and Figure 8B In the example arrangement shown, the gaze tracking device 815 is provided in the same arm portion 830 as the display device 804, so that the user's eye gaze can be tracked not only relative to objects in the physical environment, but also relative to the content output displayed by the display device 804. In some examples, the gaze tracking device 815 can be provided in each of the two arm portions 830 to provide gaze tracking for each of the user's two eyes. In some examples, the display device 804 can be provided in each of the two arm portions 830 to provide a binocular display of visual content.

[0152] Wearable device 800 is illustrated as glasses, such as smart glasses, augmented reality (AR) glasses, or virtual reality (VR) glasses. More generally, wearable device 800 may represent any head mounted device (HMD), including, for example, a hat, a helmet, or a headband. Even more generally, wearable device 800 and computing device 706 may represent any wearable device, handheld computing device, or combination thereof.

[0153] Wearable device 800 and similar wearable devices or handheld devices (such as Figure 7 The use of those shown) realizes Figure 14. For example, such wearable devices and handheld devices may be highly portable and therefore available to user 702 in many different scenarios. At the same time, the available display area of ​​such devices may be limited. For example, the display area 805 of the wearable device 800 may be a relatively small display area, constrained by the overall size and form factor of the wearable device 800.

[0154] Thus, user 702 may benefit from the use of various summarization techniques described herein. For example, user 702 may interact with a single speaker, such as a lecturer or a participant in a conversation. User 702 and the single speaker may have varying degrees of interactivity or back-and-forth, and there may also be two or more additional speakers.

[0155] Using the described techniques, as interactions occur, dynamic, real-time digest summaries can be provided to user 702 during all such interactions. For example, speakers can speak for short or long periods of time in conjunction with (e.g., in response to) dialogue provided by user 702. During all such interactions, user 702 can be provided with useful and convenient summaries of the words spoken by individual speakers.

[0156] As described, the dynamic, real-time summary may have a compression ratio and complexity that is dynamically updated, or may otherwise be dynamically adjusted over time and during the course of a conversation or other interaction. Thus, a user 101 / 702 may be provided with a meaningful, context-specific summary that reduces the cognitive burden on the user 101 / 702 and promotes meaningful interaction, even in situations where one or more participants in the interaction are non-native speakers, are currently speaking a different language, or are an expert in a field who is speaking to a novice in the field.

[0157] In a first example implementation (referred to herein as Example 1), a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and includes instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream including summary text; Rendering a stream selector icon on the user interface (UI); receiving, via a stream selector icon, a selection of at least one of a transcription stream and a summary stream; and Display of at least one selected stream is caused to be performed on the UI.

[0158] Example 2 includes the computer program product of Example 1, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: Renders the stream selector icon as a toggle switch with a Transcription position for selecting a transcribed stream and a Summary position for selecting a Summary stream.

[0159] Example 3 includes the computer program product of Example 1 or 2, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: Using the UI and when the at least one selected stream includes a summary stream, rendering a stream status indicator that provides a status of the summary stream as the transcription stream is currently being processed using the summary summarization ML model.

[0160] Example 4 includes the computer program product of any of the preceding examples, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: A stream type indicator is rendered using the UI, the stream type indicator identifying at least one selected stream as a transcription stream or a summary stream.

[0161] Example 5 includes the computer program product of any of the preceding examples, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: storing the transcription stream in a transcription buffer; and Store the summary stream in the summary buffer.

[0162] Example 6 includes the computer program product of any of the preceding examples, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: Rendering a scroll bar using the UI in conjunction with the at least one selected stream; and An earlier portion of the at least one selected stream is retrieved in response to movement of the scroll bar.

[0163] Example 7 includes the computer program product of any of Examples 1-3, 5, or 6, wherein the at least one selected stream includes both a transcription stream and a summary stream.

[0164] Example 8 includes the computer program product of any of the preceding examples, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: At least one selected flow is translated into a selected language for display using the UI.

[0165] Example 9 includes the computer program product of any of the preceding examples, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: Causes display of at least one selected stream on the head mounted device.

[0166] Example 10 includes the computer program product of any of the preceding examples, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: identifying at least one entity within at least one selected stream; associating at least one entity with at least one action; rendering at least one selected stream in a manner that visually highlights at least one entity; receiving a selection of at least one entity; and Invoke at least one action.

[0167] In an eleventh example implementation (referred to herein as Example 11), an apparatus includes: at least one processor; at least one display; and at least one memory storing instructions that, when executed by at least one processor, cause the apparatus to: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream including summary text; Rendering a stream selector icon on a graphical user interface (GUI); receiving, via a stream selector icon, a selection of at least one of a transcription stream and a summary stream; and Display of at least one selected stream is caused to occur on the GUI.

[0168] Example 12 includes the apparatus of Example 11, wherein the instructions, when executed by at least one processor, are further configured to cause the apparatus to: Renders the stream selector icon as a toggle switch with a Transcription position for selecting a transcribed stream and a Summary position for selecting a Summary stream.

[0169] Example 13 includes the apparatus of Example 11 or 12, wherein the instructions, when executed by at least one processor, are further configured to cause the apparatus to: Using the GUI and when the at least one selected stream includes a summary stream, rendering a stream status indicator that provides a status of the summary stream as the transcription stream is currently being processed using the summary summarization ML model.

[0170] Example 14 includes the apparatus of any one of Examples 11 to 13, wherein the instructions, when executed by at least one processor, are further configured to cause the apparatus to: A stream type indicator is rendered using the GUI, the stream type indicator identifying at least one selected stream as a transcription stream or a summary stream.

[0171] Example 15 includes the apparatus of any of Examples 11 to 14, wherein when executed by at least one processor, the instructions are further configured to cause the apparatus to: storing the transcription stream in a transcription buffer; and Store the summary stream in the summary buffer.

[0172] Example 16 includes the apparatus of any of Examples 11 to 15, wherein when executed by at least one processor, the instructions are further configured to cause the apparatus to: rendering a scroll bar using the GUI in conjunction with the at least one selected stream; and An earlier portion of the at least one selected stream is retrieved in response to movement of the scroll bar.

[0173] In a seventeenth example implementation (referred to herein as Example 17), a method includes: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream including summary text; Rendering a stream selector icon on a graphical user interface (GUI); receiving, via a stream selector icon, a selection of at least one of a transcription stream and a summary stream; and Display of at least one selected stream is caused to occur on the GUI.

[0174] Example 18 includes the method of Example 17, further comprising: Renders the stream selector icon as a toggle switch with a Transcription position for selecting a transcribed stream and a Summary position for selecting a Summary stream.

[0175] Example 19 includes the method of Example 17 or 18, further comprising: storing the transcription stream in a transcription buffer; and Store the summary stream in the summary buffer.

[0176] Example 20 includes the method of any one of Examples 17 to 19, further comprising: rendering a scroll bar using the GUI in conjunction with the at least one selected stream; and An earlier portion of the at least one selected stream is retrieved in response to movement of the scroll bar.

[0177] In a twenty-first example implementation (referred to herein as Example 21), a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and comprises instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream of the summary text; Identifying summary content associated with at least one action within the summary text; rendering summary summary text including summary summary content on a graphical user interface (GUI) along with an action indicator relating the summary summary content to at least one action; receiving, via the GUI, a selection of abstract summary content; and At least one action is performed in response to the selection.

[0178] Example 22 includes the computer program product of Example 21, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: Use entity extraction ML models to identify summary content.

[0179] Example 23 includes the computer program product of Example 21, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: identifying within the transcript content the transcript associated with the second action; rendering, on the GUI, a transcribed text including the transcribed content and a second action indicator relating the transcribed content to a second action; receiving, via the GUI, a second selection of the transcribed content; and A second action is performed in response to the second selection.

[0180] Example 24 includes the computer program product of Example 21 or 22, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: storing the transcription stream in a transcription buffer; and Store the summary stream in the summary buffer.

[0181] Example 25 includes the computer program product of any of Examples 21 to 24, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: Using the GUI in conjunction with the summary stream to render scrollbars; and An earlier portion of the summary stream is retrieved in response to movement of the scroll bar.

[0182] Example 26 includes the computer program product of Example 25, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: detecting alignment of a scroll button of the scroll bar with summary content within the GUI; and Accepts selections made using the scroll button in conjunction with pointing.

[0183] Example 27 includes the computer program product of any one of Examples 21 to 26, wherein the action indicator comprises a visual distinction of the summary summary content relative to the remaining summary summary content of the summary summary text.

[0184] Example 28 includes the computer program product of any one of Examples 21 to 25 or 27, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: rendering, using the GUI, an action selector icon that identifies a type of at least one action; and A selection is received via an action selector icon.

[0185] Example 29 includes the computer program product of any one of Examples 21 to 28, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: Rendering the GUI on a display of the head mounted device.

[0186] Example 30 includes the computer program product of any one of Examples 21 to 29, wherein when executed by at least one computing device, the instructions are further configured to cause the at least one computing device to: At least one action is performed, the at least one action comprising communicating with an application to cause the application to perform the at least one action.

[0187] In a thirty-first example implementation (referred to herein as Example 31), an apparatus includes: at least one processor; at least one display; and at least one memory storing instructions that, when executed by at least one processor, cause the apparatus to: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream of the summary text; Identifying summary content associated with at least one action within the summary text; rendering summary summary text including summary summary content on a graphical user interface (GUI) along with an action indicator relating the summary summary content to at least one action; receiving, via the GUI, a selection of abstract summary content; and At least one action is performed in response to the selection.

[0188] Example 32 includes the apparatus of Example 31, wherein the instructions, when executed by at least one processor, are further configured to cause the apparatus to: Use entity extraction ML models to identify summary content.

[0189] Example 33 includes the apparatus of Example 31 or 32, wherein when executed by at least one processor, the instructions are further configured to cause the apparatus to: identifying within the transcript content the transcript associated with the second action; rendering, on the GUI, a transcribed text including the transcribed content and a second action indicator relating the transcribed content to a second action; receiving, via the GUI, a second selection of the transcribed content; and A second action is performed in response to the second selection.

[0190] Example 34 includes the apparatus of any one of Examples 31 to 33, wherein the instructions, when executed by at least one processor, are further configured to cause the apparatus to: storing the transcription stream in a transcription buffer; and Store the summary stream in the summary buffer.

[0191] Example 35 includes the apparatus of any one of Examples 31 to 34, wherein the instructions, when executed by at least one processor, are further configured to cause the apparatus to: rendering a scroll bar using the GUI in conjunction with the at least one selected stream; and An earlier portion of the at least one selected stream is retrieved in response to movement of the scroll bar.

[0192] Example 36 includes the apparatus of Example 35, wherein the instructions, when executed by at least one processor, are further configured to cause the apparatus to: detecting alignment of a scroll button of the scroll bar with summary content within the GUI; and Accepts selections made using the scroll button in conjunction with pointing.

[0193] In a thirty-seventh example implementation (referred to herein as Example 37), a method includes: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream of the summary text; Identifying summary content associated with at least one action within the summary text; rendering summary summary text including summary summary content on a graphical user interface (GUI) along with an action indicator relating the summary summary content to at least one action; receiving, via the GUI, a selection of abstract summary content; and At least one action is performed in response to the selection.

[0194] Example 38 includes the method of Example 37, further comprising: Use entity extraction ML models to identify summary content.

[0195] Example 39 includes the method of Example 37 or 38, further comprising: storing the transcription stream in a transcription buffer; Store the summary stream in a summary buffer; Using the GUI in conjunction with the summary stream to render scrollbars; and An earlier portion of the summary stream is retrieved from corresponding ones of the transcription buffer and the summary buffer in response to movement of the scroll bar.

[0196] Example 40 includes the method of Example 39, further comprising: detecting alignment of a scroll button of the scroll bar with summary content within the GUI; and Accepts selections made using the scroll button in conjunction with pointing.

[0197] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be special-purpose or general-purpose and can be coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0198] These computer programs (also referred to as modules, programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium," "computer-readable medium," refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0199] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or an LED (light emitting diode)) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and the input from the user can be received in any form, including sound, voice, or tactile input.

[0200] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a client can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0201] A computing system may include clients and servers. Clients and computers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises from computer programs running on the respective computers and having a client-server relationship to each other.

[0202] In some implementations, one or more input devices other than the computing device (e.g., mouse, keyboard) may also be rendered in the display of an HMD such as HMD 800. The rendered input devices (e.g., rendered mouse, rendered keyboard) may be used as if rendered in the display.

[0203] A number of implementations have been described. However, it will be appreciated that various modifications can be made without departing from the spirit and scope of the description and claims.

[0204] Furthermore, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve the desired results. Furthermore, other steps may be provided, or steps may be deleted from the described flows, and other components may be added to or removed from the described systems. Therefore, other implementations are within the scope of the following claims.

[0205] Continuing with the above description, controls are provided to the user, allowing the user to make choices about whether and when the systems, programs, devices, networks, or features described herein can be able to collect user information (e.g., information about the user's social network, social behavior, or activities, occupation, the user's preferences, or the user's current location) and whether to send content or communications from the server to the user. In addition, before certain data is stored or used, it can be processed in one or more ways so that the user information is removed. For example, the user's identity can be processed so that the user's user information cannot be determined, or the user's geographic location for which location information is obtained can be generalized (such as to the city, zip code, or state level) so that the user's specific location cannot be determined. Therefore, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0206] The computer system (e.g., computing device) may be configured to communicate wirelessly with a network server over a network via a communication link established with the network server using any known wireless communication techniques and protocols, including radio frequency (RF), microwave frequency (MWF) and / or infrared frequency (IRF) wireless communication techniques and protocols suitable for communicating over a network.

[0207] According to aspects of the present disclosure, implementations of the various techniques described herein may be implemented in digital electronic circuit systems, or in computer hardware, firmware, software, or in a combination thereof. The implementation may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium) for processing by a data processing device (e.g., a programmable processor, a computer or multiple computers), or to control the operation of the data processing device. In some implementations, a tangible computer-readable storage medium may be configured to store instructions that, when executed, cause the processor to perform a process. A computer program (such as the computer program described above) may be written in any form of programming language, including a compiled language or an interpreted language, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program may be deployed to be processed on a computer or on multiple computers distributed at a site or across multiple sites and interconnected by a communication network.

[0208] The specific structural and functional details disclosed herein are merely representative for the purpose of describing the example implementations. However, the example implementations may be embodied in many alternative forms and should not be construed as being limited to only the implementations set forth herein.

[0209] The terms used herein are for the purpose of describing specific implementations only and are not intended to limit the implementations. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. It should be further understood that the terms "comprises, comprising, includes, and / or including" when used in this specification specify the presence of the features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0210] It should be understood that when an element is referred to as being "coupled," "connected" to another element, "responsive to" another element, or "on another element," the element may be directly coupled, connected to, responsive to, or on the other element, or intervening elements may also be present. In contrast, when an element is referred to as being "directly coupled," "directly connected" to another element, or "directly responsive to" another element, or "directly on another element," there are no intervening elements. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0211] For ease of description, spatially relative terms such as "below", "below", "below", "above", "on", etc. may be used herein to describe the relationship of one element or feature to another element or feature as illustrated in the figure. It should be understood that the spatially relative terms are intended to cover different orientations of the device in use or operation in addition to the orientation depicted in the figure. For example, if the device in the figure is flipped, the elements described as "below" or "below" other elements or features will have an orientation "above" the other elements or features. Therefore, the term "below" can cover both orientations of above and below. The device can be oriented in other ways (rotated 130 degrees or in other orientations), and the spatially relative descriptors used herein can be interpreted accordingly.

[0212] Example implementations of the concepts are described herein with reference to cross-sectional illustrations, which are schematic illustrations of idealized implementations (and intermediate structures) of the example implementations. Therefore, variations in the shapes of the illustrations as a result of, for example, manufacturing techniques and / or tolerances are expected. Therefore, example implementations of the described concepts should not be interpreted as limited to the specific shapes of the areas illustrated herein, but will include shape deviations, for example, due to manufacturing. Therefore, the areas illustrated in the figures are schematic in nature, and their shapes are not intended to illustrate the actual shapes of areas of the device and are not intended to limit the scope of the example implementations.

[0213] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. Therefore, without departing from the teachings of the implementation of the present invention, a "first" element may be referred to as a "second" element.

[0214] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those of ordinary skill in the art to which these concepts belong. It should be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and / or this specification, and should not be interpreted in an idealized or overly formal sense unless explicitly defined as such herein.

[0215] Although certain features of the described implementations have been illustrated as described herein, those skilled in the art will now expect many modifications, substitutions, changes and equivalents. Therefore, it should be understood that the appended claims are intended to cover such modifications and changes that fall within the scope of the implementations. It should be understood that they are presented only by way of example and not limitation, and various changes in form and detail may be made. In addition to mutually exclusive combinations, any part of the devices and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.

Claims

1. A computer program product tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream including summary text; Rendering a stream selector icon on the user interface (UI); receiving, via the stream selector icon, a selected stream of at least one of the transcription stream and the summary stream; as well as The display of the at least one selected stream is caused to be performed on the UI.

2. The computer program product of claim 1 , wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The stream selector icon is rendered as a toggle switch having a transcribed position for selecting the transcribed stream and a summary position for selecting the summary stream.

3. The computer program product of claim 1 or 2, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: When the at least one selected stream includes the summary stream, a stream status indicator is rendered using the UI, the stream status indicator providing a status of the summary stream as the transcription stream is currently being processed using the summary summarization ML model.

4. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: A stream type indicator is rendered using the UI, the stream type indicator identifying the at least one selected stream as the transcription stream or the summary stream.

5. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: storing the transcription stream in a transcription buffer; and The summary stream is stored in a summary buffer.

6. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: rendering a scroll bar using the UI in conjunction with the at least one selected stream; and An earlier portion of the at least one selected stream is retrieved in response to movement of the scroll bar.

7. The computer program product of any one of claims 1 to 3, 5 or 6, wherein the at least one selected stream comprises both the transcription stream and the summary stream.

8. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The at least one selected flow is translated into a selected language for display using the UI.

9. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The at least one selected stream is caused to be displayed on the head mounted device.

10. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: identifying at least one entity within the at least one selected stream; associating the at least one entity with at least one action; rendering the at least one selected flow in a manner that visually highlights the at least one entity; receiving a selection of the at least one entity; as well as The at least one action is invoked.

11. An apparatus comprising: at least one processor; at least one display; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream including summary text; Rendering a stream selector icon on a graphical user interface (GUI); receiving, via the stream selector icon, a selected stream of at least one of the transcription stream and the summary stream; as well as The at least one selected stream is caused to be displayed on the GUI.

12. The apparatus of claim 11, wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: The stream selector icon is rendered as a toggle switch having a transcribed position for selecting the transcribed stream and a summary position for selecting the summary stream.

13. The apparatus of claim 11 or 12, wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: When the at least one selected stream includes the summary stream, a stream status indicator is rendered using the GUI, the stream status indicator providing a status of the summary stream as the transcription stream is currently being processed using the summary summarization ML model.

14. The apparatus of any one of claims 11 to 13, wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: A stream type indicator is rendered using the GUI, the stream type indicator identifying the at least one selected stream as the transcription stream or the summary stream.

15. The apparatus of any one of claims 11 to 14, wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: storing the transcription stream in a transcription buffer; and The summary stream is stored in a summary buffer.

16. The apparatus of any one of claims 11 to 15, wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: rendering a scroll bar using the GUI in conjunction with the at least one selected stream; and An earlier portion of the at least one selected stream is retrieved in response to movement of the scroll bar.

17. A method comprising: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream including summary text; Rendering a stream selector icon on a graphical user interface (GUI); receiving, via the stream selector icon, a selected stream of at least one of the transcription stream and the summary stream; as well as The at least one selected stream is caused to be displayed on the GUI.

18. The method of claim 17, further comprising: The stream selector icon is rendered as a toggle switch having a transcribed position for selecting the transcribed stream and a summary position for selecting the summary stream.

19. The method of claim 17 or 18, further comprising: storing the transcription stream in a transcription buffer; as well as The summary stream is stored in a summary buffer.

20. The method of any one of claims 17 to 19, further comprising: rendering a scroll bar using the GUI in conjunction with the at least one selected stream; as well as An earlier portion of the at least one selected stream is retrieved in response to movement of the scroll bar.

21. A computer program product tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream of summary text; Identifying summary content associated with at least one action within the summary text; rendering the summary summary text including the summary summary content therein on a graphical user interface (GUI) together with an action indicator relating the summary summary content to the at least one action; receiving, via the GUI, a selection of the abstract summary content; and The at least one action is performed in response to the selection.

22. The computer program product of claim 21, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: An entity extraction ML model is used to identify the summary content.

23. The computer program product of claim 21, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: identifying within the transcript content the transcript content associated with the second action; rendering the transcribed text including the transcribed content on the GUI together with a second action indicator relating the transcribed content to the second action; receiving, via the GUI, a second selection of the transcribed content; and The second action is performed in response to the second selection.

24. The computer program product of claim 21 or 22, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: storing the transcription stream in a transcription buffer; and The summary stream is stored in a summary buffer.

25. The computer program product of any one of claims 21 to 24, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: rendering a scroll bar using the GUI in conjunction with the summary stream; and An earlier portion of the summary stream is retrieved in response to movement of the scroll bar.

26. The computer program product of claim 25, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: detecting alignment of a scroll button of the scroll bar with the digest summary content within the GUI; and The selection made using the scroll button in conjunction with the pointing is received.

27. The computer program product of any one of claims 21 to 26, wherein the action indicator comprises a visual distinction of the abstract summary content relative to a remaining abstract summary content of the abstract summary text.

28. The computer program product of any one of claims 21 to 25 or 27, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: rendering, using the GUI, an action selector icon identifying a type of the at least one action; and The selection is received via the action selector icon.

29. The computer program product of any one of claims 21 to 28, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The GUI is rendered on a display of the head mounted device.

30. The computer program product of any one of claims 21 to 29, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The at least one action is performed, the at least one action comprising communicating with an application to cause the application to perform the at least one action.

31. An apparatus comprising: at least one processor; at least one display; as well as at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus to: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream of summary text; Identifying summary content associated with at least one action within the summary text; rendering the summary summary text including the summary summary content therein on a graphical user interface (GUI) together with an action indicator relating the summary summary content to the at least one action; receiving, via the GUI, a selection of the abstract summary content; and The at least one action is performed in response to the selection.

32. The apparatus of claim 31 , wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: An entity extraction ML model is used to identify the summary content.

33. The apparatus of claim 31 or 32, wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: identifying within the transcript content the transcript content associated with the second action; rendering the transcribed text including the transcribed content on the GUI together with a second action indicator relating the transcribed content to the second action; receiving, via the GUI, a second selection of the transcribed content; and The second action is performed in response to the second selection.

34. The apparatus of any one of claims 31 to 33, wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: storing the transcription stream in a transcription buffer; and The summary stream is stored in a summary buffer.

35. The apparatus of any one of claims 31 to 34, wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: rendering a scroll bar using the GUI in conjunction with the at least one selected stream; and An earlier portion of the at least one selected stream is retrieved in response to movement of the scroll bar.

36. The apparatus of claim 35, wherein when executed by the at least one processor, the instructions are further configured to cause the apparatus to: detecting alignment of a scroll button of the scroll bar with the digest summary content within the GUI; and The selection made using the scroll button in conjunction with the pointing is received.

37. A method comprising: receiving a transcription stream including transcribed text; Inputting the transcription stream into a summary machine learning (ML) model to obtain a summary stream of summary text; Identifying summary content associated with at least one action within the summary text; rendering the summary summary text including the summary summary content therein on a graphical user interface (GUI) together with an action indicator relating the summary summary content to the at least one action; receiving, via the GUI, a selection of the abstract summary content; and The at least one action is performed in response to the selection.

38. The method of claim 37, further comprising: An entity extraction ML model is used to identify the summary content.

39. The method of claim 37 or 38, further comprising: storing the transcription stream in a transcription buffer; storing the summary stream in a summary buffer; Rendering a scroll bar using the GUI in conjunction with the summary stream; as well as An earlier portion of the summary stream is retrieved from a corresponding buffer in the transcription buffer and the summary buffer in response to movement of the scroll bar.

40. The method of claim 39, further comprising: detecting alignment of a scroll button of the scroll bar with the digest summary content within the GUI; as well as The selection made using the scroll button in conjunction with the pointing is received.

Images