Digest generation of live digests with user and device customization

By summarizing the machine learning model to process speech transcription text, combining user preferences and device characteristics, dynamically generate real-time summary, solving the problems of large data capacity and short battery life in the existing technology, and achieving efficient and personalized real-time summary generation and show.

CN119096246APending Publication Date: 2024-12-06GOOGLE LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380039288.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-05-10
Filing Date
2023-05-10
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art is difficult to effectively reduce text data capacity when generating live summary, resulting in shortened device battery life and poor user experience, and the inability to dynamically adjust the summary to meet the needs of real-time sessions.

Method used

The summary machine learning model is used to process speech transcription text, combine user preferences and device characteristics, and dynamically generate and display live summary, reduce data transmission volume and optimize display size to adapt to changes in real-time sessions.

Benefits of technology

It realizes efficient reduction of data transmission in real-time conversations, extends device battery life, improves user experience, and provides personalized live summary to adapt to different scenarios and language needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119096246A_ABST
    Figure CN119096246A_ABST
Patent Text Reader

Abstract

The described techniques may be used to receive a transcriptional stream including transcriptional text that has been transcribed from speech, and to receive a summary request for a summary to be provided on a display of a device. The extracted text is identified from the transcriptional text in response to the summary request. The extracted text may be processed using a summary machine learning (ML) model to obtain a summary of the extracted text, and the summary may be displayed on the display of the device. When an image is captured, an enhanced summary may be generated that includes an image and a visual indication of one or more of an emotion, an entity, or an intent associated with the image, the summary, or the extracted text.
Need to check novelty before this filing date? Find Prior Art

Description

Related Applications

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 364,478, filed on May 10, 2022, the disclosure of which is incorporated herein by reference in its entirety.

[0002] The present application also incorporates by reference the disclosures of related co-pending applications: U.S. Application No. 18 / 315,113, filed on May 10, 2023; “Multi-Stage Summarization for Customized, Contextual Summaries” (Agent Docket No. 0120-533WO1), filed on May 10, 2023; “Dynamic Summary Adjustments for Live Summaries” (Agent Docket No. 0120-534WO1), filed on May 10, 2023; “Summary Generation for Live Summaries with User and Device Customization” (Agent Docket No. 0120-535WO1), filed on May 10, 2023; “Summarization with User Interface (UI) Stream Control and Actionable Information Streaming” (Agent Docket No. 0120-536WO1), filed on May 10, 2023. Extraction with User Interface (UI) Flow Control and Actionable Information Extraction ()" (Agent Docket No. 0120-541WO1) and "Incremental Streaming for Live Summaries" (Agent Docket No. 0120-589WO1), filed on May 10, 2023. Technical Field

[0003] This description involves summarization using machine learning (ML) models. Background Art

[0004] A volume of text (such as a document or article) often includes content that is useless or unnecessary to the consumer of the volume of text. Additionally or alternatively, a user may not be willing to spend time (or may not have enough time) to consume the entire volume of text.

[0005] Summarization generally refers to a technique that attempts to reduce a volume of text to obtain a reduced volume of text that retains most of the information of the volume of text in the summary. Thus, users can consume information in a more efficient and desirable way. In order to be able to perform the necessary processing on the text, the latter can be represented by electronic data (text data). For example, an ML model can be trained to input text and output a summary of the text. Summary of the invention

[0006] The described technology processes input text data to reduce the data capacity of the input text data and obtains output text data expressing a summary of the content of the input text data. The reduced output text data capacity obtained can be consistent with the size of the display to optimize the size of the output text data relative to the size of the display. In addition, compared with prior art and methods, the described technology can achieve this customized data capacity reduction with a reduced delay.

[0007] In a general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and includes instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to receive a transcription stream including transcribed text that has been transcribed from speech; receive a summary request for a summary to be provided on a display of the device; and identify extracted text from the transcribed text in response to the summary request. When executed by the at least one computing device, the instructions are configured to cause the at least one computing device to process the extracted text using a summarization machine learning (ML) model to obtain a summary of the extracted text, and display the summary on the display of the device.

[0008] According to another general aspect, an apparatus includes at least one processor, at least one memory, at least one input device, and at least one display, wherein instructions stored using the at least one memory, when executed by the at least one processor, cause the apparatus to receive a transcription stream including transcribed text that has been transcribed from speech; receive, via the input device, a summary request for a summary to be provided on the at least one display; identify extracted text from the transcribed text in response to the summary request; process the extracted text using a summary machine learning (ML) model to obtain a summary of the extracted text; and display the summary on the at least one display.

[0009] According to another general aspect, a method includes: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving a summary request for a summary to be provided on a display of a device; identifying extracted text from the transcribed text in response to the summary request; processing the extracted text using a summary machine learning (ML) model to obtain a summary of the extracted text; and displaying the summary on the display of the device.

[0010] According to another general aspect, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and includes instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to receive a transcription stream including transcribed text that has been transcribed from speech and receive an image associated with the receipt of the transcription stream. When executed by the at least one computing device, the instructions are configured to cause the at least one computing device to process the transcription stream using a summary machine learning (ML) model to obtain a summary stream, including processing the transcribed text to obtain a summary; combining the image and the summary to obtain an enhanced summary; and displaying the enhanced summary.

[0011] According to another general aspect, an apparatus includes at least one processor, at least one memory, at least one input device, and at least one display, wherein instructions stored using the at least one memory, when executed by the at least one processor, cause the apparatus to receive a transcription stream including transcribed text that has been transcribed from speech; receive an image associated with the receipt of the transcription stream; process the transcription stream using a summary machine learning (ML) model to obtain a summary stream, including processing the transcribed text to obtain a summary; combine the image and the summary to obtain an enhanced summary; and display the enhanced summary using the at least one display.

[0012] According to another general aspect, a method includes: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving an image associated with the receipt of the transcription stream; processing the transcription stream using a summary machine learning (ML) model to obtain a summary stream, including processing the transcription text to obtain a summary; combining the image and the summary to obtain an enhanced summary; and displaying the enhanced summary.

[0013] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a block diagram of a system for summarizing live summaries with user and device customization.

[0015] Figure 2 It shows Figure 1 A flowchart of an example operation of a system.

[0016] Figure 3 It shows Figure 1 A block diagram of a more detailed example implementation of a system.

[0017] Figure 4 It is shown that Figure 3 Flowchart of example operations corresponding to the examples.

[0018] Figure 5 Shown for Figure 3 and Figure 4 The example shows the layout of the example.

[0019] Figure 6 Shown for Figure 3 and Figure 4 An example summary of an example of the selection process.

[0020] Figure 7 It is a third-person view of the user in the surrounding computing environment.

[0021] Fig. 8A and Figure 8B Shown are front and back views of an example implementation of a pair of smart glasses.

[0022] Fig. 9 is used for visual enhancement of transcribed text Figure 1 A block diagram of an alternative implementation of a system.

[0023] Fig.10 It shows Figure 1 A flowchart of an example operation of a system.

[0024] Fig.11 It shows Fig. 9 A block diagram of a more detailed example implementation of a system.

[0025] Fig.12 It is shown that Fig.11 Flowchart of example operations corresponding to the examples.

[0026] Fig.13 The transcribed text is shown using Fig.11 A first example implementation of visual enhancement of an implementation of.

[0027] Fig.14 The transcribed text is shown using Fig.11 A second example implementation of visual enhancement of an implementation of .

[0028] Fig.15 An example screenshot is shown where a scroll bar is used to identify text for visual enhancement. DETAILED DESCRIPTION

[0029] The described systems and techniques enable customized summary generation during a live conversation between a speaker and a user. Input speech (audio data) received at a device during the live conversation can be transcribed, and the resulting transcribed text (text data) can be provided as a caption using a display of the device (or another device). In response to a summary request or other summary trigger, a recent portion of the transcribed text can be extracted and processed using at least one trained summary model or summarizer to provide a summary of the speech.

[0030] Conventional summary generation uses a text prompt to trigger summarization, such as "summarize the following text:". In contrast, the described technology summarizes speech-to-text results during a live conversation, dialog, or other interaction between a speaker and a user. For example, such live summarization can be triggered in response to a summary request received from a user and / or based on the speaker's speech content, user interface constraints of a device / display for providing the transcription / summary, and user preferences regarding whether, when, and how to generate summaries (e.g., determined based on a user-selected device setting or other user operation of the device). Such technology achieves reduced computational workload and intelligent switching between live transcription and live summarization.

[0031] In a specific example, augmented reality (AR) glasses and virtual reality (VR) headsets with video-see-through capabilities are becoming increasingly popular. Such devices have many advantages over traditional smartphones and tablet computers, including, for example, hands-free access to information and the ability to overlay digital content (e.g., real-time text descriptions) on the real world. However, one of the main challenges of such devices is that the field of view is typically limited (e.g., within 20 degrees to ensure all-day battery life), thereby limiting how much text can be reasonably presented on the corresponding display.

[0032] The type of adaptive speech summarization described herein helps address resolution / display limited issues and difficulties, for example, by reducing the amount of data transmitted from a paired device (e.g., a smartphone) to the AR glasses. As described in detail below, this reduction in the amount of data transmitted can be achieved by automatically summarizing the key points of the conversation using a Transformer-based language model, or by compressing the audio file itself.

[0033] As a result, such AR / VR devices may be more comfortable to wear for long periods of time, and the battery life of the device may be improved. Reducing the amount of data that needs to be transmitted not only has the potential to improve the battery life of the device, but also provides new opportunities for personalized experience for each user, potentially making AR glasses more comfortable, user-friendly, and accurate. For example, the volume of the audio can be adjusted dynamically, and / or certain keywords or phrases can be highlighted.

[0034] The described techniques provide methods for adaptively summarizing and compressing speech, for example, in response to a summary request received from a user, and / or in response to some other summarization trigger. For example, a summary request may be received manually from a user via a hardware input device such as a touch screen or capacitive touch user interface, a physical button, a hand gesture or head pose, or other input appropriate to the form factor of the device being used. In other examples, a summary request may be received verbally, such as by a user stating the word "summarize".

[0035] Additionally or alternatively, a summary trigger may be detected based on speech characteristics of the speech being analyzed. For example, speech containing a defined number of words (e.g., within a specified time interval), such as 200 words, may be automatically summarized, while speech with less than the defined number of words may not be summarized (unless requested by the user). In a further example, if the transcribed speech can be appropriately displayed without summarization, speech with less than a minimum number of words (e.g., 30 words) may not be summarized even if a summary request is received from the user.

[0036] As described above, the relevant speech characteristics can be expressed as speech rate (e.g., words per second or per minute) rather than word count. Other criteria can also be used, such as using a pause of sufficient length (e.g., 2 seconds) detected within the speech as a summary trigger. Combinations of such summary triggers can also be used, as well as considerations related to the resolution / size of the relevant display and / or user preferences for whether / when / how to receive summaries, as described above. In addition, user selections and actions can be used over time to fine-tune the summarization model being used so that users receive summaries in a personalized and customized manner.

[0037] The wearer or other user may have the ability to explicitly select between summary and transcription modes. For example, the user may switch between transcription mode and summary mode. In other examples, the user may be provided with a summary and, if the user is not satisfied with the summary, be able to switch back to transcription mode.

[0038] In a particular example related to AR / VR glasses with a defined field of view (FOV) (e.g., a 100° FOV), the summary may be presented in the wearer's peripheral vision by default, but when the summary is determined to be important (e.g., as determined using the example techniques described above), the summary may be moved toward the center of the field of view (e.g., the fovea).

[0039] Thus, the described techniques can be used to utilize adaptive speech summarization and compression on AR / VR glasses and other devices, where display technologies include but are not limited to all head-mounted displays (HMDs), wearable devices (e.g., watches, fitness bands / trackers), and other computing devices (e.g., smartphones, laptops, or desktop computers).

[0040] In addition to addressing the limited resolution / display issues in systems such as those described above, the summarization techniques described can also be used to address other challenges of live conversations. For example, adding summarization to everyday conversations and other live interactions could have many potential benefits.

[0041] For example, such summarization can reinforce the speaker's statement while improving speech redundancy and unclear expression problems (e.g., filler words, stuttering, etc.). In other examples, summarization techniques can help understand a fast-moving conversation (e.g., a fast-paced speaker) by reducing the amount of information presented at one time. In other examples, the described technology can help a user remember the main points of a long speech and otherwise track the status and / or overview of the conversation (including taking notes in the context of a lecture), even when the speech includes various digressions relative to the main topic being discussed. Thus, a user can be helped to keep track of a conversation or other dialogue.

[0042] Thus, a user may be provided with a text description summary stream that is updated, for example, as the speaker speaks. The described techniques may then utilize the user preferences of the user, the voice characteristics of the speaker, and / or the device characteristics of the device to dynamically adjust the summary characteristics of the summary stream over time during the live session. Thus, the user may smoothly experience the live session, wherein the dynamically adjusted summary stream may help the user understand the live session.

[0043] Thus, the described technology can help, for example, when a user is deaf or hard of hearing, because a summary stream can be provided to the user visually on a display. Similarly, when a user attempts to have a conversation with a speaker in a foreign language, a summary stream in the user's native language can be provided to the user.

[0044] The described techniques can be implemented for almost any type of verbal input text (text data). For example, automatic speech recognition (ASR) or other transcription techniques can be used to provide a live transcription of the detected speech, which can then be provided or made available to the user as a transcription stream. The described techniques can then be used to simultaneously provide the above-mentioned live, dynamically adjusted summary stream type, i.e., provide the summary stream in parallel with the transcription stream.

[0045] For example, a user wearing smart glasses or a smart watch or using a smart phone can obtain either or both of the transcription stream and the summary stream while listening to the speaker. In other examples, either or both of the transcription stream and the summary stream can be provided to users watching a video or participating in a video conference.

[0046] The described techniques thus overcome various shortcomings and deficiencies of existing summarization techniques, while also enabling new implementations and use cases. For example, existing summarization techniques may overly reduce the input text, may not sufficiently reduce the input text, may include irrelevant text, or may include inaccurate information. In the above scenario where it is necessary to provide a transcription stream and a summary stream in parallel, existing summarization techniques (in addition to the shortcomings just mentioned) may not be able to generate the desired summary quickly enough, or may attempt to generate a summary at an inappropriate time (e.g., before the speaker has finished discussing the topic). In addition, existing techniques may generate summaries that are too long (or inappropriate) to be efficiently displayed on the available display area of ​​the device being used (e.g., smart glasses).

[0047] In contrast, the described technology solves the above and other problems by analyzing spoken input while accessing user preferences and device characteristics over a period of time during a live session, for example. Thus, the described technology is well suited for generating dynamic, real-time summaries that are also generated and adjusted over time to the live transcript available to the user during one or more live sessions as the speaker speaks.

[0048] Figure 1 is a block diagram of a system for summarizing live summaries with user and device customization. Figure 1In the example of , the summary stream manager 102 processes the speech 104 (audio data, also referred to as spoken input) of the speaker 100 to obtain a summary 106, which is provided to the user 101 as part of a live, dynamically adjusted summary stream 134 (data stream). As described above, the speech 104 can include almost any spoken words or other spoken input. For example, the speech 104 can be a lecture, a speech, a conversation, an interview, a conversation, or any other spoken word interaction of two or more participants. Such an interaction can be largely one-sided (monologue), such as in the case of a lecture, or it can be an exchange of opinions between the speaker 100 and the user 101.

[0049] For example, a conversation may be conducted between speaker 100 and user 101, and the conversation may be facilitated by summary stream manager 102. As just mentioned, in other examples, speaker 100 may represent a lecturer, and user 101 may represent a lecture attendee, such that summary stream manager 102 facilitates the lecture's utility to user 101. Speaker 100 and user 101 may be co-located and conduct a face-to-face conversation, or may be remote from one another and communicate via web conferencing.

[0050] In other examples, the speaker 100 may first record the speech 104, and the user 101 may view (and receive) the summary 106 of the recorded audio and / or video at a later time. In this sense, the term 'live conversation' should be understood primarily from the perspective of the user 101. For example, as just mentioned, the user 101 may live listen to a previously recorded video of the speaker 100 and obtain a live summary stream 134 of the type described herein.

[0051] therefore, Figure 1 It should be understood that the summary stream manager 102 is capable of providing the summary 106 in a standalone or static manner in response to discrete instances of speech 104 (e.g., summarizing the audio of a single recorded video). Figure 1 It is also shown that summary stream manager 102 is capable of receiving speech from speaker 100 during a first time interval and outputting summary 106 to user 101, and then repeating such speech-to-summary operations during a subsequent second time interval to provide a dynamic summary of the type described above and described in detail below with reference to summary stream 134. In other words, as shown and described, summary 106 may be understood as representing a single discrete summary of corresponding discrete speech from speaker 100 during a single time interval of a larger time period or time window of a conversation.

[0052] As also described in detail below, summary stream manager 102 can be implemented in conjunction with any suitable device 138, such as a handheld computing device, smart glasses, earbuds, or a smart watch. For example, summary stream manager 102 can be implemented in conjunction with one or more devices in which speech 104 is received using a microphone or other input device and speech 104 is displayed using an audio output, a visual display (e.g., Figure 1 The summary 106 and summary stream 134 may be presented or provided via a display 140 in the display 140 and / or other output device.

[0053] Summary Stream Manager 102 Figure 1 In a simplified example of FIG. 1 , the summary flow manager 102 is shown as a single component including multiple subcomponents. However, as also described below, the summary flow manager 102 may be implemented using multiple devices that communicate with each other.

[0054] like Figure 1 As shown, the summary flow manager 102 may include or utilize Figure 1 Device characteristics 108 of one or more devices represented by device 138 in FIG. For example, device characteristics may include the display size of display 140, available fonts or formats, or available scrolling rates of device 138 / display 140.

[0055] User preferences 110 may include any user preferences for receiving summary stream 134 (e.g., set by a device selected by the user or reflected by other user operations on the device). For example, user preferences 110 may include a user preference for a slow, medium, or fast scroll rate of summary stream 134 on display 140. User preferences 110 may also specify a preferred font / format, or a preferred device among multiple available devices. User preferences 110 may be manually entered by user 101, and / or inferred by summary stream manager 102 based on actions of user 101.

[0056] The training data 112 generally represents any training data that can be processed by the training engine 114 to train one or more machine learning (ML) models, as described herein. The training data 112 can represent one or more available repositories of labeled training data for training such ML models, and / or can represent training data compiled by a designer of the summary flow manager 102.

[0057] The speech analyzer 116 may be configured to receive the speech 104, for example, via a microphone or other input from the device 138, and process the speech 104 to determine relevant speech characteristics (as reflected by audio data representing the speech). For example, the speech analyzer 116 may calculate or otherwise determine the rate, tone, volume, pitch, stress, or any other characteristic of the speech 104. The speech analyzer 116 may also identify the speaker 100, either individually or as a category / type of speaker. For example, the speech analyzer 116 may identify the speaker 100 as a friend of the user 101, or a work colleague or teacher of the user 101. The speech analyzer 116 may also identify the language spoken by the speaker 100.

[0058] The input handler 118 may be configured to receive or recognize any of the user preferences 110 discussed above, as well as receive summary requests from the user 101, as described in detail below. For example, the input handler 118 may provide interaction with the user 101, such as via the display 140, to receive manually submitted preferences and summary requests. Such manually submitted preferences or summary requests may be received from an input device 142 associated with the display 140 and / or the device 138, wherein the input device 142 may include, for example, a touch screen, a scroll bar, a button, a switch, a microphone, a gesture detector, or any other suitable input device, or a combination thereof. The input handler 118 may be implemented using a heuristic approach, or may be implemented as a trained ML model trained using the training engine 114.

[0059] The text extractor 120 may be configured, for example, to extract the transcribed text to be summarized from the transcribed stream 130, as described in detail below. For example, in response to a summary request received via the input device 142 and the input handler 118, the text extractor 120 may extract the most recent transcribed text from the transcribed stream 130. For example, the text extractor 120 may retrieve the transcribed stream 130 for the most recent ten seconds, or five seconds, or other suitable time intervals. In other examples, the text extractor 120 may retrieve the transcribed text based on detected characteristics of the transcribed stream 130, such as detected pauses and / or punctuation marks within the transcribed speech. The text extractor 120 may be implemented using a heuristic approach, or may be implemented as a trained ML model trained using the training engine 114.

[0060] Display optimizer 122 may be configured to optimize the use of display 140 in generating and displaying transcript stream 130 and / or summary stream 134. For example, display optimizer 122 may be used to adapt summary 106 for display using display 140, such that, for example, summary 106 is neither too small nor too large for display 140. Display optimizer 122 may be implemented using a heuristic approach, or may be implemented as a trained ML model trained using training engine 114.

[0061] The transcription generator 124 may be configured to convert the spoken words of the speech 104 into Figure 1 1 is a transcribed text shown as transcript 126. For example, the transcription generator 124 may include an automatic speech recognition (ASR) engine or a speech-to-text (STT) engine.

[0062] The transcription generator 124 may include many different text generation methods, including additional processing of the generated text. For example, the transcription generator 124 may provide a timestamp for the generated text, a confidence level in the generated text, and inferred punctuation for the generated text. For example, the transcription generator 124 may also utilize natural language understanding (NLU) and / or natural language processing (NLP) models or related techniques to identify semantic information (e.g., sentences or phrases), identify topics, or otherwise provide metadata for the generated text.

[0063] The transcription generator 124 may provide various other types of information in conjunction with the transcribed text, possibly using related hardware / software. For example, the transcription generator 124 may analyze the input audio stream to distinguish different speakers, or characterize the duration, pitch, speed, or volume of the input audio, or other audio characteristics. For example, in some implementations, the transcription generator 124 may be understood to implement some or all of the speech analyzer 116.

[0064] exist Figure 1 , the transcription generator 124 can utilize the transcription buffer 128 to output the transcription stream 130. That is, for example, the transcription generator 124 can process the live conversation, discussion, or other speech in real time and while the speech is ongoing. Thus, the transcription 126 represents the transcription of segments or instances of the transcribed text within a time interval occurring within a larger time period or time window of the conversation. For example, the summary 106 can represent a summary of the transcription 126, where the transcription 126 represents the transcription of the first 10 seconds of the speech 104, for example.

[0065] For example, while the speaker 100 is speaking, the transcription generator 124 may output the transcribed text for storage in the transcription buffer 128. The transcribed text may be designated as intermediate text or final text within the transcription buffer 128 and then available as the transcription 126 / transcription stream 130. For example, the transcription generator 124 may detect the end of a sentence, a switch in speakers, a pause of a predefined length, or other detected audio characteristics to designate a final transcription to be included in the transcription stream 130. In other examples, the transcription generator 124 may wait until a defined or detected time interval has elapsed to designate a final transcription of the audio.

[0066] Thus, the transcript stream 130 may be processed by the summarizer 136, e.g., in response to a summary request received from the input handler 118, to populate the summary buffer 132 and otherwise output the summary 106 / summary stream 134. Although the transcript buffer 128 and the summary buffer 132 are described herein as memories for providing short-term storage of the transcript stream 130 and the summary stream 134, respectively, it should be understood that the same or other suitable memories may be used for long-term storage of some or all of the transcript stream 130 and the summary stream 134. For example, the user 101 may wish to capture a summary of a lecture that the user 101 attends for later review.

[0067] Summarizer 136 may represent any trained model or algorithm designed to perform summarization. For example, summarizer 136 may be implemented as a sequence-to-sequence generating large learning model (LLM).

[0068] As described above, the summarizer 136 may be invoked in response to a summary request received via the input handler 118 (e.g., from the input device 142). Over time, the summarizer 136 may be fine-tuned to anticipate the user's 101 need or desire for a summary 106 based on earlier selections or actions by the user 101. For example, the summarizer 136 may be fine-tuned to correlate characteristics of the transcription stream 130 at one or more previous summary requests to determine whether to generate a summary 106 at the current time. For example, the text extractor 120 and / or the summarizer 136 may detect sentence endings, speech pauses, or the rate (or other characteristics) of the audio to determine whether / when to invoke the summarizer 136.

[0069] For example, the text extractor 120 may be configured to analyze the selections made or actions taken by the user 101 with respect to the transcription stream 130 and / or the summary stream 134 in order to determine or infer the user preference 110 for whether and when to invoke the summarizer 136 for summarization. For example, the input handler 118 may detect that the user 101 frequently requests summaries in corresponding scenes or contexts, and may update the user preference 110 to reflect a higher likelihood of generating summaries in such scenes / contexts in the future.

[0070] In some examples, upon receiving a summary request via input device 142, text extractor 120 may extract text from transcription stream 130 (such as transcript 126) from times before and / or after the time of the summary request. For example, if a summary request is received at time t=0, transcript 126 may include text from time t=-10 (i.e., 10 seconds before), and / or text from time t=10 (i.e., 10 seconds after). For example, the transcribed text before the summary request may be obtained from transcription buffer 128.

[0071] In further examples, the display optimizer 122 may be configured to control various display characteristics used to provide the transcript stream 130 and / or the summary stream 134. For example, the display optimizer 122 may provide the user 101 with the option to view (e.g., switch between) either or both of the transcript stream 130 and the summary stream 134.

[0072] The display optimizer 122 may also be configured to display various indicators related to the transcript stream 130 and the summary stream 134. For example, while the summarizer 136 is processing a corresponding portion of the transcript stream 130, the display optimizer 122 may display a summary indicator that notifies the user 101 that the current portion of the summary stream 134 is being generated. As described above, the display optimizer 122 may also control the size, spacing, font, format, and / or speed (e.g., scrolling speed) of the transcript stream 130 and the summary stream 134.

[0073] In the following description, compression ratio refers to a measure of the reduction of summary 106 relative to the corresponding input speech of speech 104. For example, in a simple example, if summary 106 includes 50 words and is generated from speech 104 including 100 words, the corresponding compression ratio would be 50% or 0.5. Various techniques can be used to calculate compression ratio in addition to or instead of word counts. For example, compression ratio can be expressed as character counts instead of word counts, or can be implemented as word counts, but without stop words.

[0074] Summarizer 136 may be configured to directly or indirectly implement an appropriate compression rate to ensure that summary 106, and summary stream 134 in general, conforms to requirements of, for example, input handler 118, text extractor 120, and display optimizer 122. For example, text extractor 120 may extract a certain amount of text for summarization, and display optimizer 122 may require that the resulting summary include a maximum of 4 lines of 5 words per line in order to be properly displayed using display 140. Summarizer 136 may then use an appropriate compression rate to ensure that the extracted text is summarized accordingly.

[0075] In addition, the flow manager 102 can provide additional processing for the summary flow 134. For example, the flow manager 102 can identify and extract actionable content, such as calendar items, emails, or phone calls, within the summary flow 134. In some implementations, the flow manager 102 can be configured to facilitate or enact corresponding actions based on the content of the summary flow 134, such as generating a calendar item, or sending an email or text message.

[0076] exist Figure 1106 , the transcript stream 130 is shown separately from the summary stream 134 and from the display 140 . However, as described above, the transcript stream 130 may be displayed on the display simultaneously with the summary stream 134 or in place of the summary stream 134 . In addition, the transcript stream 130 and the summary stream 134 may be implemented as a single (e.g., interleaved) text stream. That is, for example, the transcript stream 130 may be displayed for a period of time, then a summary request may be received via the input device 142 , and a corresponding summary (e.g., summary 106 ) may be generated and displayed. In other words, the output stream of the display 140 may alternate between displaying the transcript stream 130 and the summary stream 134 .

[0077] In the simplified example of the stream manager 102, the various subcomponents 108-136 are each shown in the singular, but should be understood to represent at least one instance of each subcomponent. For example, two or more training engines, represented by the training engine 114, may be used to implement various types of training for training and deploying various ML models used in or with the summary stream manager 102.

[0078] exist Figure 1 , summary flow manager 102 is shown as being implemented and executed using device 138. For example, device 138 may represent a handheld computing device such as a smartphone, or a wearable computing device such as smart glasses, smart earbuds, or a smart watch.

[0079] Device 138 may also represent a cloud or network resource in communication with a local device, such as one or more of the devices just mentioned. For example, various types of training data and training engine 114 may be implemented remotely from user 101 operating a local device, while the remaining illustrated components of the summary manager are implemented at one or more of the local devices.

[0080] Summary 106 and / or summary stream 134 are shown as being output to display 140. For example, display 140 may be a display of device 138, or may represent a display of a separate device in communication with device 138. For example, device 138 may represent a smartphone, and display 140 may be the display of the smartphone itself, or a display of smart glasses or a smart watch worn by user 101 and in wireless communication with device 138.

[0081] For example, the following Figure 7 , Fig. 8A and Figure 8B More detailed examples of devices, displays, and network architectures are provided. Additionally, summary 106 and summary stream 134 (as well as transcription 126 and transcription stream 130) may be output via audio, such as using smart earbuds of the type described above.

[0082] Figure 2It shows Figure 1 A flowchart of an example operation of the system. Figure 2 In the example of FIG. 2 , operations 202-210 are shown as separate, continuous operations. However, in various example implementations, operations 202-210 may be implemented in an order different from that shown, in an overlapping or parallel manner, and / or in a nested, iterative, cyclic or branching manner. In addition, various operations or sub-operations may be included, omitted or replaced.

[0083] exist Figure 2 In the embodiment of the present invention, a transcription stream 130 can be received, which includes transcribed text (text data, such as transcription 126) that has been transcribed from speech 104 (audio data) (202). For example, the transcription stream 130 (data stream) can be received from an ASR engine or any other example of a transcription generator 124 as described herein. The transcription stream 130 can be received at an HMD and displayed on a display of such HMD, such as by Figure 1 138 and display 140. For example, the transcription stream 130 can be displayed on the display 140 as a streaming text description that captures the speech 104 for consumption by the user 101. In other examples, the transcription stream 130 can be captured in a background process and does not need to be displayed.

[0084] A summary request for a summary to be provided on a display of the device may be received (204). For example, a summary request may be received from user 101 via input device 142, which is associated with Figure 1 140, but may also be associated with device 138 when device 138 is detached from display 140 (e.g., in wireless communication with the display). As described herein, input device 142 may represent, for example, a gesture recognition detector, a scroll wheel, a touch screen, a microphone, or a mouse, so that any suitable input function for capturing a summary request may be used, including, for example, a tap, a gesture, a nod, a click, a verbal or voice command, or a touch. For example, user 101 may be listening to speaker 100 speaking and may feel that information within speech 104 is being missed, or may miss upcoming speech (e.g., if speaker 100 speaks very quickly, at a relatively low volume, or in a different language).

[0085] As described herein, summarization may be initiated by user 101, and / or by summary stream manager 102 in response to, for example, detected characteristics of speech 104, possibly with reference to user preferences 110 and / or device characteristics 108. Such speech characteristics may include, for example, the rate or volume of speech 104, or in response to pauses in speaker 100's speech 104, or in response to punctuation detected by transcription generator 124.

[0086] For example, summary stream manager 102 may determine and update automatic summary requests over time as user 101 uses summary stream manager 102. For example, if user 101 frequently requests summaries in certain situations, such as the rapid speech type described above, summarizer 136 may be fine-tuned to recognize such speech rates in the future and automatically initiate summary generation.

[0087] Extracted text may be identified from the transcribed text in response to the summary request (206). For example, transcript 126 may be extracted from transcript stream 130. In some examples, transcript 126 may be obtained from transcript buffer 128, including transcribed text that may have been buffered prior to receiving the summary request. In other examples, transcript 126 may be captured after receiving the summary request. In other words, if the summary request is received at time t=0, the extracted text may include transcribed text that occurred before t=0 and was buffered in transcript buffer 128, and / or may include transcribed text that occurred after t=0 and was identified for summarization in conjunction with a text transcription operation. For example, the following description of the transcribed text may include transcribed text that occurred before t=0 and was buffered in transcript buffer 128. Figure 4 Additional example extraction operations for obtaining extracted text are described in more detail.

[0088] The extracted text may be processed using a summarization machine learning (ML) model to obtain a summary of the extracted text (208). For example, summarizer 136 may process transcription 126 to output summary 106. The summary may be displayed on a display 140 of a (same or separate) device 138 (210).

[0089] In an example implementation, summarizer 136 may provide summaries 106 to summary stream 134, which may be displayed in parallel or alternately with transcript stream 130. In other words, user 101 may be able to view transcript stream 130 interspersed with one or more summaries of summary stream 134, or may switch between transcript stream 130 and summary stream 134, for example, as desired.

[0090] Many different display techniques may be used, some of which are described herein. For example, summary 106 may be truncated or otherwise processed to fit display 140 in a desired manner, for example, based on device characteristics 108 and user preferences 110 .

[0091] In other examples, other display resizing techniques may be used. For example, summarizer 136 may have summary size constraints, such as a minimum and / or maximum number of words to be included in summary 106, so that provided summary 106 may optimally utilize the available screen size of display 140. In some implementations, such as with respect to Figure 6As described, summarizer 136 may generate multiple summaries and may select the best available summary as summary 106 , for example, based on device characteristics 108 and user preferences 110 .

[0092] Other example display technologies may also be used. For example, in an HMD (such as, Fig. 8A and Figure 8B In the context of smart glasses (such as those shown in FIG. 4 ), summary 106 may be provided at the periphery of the available screen size unless some other display parameter specifies that summary 106 should be moved to be displayed at the central display portion.

[0093] Figure 3 It shows Figure 1 A block diagram of a more detailed example implementation of a system of FIG. Figure 3 In the example of , a live transcription / translation result 302 may be received. For ease of illustration, a transcription example 304 is provided as “Hello! Good morning! Thank you very much for coming today. We have a guest speaker this morning, Dr. Alan Kay. He is a Fellow of the American Academy of Arts and Sciences, National Academy of Engineering, and Royal Society of Arts”, which is shown as being summarized in an abstract 306, which includes the abstract “Hello! Good morning! We have a guest speaker this morning, Dr. Alan Kay.”

[0094] Figure 3It is shown that the summary 306 can be triggered using heuristics 308, user preferences 310, or a fine-tuned ML model 312. For example, with respect to the heuristics 308, the summary 306 can be generated in response to a first heuristic 314, which is defined as triggering the summary 324 when the amount of text within the field of view of the display used to provide the transcription / translation result 302 exceeds a certain character count of N characters (e.g., 100 characters). The summary 306 can be generated in response to a second heuristic 316, which is defined as triggering the summary 324 when the word rate per minute exceeds a certain value X (e.g., 80 words per minute).

[0095] In response to the third heuristic method 318, when the number of words per minute is less than the defined value Y (e.g., 30 words per minute), the transcription / translation 320 can be retained. In other words, for example, even if the user requests a summary, Figure 3 The system may also retain the transcription / translation results according to a third heuristic method 318. For example, if the speaker speaks slowly, there may be no benefit in attempting to summarize, and / or information may be unnecessarily lost in such a summary.

[0096] Summary 306 may be generated in response to fourth heuristic 322, which is defined as triggering summary 324 when there is no speech (e.g., a pause) for at least a certain number of seconds Z (e.g., 2 seconds). Summary 306 may be generated in response to fifth heuristic 326, which is defined as triggering summary 324 when it is determined that the current sentence is complete.

[0097] With respect to user preferences 310, summary 324 may be triggered in response to a user action 328 taken with respect to the HMD being worn, such as a tap, double tap, or swiping forward / backward on a sensor or scroll bar on the HMD frame. Similarly, a user action 330 of tapping on a smartwatch paired with the HMD may trigger summary 324. A user action 332 of verbally providing a 'summarize' command may trigger summary 324. Additionally, a user action 334 of using a head gesture (e.g., nodding) or a pinch gesture function of the HMD may trigger summary 324.

[0098] When using the fine-tuned ML model 312, user feedback 338 (e.g., user preferences 310) can be used to understand what and when to provide summaries 324. For example, once such fine-tuning has been completed, automatically identified triggers 336 can be used to trigger summaries 324, e.g., topics that are considered worthy of summary based on the fine-tuning, including talking about action items, shopping lists, tour guide reviews, long lectures, or content determined to be unfamiliar to the user 101.

[0099] Figure 4 It is shown that Figure 3 Flowchart of example operations corresponding to the example of Figure 4 In the example of , a live transcript can be received and buffered (402). In some implementations, the generated transcript can be automatically or preferentially displayed (404).

[0100] If a summary request is received from the user (406), then if the corresponding speech rate is below a minimum value (408), the transcription can continue without summarizing (402). Otherwise, the corresponding text can be extracted from the transcription and summarized (412).

[0101] The generated summary may be adapted for an associated display or screen (414) and displayed (416). For example, Figure 5 and Figure 6 Various examples are shown that illustrate truncating summaries and / or generating summaries of appropriate sizes to fit the summaries in a relevant display. Figure 8B A clear example of using an HMD to display a summary is shown.

[0102] A summary event including at least a summary and a corresponding summary request may be stored as labeled training data, possibly in conjunction with any other suitable metadata that may be helpful to include in the labeled training data (418). For example, a summary context or type may be stored, or a response from user 101 (e.g., view summary or delete summary) may be captured. Thus, the resulting labeled training data may be used to fine-tune summarizer 136. Thus, for example, future summary requests that do not require a user action by user 101 to trigger a summary may be received, and the corresponding extracted transcript text may be extracted and summarized using the fine-tuned summarizer 136.

[0103] As described herein, extraction of transcribed text for summarization may be performed with respect to time t=0 when a summary request is received (406) or some other summary trigger is detected (as described above with respect to heuristics 308) (410). For example, extraction may capture t-10 seconds of previous text from transcription buffer 128, or may extract text up to the nearest period (or other punctuation) or pause. In other examples, extraction may occur with respect to current / future transcribed text, such as when a summary request triggers summarization of a subsequent 10 seconds (t+10) or subsequent sentence or sentences.

[0104] Such extraction parameters may be predefined and stored using user preferences 110, and / or may be determined with respect to fine-tuning of summarizer 136. In other examples, multiple extraction options may be available to user 101. For example, when using an HMD with a sensor or scroll bar, user 101 may slide / scroll the bar forward to summarize subsequently received transcriptions, and may slide / scroll the bar backward to summarize previously received transcriptions from transcription buffer 128.

[0105] As described above, the extraction parameters can be determined using labeled training data. For example, <previoussentences>→<Should [NOT] summarize> in <sentences> / <bullet points> The format of " determines the token. The fixed delimiter "→" informs the model when the prompt ends and when completion begins. A fixed stop sequence (e.g., "\n") can be used to indicate the completion of the current example.

[0106] For example, such training data can be generated as: {"prompt": "<Previous Sentences> →", "completion": "<Should not summarize> ”; "<Should summarize> in two sentences”; "<Should summarize> in one sentence"; "<Should summarize> as bullet points”; "<Should summarize> as bullet points, with no more than five key items”; ... \𝑛”}

[0107] Human annotators can be hired to tag appropriate start and end sentences. The resulting small amount of training data can then be fed to the relevant model to fine-tune the model to output summaries.

[0108] Additionally or alternatively, a timestamp can be added when summarization is triggered. The following session can then be fed into the model to generate summaries, where the model emphasizes recent sentences. For example, timestamps can be added to the original input data every 10 seconds, which will enable the fine-tuned LLM model to learn how to emphasize recent sentences and determine the above-mentioned type of tags or similar tagging schemes.

[0109] During the above-described type of training, when errors occur in the generated summary compared to the ground-truth summary of the labeled training data, the training engine 114 can use the type and / or degree of the error in subsequent training iterations to adjust the weights or other parameters of the summarizer 136. Thus, over multiple iterations, the training engine 114 can adjust the weights or other parameters so that the summarizer 136 processes the speech 104 and generates corresponding summaries in a desired manner after deployment.

[0110] In a more specific example implementation, the described example techniques are able to model user preferences / actions, device characteristics, and speech features as intermediate representations, and then use such intermediate representations as additional inputs along with the original ASR transcription to produce useful and usable summaries. For example, as described below with respect to Figure 6 As shown in more detail, the described techniques can dynamically control the summary to be more or less concise based at least on the aforementioned factors or similar suitable factors.

[0111] Figure 5 Shows about Figure 3 Example of a transcript of 304 for Figure 3 and Figure 4 The example shows the layout of the example. Figure 5 In the example of , the first screenshot 502 includes a title portion 504 and a body portion 506. As shown, the title portion 504 can be used to display one or more icons and / or related meta information or metadata, while the body portion 506 can include a summary of the example transcription 304. Figure 5 Further shown are a second screenshot 508 with a title 510 and a body portion 512 having an alternative summary of the transcript 304 , and a third screenshot 514 with a title 516 and a body portion 518 having another alternative summary of the transcript 304 .

[0112] Figure 5 It is shown that multiple summaries can be generated from a single transcription 304 to accommodate the available screen size, user preferences, and summary context. For example, the body portion 506 of the screenshot 502 includes the summary "Hello! Good morning! We have a guest speaker this morning, Dr. Alan Kay."

[0113] For example, the foregoing summary may be selected and formatted to fit the available screen space in screenshot 502. For example, the foregoing summary may be limited to a specified number of lines (e.g., 1-4 lines), have a specified number of words per line (e.g., a maximum of 4 words per line), and may be required to end with punctuation (e.g., a period, as shown).

[0114] In another example, the body portion 512 of screenshot 508 includes a summary "Hi! Thank you very much for coming. Today we have a guest speaker, Dr. Alan Kay." As shown, the available screen space is slightly larger than the example of screenshot 502, and the preceding summary provides 5 lines of text and includes more of the original transcription 304.

[0115] In another example, the body portion 518 of screenshot 514 includes a summary "Hello! Good morning everyone. Thank you for coming today. We have a special guest speaker, Dr. Alan Kay…" The preceding summary shows an example in which more of the available screen space is utilized and the summary is provided in a scrolling manner, as indicated by the ellipsis at the end of the summary.

[0116] Can be used Figure 1 The screenshots 502, 508, 514 may be constrained or otherwise defined by one or more of the device characteristics 108 and / or user preferences 110 in the display. For example, the device characteristics 108 may specify, for example, a maximum number of lines and / or words per line, which may directly or indirectly affect other parameters such as font size. The device characteristics 108 may also specify a minimum or maximum scroll rate for the associated display, as well as any other display parameters and associated minimum, maximum, or optimal values.

[0117] The summary may be provided in any desired manner to optimize readability or other preferences of the user 101. For example, the summary may be limited to be displayed in the middle of a sentence, at the end of a clause or phrase. For example, the generated summary may be truncated at the nearest punctuation mark, and the remainder of the summary (if any) may be provided on a subsequent screen.

[0118] The titles 504, 510, 516 may include virtually any information that may be useful to the user 101 in interpreting, understanding, or otherwise using the summary stream provided in the corresponding body portion 506, 512, 518. For example, as shown in the example titles 504, 510, 516, the title may indicate that the displayed text is part of a summary. In other examples, the title may indicate that the corresponding body portion is a transcription and / or a translation.

[0119] In a further example, one of the headings 504, 508, 514 may indicate that the summarization operation is being processed and / or has been processed (e.g., the transcript 304 is currently being processed by the summarizer 136). For example, in addition to indicating that summarization is being performed, there may be a delay associated with inputting the transcript 304 and outputting the corresponding summary, and the headings 504, 508, 516 may be used to convey the corresponding summary status to the user 101 until the summary is ready to be included in the corresponding body portion 506, 512, 518.

[0120] Figure 6 Shown for Figure 3 and Figure 4 An example summary of the example selection process. Figure 6 In the example of , input transcription 602 is shown as "Last week I went to the theater, I had a very good seat, the play was very interesting, but I did not enjoy it, a young man and a young woman were sitting behind me they were talking loudly. I got very angry, I could not hear the actor. I turned around, I looked at the man and the young woman angrily. They did not pay any attention, in the end I turned around again. I can't hear a word!!!! I said angrily. It's none of your business, the young man said rudely. This is a private conversation! "

[0121] Figure 6 It is further shown that a number of summaries 604-618 may be generated from the input transcript 602. For example, the summary 604 includes "I saw." The summary 606 includes "I went to the theater and was annoyed by a couple." The summary 608 includes "I went to the theater. The play was interesting but I did not enjoy it because there were two people talking loudly behind me." The summary 610 includes the list item "1) Go to theater." Abstract 612 includes "I went to the theater and had a good seat, but was unable to hear the play." Abstract 614 includes "I went to the theater and was annoyed by a couple." Abstract 616 includes "Last week, I went to the theater. I had a very good seat. The play was very interesting, but I did not enjoy it." Abstract 618 includes list item "1) Go to theater."

[0122] therefore, Figure 6 It is shown that multiple summaries can be generated for a single input transcription. For example, the summarizer 136 can generate multiple such summaries and then select the summary that is most suitable with respect to the device characteristics 108 and / or user preferences 110. In such an example, the remaining summaries can be discarded without the user 101 reviewing them.

[0123] In other examples, two or more summaries may be provided to user 101, and the user may select a desired summary. For example, as described above, user 101 may wish to retain one or more summaries in long-term storage, e.g., for later use after a session or lecture. In these or similar situations, user 101 may wish to select the most desired summary for long-term storage.

[0124] Figure 7 is a user 702 in the surrounding environment 7000 (similar to Figure 1 101), where one or more external computing systems are displayed as additional resources 752, which can be accessed by user 702 via network 7200. Figure 7 A plurality of different wearable devices operable by user 702 on one or more body parts of user 702 are shown, including a first wearable device 750 in the form of glasses worn on the user's head, a second wearable device 754 in the form of earplugs worn on one or both ears of user 702, a third wearable device 756 in the form of a watch worn on the user's wrist, and a computing device 706 held by user 702. Figure 7 , computing device 706 is shown as a handheld computing device, but may also be understood to represent any personal computing device, such as a tablet computer or a personal computer.

[0125] In some examples, the first wearable device 750 is in the form of a pair of smart glasses, including, for example, a display, one or more image sensors that can capture images of the surrounding environment, audio input / output devices, user input capabilities, computing / processing capabilities, etc. Fig. 8A and Figure 8B Additional examples of the first wearable device 750 are provided.

[0126] In some examples, the second wearable device 754 is in the form of an ear-worn computing device, such as headphones or earbuds, which may include audio input / output functionality, an image sensor that can capture images of the surrounding environment 7000, computing / processing functionality, user input capabilities, etc. In some examples, the third wearable device 756 is in the form of a smart watch or smart bracelet, which includes, for example, a display, an image sensor that can capture images of the surrounding environment, audio input / output capabilities, computing / processing capabilities, user input capabilities, etc. In some examples, the handheld computing device 706 may include a display, one or more image sensors that can capture images of the surrounding environment, audio input / output capabilities, computing / processing capabilities, user input capabilities, etc., such as in a smartphone. In some examples, the example wearable devices 750, 754, 756 and the example handheld computing device 706 can communicate with each other and / or with the external computing system 752 to exchange information, receive and send input and / or output, etc. The principles described herein can be applied to Figure 7 Other types of wearable devices not specifically shown or described herein.

[0127] User 702 may choose to use one or more of devices 706, 750, 754, or 756, possibly in conjunction with external resource 752, to implement the above description of Figures 1 to 6 C. For example, user 702 may use an application executed on device 706 and / or smart glasses 750 to receive, transcribe, and display Figure 1 Transcription flow 130 and / or Figure 1 Summary of Stream 134.

[0128] As described above, device 706 can access additional resources 752 to facilitate various summarization techniques or related techniques described herein. In some examples, additional resources 752 can be partially or entirely available locally on device 706. In some examples, some of additional resources 752 can be available locally on device 706, and some of additional resources 752 can be available to device 706 via network 7200. As shown, additional resources 752 can include, for example, server computer systems, processors, databases, storage devices, etc. In some examples, processors can include training engines, transcription engines, translation engines, rendering engines, and other such processors. In some examples, additional resources can include ML models, such as Figure 1 and / or Figure 3 Various ML models with different architectures.

[0129] The device 706 can operate under the control of the control system 760. The device 706 can communicate with one or more external devices directly (via wired and / or wireless communication) or via the network 7200. In some examples, the one or more external devices may include various wearable computing devices in the wearable computing devices 750, 754, 756 shown, another mobile computing device similar to the device 706, and the like. In some implementations, the device 706 includes a communication module 762 to facilitate external communication. In some implementations, the device 706 includes a sensing system 764, which includes various sensing system components. The sensing system components may include, for example, one or more image sensors 765, one or more position / orientation sensors 764 (including, for example, inertial measurement units, accelerometers, gyroscopes, magnetometers, and other such sensors), one or more audio sensors 766 that can detect audio input, one or more touch input sensors 768 that can detect touch input, and other such sensors. The device 706 may include more or fewer sensing devices and / or a combination of sensing devices.

[0130] The captured still and / or moving images may be displayed by a display device of an output system 772 and / or sent externally via a communication module 762 and a network 7200, and / or stored in a memory 770 of the device 706. The device 706 may include one or more processors 774. The processor 774 may include various modules or engines configured to perform various functions. In some examples, the processor 774 may include, for example, a training engine, a transcription engine, a translation engine, a rendering engine, and other such processors. The processor 774 may be formed in a substrate configured to execute one or more machine executable instructions or multiple pieces of software, firmware, or a combination thereof. The processor 774 may be semiconductor-based, including semiconductor materials that can execute digital logic. The memory 770 may include any type of storage device or non-transitory computer-readable storage medium that stores information in a format that can be read and / or executed by the processor 774. The memory 770 may store applications and modules that perform certain operations when executed by the processor 774. In some examples, the applications and modules may be stored in an external storage device and loaded into the memory 770.

[0131] Although not in Figure 7 706, but it should be understood that the various resources of the computing device 706 can be implemented in whole or in part within one or more of the various wearable devices, including the smart glasses 750, earbuds 754, and smartwatch 756 shown, which can communicate with each other to provide the various features and functions described herein. For example, the memory 770 can be used to implement the transcription buffer 128 and the summary buffer 132.

[0132] exist Figure 7 In the present invention, any audio and / or video output can be used to provide summaries of the type described herein and associated features. For example, the described technology can be implemented in any product where improved speech-to-text conversion would be helpful and high-quality summaries would be beneficial. In addition to head-mounted displays, wearable devices, and mobile devices, the described technology can also be used for teleconferencing and web applications (including, for example, providing text descriptions / summaries in web conferencing software and / or pre-recorded videos).

[0133] The described techniques can also be used in conjunction with the translation capabilities of, for example, additional resources 752. For example, user 702 can hear a translation from a separate speaker (corresponding to Figure 1 The embodiment of the present invention is a conversation between a speaker 100 of the user 702, who may be close to the user 702, or far away from the user), wherein the speaker may be speaking in a first language. The translation engine of the processor of the additional resource 752 can automatically translate the conversation into the native language of the user 702, and can also summarize the translated conversation using the technology described herein.

[0134] Figure 7 The architecture can be used to implement or access one or more large language models (LLMs) that can be used to implement a summarizer for use in the previous example. For example, the Pathways Language Model (PaLM) and / or the Language Model for Conversational Applications (LaMDA) provided by Google can be used. For example, these and other LLMs can be implemented using a neural network architecture called a transformer architecture, which is designed to process data sequentially and determine relationships / connections within the training input using a self-attention mechanism that assigns a score (or weight) to each term (or token) and then uses the score to determine the output for the currently received input.

[0135] For the purpose of discussion and illustration, Fig. 8A and Figure 8B An example head-mounted wearable device 800 in the form of a pair of smart glasses is shown. The example head-mounted wearable device 800 includes a frame 802 having an edge portion 803 surrounding a glass portion or lens 807 and an arm portion 830 coupled to a corresponding edge portion 803. In some examples, the lens 807 can be a corrective / prescription lens. In some examples, the lens 807 can be a glass portion that does not necessarily include a corrective / prescription parameter. A bridge portion 809 can connect the edge portion 803 of the frame 802. In Fig. 8A and Figure 8B In the example shown in , for purposes of discussion and illustration only, the wearable device 800 is in the form of a pair of smart glasses or augmented reality glasses.

[0136] In some examples, the wearable device 800 includes a display device 804 that can output visual content, for example, at an output coupler that provides a visual display area 805 so that the user can see the visual content. Fig. 8A and Figure 8B In the example shown, the display device 804 is disposed in one of the two arm portions 830, which is for discussion and illustration purposes only. The display device 804 can be disposed in each of the two arm portions 830 to provide binocular output of the content. In some examples, the display device 804 can be a perspective near-eye display. In some examples, the display device 804 can be configured to project light from a display source onto a portion of a teleprompter glass that acts as a beam splitter placed at an angle (e.g., 30 to 45 degrees). The beam splitter can achieve reflection values ​​and transmission values ​​that allow light from the display source to be partially reflected and the remaining light to be transmitted. This optical design can allow a user to see two physical items in the world, for example, through a lens 807, next to the content (e.g., digital images, user interface elements, virtual content, etc.) output by the display device 804. In some implementations, waveguide optics can be used to depict content on the display device 804.

[0137] like Fig. 8A and Figure 8B The example wearable device 800 shown in the form of smart glasses includes one or more of an audio output device 806 (such as, for example, one or more speakers), an illumination device 808, a sensing system 810, a control system 812, at least one processor 814, and an outward-facing image sensor 816 (e.g., a camera). In some examples, the sensing system 810 may include various sensing devices, and the control system 812 may include various control system devices, including, for example, at least one processor 814 operably coupled to components of the control system 812. In some examples, the control system 812 may include a communication module that provides communication and information exchange between the wearable device 800 and other external devices. In some examples, the head-mounted wearable device 800 includes a gaze tracking device 815 to detect and track eye gaze direction and movement. The data captured by the gaze tracking device 815 can be processed to detect and track gaze direction and movement as user input. Fig. 8A and Figure 8B In the example shown, the gaze tracking device 815 is disposed in one of the two arm portions 830, which is for discussion and illustration purposes only. Fig. 8A and Figure 8B In the example arrangement shown, the gaze tracking device 815 is disposed in the same arm portion 830 as the display device 804, so that the user's eye gaze can be tracked not only with respect to objects in the physical environment, but also with respect to content output for display by the display device 804. In some examples, the gaze tracking device 815 can be disposed in each of the two arm portions 830 to provide gaze tracking for each of the user's two eyes. In some examples, the display device 804 can be disposed in each of the two arm portions 830 to provide a binocular display of visual content.

[0138] Wearable device 800 is shown as glasses, such as smart glasses, augmented reality (AR) glasses, or virtual reality (VR) glasses. More generally, wearable device 800 can represent any head mounted device (HMD), including, for example, a hat, a helmet, or a headband. More generally, wearable device 800 and computing device 706 can represent any wearable device, handheld computing device, or combination thereof.

[0139] Using wearable device 800 and similar wearable or handheld devices such as Figure 7 Those devices shown in Figures 1 to 4 The invention provides useful and convenient use case scenarios for implementations of the system. For example, such wearable devices and handheld devices can be highly portable and therefore usable by the user 702 in many different scenarios. At the same time, the available display area of ​​such devices may be limited. For example, the display area 805 of the wearable device 800 can be a relatively small display area, limited by the overall size and form factor of the wearable device 800.

[0140] Thus, user 702 can benefit from using the various summarization techniques described herein. For example, user 702 can interact with a separate speaker, such as a lecturer or a participant in a conversation. User 702 and the separate speaker can have varying degrees of interactivity or back and forth, and there can also be two or more additional speakers.

[0141] Using the described techniques, dynamic, real-time summaries can be provided to user 702 during all such interactions as they occur. For example, speakers can speak for short or long periods of time in conjunction with (e.g., in response to) dialogue provided by user 702. During all such interactions, user 702 can be provided with useful and convenient summaries of the words spoken by the separate speakers.

[0142] As described above, dynamic, real-time summaries can be dynamically adjusted over time and during a conversation or other interaction. Thus, a user 101 / 702 can be provided with a meaningful, situation-specific summary that reduces the cognitive load of the user 101 / 702 and promotes meaningful interactions, even when one or more participants in the interaction are not native speakers, are currently speaking a different language, or are an expert in a field talking to a novice in the field.

[0143] Fig. 9 is used for visual enhancement of transcribed text Figure 1 A block diagram of an alternative implementation of a system. Fig. 9 In the example of Figure 1 Summary of alternative implementations of flow manager 102 and may include Figure 1 Any one or more of the various components or subcomponents described in the context of the examples of FIG. Fig. 9 Such components or subcomponents are not described here beyond the operation of the implementation method.

[0144] exist Fig. 9 In the example of , device 138 includes or has access to image sensor 904. For example, image sensor 904 may represent Figure 7 The image sensor 765, or Fig. 8A and Figure 8B image sensor 816 , or any suitable camera available to user 101 .

[0145] In addition, summary stream manager 902 is configured to provide various types of enhancements to the text generated by summary stream manager 902 such that Figure 1 Abstract 106 in Fig. 9 906. Similarly, Figure 1 Transcription 126 in Fig. 9 906 or enhanced transcription 910. A detailed description of example types of enhancements is provided below, but in general, such enhancements may include, for example, sentiment enhancements, entity enhancements, and / or intent enhancements. In other words, for example, enhanced summary 906 or enhanced transcription 910 may be enhanced to highlight or emphasize the sentiment of speaker 100 or user 101, important or meaningful entities mentioned within the generated text, or the intent of speech 104. As described below, any such enhancement may be provided by altering (e.g., visually highlighting) the text of enhanced summary 906 or enhanced transcription 910, or by providing additional information in addition to the generated text (including, for example, part or all of an image captured by image sensor 904).

[0146] In addition, Fig. 9 , the enhanced image 908 represents the image captured by the image sensor 904 and Fig. 9 The image is enhanced by the operation of the summary stream manager 902. For example, as described in detail below, part or all of the enhanced summary 906 may be added to, combined with, superimposed on, or otherwise provided with an image from the image sensor 904 to produce an enhanced image 908. The user 101 may then easily share or use the enhanced image 908 in any desired manner, including, for example, sending the enhanced image 908 to a friend or colleague to share or discuss the speech 104 or the associated experience of the user 101 occurring in conjunction with the speech 104.

[0147] In some implementations, the image captured by the image sensor 904 can be added to the summary generated by the summarizer 136 (e.g., Figure 1 106), thus forming part of the enhanced summary 906. Similarly, the image captured by the image sensor 904 can be added to the transcription generated by the transcription generator 124 (e.g., Figure 1 126), thus forming part of enhanced transcript 910. In other examples, enhanced image 908 may be formed by adding a transcript (e.g., transcript 126) to an image captured by image sensor 904.

[0148] In more detail, as shown, summary stream manager 902 may include image handler 912. Image handler 912 may be configured to interact or interface with image sensor 904 and / or any memory for storing images captured by image sensor 904. For example, when summary stream manager 902 has been activated to capture speech 104 and generate transcription stream 130 and summary stream 134, image handler 912 may be automatically activated and connected to image sensor 904, so that any images captured by the image sensor while summary stream manager 902 is active may be received by image handler 912 by default.

[0149] In other examples, images captured by the image sensor while the summary stream manager 902 is active may be manually specified by the user 101 for use in forming one or more of the enhanced summary 906, the enhanced image 908, and / or the enhanced transcription 910. For example, while the speaker 100 is producing speech 104, the user 101 may capture an image using the image sensor 904, and the image handler 912 may display the captured image using the display 140 with an option to combine the captured image with a simultaneously captured summary or transcription. Additional options may be displayed to enable the user 101 to combine the captured image with the simultaneously captured summary or transcription in a desired manner, for example, by positioning some or all of the summary at a desired location within the image and / or by editing the summary (or transcription).

[0150] The image entity model 914 may represent an ML model trained by the training engine 114 to extract one or more entities from an image captured by the image sensor 904. For example, the image entity model 914 may identify entities within an image received by the image handler 912. For example, the image entity model 914 may represent a neural network, such as a convolutional neural network (CNN), an image classifier, a segmentation model, and / or any suitable object or entity recognition model.

[0151] Somewhat similarly, text entity extractor 916 may be configured to identify transcribed text (e.g., Figure 1 126) or the summary text output by the summarizer 136 (e.g., Figure 1 The text entity extractor 916 may be any trained model or algorithm designed to identify and extract content that may be useful or important with respect to the summary / transcription enhancement described herein. For example, such extracted content may include any type, category, or information instance that may be constructed in a known manner. Any type of fact, phrase, or other key information may be identified for extraction. Some specific but non-limiting examples of such content may include, for example, named entities such as people, things, dates, times, events, locations, and the like.

[0152] During operation, as the transcript stream 130 and / or summary stream 134 are received, the text entity extractor 916 may continuously identify one or more such entities within the transcript stream 130 and / or summary stream 134. The text entity extractor 916 may be configured to assign a probability to each identified entity as to whether it will (should) be included in the enhanced summary 906, the enhanced image, and / or the enhanced transcript 910.

[0153] In the case of the image entity model 914 and / or the text entity extractor 916, the training data 112 may therefore include many examples of training images and training text, respectively, from which various types and instances of content have been identified and associated with tags that produce the desired object or entity identification. Fig.13 and Fig.14 Provide a more specific example.

[0154] The sentiment analyzer 918 may be configured to analyze the text from the transcription generator 124 and associate one or more sentiments with corresponding portions of the text (e.g., corresponding words, phrases, sentences, or paragraphs). Fig.13 As shown and described in more detail, the sentiment analyzer 918 can be associated with a set of predefined emotions (e.g., joy, anger, disgust, fear, surprise, etc.), and each of these emotions can be associated with a probability of being present (or absent) at a given point in time when processing the transcription stream 130 and / or summary stream 134.

[0155] The intent extractor 920 can be trained and configured to analyze the intent associated with the corresponding portion of the transcription stream 130 at a given point in time. For example, the intent extractor 920 can be trained as a classifier to classify the current intent of the speaker 100 from a plurality of known intents, such as including view, buy, show, give / receive, visit, help, etc. The intent extractor 920 can be obtained by fine-tuning a pre-trained LLM for intent detection.

[0156] Rendering engine 922 may be configured to render various icons, files, or other content on display 140. For example, rendering engine 922 may render any of transcript stream 130, summary stream 134, enhanced transcript 910, enhanced summary 906, images captured by image sensor 904, and / or enhanced image 908. For example, the rendering engine 922 may render any of the transcript stream 130, summary stream 134, enhanced transcript 910, enhanced summary 906, images captured by image sensor 904, and / or enhanced image 908. Figure 8B The visual display area 805 is used to render any of the above items. Figure 13 to Figure 1 6 provides additional rendering examples.

[0157] Therefore, in Fig. 9 In a simplified example implementation of , while talking to speaker 100 about adopting a dog, user 101 can use image sensor 904 to capture an image of speaker 100 and a dog to send to a friend. Transcription generator 124 can generate a transcription stream including the sentence "I just adopted him from the shelter on the way home from get off work yesterday!" Summarizer 136 can output summary stream 134 including the summary "I got a new dog!"

[0158] The text entity extractor 916 may analyze the transcription stream 130 to identify entities such as “I,” “adopted,” “him,” and “shelter,” and / or may analyze the summary stream 134 to identify the entities “I” and “newdog.” More specifically, the text entity extractor 916 may identify all entities within a sentence and may assign a value indicating the likelihood of importance of each entity, e.g., in a range between 0 and 1, for inclusion or consideration in generating the enhanced transcription 910, the enhanced summary 906, and / or the enhanced image 908. For example, the text entity extractor 916 may view the example sentence as "I (0.9) just adopted (0.85) him (0.85) from the shelter (0.7) on the wayhome (0.35) from work (0.35) yesterday (0.5)!"

[0159] The sentiment analyzer 918 may analyze the transcription stream 130 and / or the summary stream 134 to identify the emotions of excitement and joy. The intent extractor 920 may analyze the transcription stream 130 and / or the summary stream 134 to identify the intent of "to share".

[0160] If user 101 uses image sensor 904 to capture an image of speaker 100 with the speaker's new dog, image processor 912 can trigger the above analysis, and rendering engine 922 can then generate enhanced summary 906. For example, enhanced summary 906 can include the captured image with a caption "I got a new dog!" and a corresponding emoji representing joy / excitement. Image entity model 914 can identify the entities of the person and dog within the captured image, so that rendering engine 922 can position the emoji and / or caption near the image of the dog within enhanced summary 906. User 101 can then easily share enhanced summary 906 via any appropriate sharing technology, including, for example, mobile phone text messaging or social media.

[0161] In some examples, as described below with respect to Fig.15 As described above, user 101 may decide to send such a message at a later point in time during the conversation after the relevant sentence / summary has appeared. In such a case, user 101 may use transcription buffer 128 and / or summary buffer 132 to send the desired message. For example, at the current time, user 101 may scroll back through summary stream 134 to an earlier point in time when the relevant summary was generated. When the summary "I got a new dog" is displayed on display 134, the image sensor 904 is used to capture the desired image at the current time so that rendering engine 922 can render the above-mentioned enhanced summary 906.

[0162] In an example implementation, the text entity extractor 916, sentiment analyzer 918, intent extractor 920, and summarizer 136 can be trained independently or in groups of two or more. As described above, and described in more detail below, the training of each model can be performed with respect to input text, such as an example representing the transcription stream 130, and the true value output of the model being trained. In examples where two or more models are trained together or as a combined model, the input text and the true value output of each output model / type, as well as the true value output by the summarizer 136, can be used to enhance the summary.

[0163] For example, to train the textual entity extractor 916, the training data 112 may include training records that include training input text and corresponding real-value textual entity types. Thus, the textual entity extractor 916 may be used to generate entities from the training input text, which may then be compared to real-value entities. For example, when creating the training data 112, the real-value entities may be obtained from early markings of the training input text.

[0164] When errors occur between generated entities and true value entities, the training engine 114 can use the type and / or degree of the error in subsequent training iterations to adjust weights or other parameters of the text entity extractor 916. Thus, over multiple iterations, the training engine 114 can adjust weights or other parameters so that the text entity extractor 916 processes the transcription stream 130 and identifies the included entities with an acceptable level of accuracy after deployment.

[0165] Similar comments apply to the training of the image entity model 914, the sentiment analyzer 918, the intent extractor 920, and the summarizer 136. Thus, as described below, for example, Figure 3 As shown and described, the image entity model 914, text entity extractor 916, sentiment analyzer 918, intent extractor 920, and summarizer 136 can be trained and deployed in a variety of different architectures. For example, the various models can be trained in a modular approach, where each model is trained separately as just described, and the output of each model can be provided as a text output that can be input by one or more subsequent models to obtain, for example, enhanced summary 906.

[0166] For example, the model output of any of the image entity model 914, the text entity extractor 916, the sentiment analyzer 918, and the intent extractor 920 can operate on the output of the transcription generator 124 and provide the output to the summarizer 136. In such an example, any or all of such model outputs can constrain and improve the output of the summarizer 136. That is, in such an example, the summarizer 136 can be trained to emphasize entities, emotions, and / or intents identified by available model outputs in the model output within the summary stream 134. In such an example, when the summarizer 136 is trained using the training engine 114, the training data 112 can include true value entities, emotions, and / or intents and true value summaries that reflect such entities, emotions, and / or intents. Once deployed, the summarizer 136 can therefore provide a summary stream 134 in a manner that reflects (e.g., emphasizes) or visually distinguishes such entities (including image entities and / or text entities), emotions, and / or intents, for example using the Markdown language.

[0167] In other examples, when using a transformer-based LLM, the input can be set as a vector embedding, and a single combined model can be trained to input only the transcription stream 130 and the image captured by the image sensor 904, and output, for example, an enhanced summary 906 (or an enhanced transcription 910 or an enhanced image 908).

[0168] Many other implementations are possible. For example, the model outputs may be defined in terms of a knowledge graph representation, rather than in text or as vector embeddings. Additionally, as described above, two or more of the included models may be jointly trained, for example, using such a knowledge graph representation or other output scheme. Further, additional or alternative models may be used, for example, additional or alternative classifiers may be used, and one or more of the illustrated models may be modified or omitted.

[0169] Fig.10 It shows Figure 1 A flowchart of an example operation of the system. Fig.10 In the example of FIG. 1 , operations 1002-1010 are shown as separate, continuous operations. However, in various example implementations, operations 202-210 may be implemented in an order different from that shown, in an overlapping or parallel manner, and / or in a nested, iterative, cyclic or branching manner. In addition, various operations or sub-operations may be included, omitted or replaced.

[0170] exist Fig.10 In the embodiment of the present invention, a transcription stream 130 including transcribed text that has been transcribed from speech 104 can be received (1002). For example, as described above, the transcription generator 124 can generate the transcription stream 130.

[0171] The image may be received in association with—eg, in conjunction with—the receipt of the transcribed stream ( 1004 ). For example, the image handler 912 may be configured to receive the image from the image sensor 904 .

[0172] The transcription stream 130 may be processed using a summary machine learning (ML) model to obtain a summary stream 134 , including processing the transcribed text to obtain a summary ( 1006 ). For example, the summarizer 136 may process the transcription stream 130 to obtain the summary stream 134 .

[0173] The image and summary may be combined to obtain an enhanced summary 906 (1008), and the enhanced summary may be displayed (1010). For example, the rendering engine 922 may combine the outputs of the summarizer 136 and the image handler 912, possibly in conjunction with the outputs of one or more of the image entity model 914, the text entity extractor 916, the sentiment analyzer 918, and the intent extractor 920, to provide the enhanced summary 906 (and / or the enhanced transcription 910 or the enhanced image 908).

[0174] In some implementations, the capture of an image may provide a trigger for generating an enhanced summary 906. For example, when an image is captured at the current time, a time interval before and / or after the image capture trigger may be defined, and an enhanced summary 906 or enhanced transcription 910 may be defined with respect to the text generated corresponding to (falling within) the defined time interval. When an image is captured after the user 101 scrolls backward through the transcript stream 130 or summary stream 134, the image capture trigger may define similar time intervals before and after the timestamp of the end of the scrolling to generate an enhanced summary 906 or enhanced transcription 910.

[0175] Additionally, as described above, output from any of the image entity model 914, text entity extractor 916, sentiment analyzer 918, and / or intent extractor 920 may be included in the enhanced summary 906, the enhanced transcription 910, or the enhanced image 908. For example, the identified entities (image entities or text entities) may be highlighted, pointed to with arrows or other visual indicators, provided in a different font color or a different font, or provided as links to external information (e.g., corresponding Internet search results). Emotions may be expressed by using corresponding emojis or by adding text, such as "So exciting!" A determined intent may be added, such as "Just wanted to share this with you."

[0176] Fig.11 It shows Fig. 9 A block diagram of a more detailed example implementation of a system. Fig.11 A modular solution 1102 and an end-to-end solution 1104 are shown. In the modular solution 1102, audio 1106 is received at a microphone, while audio is received from, for example, a Figure 7 , Fig. 8A and Figure 8B The camera on the AR glasses shown receives image 1108 as a stereo image or image pair from the video see-through VR goggles.

[0177] As shown in the example, audio 1106 may be received and processed, including through STT, ASR or similar processing, and / or summarized by a summarizer 136, all of which may be performed in Fig.11 1106, including assigning a probability of whether each identified entity should be included when generating a subsequent enhanced output. For example, the entity extraction model 1110 can be run locally on the head-mounted wearable device 800, including using a local device that wirelessly communicates with the head-mounted wearable device 800.

[0178] The sentiment analysis model 1112 can process the audio 1106 in parallel over time to determine and characterize one or more emotions associated with the audio 1106. That is, the sentiment analysis model 1112 can continuously or periodically update the currently associated emotions or most likely emotions, for example, as the transcription stream 130 and summary stream 134 are received over time in conjunction with the receipt of the speech 104. The sentiment analysis model 1112 can be executed, for example, as a cloud-based model.

[0179] Similarly, the intent extraction model 1114 can be executed in parallel with the entity extraction model 1110 and the sentiment analysis model 1112. While processing the audio 1106, the intent extraction model 1114 can determine the current intent of the speaker 100 when feasible. For example, in some cases, the intent can be determined after the speaker 100 has spoken only a few words, while in other cases, there may be no obvious intent, or a longer segment of the speech 104 may be required to determine the intent. Therefore, the intent extraction model 1114 can operate in parallel with the entity extraction model 1110 and the sentiment analysis model 1112, but can output the determined intent only when available.

[0180] Therefore, it is possible to base the Figure 1 The user preference 110 of FIG. 110 is to provide a highlighted live transcript 1116 as an enhanced transcript / summary on, for example, AR / VR glasses (such as, head-mounted wearable device 800). For example, user 101 may wish to continuously view a transcript stream 130 and / or a summary stream 134 having recognized entities determined by entity extraction model 1110, indications of sentiment determined by sentiment analysis model 1112, and / or intents detected from intent extraction model 1114. User 101 may individually disable such outputs from being displayed in conjunction with transcript stream 130 or summary stream 134 if user 101 finds one or more of these outputs distracting or undesirable.

[0181] In some such cases, summary flow manager 902 may continue to execute entity extraction model 1110, sentiment analysis model 1112, and intent extraction model 1114 and obtain corresponding outputs. Such outputs may be stored in transcription buffer 128 or summary buffer 132, for example, for potential subsequent summary enhancement operations. In other examples, for example, to save resources or otherwise based on user preferences, one or more of entity extraction model 1110, sentiment analysis model 1112, and intent extraction model 1114 may be deactivated or not used until a suitable trigger is detected, such as receiving image 1108 at image handler 912, as described above and below with respect to Fig.12 Describe in more detail.

[0182] exist Fig.11 In the example of modular solution 1102, an enhanced image 1118 may be provided by combining a highlighted live transcript 1116 with the image 1108. For example, Fig.11 As shown, the augmented image 1118 may include the image 1108 with augmentations including one or more of an AR sticker of an emoji, a key entity knowledge card, and / or an intent provided as a tagline for the image 1108 .

[0183] Furthermore, in the example of modular solution 1102, Fig. 9 It will be understood from the above description that the summarizer 136 can be configured to output to one or more of the models 1110, 1112, 1114 so that the models 1110, 1112, 1114 operate on the summarized content. Conversely, in other examples, one or more of the models 1110, 1112, 1114 can be output to the summarizer 136 so that the summarizer 136 can generate the summarized content with the aid of or based on the corresponding output of one or more models 1110, 1112, 1114.

[0184] In addition, Fig.11 In the end-to-end solution 1104, the semantic-aware model 1124 can be implemented using the transformer architecture described above, where all inputs are set as vector embeddings. The image 1120 and audio 1122 (possibly after STT / ASR processing and / or summarization processing) can then be directly input to the semantic-aware model 1124. In this way, the highlighted live transcription 1116 can be provided directly by the semantic-aware model 1124 as an enhanced transcription / summary. The enhanced image 1118 can be provided directly by the semantic-aware model 1124 and / or indirectly based on the highlighted live transcription 1116.

[0185] For example, a method for separately training the entity extraction model 1110, the sentiment analysis model 1112, and the intent extraction model 1114 as well as the summarizer 136 and the image entity model 914 (not shown in FIG. 1 ) may be used. Fig.11 1120 and 1122). In order to perform these multiple types of training, the training input can be provided with labeled and true value outputs of various types of desired outputs. Then, by using vector embeddings to express the training data, the semantic-aware model 1124 can be trained on common expressions—i.e., in common language—across the different types of outputs required. Once deployed, the semantic-aware model 1124 can use vector embeddings to express live inputs (e.g., images 1120 and audio 1122) to produce the various types of outputs required.

[0186] Fig.12 It is shown that Fig.11 Flowchart of example operations corresponding to the example of Fig.12 In the example of , a live transcript such as transcript stream 130 is received and buffered (1202). The live transcript is summarized and buffered (1204).

[0187] The transcription (1206) can be enhanced with entity, sentiment, and / or intent information. Additionally or alternatively, the summary (1208) can be similarly enhanced with entity, sentiment, and / or intent information. As described above, the transcription and summary can occur simultaneously with the enhancement, in which case the enhancement can be displayed or not. In other examples, the enhancement can be added later to, for example, a selected, buffered transcription / summary, in which case the transcription / summary can initially be provided without the enhancement. That is, the transcription and / or summary can be displayed without or including entity, sentiment, and / or intent information, for example, depending on user preferences, and the entity, sentiment, and / or intent information can be added later to the content within, for example, the transcription buffer 128 or the summary buffer 132.

[0188] If, for example, the image handler 912 does not receive an image, then the live transcription and subsequent / parallel operations may continue (1210). After the image is received at the image handler 912, another enhancement trigger, such as a voice command made by the user 101, may be combined. Fig. 9 The image entity model 914 can extract image objects or other entities from the received image (1212).

[0189] The enhanced text may then be extracted from either or both of the enhanced transcription and / or the enhanced summary, depending on the user's preference (1214). Thus, an enhanced transcription, an enhanced summary, or an enhanced image may be generated (1216).

[0190] exist Fig.12 In the example of , the receipt of an image is used as an enhanced trigger, for example to construct a message to be sent or other composite file to be saved or otherwise used by user 101. However, in other examples, as just mentioned, additional or alternative triggers may be used. In addition to the voice commands just mentioned, for example, regarding Figure 1 The input device 142 may be any input technique described herein, including predefined gestures, taps, or other suitable selection techniques.

[0191] User preferences may define various characteristics related to, for example, how much or what content to include in a specified augmentation operation. For example, user 101 may specify which of entity, sentiment, or intent information should be included. In other examples, user preferences may determine the amount or type of preceding and following text that should be used (relative to the time when the image request was received). User preferences may also determine available and / or automatic uses of the generated augmented content, including options for messaging, email, or posting to social media.

[0192] Any or all such user preferences may be further constrained or defined by the content or context of the associated text / image. For example, different options may be defined depending on the time, location, or other settings of the user 101 when the image was captured. As described above, the user preferences may also define enhancement time intervals for one or more enhancement triggers that include the receipt of an image. For example, an enhancement trigger may define a time interval between ten seconds before the enhancement trigger and ten seconds after the enhancement trigger, or other suitable intervals.

[0193] Figures 13 to 15 Shown with Figures 9 to 12 As described above, such as Figure 7 and Fig. 8A The AR glasses of the example of / 8B implement speech-to-text, translation, and image-to-text systems that play an everyday role in the user's daily life. The described technology is able to convey emotions, such as by including emojis in addition to or instead of text summaries. As described above, when taking a photo, the user 101 can simultaneously make relevant statements about the captured image and see corresponding emoji stickers placed on the resulting image. Emojis can also be used to express answers to questions, such as thumbs up / thumbs down or "I don't know" emojis.

[0194] The described techniques for extracting entities from transcribed, translated, or summarized text can highlight entities with such text, while intent analysis can emphasize the intent of speech 104. By automatically identifying and semantically annotating objects and scenes in images of transcribed text, the descriptions provide additional information that helps users understand the text.

[0195] exist Fig.13 In the example of FIG. 8 , user 101 can be understood to represent a tourist wearing head-mounted wearable device 800, while speaker 100 represents a tour guide speaking in Spanish. Audio 1302 represents the Spanish audio of the tour guide, and STT engine 1304 is shown transcribing and translating audio 1302 into transcribed text 1306. As can be understood from the above, text 1306 can also be summarized by summarizer 136.

[0196] Entity detection 1308 shows the identification of important entities within text 1306, and the probability of importance of each identified entity. Sentiment analysis 1310 shows an array of emotions for anger, disgust, fear, joy, neutral, sadness, and surprise, and a corresponding bar graph showing the current relative likelihood of each emotion with respect to audio 1302.

[0197] Similarly, intent detection 1312 provides a determined intent associated with audio 1302. Fig.13 In the example of , the detected intent 1312 "pour their wealth" is determined with respect to the subject of the audio (i.e., Walt Disney and John Paul Getty), rather than with respect to the intent of the speaker 100 (e.g., a tour guide).

[0198] Thus, enhanced text 1314 may be provided. In the enhanced text 1314, the most important entities from the entity detection 1308, such as locations and names, are visually highlighted, e.g., rendered in a different color. The image 1316 may be enhanced with emoji 1318. In additional or alternative examples, the enhanced text 1320 may have the detected intent visually highlighted, e.g., by bolding / italicizing or other suitable techniques.

[0199] exist Fig.14 In the example of FIG. 1 , it is assumed that user 101 is also speaker 100 and is solving an example math problem. While viewing the math problem using head mounted wearable device 800 , user 101 may verbally express that the math problem cannot be solved and capture an image 1402 of the math problem.

[0200] Then, as shown, a transcription or summary text 1404 "I don't understand this equation. Can you help?" can be generated, along with an emoji 1406 visually indicating the lack of understanding. Additionally, based on user preferences / user selections, the share option 1408 can include an optional email address that the user 101 wishes to use to contact someone for help with the math problem 1402.

[0201] therefore, Fig.14 It can be understood to represent an enhanced image, where the image 1402 is enhanced with text 1404, emoji 1406, and sharing options 1408. Fig.14 It may also be accurately described as enhanced text (eg, an enhanced transcription or an enhanced summary) in which the image 1402 , emoji 1406 , and sharing options 1408 are added to the text 1404 .

[0202] although Fig.14 It is shown with respect to a mathematical problem, but it should be understood that any text may be used in conjunction with, or as the subject of, a captured image. For example, text from a book or article may be captured, and text recognition may be used in conjunction with the described image processing techniques to reflect or include the recognized text within the enhanced summary 906 or enhanced image 908.

[0203] exist Fig.15 15, the screenshot 1502 includes a text body 1504 that may represent a transcribed or summarized text. A scroll bar 1506 includes a scroll button 1508. As described above, the scroll bar 1506 may be used to scroll through the text within the transcription buffer 128 or the summary buffer 132, depending on whether the user 101 has selected the transcription or summary mode. Thus, a timestamp 1512 may be provided in conjunction with the scroll bar 1506 to indicate how far the user 101 has scrolled back relative to the text provided. In the screenshot 1502, the timestamp 1512 indicates that the user 101 has scrolled back 15 seconds from the most recently available text.

[0204] In the example, user 101 is having a conversation with a friend named Bob who is speaker 100. At some point in the conversation, user 101 decides to send a message about a comment speaker 100 made at an earlier point in the conversation. Therefore, user 101 uses scroll bar 1506 to scroll back to the relevant point in the conversation. Fig.15 In the example of , text 1504 is not highlighted with entity, sentiment, or intent information, although such information may have been determined in conjunction with the generation of text 1504. In other examples, entity, sentiment, or intent information can be used to visually enhance text 1504 when generating text 1504, as described in the above examples.

[0205] To select text to use as the basis for a desired message, user 101 may move / scroll scroll button 11508 to align with a desired portion of text 1504. In other words, in the example, horizontal alignment of scroll button 1508 with the desired text indicates that user 101 wishes to generate a message with respect to the aligned text. Aligned text may be defined as text within a time interval before and after timestamp 1512.

[0206] The scroll button 1508 may be a selectable or operable icon provided by the rendering engine 922. For example, the scroll button 1508 may be used to initiate image capture using the image sensor 904, wherein the resulting image is automatically provided to the image handler 912. Fig.12 As described, such image capture may be used as an enhanced trigger to generate a message to be sent.

[0207] Therefore, in Fig.15 , a message 1510 including an enhanced summary or an enhanced image includes a summary 1513 (shown as "Bob got a new dog!"), an image of a dog 1514, and an emoji 1516 indicating happiness or joy. The message 1510 may be automatically generated and sent to pre-specified recipients, or the user 101 may have the option of selecting the desired recipients and communication channels. The user 101 may also be provided with the ability to modify the message 1510 before sending it or to save the message 1510 for later use (e.g., to send it later).

[0208] exist Fig.15 In the various examples above, the operation of the image entity model 914 can identify relevant objects or entities within the image, such as the dog in image 1514. Such image entities can be within the enhanced summary or enhanced image, such as in Fig.15 The enhanced summary 1510 is visually distinguished.

[0209] Although multiple embodiments have been described, more example implementations are possible. For example, the user 101 may be provided with a selection of enhanced content to include. For example, when adding emojis to express emotions, the user 101 may be provided with the option to select a favorite emoji. In these and other scenarios, the user's selections may be used to fine-tune the training of related models. Similarly, the user 101 may set thresholds to determine the number or likelihood of entities recognized in a transcription or summary (e.g., a higher or lower confidence threshold for entity detection).

[0210] In a first example implementation, referred to herein as Example 1, a computer program product is tangibly embodied on a non-transitory computer-readable storage medium and includes instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving a summary request for a summary to be provided on a display of the device; identifying extracted text from the transcript in response to the summary request; processing the extracted text using a summarization machine learning (ML) model to obtain a summary of the extracted text; and The summary is displayed on the display of the device.

[0211] Example 2 includes the computer program product of Example 1, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The summary request is received from a user of the device via an input device of the device.

[0212] Example 3 includes the computer program product of Example 1 or 2, wherein the input device includes at least one of a touch screen, a gesture recognition device, a scroll bar, a button, or a microphone.

[0213] Example 4 includes a computer program product as described in any of the preceding examples, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The summary request is received as a voice command from a user of the device via a microphone of the device.

[0214] Example 5 includes a computer program product as described in any of the preceding examples, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The transcription stream is received from a speech recognition engine.

[0215] Example 6 includes the computer program product of any of the preceding examples, wherein the device comprises a head mounted display (HMD), and the display comprises an HMD display, and wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: The transcribed stream is displayed using the HMD display.

[0216] Example 7 includes the computer program product of any of the preceding examples, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: identifying the extracted text as including text received at the device prior to the summary request; and The extracted text is extracted from the transcription buffer.

[0217] Example 8 includes the computer program product of any of the preceding examples, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: identifying the extracted text as including text received after the summary request; and The extracted text is extracted from the transcription stream following the summary request.

[0218] Example 9 includes the computer program product of any of the preceding examples, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: generating at least two summaries including the summary using the summarization ML model; and The summary is selected from the at least two summaries based on device characteristics of the device and user preferences of a user of the device.

[0219] Example 10 includes the computer program product of any of the preceding examples, wherein the device comprises a head mounted display (HMD), and the display comprises an HMD display, and wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: The summary is displayed using the HMD display.

[0220] In an eleventh example, referred to herein as Example 11, an apparatus includes: at least one processor; at least one memory; at least one input device; and at least one display, wherein when executed by the at least one processor, using the instructions stored in the at least one memory causes the device to: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving, via an input device, a summary request for a summary to be provided on the at least one display; identifying extracted text from the transcript in response to the summary request; processing the extracted text using a summarization machine learning (ML) model to obtain a summary of the extracted text; and The summary is displayed on the at least one display.

[0221] Example 12 includes the apparatus of Example 11, wherein the apparatus comprises a head mounted display (HMD).

[0222] Example 13 includes the apparatus of Example 11 or 12, wherein the apparatus is configured to receive the transcribed stream and the summary from a second apparatus in communication with the apparatus.

[0223] Example 14 includes a device as described in any of the above examples 11 to 13, wherein the input device includes at least one of a touch screen, a gesture recognition device, a scroll bar, a button or a microphone.

[0224] Example 15 includes the apparatus of Example 14, wherein the input device comprises a microphone, and the summary request is received as a voice command from a user of the apparatus via the microphone.

[0225] In a sixteenth example, referred to herein as Example 16, a method includes: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving a summary request for a summary to be provided on a display of the device; identifying extracted text from the transcript in response to the summary request; processing the extracted text using a summarization machine learning (ML) model to obtain a summary of the extracted text; and The summary is displayed on the display of the device.

[0226] Example 17 includes the method of Example 16, further comprising: storing the summary together with the summary request as labeled training data; and The summarization ML model is trained using the labeled training data.

[0227] Example 18 includes the method of Example 17, further comprising: detecting a second summary request using the summary ML model after the training; and The second extracted text is summarized using the summarization ML model.

[0228] Example 19 includes the method of Example 16, wherein the apparatus comprises a head mounted display (HMD), and the display comprises an HMD display, and further comprising: The summary is displayed using the HMD display.

[0229] Example 20 includes the method of Example 16, further comprising: generating at least two summaries including the summary using the summarization ML model; and The summary is selected from the at least two summaries based on device characteristics of the device and user preferences of a user of the device.

[0230] A twenty-first example, referred to herein as Example 21, includes a computer program product tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving an image associated with receipt of the transcribed stream; processing the transcription stream using a summary machine learning (ML) model to obtain a summary stream, including processing the transcription text to obtain a summary; combining the image and the summary to obtain an enhanced summary; and Displays a summary of the enhancement.

[0231] Example 22 includes the computer program product of Example 21, wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: selecting a time interval based on receiving the image; and The summary is extracted from a portion of the summary stream corresponding to the time interval.

[0232] Example 23 includes the computer program product of Example 21 or 22, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: processing the transcript using a text entity extractor ML model to identify entities within the transcript; and The enhanced summary is displayed in which the entity is visually distinguished.

[0233] Example 24 includes the computer program product of any of Examples 21 to 23, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: processing the transcript using a sentiment analyzer ML model to identify a sentiment associated with the transcript; and The enhanced summary is displayed in which the emotion is visually distinguished.

[0234] Example 25 includes the computer program product of Example 24, wherein the emotion is indicated by including a corresponding emoji.

[0235] Example 26 includes the computer program product of any of the preceding Examples 21 to 25, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: Processing the image using an image entity extractor ML model to identify entities within the image; and The enhanced summary is displayed in which the entity is visually distinguished.

[0236] Example 27 includes the computer program product of any of the preceding Examples 21 to 26, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: processing the transcript using an intent extractor ML model to identify an intent associated with the transcript; and The enhanced summary is displayed in which the intent is visually distinguished.

[0237] Example 28 includes the computer program product of any of the preceding Examples 21 to 27, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: combining the image and the transcribed text to obtain an enhanced transcription; and showed that the enhanced transcription.

[0238] Example 29 includes the computer program product of any of Examples 21 to 28, wherein the at least one computing device includes a head mounted display (HMD), and wherein the instructions, when executed by the at least one computing device, are further configured to cause the at least one computing device to: The augmented summary is displayed using the HMD.

[0239] Example 30 includes the computer program product of any of the preceding Examples 21 to 29, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: displaying at least one of the transcription stream and the summary stream together with a scroll bar having scroll buttons; receiving movement of the scroll button, the movement aligning the scroll button with text of the transcript stream or the summary stream; and The enhanced summary is generated based on selection of the scroll button while aligned with the text.

[0240] A thirty-first example, referred to herein as Example 31, includes an apparatus comprising: at least one processor; at least one memory; at least one input device; and at least one display, wherein when executed by the at least one processor, using the instructions stored in the at least one memory causes the device to: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving an image associated with receipt of the transcribed stream; processing the transcription stream using a summary machine learning (ML) model to obtain a summary stream, including processing the transcription text to obtain a summary; combining the image and the summary to obtain an enhanced summary; and The enhanced summary is displayed using the at least one display.

[0241] Example 32 includes the apparatus of Example 31, wherein the apparatus comprises a head mounted display (HMD).

[0242] Example 33 includes the apparatus of Example 31 or 32, wherein the instructions, when executed by the at least one processor, cause the apparatus to: selecting a time interval based on receiving the image; and The summary is extracted from a portion of the summary stream corresponding to the time interval.

[0243] Example 34 includes the apparatus of any of the preceding Examples 31 to 33, wherein the instructions, when executed by the at least one processor, cause the apparatus to: processing the transcript using a text entity extractor ML model to identify entities within the transcript; and The enhanced summary is displayed in which the entity is visually distinguished.

[0244] Example 35 includes the apparatus of any of the preceding Examples 31 to 34, wherein the instructions, when executed by the at least one processor, cause the apparatus to: processing the transcript using a sentiment analyzer ML model to identify a sentiment associated with the transcript; and The enhanced summary is displayed with the emotion indicated therein.

[0245] Example 36 includes the apparatus of any of the preceding Examples 31 to 35, wherein the instructions, when executed by the at least one processor, cause the apparatus to: Processing the image using an image entity extractor ML model to identify entities within the image; and The enhanced summary is displayed in which the entity is visually distinguished.

[0246] Example 37 includes the apparatus of any of the preceding Examples 31 to 36, wherein the instructions, when executed by the at least one processor, cause the apparatus to: processing the transcript using an intent extractor ML model to identify an intent associated with the transcript; and The enhanced summary is displayed in which the intent is visually distinguished.

[0247] A thirty-eighth example, referred to herein as Example 38, includes a method comprising: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving an image associated with receipt of the transcribed stream; processing the transcription stream using a summary machine learning (ML) model to obtain a summary stream, including processing the transcription text to obtain a summary; combining the image and the summary to obtain an enhanced summary; and Displays a summary of the enhancement.

[0248] Example 39 includes the method of Example 38, further comprising: The enhanced summary is displayed on a display of a head mounted device (HMD).

[0249] Example 40 includes the method of Example 38 or 39, further comprising: selecting a time interval based on receiving the image; and The summary is extracted from a portion of the summary stream corresponding to the time interval.

[0250] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor that can be either special purpose or general purpose and can be coupled to receive data and instructions from and send data and instructions to a storage system, at least one input device, and at least one output device.

[0251] These computer programs (also referred to as modules, programs, software, software applications or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium", "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0252] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, or an LED (light emitting diode)) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including sound, voice, or tactile input.

[0253] The systems and techniques described herein may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back-end, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

[0254] A computing system may include clients and servers. Clients and computers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises from computer programs running on the respective computers and having a client-server relationship to each other.

[0255] In some implementations, one or more input devices other than the computing device (e.g., mouse, keyboard) may also be presented in the display of an HMD such as HMD 800. The presented input devices (e.g., presented mouse, presented keyboard) may be used as presented in the display.

[0256] A number of implementations have been described. However, it will be appreciated that various modifications can be made without departing from the spirit and scope of the description and claims.

[0257] Furthermore, the logic flows depicted in the figures do not require the particular order shown or sequential order to achieve the desired results. In addition, other steps can be provided, or steps can be deleted from the described flows, and other components can be added to or removed from the described systems. Therefore, other implementations are within the scope of the following claims.

[0258] Continuing with the above description, controls are provided to the user, allowing the user to make choices about whether and when the systems, programs, devices, networks, or features described herein can be able to collect user information (e.g., information about the user's social network, social behavior, or activities, occupation, user preferences, or the user's current location) and whether to send content or communications from the server to the user. In addition, before storing or using certain data, it can be processed in one or more ways so that the user information is removed. For example, the user identity can be processed so that the user information of the user cannot be determined, or the user's geographic location for which location information is obtained can be generalized (such as to the city, zip code, or state level) so that the specific location of the user cannot be determined. Therefore, the user can control what information is collected about the user, how the information is used, and what information is provided to the user.

[0259] The computer system (e.g., computing device) can be configured to communicate wirelessly with a network server over a network via a communication link established with the network server using any known wireless communication technologies and protocols, including radio frequency (RF), microwave frequency (MWF) and / or infrared frequency (IRF) wireless communication technologies and protocols suitable for communicating over a network.

[0260] According to various aspects of the present disclosure, implementations of the various techniques described herein may be implemented in digital electronic circuit systems, or in computer hardware, firmware, software, or in a combination thereof. The implementation may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium), for executing or controlling the operation of the data processing device by a data processing device (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that, when executed, cause the processor to perform a certain process. Computer programs such as the above-described computer programs may be written in any form of programming language, including compiled languages ​​or interpreted languages, and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. The computer program may be deployed to be processed on one computer or on multiple computers distributed at one site or across multiple sites and interconnected by a communication network.

[0261] The specific structural and functional details disclosed herein are merely representative for the purpose of describing the example implementations. However, the example implementations may be embodied in many alternative forms and should not be construed as being limited to only the implementations set forth herein.

[0262] The terms used herein are for the purpose of describing specific implementations only and are not intended to limit the implementations. As used herein, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the terms "comprises, comprising, includes, and / or including" when used in this specification specify the presence of stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0263] It should be understood that when an element is referred to as being "coupled" to another element, "connected" to another element, or "responsive to" another element or "on" another element, the element may be directly coupled to, connected to, responsive to or on another element, or intervening elements may also be present. Conversely, when an element is referred to as being "directly coupled" to another element, "directly connected" to another element, or "directly responsive to" another element or "directly on" another element, there are no intervening elements. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0264] For ease of description, spatially relative terms such as "below", "below", "below", "above", "on", etc. may be used herein to describe the relationship of one element or feature to another element or feature as shown in the figures. It should be understood that the spatially relative terms are intended to cover different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figure is flipped, then the elements described as being "below" or "below" other elements or features are then oriented to be "above" the other elements or features. Therefore, the term "below" can include both the above and below orientations. The device can be oriented in other ways (rotated 130 degrees or in other orientations), and the spatially relative descriptors used herein can be interpreted accordingly.

[0265] Example implementations of the concepts are described herein with reference to cross-sectional illustrations, which are schematic diagrams of idealized implementations (and intermediate structures) of the example implementations. Therefore, variations in the shapes of the illustrations as a result of, for example, manufacturing techniques and / or tolerances are expected. Therefore, the example implementations of the described concepts should not be interpreted as limited to the specific shapes of the zones shown herein, but include shape deviations, for example, due to manufacturing. Therefore, the zones shown in the figures are schematic in nature, and their shapes are not intended to illustrate the actual shapes of the zones of the device and are not intended to limit the scope of the example implementations.

[0266] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. Therefore, a "first" element may be referred to as a "second" element without departing from the teachings of the implementation of the present invention.

[0267] Unless otherwise defined, the terms (including technical and scientific terms) used herein have the same meaning as those commonly understood by those skilled in the art to which these concepts belong. It should be further understood that the terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and / or this specification, and should not be interpreted in an idealized or overly formal sense unless explicitly defined as such herein.

[0268] Although certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Therefore, it should be understood that the appended claims are intended to cover such modifications and changes that fall within the scope of the implementations. It should be understood that they are presented only by way of example and not limitation, and various changes in form and detail may be made. In addition to mutually exclusive combinations, any part of the devices and / or methods described herein may be combined in any combination. The implementations described herein may include various combinations and / or sub-combinations of the functions, components and / or features of the different implementations described.< / sentences> < / previoussentences>

Claims

1. A computer program product tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving a summary request for a summary to be provided on a display of the device; identifying extracted text from the transcribed text in response to the summary request; processing the extracted text using a summarization machine learning (ML) model to obtain a summary of the extracted text; and The summary is displayed on the display of the device.

2. The computer program product of claim 1 , wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The summary request is received from a user of the device via an input device of the device.

3. The computer program product of claim 1 or 2, wherein the input device comprises at least one of a touch screen, a gesture recognition device, a scroll bar, a button, or a microphone.

4. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The summary request is received as a voice command from a user of the device via a microphone of the device.

5. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The transcription stream is received from a speech recognition engine.

6. The computer program product of any one of the preceding claims, wherein the device comprises a head mounted display (HMD) and the display comprises an HMD display, and wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The transcribed stream is displayed using the HMD display.

7. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: identifying the extracted text as including text received at the device prior to the summary request; and The extracted text is extracted from a transcription buffer.

8. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: identifying the extracted text as including text received after the summary request; and The extracted text is extracted from the transcription stream following the summary request.

9. The computer program product of any one of the preceding claims, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: generating at least two summaries including the summary using the summarization ML model; and The summary is selected from the at least two summaries based on device characteristics of the device and user preferences of a user of the device.

10. The computer program product of any one of the preceding claims, wherein the device comprises a head mounted display (HMD) and the display comprises an HMD display, and wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The summary is displayed using the HMD display.

11. An apparatus comprising: at least one processor; at least one memory; at least one input device; as well as at least one display, wherein when executed by the at least one processor, using the instructions stored in the at least one memory causes the apparatus to: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving, via the input device, a summary request for a summary to be provided on the at least one display; identifying extracted text from the transcribed text in response to the summary request; processing the extracted text using a summarization machine learning (ML) model to obtain a summary of the extracted text; as well as The summary is displayed on the at least one display.

12. The apparatus of claim 11, wherein the apparatus comprises a head mounted display (HMD).

13. The apparatus of claim 11 or 12, wherein the apparatus is configured to receive the transcribed stream and the summary from a second apparatus in communication with the apparatus.

14. The device of any one of the preceding claims 11 to 13, wherein the input device comprises at least one of a touch screen, a gesture recognition device, a scroll bar, a button or a microphone.

15. The apparatus of claim 14, wherein the input device comprises a microphone, and the summary request is received as a voice command from a user of the apparatus via the microphone.

16. A method comprising: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving a summary request for a summary to be provided on a display of the device; identifying extracted text from the transcribed text in response to the summary request; processing the extracted text using a summarization machine learning (ML) model to obtain a summary of the extracted text; and The summary is displayed on the display of the device.

17. The method of claim 16, further comprising: storing the summary together with the summary request as labeled training data; as well as The summary ML model is trained using the labeled training data.

18. The method of claim 17, further comprising: detecting a second summary request using the summary ML model after the training; as well as The second extracted text is summarized using the summarization ML model.

19. The method of claim 16, wherein the device comprises a head mounted display (HMD) and the display comprises an HMD display, and the method further comprises: The summary is displayed using the HMD display.

20. The method of claim 16, further comprising: generating at least two summaries including the summary using the summarization ML model; as well as The summary is selected from the at least two summaries based on device characteristics of the device and user preferences of a user of the device.

21. A computer program product tangibly embodied on a non-transitory computer-readable storage medium and comprising instructions that, when executed by at least one computing device, are configured to cause the at least one computing device to: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving an image associated with receipt of the transcribed stream; processing the transcription stream using a summary machine learning (ML) model to obtain a summary stream, including processing the transcription text to obtain a summary; combining the image and the summary to obtain an enhanced summary; as well as A summary of the enhancements is displayed.

22. The computer program product of claim 21, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: selecting a time interval based on receiving the image; and The summary is extracted from a portion of the summary stream corresponding to the time interval.

23. The computer program product of claim 21, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: processing the transcript using a text entity extractor ML model to identify entities within the transcript; and The enhanced summary is displayed in which the entity is visually distinguished.

24. The computer program product of claim 21, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: processing the transcribed text using a sentiment analyzer ML model to identify a sentiment associated with the transcribed text; and The enhanced summary is displayed in which the emotions are visually distinguished.

25. The computer program product of claim 24, wherein the emotion is indicated by including a corresponding emoji.

26. The computer program product of claim 21, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: Processing the image using an image entity extractor ML model to identify entities within the image; and The enhanced summary is displayed in which the entity is visually distinguished.

27. The computer program product of claim 21, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: processing the transcribed text using an intent extractor ML model to identify an intent associated with the transcribed text; and The enhanced summary is displayed in which the intent is visually distinguished.

28. The computer program product of claim 21, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: combining the image and the transcribed text to obtain an enhanced transcription; and The enhanced transcription is shown.

29. The computer program product of claim 21, wherein the at least one computing device comprises a head mounted display (HMD), and wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: The enhanced summary is displayed using the HMD.

30. The computer program product of claim 21, wherein when executed by the at least one computing device, the instructions are further configured to cause the at least one computing device to: displaying at least one of the transcription stream and the summary stream together with a scroll bar having scroll buttons; receiving a movement of the scroll button, the movement aligning the scroll button with text of the transcription stream or the summary stream; as well as The enhanced summary is generated based on selection of the scroll button while aligned with the text.

31. An apparatus comprising: at least one processor; at least one memory; at least one input device; as well as at least one display, wherein when executed by the at least one processor, using the instructions stored in the at least one memory causes the apparatus to: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving an image associated with receipt of the transcribed stream; processing the transcription stream using a summary machine learning (ML) model to obtain a summary stream, including processing the transcription text to obtain a summary; combining the image and the summary to obtain an enhanced summary; as well as The enhanced summary is displayed using the at least one display.

32. The apparatus of claim 31 , wherein the apparatus comprises a head mounted display (HMD).

33. The apparatus of claim 31 or 32, wherein when executed by the at least one processor, the instructions cause the apparatus to: selecting a time interval based on receiving the image; and The summary is extracted from a portion of the summary stream corresponding to the time interval.

34. The apparatus of any one of claims 31 to 33, wherein when executed by the at least one processor, the instructions cause the apparatus to: processing the transcript using a text entity extractor ML model to identify entities within the transcript; and The enhanced summary is displayed in which the entity is visually distinguished.

35. The apparatus of any one of claims 31 to 34, wherein the instructions, when executed by the at least one processor, cause the apparatus to: processing the transcribed text using a sentiment analyzer ML model to identify a sentiment associated with the transcribed text; and The enhanced summary is displayed with the emotion indicated therein.

36. The apparatus of any one of claims 31 to 35, wherein when executed by the at least one processor, the instructions cause the apparatus to: Processing the image using an image entity extractor ML model to identify entities within the image; and The enhanced summary is displayed in which the entity is visually distinguished.

37. The apparatus of any one of claims 31 to 36, wherein when executed by the at least one processor, the instructions cause the apparatus to: processing the transcribed text using an intent extractor ML model to identify an intent associated with the transcribed text; and The enhanced summary is displayed in which the intent is visually distinguished.

38. A method comprising: receiving a transcription stream including transcribed text that has been transcribed from speech; receiving an image associated with receipt of the transcribed stream; processing the transcription stream using a summary machine learning (ML) model to obtain a summary stream, including processing the transcription text to obtain a summary; combining the image and the summary to obtain an enhanced summary; as well as A summary of the enhancements is displayed.

39. The method of claim 38, further comprising: The augmented summary is displayed on a display of a head mounted device (HMD).

40. The method of claim 38 or 39, further comprising: selecting a time interval based on receiving the image; as well as The summary is extracted from a portion of the summary stream corresponding to the time interval.