System and method for enabling modification of automatically orchestrated transcription in intelligent dictation

Abstract

Implementations described herein generally relate to generating modified selectable elements in an intelligent dictation session with an automated assistant that may be provided for presentation to a user. The modifying selectable element, when selected, may cause a transcription to be modified, the transcription including text data generated based on processing audio data capturing a spoken utterance and being automatically orchestrated. Transcription may be automatically orchestrated to include separation, punctuation, initial capitalization, indentation, segmentation, and / or other orchestration operations that the user does not specify when providing spoken utterances. Thus, subsequent selection of the modifying selectable elements may cause the auto-orchestration operations and / or text data located proximate to the auto-orchestration operations to be modified. Implementations described herein also relate to generating transcriptional and / or modifying selectable elements on behalf of a third-party software application.

Description

Background Art

[0001] A human may engage in a human-computer dialogue session with an interactive software application referred to herein as an "automated assistant" (also referred to as a "chatbot," "interactive personal assistant," "intelligent personal assistant," "personal voice assistant," "conversational agent," etc.). Automated assistants typically rely on a pipeline of components to interpret and respond to various inputs from humans (who may be referred to as "users" when interacting with automated assistants). For example, an automatic speech recognition (ASR) engine may be used to process audio data that captures a user's spoken utterances, and generate ASR outputs, such as text data corresponding to the spoken utterances (i.e., sequences of terms and / or other tokens). Further, a natural language understanding (NLU) engine may be used to process the ASR outputs, and generate NLU outputs, such as the intent of the user when providing the spoken utterances, and optionally slot values ​​for parameters associated with the intent. Furthermore, a fulfillment engine may be used to process the NLU outputs, and generate fulfillment outputs, such as structured requests for obtaining content in response to the spoken utterances or for causing an action to be performed in response to the spoken utterances.

[0002] In some cases, an automated assistant may be utilized to dictate transcription for a variety of purposes. For example, a user may provide spoken utterances to an automated assistant, and the automated assistant may process the spoken utterances using an ASR engine to generate transcriptions for inclusion in text messages, emails, notes, and the like. In some of these cases, the user typically must include specific formatting operations in the spoken utterances (e.g., by saying “period”, by saying “capitalize”, etc.). However, including these specific formatting operations in the spoken utterances increases the amount of user input provided by the user and / or the length of the user input, and requires the automated assistant to perform additional processing on the spoken utterances to determine the user's intent when providing these specific formatting operations, thereby unnecessarily wasting computing resources. In addition, including these specific formatting operations in the spoken utterances is unnatural for the user, and may cause the user to lose their train of thought when dictating the transcription, which may cause the user to provide further user input and extend the human-computer dialogue, thereby unnecessarily wasting computing resources.

[0003] Although many automated assistants are now able to automatically format text data for transcription (i.e., without the user having to include these specific formatting operations in the spoken utterance), these automated assistants may still make mistakes. Therefore, in some of these cases, the user must often manually manipulate the text data with additional formatting operations to format the text data with the desired formatting (i.e., desired separations, punctuation, capitalization, indentation, segmentation, etc.). However, the user having to manually manipulate the text data also increases the amount of user input provided by the user, thereby wasting computing resources unnecessarily. Therefore, there is a need in the art for improved techniques related to quickly and efficiently modifying the automatic formatting of text data in intelligent dictation. Summary of the invention

[0004] Some implementations described herein relate to an automated assistant that can generate text data corresponding to a spoken utterance provided by a user of a client device, generate a transcription that is automatically arranged (i.e., in a manner that is not explicitly specified by the user in the spoken utterance), and generate a modification selectable element that causes the automatically arranged transcription to be modified when subsequently presented and selected by the user. For example, a processor of the client device can receive audio data capturing the spoken utterance via a microphone of the client device, process the audio data capturing the spoken utterance using an automatic speech recognition (ASR) model to generate text data, generate an automatically arranged transcription based on at least the text data corresponding to the spoken utterance, cause the transcription to be provided for presentation to the user via a display of the client device, and generate a modification selectable element. Further, the processor of the client device can receive a touch input from the user directed to the transcription. In addition, and in response to receiving the touch input from the user directed to the transcription and automatically arranged based on the transcription of the spoken utterance, the processor of the client device can cause the modification selectable element to be provided for presentation to the user via the display of the client device. Thus, implementations described herein are directed to improved techniques for quickly and efficiently modifying automatically composed transcriptions. Thus, an automated assistant can guide a user to end a dictation session in a faster and more efficient manner by reducing the overall duration of the dictation session using modification selectable elements.

[0005] In various implementations, and upon selecting to modify the selectable element, the modification of the automatic arrangement of the transcription can be based on an automatic arrangement operation for automatically arranging the transcription. For example, the processor of the client device can use one or more automatic arrangement machine learning (ML) models to process audio data capturing the spoken utterance and / or text data corresponding to the spoken utterance to generate automatic arrangement data. The one or more automatic arrangement ML models can include: an automatic punctuation ML model that is trained to identify automatic punctuation for the transcription for inclusion in the automatic arrangement data; an automatic capitalization ML model that is used to identify automatic capitalization for the transcription for inclusion in the automatic arrangement data; an automatic segmentation ML model that is trained to identify automatic segmentation for the transcription for inclusion in the automatic arrangement data; an automatic line wrapping ML model that is trained to identify automatic line wrapping for the transcription for inclusion in the automatic arrangement data; and / or any other ML model that can be used in generating the automatic arrangement of the transcription.

[0006] For example, assume that a user is utilizing an automated assistant (e.g., executing at least in part at a user's client device) to dictate the text of an email, and assume that the user intended the spoken utterance to be dictated as “You can use any resources to complete the exam.” However, assume that the automated assistant generates the transcription “You can use any resources. To complete the exam.” Notably, in this example, the automated assistant inadvertently included a period after the given term “resources,” and inadvertently capitalized the first letter of the subsequent term “to” following the given term “resources” and the inadvertent period. Thus, rather than the user having to manually remove the period inadvertently included after "resources" and manually lowercase the inadvertently capitalized "to", the user can simply direct a touch input toward the unintentional automatic arrangement in the transcription and be provided with a modification selectable element which, when subsequently selected by the user, automatically causes the automatically arranged transcription to be modified to what the user originally intended by removing the period inadvertently included after the given term "resources" and by lowercaseing the subsequent term "to" that was inadvertently capitalized.

[0007] In this example, if it is a non-inherent capitalization that is capitalized only based on following an inadvertently included period, the selection of the modifying optional element can only cause the subsequent term "to" that is inadvertently capitalized to be made lowercase. Conversely, assume that the word following the inadvertently included period is inherently capitalized, such as an entity reference to an entity (e.g., a reference to a person, organization, location, etc.). For example, based on grammar rules, the personal pronoun "I (我)" as an entity reference to a user can be considered inherently capitalized, while other personal pronouns (e.g., "you (你)", "she(她)", "her (她)", etc.) may not be considered inherently capitalized, and proper nouns can be considered entity references to entities (e.g., "Boston Red Sox (波士顿红袜队)" can be a proper noun, which is an entity reference to a professional baseball team located in Boston, Massachusetts, "John (约翰)" can be a proper noun, which is an entity reference to a person), and so on.

[0008] As another example, further assume that the user wants the additional spoken utterance to be dictated as “Please let meknow if you have any questions. [paragraph break] Regards, Teacher.” However, assume that the automated assistant generates the transcription “[paragraphbreak] Please let me know if you have any questions. [paragraph break]Regards, Teacher,” resulting in the transcription “You can use any resources to complete the exam. [paragraph break] Pleaselet me know if you have any questions. [paragraph break] Regards, Teacher” (e.g., when combined with the transcription in the previous example). Notably, in this example, the user may not have intended a segment between the spoken utterances "You can use any resources to complete the exam" and "Please let me know if you have any questions," but did intend a segment between "Please let me know if you have any questions" and "Regards, Teacher." Similarly, in this example, the automated assistant inadvertently included an additional segment, which was beyond what the user intended. Thus, rather than the user having to manually remove the inadvertently included segment, the user can simply direct a touch input toward the inadvertent automatic arrangement in the transcription and be provided with a modification selectable element, which, when subsequently selected by the user, automatically causes the automatically arranged transcription to be modified to what the user originally intended by removing the inadvertently included segment.

[0009] Although the above examples are described with respect to the modification of unintentional punctuation and unintentional separation, it should be understood that these examples are provided to illustrate the technology of the present disclosure and are not intended to be limiting. Rather, it should be understood that the technology of the present disclosure can be used to provide modification selectable elements to perform any editing operation for automatically editing a transcription.

[0010] In various implementations, touch input received from a user and directed to a transcription of a spoken utterance may be directed to various portions of the transcription so that a modification selectable element is provided for presentation to the user. Returning to the above example in which the user wants the spoken utterance to be dictated as "You can use any resources to complete the exam." but the spoken utterance has been inadvertently automatically formatted as the transcription "You can use any resources. To complete the exam." In this example, the user may direct the touch input to a given term "resources," a subsequent term "to," and / or a period inadvertently included between a given term "resources" and a subsequent term "to." In other words, the modification selectable element may be provided for presentation to the user only in response to determining that the touch input is located proximate to an automatically formatted portion of the transcription. Therefore, it should be understood that the three locations in which a touch input may be provided to cause a modification selectable element to be provided for presentation to the user as described above are for example and are not intended to be limiting. Rather, it should be appreciated that the location to which a touch input may be directed to cause a modifying selectable element to be provided for presentation to the user may vary based not only on the textual data included in the transcription, but also on the manner in which the textual data is automatically arranged in the transcription.

[0011] In various implementations, the processor may generate a modification selectable element only for the automatically arranged portion of the transcription. Return to the above example in which the transcription has been arranged as "You can use any resources. To complete the exam." In contrast, suppose that during dictation, the user said "period" after saying "resources." In this example, and despite incorrect grammar (e.g., because the transcription includes two separated sentences), the processor may avoid generating a modification selectable element because the user explicitly stated "period" after resources. Nevertheless, other techniques may also be utilized to alert the user to this grammatical error. It is noteworthy that other portions of the same transcription may still be automatically arranged, and the processor may still generate modification selectable elements for those other automatically arranged portions of the transcription.

[0012] Other implementations described herein relate to an automated assistant that can generate transcriptions and / or modify selectable elements as a service of a third-party software application executed at least in part at a client device. For example, a processor of a client device can generate an automatically arranged transcription in the same or similar manner as described above based on text data corresponding to at least a spoken utterance. In some versions of these implementations, the processor can cause the transcription to be provided to be presented to the user via a display of the client device as described above, and in other versions of these implementations, the processor can cause the automatically arranged transcription to be provided to a third-party software application, thereby causing the third-party software application to provide the automatically arranged transcription to be presented to the user via the display of the client device. In some versions of these implementations, the processor can generate a modified selectable element, and can cause the modified selectable element to be provided to a third-party software application, thereby causing the third-party software application to provide a modified selectable element to be presented to the user in response to receiving a touch input from the user. In other versions of those implementations, the processor can provide automatic arrangement data to the third-party software application so that the third-party software application can generate a modified selectable element. Thus, implementations described herein are also directed to improved techniques for third-party software applications to quickly and efficiently modify automatically composed transcriptions.

[0013] By using the techniques described herein, one or more technical advantages may be achieved. As a non-limiting example, the techniques described herein enable an automated assistant to more quickly and efficiently correct unintentional automatic arrangement of a transcription via modification of a selectable element that uses a single click or a double click to correct the unintentional automatic arrangement of the transcription. Thus, the amount of user input received at the client device is reduced, and the total duration required to correctly format the dictated transcription is reduced. Further, these techniques guide human-computer interaction by providing a mechanism to quickly and efficiently correct unintentional automatic arrangement of a transcription, thereby reducing the duration of the human-computer interaction. Thus, the claimed techniques produce rapid and efficient modification of the automatic arrangement of a transcription in intelligent dictation.

[0014] The above description is provided as an overview of only some implementations of the present disclosure. Further descriptions of those implementations and other implementations are described in greater detail herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an example environment in which implementations disclosed herein may be implemented.

[0016] Figure 2A , Figure 2B , Figure 2C , Figure 2D , Figure 2E and Figure 2F Various non-limiting examples of automatically formatting text data in a transcription and correcting unintentional automatic formatting in a transcription are shown according to various implementations.

[0017] Figure 3 is a flow chart illustrating an example method of automatically formatting text data in a transcription and correcting unintentional automatic formatting in the transcription according to various implementations.

[0018] Figure 4 is a flow chart illustrating an example method of providing automatic formatting of text data in a transcription and correction of unintentional automatic formatting in a transcription as a service to a third-party software application, according to various implementations.

[0019] Figure 5 An example architecture of a computing device according to various implementations is shown. DETAILED DESCRIPTION

[0020] Steering Figure 1 , depicts a block diagram of an example environment that illustrates various aspects of the present disclosure and in which implementations disclosed herein may be implemented. The example environment includes a client device 110 and a dictation system 160. In some implementations, the dictation system 160 may be implemented locally on the client device 110. In additional or alternative implementations, the dictation system 160 may be implemented remotely from the client device 110, such as Figure 1 In these implementations, client device 110 and dictation system 160 may be communicatively coupled to one another via one or more networks 190, such as one or more wired or wireless local area networks (“LANs,” including Wi-Fi®, mesh networks, Bluetooth®, near field communications, etc.) or wide area networks (“WANs,” including the Internet).

[0021] The client device 110 may be, for example, one or more of a desktop computer, a laptop computer, a tablet computer, a mobile phone, a computing device of a vehicle (e.g., an in-vehicle communication system, an in-vehicle entertainment system, an in-vehicle navigation system), a stand-alone interactive speaker (optionally with a display), a smart home appliance (such as a smart TV), and / or a user's wearable device including a computing device (e.g., a user's watch with a computing device, a user's glasses with a computing device, a virtual or augmented reality computing device, etc.). Additional and / or alternative client devices may be provided.

[0022] Client device 110 may execute automated assistant client 113. An instance of automated assistant client 113 may be an application separate from (e.g., installed “on top of”) an operating system of client device 110, or may alternatively be implemented directly by the operating system of client device 110. Automated assistant client 113 may interact with dictation system 160 locally on client device 110 via dictation system client 114 (e.g., an instance of dictation system 160 that may be implemented locally on client device 110) and / or via network 190 (e.g., such as a Figure 1 15 ) remotely from client device 110 to dictation system 160. Automated assistant client 113 (and optionally by way of its interaction with dictation system 160) may form what from the user's perspective appears to be a logical instance of automated assistant 115, which the user may utilize to engage in a human-computer conversation. Figure 1 10 and is surrounded by a dashed line that includes an automated assistant client 113 of a client device 110 and a dictation system 160. Thus, it should be understood that a user interacting with an automated assistant client 113 executing on a client device 110 may actually be interacting with his or her own logical instance of an automated assistant 115 (or a logical instance of an automated assistant 115 shared among a family or other group of users). For the sake of brevity and simplicity, automated assistant 115 as used herein will refer to automated assistant client 113 executing on and / or remotely from client device 110. Further, dictation system 160 as used herein will refer to dictation system client 114 executing at and / or remotely from client device 110.

[0023] In various implementations, the client device 110 may include a user input engine 111 configured to detect user input provided by a user of the client device 110 using one or more user interface input devices. For example, the client device 110 may be equipped with one or more microphones configured to detect sound waves (or pressure waves), such as spoken words of a user or other sounds in the environment of the client device 110, and generate corresponding audio data that captures the spoken words of the user. Additionally or alternatively, the client device 110 may be equipped with one or more visual components configured to capture objects and / or movements (e.g., gestures) detected in the field of view of one or more visual components of the visual components and generate visual data that captures the objects and / or movements. Additionally or alternatively, the client device 110 may be equipped with one or more touch-sensitive components (e.g., a keyboard and mouse, a stylus, a touch screen, a touch panel, one or more hardware buttons, etc.), which are configured to capture touch input directed to the client device 110 and generate corresponding touch data that captures the touch input.

[0024] In various implementations, the client device 110 may include a rendering engine 112 configured to provide content for audible and / or visual presentation to a user of the client device 110 using one or more user interface output devices. For example, the client device 110 may be equipped with one or more speakers that enable content to be provided for audible presentation to a user via the client device 110. Additionally or alternatively, the client device 110 may be equipped with a display or projector that enables content to be provided for visual presentation to a user via the client device 110.

[0025] Further, the client device 110 and / or the dictation system 160 may include one or more memories for storing data and / or software applications 195, one or more processors for accessing data and executing software applications 195, and other components that facilitate communication over one or more of the networks 190. In some implementations, one or more of the software applications 195 may be installed locally on the client device 110, while in other implementations, one or more of the software applications 195 may be hosted remotely from the client device 110 (e.g., by one or more servers) but accessible to the client device 110 over one or more of the networks 190.

[0026] In some implementations, the operations performed by automated assistant 115 may be implemented locally on client device 110. As described above, in various implementations, client device 110 may operate automated assistant client 113. Figure 1 As shown in FIG. 1 , the automated assistant client 113 may include an automatic speech recognition (ASR) engine 120A1, a natural language understanding (NLU) engine 130A1, and a text-to-speech (TTS) engine 140A1. In some implementations, the operations performed by the automated assistant 115 may be distributed across multiple computer systems, such as when the dictation system 160 is transmitted from a plurality of computer systems such as a computer system 160. Figure 1 In these implementations, automated assistant 115 can utilize ASR engine 120A2, NLU engine 130A2, and TTS engine 140A2 of dictation system 160.

[0027] Each of these engines can be configured to perform one or more functions. For example, the ASR engine 120A1 and / or 120A2 can use an ASR model (e.g., a streaming ASR model or other ASR model) stored in a machine learning (ML) model database 199 to process audio data that captures spoken speech and is generated by a microphone of the client device 110 to generate an ASR output. Further, the NLU engine 130A1 and / or 130A2 can use an NLU model stored in the ML model database 199 to process the ASR output to generate an NLU output. In addition, the TTS engine 140A1 and / or 140A2 can use a TTS model stored in the ML model database 199 to process text data (e.g., text formulated by the automated assistant 115) to generate synthesized speech audio data including computer-generated synthesized speech.

[0028] In various implementations, the ASR output may include, for example, one or more speech hypotheses predicted to correspond to a user's spoken utterance captured in the audio data, one or more corresponding prediction values ​​(e.g., probabilities, log-likelihoods, and / or other values) for each of the one or more speech hypotheses, a plurality of phonemes predicted to correspond to the user's spoken utterance captured in the audio data, and / or other ASR outputs. In some versions of those implementations, the ASR engines 120A1 and / or 120A2 may select one or more of the speech hypotheses as recognized text corresponding to the spoken utterance (e.g., based on the corresponding prediction values). The recognized text corresponding to the spoken utterance may be considered to be text data corresponding to the spoken utterance.

[0029] In various implementations, the NLU output may include, for example, annotated recognized text, which includes one or more annotations of the recognized text for one or more (e.g., all) terms in the terms of the text data. For example, the NLU engine 130A1 and / or 130A2 may include a part-of-speech tagger (not depicted) that is configured to annotate terms by the grammatical role of the terms. In addition or alternatively, the NLU engine 130A1 and / or 130A2 may include an entity tagger (not depicted) that is configured to annotate entity references in one or more fragments of the recognized text, such as references to people (including, for example, literary characters, celebrities, public figures, user's contact entries, etc.), organizations, locations (real and fictional), etc. In some implementations, data about entities may be stored in one or more databases, such as in a knowledge graph (not depicted). In some implementations, the knowledge graph may include nodes representing known entities (and in some cases, entity attributes), and edges connecting nodes and representing relationships between entities. The entity tagger may annotate references to entities at a high level of granularity (e.g., to enable identification of all references to a class of entities such as persons) and / or at a lower level of granularity (e.g., to enable identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of the natural language input to resolve the specific entity and / or may optionally communicate with a knowledge graph or other entity database to resolve the specific entity.

[0030] Additionally or alternatively, the NLU engines 130A1 and / or 130A2 may include a coreference resolver (not depicted) configured to group or "cluster" references to the same entity based on one or more context clues. For example, the coreference resolver may be utilized to resolve the term "them" in the natural language input "buy them" as "buy theatre tickets" based on the mention of "theatre tickets" in a client device notification rendered immediately before receiving the natural language input "buy them". In some implementations, one or more components of the NLU engines 130A1 and / or 130A2 may rely on annotations from one or more other components of the NLU engines 130A1 and / or 130A2. For example, in some implementations, the entity tagger may rely on annotations from the coreference resolver to annotate all mentions of a particular entity. Also, for example, in some implementations, the coreference resolver may rely on annotations from the entity tagger to cluster references to the same entity.

[0031] As described herein, automated assistant 115 can generate a transcription that is based on a spoken utterance provided by a user of client device 110 and that is automatically formatted for the user and / or manually formatted by the user based on the spoken utterance. In implementations where the transcription is automatically formatted, automated assistant 115 can generate a modification text element that enables unintended automatic formatting in the transcription to be quickly and efficiently modified by utilizing dictation engine 150. In various implementations, dictation engine 150 may include text data engine 151, formatting data engine 152, transcription engine 153, and modification engine 154, such as Figure 1 As depicted in .

[0032] Now go to FIG. 2A to FIG. 2F , automatically formatting text data in a transcription and correcting unintentional automatic formatting in the transcription, various non-limiting examples. Figure 2A, assume that a user 100 of a client device 110 directs a spoken utterance 200, "Assistant, email my students 'You can use any resources to complete the exam. Please let me know if you have any questions. Regards, Teacher," to an automated assistant 115 executing at least in part at the client device 110. The spoken utterance 200 may be captured in audio data 202 generated by a microphone of the client device 110. In various implementations, the audio data 202 may be processed only in response to determining that the audio data 202 invokes the automated assistant 115. For example, a first portion 202A of the audio data 202 may capture a particular word or phrase, such as "Assistant." The automated assistant 115 may process the first portion 202A of the audio data 202 using a hot word detection model (e.g., stored in the ML model database 199) to determine that the spoken utterance 200 provided by the user 100 is directed to the automated assistant 115 based on the inclusion of a particular word or phrase in the spoken utterance 200. Although the particular word or phrase is described as being captured in the first portion 202A of the audio data 200, it should be understood that this is for example purposes, and the particular word or phrase may be captured in other portions of the audio data 202, such as those corresponding to the middle of the spoken utterance 200, the end of the spoken utterance 200. As another example, the user 100 may tap a button (e.g., a hardware button or a software button) that invokes the automated assistant 115. As yet another example, the user may direct their gaze toward the client device 110 (e.g., captured in visual data generated by a visual component of the client device 110), and optionally accompanied by mouth movement, lip movement, head movement, etc. (e.g., also captured in visual data generated by a visual component of the client device 110), to invoke the automated assistant 115.

[0033] In various implementations, the audio data 202 may be processed to determine the intent of the user 100 when providing the spoken utterance 200. For example, the first portion 202A of the audio data 202 may also capture a request or command directed to the automated assistant 115. The automated assistant 115 may cause the ASR engine 120A1 and / or 120A2 and / or the NLU engine 130A1 and / or 130A2 to process the first portion 202A of the audio data 202 using an ASR model and / or an NLU model (e.g., stored in the ML model database 199) to recognize the command "email my students". The command "email my students" may include an "email" intent that may cause an email application accessible at the client device 110 to be launched, and a slot value of "students" for the "to" field parameter of the "email" intent. Thus, the automated assistant 115 may cause the email application to launch to a state including an email draft for "students" sent to the user 100, which may be provided for presentation to the user 100 of the client device via the display 180 of the client device 110. Although the first portion 202A of the audio data is described as capturing both hot words (e.g., "Assistant") and commands (e.g., "email my students"), it should be understood that this is for example and is not intended to be limiting. Rather, it should be understood that hot words and commands may be captured in one or more of a plurality of different portions of the audio data 202.

[0034] Further, any subsequent portions of the spoken utterance 200 captured in the audio data 202 can be incorporated into the transcription corresponding to the email draft as part of the dictation session between the user 100 and the automated assistant 115 utilizing the dictation engine 150. The transcription can be generated based on text data generated based on processing the audio data 202 capturing the spoken utterance 200. In some implementations, the transcription can be arranged according to a manual arrangement specified by the user 100 (e.g., by the user actually saying "period" after saying "exam" in the spoken utterance 200), while in other implementations, the transcription can be arranged automatically when the user 100 does not specify any manual arrangement.

[0035] Continuing with the above example, assume that the second portion 202B of the audio data 202 captures "You can use any resources to complete the exam" from the spoken utterance 200, and assume that the third portion 202C of the audio data 202 captures "Please let me know if you have any questions. Regards, Teacher". The text data engine 151 can generate text data 251 corresponding to the spoken utterance 200 by causing the ASR engine 120A1 and / or 120A2 to process the second portion 202B and the third portion 202C of the audio data 202 using the ASR model. Figure 2B , the text data 251 may include recognized text corresponding to the terms and / or phrases captured in the second portion 202B and the third portion 202C of the audio data 202. Although the terms and / or phrases included in the text data 251 are shown as being provided in Figure 2B 100 is presented to the user 100 at the display 180 of the client device 110 as depicted in FIG, but it should be understood that this is for illustration purposes. Figure 2B The text data 251 depicted in FIG. 2 includes only words and / or phrases without punctuation, but it should be understood that this is for example and is not intended to be limiting. For example, some ASR models can generate text data including punctuation.

[0036] Further, the arrangement data engine 152 can generate arrangement data 252 for automatically arranging the text data 251 in the transcription of the email draft. In an implementation in which the user 100 specifies a manual arrangement, the arrangement data 252 can include data that captures the manual arrangement specified by the user 100. However, in an implementation in which the user 100 does not specify any manual arrangement, the arrangement data 252 can generate data that specifies the automatic arrangement of the text data 251. Specifically refer to Figure 2C , the layout data 252 may include various layout operations for automatically layout text content, such as for indicating that the following letters should be capitalized. <cap>, used to indicate that a period should be inserted <per>, used to indicate that a segment should be inserted <pb>, used to indicate that a comma should be inserted <com>, used to indicate that a line break should be inserted <lb>, and / or other layout operations for automatically arranging text data 251. Although the layout operations included in the layout data 252 depicted with respect to the text data 251 are shown as being provided to Figure 2C 100 is presented to the user 100 at the display 180 of the client device 110 as depicted in FIG, but it should be understood that this is for illustration purposes. Figure 2C The orchestration operations included in the orchestration data 252 depicted in FIG. 2 are specific orchestration operations, but it should be understood that this is for example only and is not intended to be limiting.

[0037] For example, in an implementation where the user 100 does not specify any manual orchestration, the orchestration data engine 152 may cause the text data 251, the second portion 202B of the audio data 202, and / or the third portion 202C of the audio data 202 to be processed using an automatic orchestration ML model (e.g., stored in the ML model database 199) to generate orchestration data 252. The automatic orchestration ML model may include any ML model that can be used to automatically orchestrate the text data 251 in the transcription. The automatic orchestration ML model used by the orchestration data engine 152 to automatically orchestrate the transcription can be based on the type of transcription to be dictated. For example, if the transcription to be dictated corresponds to an email, the orchestration data engine 152 can automatically orchestrate the transcription using a first set of orchestration ML models. In contrast, if the transcription to be dictated corresponds to a text message, then the orchestration data engine 152 may automatically orchestrate the transcription using a second set of orchestration ML models that includes and / or omits at least one orchestration ML model when compared to the first set of orchestration ML models (e.g., a segmentation ML model may not be used in generating a transcription of a text message because text messages are typically shorter in length than emails and users typically do not include segments in text messages).

[0038] Continuing with the above example, the automatic arrangement ML model may include an automatic punctuation ML model trained to recognize automatic punctuation marks for transcription included in the arrangement data 252, such as after "resources". <per>, after "exam" <per>, "per" after "questions" and "regards" after <com>an automatic capitalization ML model for identifying automatic capitalization for transcriptions included in the choreography data 252, such as for capitalizing the "y" in "you" <cap>, used to capitalize the first letter of "t" in "to" <cap>, used to capitalize the first letter of "p" in "please" <cap>, used to capitalize the first letter of "r" in "regards" <cap>and capitalize the "t" in "teacher" <cap>; an automatic segmentation ML model trained to recognize automatic segments for transcription included in the choreographed data 252, such as after “exam” <per>After <pb>and after "questions" <per>After <pb>A word wrap ML model trained to recognize word wraps for transcriptions included in auto-formatted data, such as after "regards" <com>Behind <lb>; and / or any other ML model that may be used in generating automatic arrangement data for the instance, according to which the transcription may be automatically arranged. Notably, in some implementations, these automatic arrangement ML models may be stored locally in on-device storage of the client device 110 (e.g., the ML model database 199 may be local to the client device 110), while in other implementations, these automatic arrangement ML models may be stored remotely from the client device 110 (e.g., at the dictation system 160).

[0039] Specific reference Figure 2D , the transcription engine 153 can be based on Figure 2B The text data 251 and from Figure 2C However, assume that when providing the spoken utterance 200, the user 100 does not want the transcription 250D to include the word “resources” after “resources”. <per>, used for the "t" in "to" <cap>or after "exam" <per>After <pb>, as shown in transcription 250D. Nevertheless, modification engine 154 can generate one or more modification selectable elements for transcription 250D based on transcription 250D being automatically formatted. Specifically, modification engine 154 can generate one or more modification selectable elements that, when selected, cause the formatting operations included in text data 251 and / or formatting data 252 to be modified, thereby modifying transcription 250D.

[0040] Continuing with the above example, it is further assumed that the orchestration operations included in text data 251 and / or orchestration data 252 are stored in one or more databases accessible to the client device (e.g., in transcription database 150A). Modification engine 154 can access the orchestration operations included in text data 251 and / or orchestration data 252 to generate one or more modification selectable elements for transcription 250D. For example, for each orchestration operation that was performed in order to automatically orchestrate transcription 250D, modification engine 154 can generate a corresponding modification selectable element that causes one or more corresponding orchestration operations to be undone.

[0041] As a non-limiting example, the modification engine 154 may be based on the <per>and the "t" in "to" <cap>The transcription 250D is automatically arranged to generate the first modification selectable element 254D, so that when the user 100 selects the first modification selectable element 254D, the text after "resources" <per>can be removed and used for the "t" in "to" <cap>Can be changed to indicate that the "t" in "to" should be lowercase <decap>The first modification selectable element 254D may optionally include a preview of the modifications to be made to the transcript 250D, such as Figure 2D In this example, the first modified selectable element 254D may be in response to receiving a pointer to the word "resources" and after "resources". <per>and / or touch input of the word "to" (e.g., as in Figure 2D 101 in the example) is provided for presentation to the user 100 via the display 180. Thus, when the user 100 selects the first modification selectable element 254D, the transcript 250D may be modified to remove the "resources" after the "resources" <per>, and change the first letter of "t" in "to" to lowercase, such as Figure 2E 250E in the modified transcription 250E. Notably, in this example, if "to" is another word corresponding to a word that is inherently capitalized (e.g., as indicated by data stored in the transcription database 150A), the "t" in "to" may not be lowercase.

[0042] As another non-limiting example, and with specific reference to Figure 2E , modification engine 154 can be based on the following "exam" <pb>The transcript 250D (and the modified transcript 250E) is automatically arranged to generate a second modified selectable element 254E, so that when the user 100 selects the second modified selectable element 254E, the words after "exam" are displayed. <pb>can be removed. The second modification selectable element 254E can optionally include a preview of the modifications to be made to the modified transcript 250E, such as Figure 2E In this example, the second modifying selectable element 254E may be responsive to receiving a touch input directed to the space occupied by the segment (e.g., as in Figure 2E 101) is provided for presentation to the user 100 via the display 180. Thus, when the user 100 selects the second modification selectable element 254E, the modified transcript 250E may be further modified to remove the "exam" after the "exam". <pb>,like Figure 2F , which makes the text data arranged in the transcription appear as originally intended by the user 100 when providing the spoken utterance 200.

[0043] Thus, the techniques described herein enable the automatically composed transcript 250D to be modified quickly and efficiently by utilizing modification selectable elements. FIG. 2A to FIG. 2F The described examples describe specific layout operations, specific modification selectable elements, and directing touch inputs to cause specific locations of specific modification selectable elements, but it should be understood that this is for the purpose of example and is not intended to be limiting. Rather, it should be understood that the layout operations, modification selectable elements, and touch inputs described above can vary depending on the spoken utterance provided by the user and how automated assistant 115 determines how to layout text data corresponding to the spoken utterance.

[0044] Now go to Figure 3 , depicts a flow chart illustrating an example method 300 for automatically formatting text data in a transcription and correcting unintentional automatic formatting in the transcription. For convenience, the operations of the method 300 are described with reference to a system that performs these operations. The system of the method 300 includes a computing device (e.g., Figure 1 and FIG. 2A to FIG. 2F The client device 110, Figure 5 The method 300 may be performed by one or more processors, memories, and / or other components of a computing device 510, one or more servers, and / or other computing devices. In addition, although the operations of method 300 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0045] At box 352, the system receives audio data capturing spoken utterances of a user of the client device, the audio data being generated by one or more microphones of the client device. At box 354, the system processes the audio data using the ASR model to generate text data corresponding to the spoken utterances. The system may return to the operations of box 352 to receive additional audio data capturing additional spoken utterances of the user. In other words, the system may conduct a dictation session with the user by enabling the user to continuously provide spoken utterances. Furthermore, the system may continuously process the audio data capturing the spoken utterances while the system generates text data corresponding to the spoken utterances according to the ASR model. Figure 3 The method 300 proceeds to the operation of block 356 simultaneously. Therefore, it should be understood that the dictation can be performed in a parallel manner based on the spoken utterance provided by the user during the dictation session. Figure 3 Multiple iterations of method 300.

[0046] At block 356, the system determines whether the user has specified manual arrangement of text data in the spoken utterance. For example, the system may determine whether the user has specified "period," "comma," "capitalize," "new paragraph," "indent," and / or any other command for manual arrangement of text data in the spoken utterance. If, in an iteration of block 356, the system determines that the user has not specified manual arrangement of text data in the spoken utterance, the system may proceed to the operation of block 358.

[0047] At box 358, the system generates an automatically formatted transcription of the spoken utterance based on the text data. At box 360, the system causes the transcription to be provided for presentation to the user via a display of the client device. For example, the system can not only generate a transcription based on the text data corresponding to the spoken utterance, but the system can also automatically format the text data corresponding to the spoken utterance based on the formatting data generated by the system. The system can use various automatic formatting ML models to process the audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance to generate the formatting data (e.g., as described in relation to the audio data). Figure 1 and FIG. 2A to FIG. 2F Further, the system may cause the transcription to be provided for presentation to the user via a display of the client device and optionally in a streaming manner (e.g., as the user continues to provide spoken utterances and / or additional spoken utterances).

[0048] At block 362, the system generates a modification selectable element that, when selected, causes the automatically arranged transcription to be modified. The modification selectable element may be specific to one or more given arrangement operations used in the automatically arranged transcription (e.g., as described in relation to Figure 1 and FIG. 2A to FIG. 2F described).

[0049] At block 364, the system determines whether a touch input directed to the transcription was received. Specifically, the system may determine whether a touch input directed to an automatically arranged portion of the transcription was received. If the system determines in an iteration of block 364 that a touch input directed to the transcription was not received, the system may continue to monitor for touch input in iterations of block 364. Notably, the system may continue to monitor for touch input directed to an automatically arranged portion of the transcription for the duration of the dictation system between the user and the system.

[0050] If, in an iteration of box 364, the system determines that touch input directed to the transcription is received, the system can proceed to the operation of box 366. At box 366, the system causes the modification selectable element to be provided for presentation to the user via a display of the client device. For example, the modification selectable element can be provided for presentation to the user in a manner that the modification selectable element covers a portion of the transcription. Also, for example, the modification selectable element can be provided for presentation to the user in a manner that the modification selectable element is provided at a different portion of the display. In some implementations, the modification selectable element can include a preview of modifications to the transcription that will be made in response to selection of the modification selectable element. In other implementations, the modification selectable element can include an indication of an editing operation that can be performed in response to selection of the modification selectable element (e.g., "select this to remove the paragraph break").

[0051] At block 368, the system determines whether a touch input directed to the modifying selectable element is received. If the system determines in an iteration of block 368 that a touch input directed to the modifying selectable element is not received, the system can continue to monitor for touch input in iterations of block 368. In various implementations, a touch input directed to the display that is not a selection of the modifying selectable element can cause the modifying selectable element to be removed from the display of the client device. In these implementations, Figure 3 The method 300 may return to the operations of block 364 to allow the user to cause modification of the selectable element to be provided for presentation to the user based on the touch input described above with respect to the operations of block 364 .

[0052] If, in an iteration of block 368, the system determines that a touch input directed to a modifying selectable element is received, the system can proceed to operations at block 370. At block 370, the system causes the automatically formatted transcription to be modified according to the modifying selectable element. The modification to the transcription can be based on text data and / or formatting operations associated with the modifying selectable element (e.g., such as the text data associated with the modifying selectable element). Figure 1 and FIG. 2A to FIG. 2F Thus, in implementations in which the system automatically arranges the transcription for the user, the system may also provide a mechanism that enables the user to quickly and efficiently modify the automatically arranged transcription (e.g., modify selectable elements).

[0053] If, in an iteration of box 356, the system determines that the user has specified a manual arrangement of text data in the spoken utterance, the system can proceed to the operation of box 372. At box 372, the system generates a transcription of the spoken utterance based on the text data and based on the manual arrangement of the text data. At box 374, the system causes the transcription to be provided for presentation to the user via a display of the client device. For example, the system can not only generate a transcription based on the text data corresponding to the spoken utterance, but the system can also arrange the text data corresponding to the spoken utterance based on the manual arrangement of text data specified by the user in the spoken utterance. Further, the system can cause the transcription to be provided for presentation to the user via a display of the client device and optionally in a streaming manner (e.g., while the user continues to provide the spoken utterance and / or additional spoken utterances).

[0054] At block 376, the system refrains from generating a modification selectable element. In other words, the system may generate modification selectable elements only for the automatically arranged portions of the transcription, and refrain from generating modification selectable elements for portions of the transcription for which the user has specified an arrangement. In various implementations, the transcription may include an automatically arranged portion and a portion that was manually arranged as specified by the user. In these implementations, the automatically arranged portions of the transcription may be associated with corresponding modification selectable elements, but the portions of the transcription that were manually arranged as specified by the user may not be associated with any modification selectable elements.

[0055] Now go to Figure 4 , depicts a flow chart illustrating an example method 400 for providing automatic formatting of text data in a transcription and correction of unintentional automatic formatting in the transcription as a service to a third-party software application. For convenience, the operations of the method 400 are described with reference to a system that performs the operations. The system of the method 400 includes a computing device (e.g., Figure 1 and FIG. 2A to FIG. 2F The client device 110, Figure 5 The method 400 may be performed by one or more processors, memories, and / or other components of a computing device 510, one or more servers, and / or other computing devices. In addition, although the operations of method 400 are shown in a particular order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0056] At box 452, the system receives audio data capturing spoken utterances of a user of the client device, the audio data being generated by one or more microphones of the client device. At box 454, the system processes the audio data using the ASR model to generate text data corresponding to the spoken utterances. The system may return to the operations of box 452 to receive additional audio data capturing additional spoken utterances of the user. In other words, the system may conduct a dictation session with the user by enabling the user to continuously provide spoken utterances. Furthermore, the system may continuously process the audio data capturing the spoken utterances while the system generates text data corresponding to the spoken utterances according to the ASR model. Figure 4 The method 400 proceeds to the operation of block 456 simultaneously. Therefore, it should also be understood that the dictation can be performed in a parallel manner based on the spoken words provided by the user during the dictation session. Figure 4 Multiple iterations of method 400.

[0057] At box 456, the system generates an automatically formatted transcription of the spoken utterance based on the text data. At box 458, the system causes the transcription to be provided to a third-party software application that is executed at least in part at the client device. The third-party software application causes the transcription to be provided for presentation to the user via a display of the client device. In other words, Figure 4 In method 400, the system can be provided as a service to various third-party software applications (e.g., Figure 1 The third-party software application may be a third-party keyboard application. In this example, the third-party software application may not support dictation. Therefore, the system may Figure 4 The method 400 is provided as a service to a third-party keyboard application to enable the third-party software application to support dictation.

[0058] At box 460, the system determines whether to generate a modification selectable element that, when selected, causes the automatically arranged transcription to be modified. The system may determine whether to generate the modification selectable element based on, for example, whether a third-party software application has requested the system to generate the modification selectable element. If the system determines in an iteration of box 460 that the modification selectable element is to be generated, the system may proceed to the operation of box 462. At box 462, the system generates the modification selectable element. At box 464, the system causes the modification selectable element to be provided to the third-party software application. For example, the system may not only provide the transcription as a service to the third-party software application, but the system may also provide modifications to the transcription as a service to the third-party application by providing the modification selectable element to the third-party software application.

[0059] If the system determines in an iteration of block 460 that a modification selectable element is not to be generated, the system may proceed to the operation of block 466. At block 466, the system refrains from generating the modification selectable element. At block 468, the system causes the automatic arrangement data used in automatically arranging the transcription to be provided to the third-party software application. For example, rather than generating the modification selectable element for the third-party software application, the system may simply provide data to the third-party software application to enable the third-party software application to generate the modification selectable element.

[0060] Now go to Figure 5 , depicts a block diagram of an example computing device 510 that may optionally be used to perform one or more aspects of the techniques described herein. In some implementations, one or more of the client device 110, the dictation system 160, and / or other components may include one or more components of the example computing device 510.

[0061] The computing device 510 typically includes at least one processor 514 that communicates with a number of peripheral devices via a bus subsystem 512. These peripheral devices may include a storage subsystem 524 (including, for example, a memory subsystem 525 and a file storage subsystem 526), ​​a user interface output device 520, a user interface input device 522, and a network interface subsystem 516. The input and output devices allow a user to interact with the computing device 510. The network interface subsystem 516 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0062] The user interface input device 522 may include a keyboard, a pointing device (such as a mouse, trackball, touchpad, or graphics tablet), a scanner, a touch screen incorporated into a display, an audio input device (such as a voice recognition system), a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computing device 510 or onto a communication network.

[0063] The user interface output device 520 may include a display subsystem, a printer, a fax machine, or a non-visual display, such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device, such as a liquid crystal display (LCD), a projection device, or some other mechanisms for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Generally speaking, the use of the term "output device" is intended to include devices and methods for outputting information from the computing device 510 to all possible types of users or another machine or computing device.

[0064] The storage subsystem 524 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 524 may include memory devices to perform selected aspects of the methods described herein and to implement Figure 1 The logic of the various components depicted in .

[0065] These software modules are typically executed by the processor 514 alone or in combination with other processors. The memory 525 used in the storage subsystem 524 may include multiple memories, including a main random access memory (RAM) 530 for storing instructions and data during program execution and a read-only memory (ROM) 532 in which fixed instructions are stored. The file storage subsystem 526 can provide persistent storage for program and data files, and can include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations can be stored by the file storage subsystem 526 in the storage subsystem 524, or stored in other machines accessible to the processor 514.

[0066] Bus subsystem 512 provides a mechanism for the various components and subsystems of computing device 510 to communicate with each other as intended. Although bus subsystem 512 is schematically shown as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0067] Computing device 510 can be of different types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Figure 5 The description of computing device 510 depicted in FIG. 5 is intended only as a specific example for the purpose of illustrating some implementations. Many other configurations of computing device 510 are possible, and the configurations are similar to those of FIG. Figure 5 Computing devices are depicted as having more or fewer components than those shown.

[0068] Where certain implementations discussed herein may collect or use personal information about a user (e.g., user data extracted from other electronic communications, information about a user's social network, a user's location, a user's time, a user's biometric information, and a user's activities and demographic information, relationships between users, etc.), the user is provided with one or more opportunities to control whether information is collected, whether personal information is stored, whether personal information is used, and how information about the user is collected, stored, and used. That is, the systems and methods discussed herein collect, store, and / or use user personal information only after receiving explicit authorization from the relevant user to collect, store, and / or use the user's personal information.

[0069] For example, a user is provided with control over whether a program or feature collects user information about that particular user or other users associated with the program or feature. Each user whose personal information is to be collected is presented with one or more options to allow control over the collection of information associated with that user, providing permission or authorization as to whether to collect information and as to which portions of the information are to be collected. For example, one or more such control options may be provided to a user via a communications network. In addition, specific data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. As an example, the identity of a user may be processed so that no personally identifiable information can be determined. As another example, the user's geographic location may be generalized to a larger area so that the user's specific location cannot be determined.

[0070] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing spoken utterances of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterances; generating an automatically formatted transcription of the spoken utterances based on the text data corresponding to the spoken utterances, the automatically formatted transcription including at least automatic punctuation following a given term included in the text data and automatic punctuation following a given term included in the text data causing a transcription to be provided for presentation to a user via a display of a client device; generating a modification selectable element that, when selected by a user, causes the automatically formatted transcription to be modified to remove automatic punctuation and / or automatic capitalization; receiving touch input from a user via a display of the client device, the touch input being directed to a transcription of the spoken utterance; and in response to receiving touch input from a user directed to a transcription of the spoken utterance and a transcription based on the spoken utterance being automatically formatted based on the transcription: causing the modification selectable element to be provided for presentation to a user via a display of the client device.

[0071] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0072] In some implementations, the method further includes: receiving additional touch input from the user via a display of the client device, the additional touch input being directed to modifying the selectable element; and in response to receiving the additional touch input from the user being directed to modifying the selectable element: causing the automatically formatted transcription to be modified to remove automatic punctuation and / or automatic capitalization.

[0073] In some versions of those implementations, causing an automatically formatted transcription to be modified to remove automatic punctuation and / or automatic capitalization may include: causing automatic punctuation to be removed following a given term included in the text data; and causing automatic capitalization to be removed for subsequent terms included in the text data and following the given term.

[0074] In some further versions of those implementations, causing automatic capitalization of subsequent terms included in the text data and following the given term to be removed may include replacing the automatically capitalized first letter of the subsequent terms included in the text data and following the given term with the corresponding lowercase first letter of the subsequent term.

[0075] In additional or alternative versions of those implementations, causing an automatically formatted transcription to be modified to remove automatic punctuation and / or automatic capitalization may include: causing automatic punctuation to be removed following a given term included in the text data; and avoiding causing automatic capitalization to be removed for subsequent terms included in the text data and following the given term.

[0076] In some further versions of those implementations, avoiding having automatic capitalization of subsequent terms included in the text data and following a given term removed may include: determining whether the automatic capitalization of subsequent terms included in the text data and following the given term is inherently capitalized; and in response to determining that the automatic capitalization of subsequent terms included in the text data and following the given term is inherently capitalized: avoiding having automatic capitalization of subsequent terms included in the text data and following the given term removed.

[0077] In still further versions of those implementations, the inherent capitalization may be an entity reference to an entity.

[0078] In additional or alternative versions of those implementations, modifying the selectable element may include a preview of a portion of the modified transcription that is modified to remove automatic punctuation and / or automatic capitalization.

[0079] In some implementations, the spoken utterance may not explicitly include automatic punctuation following a given term included in the transcription, and the spoken utterance may not explicitly include automatic capitalization of subsequent terms included in the transcription and following the given term.

[0080] In some implementations, the touch input directed to the transcription of the spoken utterance may be directed to an automatic punctuation mark following a given term included in the text data. In additional or alternative implementations, the touch input directed to the transcription of the spoken utterance may be directed to the given term. In additional or alternative implementations, the touch input directed to the transcription of the spoken utterance may be directed to a subsequent term included in the text data and following the given term.

[0081] In some implementations, one or more processors may be local to the user's client device.

[0082] In some implementations, the method may further include processing audio data capturing the spoken utterance and / or text data corresponding to the spoken utterance using one or more automatic arrangement machine learning (ML) models to generate automatic arrangement data for automatically arranging the transcription.

[0083] In some versions of those implementations, the one or more automatic orchestration ML models may include an automatic punctuation ML model, and using the one or more automatic orchestration ML models to process audio data capturing the spoken utterance and / or text data corresponding to the spoken utterance to generate automatic orchestration data for automatically arranging the transcription may include: using the automatic punctuation ML model to process the audio data capturing the spoken utterance and / or text data corresponding to the spoken utterance to identify automatic punctuation marks following a given term included in the text data.

[0084] In additional or alternative versions of those implementations, the one or more automatic arrangement ML models may include an automatic capitalization ML model, and using the one or more automatic arrangement ML models to process audio data capturing the spoken utterance and / or text data corresponding to the spoken utterance to generate automatic arrangement data for automatically arranging the transcription may include: using the automatic capitalization ML model to process the audio data capturing the spoken utterance and / or text data corresponding to the spoken utterance to identify automatic capitalization of subsequent terms included in the text data and following a given term.

[0085] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing the spoken utterances of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterances; determining whether the user has specified an arrangement of text data for transcription of the spoken utterances based on the audio data capturing the spoken utterances and / or based on the text data corresponding to the spoken utterances; and in response to determining that the user has not yet specified an arrangement of text data for transcription: generating an automatically arranged transcription of the spoken utterances based on the text data corresponding to the spoken utterances; causing the transcription to be provided for presentation to the user via a display of the client device; generating a modifying selectable element that, when selected by the user, causes the automatically arranged transcription to be modified; receiving a touch input from the user via the display of the client device, the touch input being directed to the transcription of the spoken utterances; and in response to receiving the touch input from the user being directed to the transcription of the spoken utterances and based on the transcription of the spoken utterances being automatically arranged based on the transcription: causing the modifying selectable element to be provided for presentation to the user via the display of the client device.

[0086] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0087] In some implementations, the method may further include, in response to determining that the user has specified an arrangement of the text data for transcription: generating a transcription of the spoken utterance based on the text data corresponding to the spoken utterance and based on the arrangement specified by the user; causing the transcription to be provided for presentation to the user via a display of the client device; and refraining from generating the modifying selectable element. In some versions of those implementations, the arrangement specified by the user may include manual punctuation provided by the user in the spoken utterance and / or manual capitalization provided by the user in the spoken utterance.

[0088] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing spoken utterances of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterances; generating an automatically formatted transcription of the spoken utterances based on the text data corresponding to the spoken utterances; causing the transcription to be provided for presentation to the user via a display of the client device; generating a modifying selectable element that, when selected by the user, causes the automatically formatted transcription to be modified; receiving touch input from the user via the display of the client device, the touch input being directed to the transcription of the spoken utterances; and in response to receiving the touch input from the user being directed to the transcription of the spoken utterances and being automatically formatted based on the transcription: causing the modifying selectable element to be provided for presentation to the user via the display of the client device.

[0089] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0090] In some implementations, the transcription may be automatically formatted to include an automatic segmentation following a given term included in the transcription. In some versions of those implementations, the method may further include: receiving additional touch input from a user via a display of the client device, the additional touch input directed to a modification selectable element; and in response to receiving the additional touch input from the user directed to the modification selectable element: causing the transcription to be modified to remove the automatic segmentation.

[0091] In some implementations, the method may further include processing the audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance using one or more automatic arrangement machine learning (ML) models to generate automatic arrangement data for automatically arranging the transcription. In some versions of those implementations, the one or more automatic arrangement ML models may include an automatic segmentation ML model, and processing the audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance using the one or more automatic arrangement ML models to generate the automatic arrangement data may include: processing the audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance using the automatic segmentation ML model to identify an automatic segmentation following a given term included in the text data.

[0092] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing spoken utterances of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterances; generating an automatically formatted transcription of the spoken utterances based on the text data corresponding to the spoken utterances; causing the transcription to be provided to a third-party software application that is at least partially executed at the client device; generating a modification selectable element that, when selected by the user, causes the automatically formatted transcription to be modified; and causing the modification selectable element to be provided to the third-party software application that is executed at the client device.

[0093] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0094] In some implementations, causing the transcription to be provided to a third-party software application executing at the client device may cause the third-party software application to provide the transcription for presentation to the user via a display of the client device. In some versions of those implementations, causing the modifying selectable element to be provided to the third-party software application executing at the client device may cause the third-party software application to provide the modifying selectable element for presentation to the user via a display of the client device in response to determining that the user has directed a touch input toward the transcription.

[0095] In some implementations, a method implemented by one or more processors is provided, and the method includes: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing spoken utterances of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterances; generating an automatically formatted transcription of the spoken utterances based on the text data corresponding to the spoken utterances; causing the transcription to be provided to a third-party software application that is executed at least in part at the client device; and causing automatically formatted data used in automatically formatting the transcription to be provided to the third-party software application that is executed at the client device.

[0096] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0097] In some implementations, causing automatic arrangement data for use in automatically arranging a transcription to be provided to a third-party software application executing at a client device can cause the third-party software application to generate a modification selectable element that, when selected by a user, causes the automatically arranged transcription to be modified.

[0098] In addition, some implementations include one or more processors (e.g., central processing units (CPUs), graphics processing units (GPUs), and / or tensor processing units (TPUs)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause the performance of any of the above methods. Some implementations also include one or more non-transitory computer-readable storage media storing computer instructions that can be executed by one or more processors to perform any of the above methods. Some implementations also include a computer program product that includes instructions that can be executed by one or more processors to perform any of the above methods.

[0099] It should be understood that all combinations of the aforementioned concepts and additional concepts described in more detail herein are considered part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are considered part of the subject matter disclosed herein.< / pb> < / pb> < / pb> < / per> < / per> < / decap> < / cap> < / per> < / cap> < / per> < / pb> < / per> < / cap> < / per> < / lb> < / com> < / pb> < / per> < / pb> < / per> < / cap> < / cap> < / cap> < / cap> < / cap> < / com> < / per> < / per> < / lb> < / com> < / pb> < / per> < / cap>

Claims

1. A method implemented by one or more processors, the method comprising: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing the spoken utterance of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterance; generating an automatically formatted transcription of the spoken utterance based on the text data corresponding to the spoken utterance, the automatically formatted transcription comprising at least automatic punctuation following a given term included in the text data and automatic capitalization of subsequent terms included in the text data and following the given term; causing the transcription to be provided for presentation to the user via a display of the client device; generating a modification selectable element which, when selected by the user, causes the automatically formatted transcription to be modified to remove the automatic punctuation and / or the automatic capitalization; receiving, via the display of the client device, a touch input from the user, the touch input directed to the transcription of the spoken utterance; as well as In response to receiving the touch input from the user directed to the transcription of the spoken utterance and the transcription based on the spoken utterance being automatically composed and generated based on the transcription: The modifying selectable element is caused to be provided for presentation to the user via the display of the client device.

2. The method of claim 1, further comprising: receiving, via the display of the client device, additional touch input from the user, the additional touch input directed to the modifying selectable element; as well as In response to receiving the additional touch input from the user directed to the modifying selectable element: The automatically formatted transcription is caused to be modified to remove the automatic punctuation and / or the automatic capitalization.

3. The method of claim 2, wherein causing the automatically formatted transcription to be modified to remove the automatic punctuation and / or the automatic capitalization comprises: causing the automatic punctuation mark following the given word included in the text data to be removed; as well as The automatic capitalization of the subsequent term included in the text data and following the given term is removed.

4. The method of claim 3, wherein causing the automatic capitalization of the subsequent terms included in the text data and following the given term to be removed comprises: The automatically capitalized first letter of the subsequent term included in the text data and following the given term is replaced with a corresponding lowercase first letter of the subsequent term.

5. The method of claim 2, wherein causing the automatically formatted transcription to be modified to remove the automatic punctuation and / or the automatic capitalization comprises: causing the automatic punctuation mark following the given word included in the text data to be removed; as well as Avoid having the automatic capitalization of the subsequent terms included in the text data and following the given term removed.

6. The method of claim 5, wherein preventing the automatic capitalization of the subsequent terms included in the text data and following the given term from being removed comprises: determining whether the automatic capitalization of the subsequent term included in the text data and following the given term is inherently capitalized; as well as In response to determining that the automatic capitalization of the subsequent term included in the text data and following the given term is the inherent capitalization: Avoid having the automatic capitalization of the subsequent terms included in the text data and following the given term removed.

7. The method of claim 6, wherein the inherent capitalization is an entity reference to an entity.

8. A method as claimed in any one of claims 2 to 7, wherein the modification selectable element comprises a preview of a portion of a modified transcription, the portion of the modified transcription being modified to remove the automatic punctuation and / or the automatic capitalization.

9. A method as claimed in any preceding claim, wherein the spoken utterance does not explicitly include the automatic punctuation following the given term included in the transcription, and wherein the spoken utterance does not explicitly include the automatic capitalization of the subsequent terms included in the transcription and following the given term.

10. A method as claimed in any preceding claim, wherein the touch input directed to the transcription of the spoken utterance is directed to the automatic punctuation following the given term included in the text data.

11. The method of any one of claims 1 to 9, wherein the touch input directed to the transcription of the spoken utterance is directed to the given term.

12. The method of any one of claims 1 to 9, wherein the touch input directed to the transcription of the spoken utterance is directed to the subsequent term included in the text data and following the given term.

13. A method as claimed in any preceding claim, wherein the one or more processors are local to the client device of the user.

14. The method of any preceding claim, further comprising: The audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance is processed using one or more automatic arrangement machine learning (ML) models to generate automatic arrangement data for automatically arranging the transcription.

15. The method of claim 14, wherein the one or more automatic arrangement ML models include an automatic punctuation ML model, and wherein using the one or more automatic arrangement ML models to process the audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance to generate the automatic arrangement data for automatically arranging the transcription comprises: The audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance are processed using the automatic punctuation ML model to identify the automatic punctuation mark following the given term included in the text data.

16. The method of claim 14, wherein the one or more automatic arrangement ML models include an automatic capitalization ML model, and wherein using the one or more automatic arrangement ML models to process the audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance to generate the automatic arrangement data for automatically arranging the transcription includes: The audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance are processed using the automatic capitalization ML model to identify the automatic capitalization of the subsequent term included in the text data and following the given term.

17. A method implemented by one or more processors, the method comprising: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing the spoken utterance of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterance; determining, based on the audio data capturing the spoken utterance and / or based on the text data corresponding to the spoken utterance, whether the user has specified an arrangement of the text data for transcription of the spoken utterance; as well as In response to determining that the user has not specified the arrangement of the text data for the transcription: generating the automatically formatted transcription of the spoken utterance based on the text data corresponding to the spoken utterance; causing the transcription to be provided for presentation to the user via a display of the client device; generating a modification selectable element which, when selected by the user, causes the automatically formatted transcription to be modified; as well as receiving, via the display of the client device, a touch input from the user, the touch input directed to the transcription of the spoken utterance; as well as In response to receiving the touch input from the user directed to the transcription of the spoken utterance and the transcription based on the spoken utterance being automatically composed and generated based on the transcription: The modifying selectable element is caused to be provided for presentation to the user via the display of the client device.

18. The method of claim 17, further comprising: In response to determining that the user has specified the arrangement of the text data for the transcription: generating the transcription of the spoken utterance based on the text data corresponding to the spoken utterance and based on the arrangement specified by the user; causing the transcription to be provided for presentation to the user via the display of the client device; as well as Avoid generating said modified selectable element.

19. The method of claim 18, wherein the arrangement specified by the user includes manual punctuation provided by the user in the spoken utterance and / or manual capitalization provided by the user in the spoken utterance.

20. A method implemented by one or more processors, the method comprising: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing the spoken utterance of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterance; generating an automatically formatted transcription of the spoken utterance based on the text data corresponding to the spoken utterance; causing the transcription to be provided for presentation to the user via a display of the client device; generating a modification selectable element which, when selected by the user, causes the automatically formatted transcription to be modified; receiving, via the display of the client device, a touch input from the user, the touch input directed to the transcription of the spoken utterance; as well as In response to receiving the touch input from the user directed to the transcription of the spoken utterance and automatically formatted based on the transcription: The modifying selectable element is caused to be provided for presentation to the user via the display of the client device.

21. The method of claim 20, wherein the transcription is automatically formatted to include automatic segmentation following a given term included in the transcription.

22. The method of claim 21, further comprising: receiving, via the display of the client device, additional touch input from the user, the additional touch input directed to the modifying selectable element; as well as In response to receiving the additional touch input from the user directed to the modifying selectable element: The transcription is caused to be modified to remove the automatic segmentation.

23. The method of any one of claims 20 to 22, further comprising: The audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance is processed using one or more automatic arrangement machine learning (ML) models to generate automatic arrangement data for automatically arranging the transcription.

24. The method of claim 23, wherein the one or more automatic orchestration ML models include an automatic segmentation ML model, and wherein using the one or more automatic orchestration ML models to process the audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance to generate the automatic orchestration data includes: The audio data capturing the spoken utterance and / or the text data corresponding to the spoken utterance are processed using the auto-segmentation ML model to identify the auto-segmentation following the given term included in the text data.

25. A method implemented by one or more processors, the method comprising: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing the spoken utterance of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterance; generating an automatically formatted transcription of the spoken utterance based on the text data corresponding to the spoken utterance; causing the transcription to be provided to a third party software application executed at least in part at the client device; generating a modification selectable element which, when selected by the user, causes the automatically formatted transcription to be modified; as well as The modifying selectable element is caused to be provided to the third party software application executing at the client device.

26. The method of claim 25, wherein causing the transcription to be provided to the third party software application executing at the client device causes the third party software application to provide the transcription for presentation to the user via a display of the client device.

27. The method of claim 26, wherein causing the modifying selectable element to be provided to the third-party software application executing at the client device causes the third-party software application to provide the modifying selectable element for presentation to the user via the display of the client device in response to determining that the user has directed touch input toward the transcript.

28. A method implemented by one or more processors, the method comprising: receiving audio data capturing spoken utterances of a user of a client device, the audio data being generated by one or more microphones of the client device; processing the audio data capturing the spoken utterance of the user using an automatic speech recognition (ASR) model to generate text data corresponding to the spoken utterance; generating an automatically formatted transcription of the spoken utterance based on the text data corresponding to the spoken utterance; causing the transcription to be provided to a third party software application executed at least in part at the client device; as well as Automatic formatting data for use in automatically formatting the transcription is caused to be provided to the third party software application executing at the client device.

29. The method of claim 28, wherein causing the automatic arrangement data for use in automatically arranging the transcription to be provided to the third party software application executing at the client device causes the third party software application to generate a modification selectable element that, when selected by the user, causes the automatically arranged transcription to be modified.

30. A client device, comprising: one or more processors; as well as A memory storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations according to any one of claims 1 to 29.

31. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the operations of any one of claims 1 to 29.

Images