Method and system for computer-generated visualization of speech - Patents.com

Abstract

Described herein are methods, systems, and apparatus for computer-generated visualization of speech. An exemplary method for computer-generated visualization of an utterance including at least one segment includes generating a graphical representation of an object corresponding to the utterance segment and displaying the graphical representation of the object on a screen of a computing device. Generating the graphical representation includes representing the duration of each segment by a length of the object, representing the intensity of each segment by a width of the object, and disposing spaces between adjacent objects in the graphical representation.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 068,734, filed August 21, 2020, which is incorporated by reference in its entirety for all purposes.

[0002] Technical Field The present invention relates generally to methods, systems, and apparatus for language learning through speech, and more particularly to methods and systems for computer-generated visualization of speech for language learners.

[0003] background Humans communicate information through vocalized expressions, typically speech. Information communicated during human speech production can be categorized into linguistic, paralinguistic, and non-linguistic information. Linguistic information is typically expressed in written form. Paralinguistic information may accompany linguistic information during speech. Non-linguistic information may be independent of linguistic information communicated during speech.

[0004] For example, in the case of English, linguistic information is associated with phonemic features that can be represented by strings of the Roman alphabet. A phoneme is a perceptually distinct unit of sound in a particular language, such as a consonant and a vowel in English. One or two Roman alphabets can be used to represent each phoneme in English. A string of alphabets constitutes a word that can contain one or more syllables, where each syllable typically contains one vowel and can also contain one or more consonants surrounding the vowel. A vowel can be observed by physical parameters such as, for example, relatively low formant frequencies (e.g., F1 and F2) that mainly govern the listener's perception of a vowel. The formant frequencies are obtained as local maxima on a spectrogram. The formant frequencies are known to represent the acoustic resonances of the human vocal tract. A consonant can be observed as aperiodic signals or as periodic signals in the high frequency region of the spectrogram. Paralinguistic information in English is usually represented by prosodic features. For example, prosodic features include stress, rhythm, and pitch. Stress can be observed as intensity. Rhythm is a temporal parameter including the duration of each phoneme or syllable and the pauses between phonemes or syllables. Pitch is the perceived height of the sound conveying speech. Pitch can be observed as fundamental frequency (e.g., F0) on a spectrogram.

[0005] Traditional visual representations of speech have relied heavily on spectrograms, which show intensity as shades on a plane defined by time and frequency axes, and on curves of extracted acoustic parameters (e.g., F0, F1, and F2) along with a phonetic representation such as the International Phonetic Alphabets (IPA). Each alphabet in the IPA corresponds to a different phoneme, and using the IPA has the advantage that the pronunciation of the phonemes is accurately represented regardless of the textual representation using the English Roman alphabet, whose variations may be represented identically by the IPA, such as "right" and "write."

[0006] However, such conventional visual representations of speech, i.e., spectrogram representation and IPA notation, have not been intuitive or user-friendly to users. A more user-friendly visual representation of speech is desired, so that users can intuitively learn the differences between recordings of reference speech (e.g., provided by native speakers and expert second language teachers) and recordings of their own speech via the visual representation of the speech.

[0007] overview A system and method for a graphical representation including at least one segment is described. According to some embodiments, a method for computer-generated visualization of an utterance including at least one segment includes generating a graphical representation of an object corresponding to the utterance segment, where generating the graphical representation includes at least representing the duration of the segment by the length of the object, representing the strength of the segment by the width of the object, and representing the pitch curve of the segment by the tilt angle of the object relative to a reference frame, and then the graphical representation of the object is displayed on a screen of a computing device. In some embodiments where the pitch curve is associated with a fundamental frequency movement, generating the graphical representation further includes representing the fundamental frequency offset of the segment by the vertical position of the object relative to the reference frame. In some embodiments, the segment is a first segment, and the method includes displaying a first object corresponding to the first segment, and displaying a second object corresponding to a second segment of speech subsequent to the first segment, such that the first object and the second object are separated by a spacing corresponding to an unvoiced period between the first segment and the second segment. In some embodiments, the method includes generating a pictorial representation including a plurality of objects, each of the plurality of objects corresponding to a respective segment of speech, and generating the pictorial representation includes, for each of the plurality of objects, representing a duration of the respective segment by a length of the object and an intensity of the respective segment by a width of the object, and disposing a spacing between adjacent objects in the pictorial representation. In some embodiments, each of the plurality of objects is defined by a border, and a spacing between the borders of two adjacent objects in the pictorial representation is based on a duration of the unvoiced period. In some embodiments, the method further includes displaying the object in a color selected based on the place and / or manner of articulation of the sound corresponding to the segment. In some embodiments, the segment includes at least one phoneme.In some embodiments, the segment includes at least one vowel in at least one phoneme. In some embodiments, the method includes displaying the object in a color selected based on a first phoneme in the segment. In some embodiments, the method includes breaking down the utterance into segments including at least one phoneme and displaying the at least one phoneme as at least one symbol associated with the object. In some embodiments, the method includes generating and displaying on a screen a first visualization of a first utterance spoken by a first speaker, the first visualization including a first set of objects on the screen corresponding to the first utterance; generating a second visualization of a second utterance spoken by a second speaker, the second visualization including a second set of objects corresponding to the second utterance; and displaying the second visualization on the screen such that a first end of the first set of objects and a first end of the second set of objects are substantially vertically aligned on the screen. In some embodiments of the method in which the computing device includes a microphone input, the method includes, subsequent to displaying the first visualization, recording a second utterance via the microphone input and generating and displaying a second visualization in response to the recorded second utterance. In some embodiments, the object has a shape selected from a rectangle, an oval, and an oval. In some embodiments, the tilt angle of the object varies along a length of the object.

[0008] Disclosed herein are embodiments of a non-transitory computer-readable medium having instructions that, when executed by one or more processors of a computing device, cause the computing device to perform a method according to any example herein. The non-transitory computer-readable medium according to any example herein may be part of a computing system, which may optionally include a display. In some embodiments, the non-transitory computer-readable medium may be provided by a memory of a computing device that displays a computer-generated visualization of the speech.

[0009] In some embodiments, instructions executable by a computing device to generate a visualization of an utterance are stored on a non-transitory computer-readable medium, the visualization including objects corresponding to segments of the utterance. In some embodiments, generating the visualization of the utterance includes representing a duration of the segment by a length of the object, representing a strength of the segment by a width of the object, and representing a pitch curve of the segment by a tilt angle of the object relative to a frame of reference. The instructions further cause the computing device to display the visualization on a screen coupled to the computing device. In some embodiments, the object is a two-dimensional object having a regular geometric shape. In some embodiments, the object has a shape selected from an oval, an ellipse, and a rectangle. In some embodiments in which the pitch curve is associated with a movement of a fundamental frequency, generating the visualization further includes representing an offset of the fundamental frequency of the segment by a vertical position of the object relative to the frame of reference. In some embodiments, where the segment is a first segment of an utterance, the instructions further cause the computing device to display a first object corresponding to the first segment and a second object corresponding to a second segment of the utterance subsequent to the first segment, the first object and the second object being separated by an interval corresponding to an unvoiced period between the first segment and the second segment. In some embodiments, the segment includes at least one phoneme. In some embodiments, the segment includes at least one vowel in the at least one phoneme. In some embodiments, the instructions further cause the computing device to display the object in a color selected based on a first phoneme in the segment. In some embodiments, the color is selected based on a place of articulation and / or a manner of articulation of a sound corresponding to the segment. In some embodiments, the instructions further cause the computing device to break down the utterance into at least one segment including at least one phoneme and to represent the at least one phoneme together with the object as a corresponding number symbol in the visualization.In some embodiments, the instructions further cause the computing device to generate and display on the screen a first visualization of a first utterance spoken by a first speaker, the first visualization including a first collection of objects on the screen corresponding to the first utterance, and to generate a second visualization of a second utterance spoken by a second speaker, the second visualization including a second collection of objects corresponding to the second utterance, and to display the second visualization on the screen such that a first end of the first collection of objects and a first end of the second collection of objects are substantially vertically aligned on the screen. In some embodiments in which the computing device is coupled to a microphone input, the instructions further cause the computing device to record the second utterance via the microphone input subsequent to displaying the first visualization, and to generate and display the second visualization in response to the recorded second utterance. In some embodiments in which the computing device is coupled to an audio output, the instructions further cause the computing device to provide an audio playback of the first utterance via the audio output and, following displaying the second visualization, provide user controls configured to enable a user to play the audio playback of the first utterance.

[0010] A system according to some embodiments herein includes a processor, a display, and a memory including instructions that, when executed by the processor, cause the processor to perform any of the operations associated with generating a visualization of an utterance described herein. In some embodiments, the operations include displaying a first object corresponding to a first segment, displaying a second object corresponding to a second segment of the utterance that follows the first segment, and disposing a space between the first object and the second object that corresponds to an unvoiced period between the first segment and the second segment. In some embodiments, the operations further include displaying the object in a color selected based on the place and / or manner of articulation of the sound corresponding to the segment. In some embodiments, the operations further include generating and displaying on a screen a first visualization of a first utterance spoken by a first speaker, the first visualization including a first collection of objects on the screen corresponding to the first utterance, generating a second visualization of a second utterance spoken by a second speaker, the second visualization including a second collection of objects corresponding to the second utterance, and displaying the second visualization on the screen such that a first end of the first collection of objects and a first end of the second collection of objects are substantially vertically aligned on the screen. The inventive subject matter herein is not limited to the embodiments outlined in this Summary section. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a simplified block diagram of an apparatus according to an embodiment of the present disclosure. [Figure 2A] FIG. 2 is a flow diagram of a speech segmentation process according to an embodiment of the present disclosure. [Figure 2B] FIG. 1 is a flow diagram for generating a visual representation of a segment according to an embodiment of the present disclosure. [Figure 2C] FIG. 1 is a timing diagram of a generated visual representation of an utterance according to an embodiment of the present disclosure. [Figure 2D] 1A-1C are diagrams illustrating different variations of visual manifestations or visual representations of speech. [Figure 2E] 1A-1C are diagrams illustrating different variations of visual manifestations or visual representations of speech. [Figure 2F] 1A-1C are diagrams illustrating different variations of visual manifestations or visual representations of speech. [Figure 2G] 1A-1C are diagrams illustrating different variations of visual manifestations or visual representations of speech. [Figure 3A] FIG. 1 is a flow diagram for generating a visual representation of a segment according to an embodiment of the present disclosure. [Figure 3B] FIG. 2 is a schematic diagram illustrating the relationship between colors, phonemes including consonants, and places of articulation associated with the consonants, according to an embodiment of the present disclosure. [Figure 3C] FIG. 1 is a timing diagram of a generated visual representation of an utterance according to an embodiment of the present disclosure. [Figure 3D] 1 is a schematic diagram of a screen including a generated visual representation of an utterance and a facial representation associated with the utterance, according to an embodiment of the present disclosure. [Figure 4A] FIG. 1 is a flow diagram for generating a visual representation of a segment according to an embodiment of the present disclosure. [Figure 4B] FIG. 2 is a timing diagram of a waveform, a spectrogram, and a generated visual representation of speech superimposed on the spectrogram, according to an embodiment of the present disclosure. [Figure 5A] FIG. 2 is a timing diagram of a waveform, a spectrogram, and a generated visual representation of speech superimposed on the spectrogram, according to an embodiment of the present disclosure. [Figure 5B] FIG. 2 is a timing diagram of a waveform, a spectrogram, and a generated visual representation of speech superimposed on the spectrogram, according to an embodiment of the present disclosure. [Figure 6A] FIG. 2 is a timing diagram of a waveform, a spectrogram, and a generated visual representation of speech superimposed on the spectrogram, according to an embodiment of the present disclosure. [Figure 6B] 1 is a schematic diagram of a generated visual representation of an utterance according to an embodiment of the present disclosure. [Figure 6C] 1 is a schematic diagram of a generated visual representation of an utterance according to an embodiment of the present disclosure. [Figure 7A] FIG. 2 is a timing diagram of a waveform, a spectrogram, and a generated visual representation of speech superimposed on the spectrogram, according to an embodiment of the present disclosure. [Figure 7B] 1 is a schematic diagram of a generated visual representation of an utterance according to an embodiment of the present disclosure. [Figure 8] 8A-C are schematic diagrams of generated visual representations of speech according to embodiments of the present disclosure. [Figure 9] FIG. 2 is a schematic diagram of a flow for modifying a visual representation of an utterance according to an embodiment of the present disclosure. [Figure 10] 10A-D are schematic diagrams of an apparatus for providing a language learning system including a generated visual representation of an utterance on its touch screen according to an embodiment of the present disclosure. [Figure 11] 11A-E are schematic diagrams of an apparatus for providing a language learning system including a generated visual representation of an utterance on its touch screen according to an embodiment of the present disclosure. [Figure 12A] 1 is a schematic diagram of an apparatus for providing a communication system including a generated visual representation of a speech utterance on its touch screen according to an embodiment of the present disclosure. [Figure 12B] 1 is a schematic diagram of an apparatus for providing a communication system including a generated visual representation of a speech utterance on its touch screen according to an embodiment of the present disclosure. [Figure 12C] 1 is a schematic diagram of an apparatus for providing a communication system including a generated visual representation of a speech utterance on its touch screen according to an embodiment of the present disclosure. [Figure 12D] 1 is a schematic diagram of an apparatus for providing a communication system including a generated visual representation of a speech utterance on its touch screen according to an embodiment of the present disclosure.

[0012] Detailed Description Various embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The following detailed description refers to the accompanying drawings, which illustrate, by way of example, certain aspects and embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present invention. Other embodiments may be utilized, and algorithmic, structural, and logical changes may be made without departing from the scope of the present invention. The various embodiments disclosed herein are not necessarily mutually exclusive, as some disclosed embodiments may be combined with one or more other disclosed embodiments to form new embodiments.

[0013] According to the present disclosure, devices, systems, and methods are disclosed for providing a computer-generated visualization of an utterance. In some embodiments, an utterance that can be detected (e.g., from a recorded utterance) and processed via now known or later developed speech recognition technology can include multiple segments and thus can be segmented into multiple segments. In some embodiments, one or more individual segments can include at least one phoneme. In some embodiments, a segment can include a syllable. In some embodiments, an utterance can be segmented into multiple segments, some of which correspond to phonemes and others to syllables. In some embodiments, the segmentation used (e.g., phoneme-based, syllable-based, or other) can rely on a confidence or accuracy metric. The utterance can also include unvoiced periods between segments of the utterance. According to some examples, a pictorial representation is generated that visualizes the utterance in a manner that may be more intuitive or user-friendly to a non-expert user and is displayed on a screen of a computing device. The iconographic representation used to visualize the speech may include one or more objects, each of which corresponds to one segment of the speech. In generating the iconographic representation, the duration of each segment of the speech is represented by the length of the object, and the intensity of that segment of the speech is represented by the width of the object. The individual objects representing the individual segments of the speech may be spaced apart from one another in the iconographic representation, with the spacing corresponding to the unvoiced period between the corresponding segments. In embodiments herein, each object has a border, and the size (e.g., length) of the spacing between the borders of two adjacent objects corresponds to the duration of the unvoiced period between the corresponding segments. In some embodiments, the objects may have a shape selected from a rectangle, an oval, an egg, or other regular geometric shape. The regular geometric shape may be a shape having symmetry about one or more axes.In some embodiments, objects do not necessarily have to be represented by regular geometric shapes, as long as they can be clearly defined (e.g., bordered / outlined by a border) and have lengths and widths to represent the duration and intensity of the corresponding segments, respectively.

[0014] In some embodiments, the pictorial representation used to visualize the speech may further include representing the pitch curve of the segment by the tilt or slant of the object relative to a reference frame or the like, which may be displayed but often does not have to be displayed. In the context of this specification, the pitch curve may represent the movement of one or more physical parameters related to the perceived height or pitch of a voice, also referred to as pitch parameters. One example of a pitch curve may be a curve representing the movement of the fundamental frequency, although the examples herein are not limited to only this pitch parameter. In some embodiments, the tilt or slant of the object may vary along the length of the object, thereby capturing or reflecting the shifts in the pitch curve associated with a given segment of speech. In further embodiments, the offset of the pitch parameter (e.g., the offset of the fundamental frequency) may be represented in the visualization by the height of the object relative to the reference frame. In some embodiments, additional information about the speech may be conveyed via the visualization, such as by selecting a color for the object based on the place and / or manner of articulation of one or more sounds corresponding to the segment. For example, different colors may be assigned to different phonemes. In some embodiments, the color of the object may be selected based on the first phoneme in the segment. In some embodiments, commonalities in the places and / or methods of articulation of different phoneme sounds (e.g., using the same articulatory organs to articulate two different phoneme sounds) may be reflected by commonalities in color (e.g., different shades of the same color, and / or multiple colors that may otherwise be grouped together in a color group). A wide variety of other combinations and variations may be used to provide an intuitive and user-friendly visualization of speech.The methods described herein for providing a computer-generated visualization of an utterance may be embodied in a computer-readable medium, for example, in the form of instructions that, when executed by a computing device, cause the computing device to generate and / or display a graphical representation of the utterance in accordance with any example herein.

[0015] FIG. 1 is a simplified block diagram of a device 10 according to an embodiment of the present disclosure. The device 10 can be implemented in part by a smartphone, a portable computing device, a laptop computer, a game console, or a desktop computer. The device 10 may be implemented by any other suitable computing device. In some embodiments, the device 10 includes a processor 11, a memory 12 coupled to the processor 11, and a display screen 13, which may be a touch screen in some examples, also coupled to the processor 11. The device may further include one or more input devices 16, an external communication interface (e.g., a wireless transmitter / receiver (Tx / Rx) 17), and one or more output devices 19 (e.g., the display screen 13 and an audio output section 15). Although the present application refers to the singular "a" or "an" when describing components of the system (e.g., components of device 10 such as processor 11 and memory 12), it will be understood that any of these components (e.g., processor and / or memory) may include one or more individual such components operatively arranged (e.g., in parallel or other suitable arrangement) to provide the functionality of the components described herein. For example, in the case of memory, memory 12 may be implemented by multiple memory devices arranged, for example, in parallel, and capable of storing the same or different types of data with the same or different storage times. In some examples, display screen 13 may be coupled to a video processor (e.g., a graphics processing unit (GPU)) that controls the display operations of display screen 13 (e.g., to control the display of graphics and video data on display screen 13). In some embodiments, display screen 13 may be a touch screen and may provide user interaction data (e.g., received via user input) to processor 11. For example, a touch-sensitive display screen 13 may detect a user's touching actions, such as tapping, swiping, etc., on a particular area on the surface of the touch screen.The touch screen can provide information about detected touch actions to the processor 11. The processor 11 can cause the device 10 to process speech and generate a visual representation of the speech, possibly in response to the touch action. Thus, the touch-sensitive display screen 13 of the device 10 can function as both an input device 16 and an output device 19. In some embodiments, the device 10 can include one or more additional input devices 16 (e.g., input device 18 and audio input 14, which can include one or more buttons, keys, pointing devices, etc.). In some embodiments, the processing of the speech is performed in part by the processor 11. In other embodiments, the speech can be processed by an external processor in communication with the processor 11 via a communication interface (e.g., wireless transmitter / receiver (Tx / Rx) 17). The wireless transmitter / receiver (Tx / Rx) 17 may facilitate communication between the device 10 and the Internet using a mobile network (e.g., 3G, 4G, 5G, LTE, Wi-Fi, etc.) or may facilitate communication between the device 10 and other devices using a peer-to-peer connection.

[0016] As shown, device 10 can include an audio input 14 and an audio output 15. Although the present application refers to "an" audio input and "an" audio output, it will be understood that any of these components (e.g., microphone input, audio output) may include one or more. For example, device 10 can include one or more audio inputs for internal and / or external microphones, one or more audio outputs for internal and / or external speakers and / or phone jacks. In some examples, audio input 14 and audio output 15 can be coupled to one or more audio signal processors that control audio signal processing of the audio input signal from audio input 14 or the audio output signal to audio output 15. Thus, audio input 14 and audio output 15 can be operatively coupled to processor 11 via an audio DSP. Processor 11 can cause device 10 to record audio data converted from the audio input signal or to play audio data by providing an audio output signal.

[0017] FIG. 2A is a flow diagram of a process 200 for visualizing speech according to some embodiments of the present disclosure that may be implemented by device 10 (e.g., at least in part by processor 11). Device 10 may receive speech input in step S20. The speech input may be a word, phrase, or other spoken or uttered by a user. The speech input may be a pre-recorded and stored utterance (e.g., a reference utterance). The speech input may be received by device 10 as an acoustic signal (i.e., a waveform signal (or simply a waveform) representing a speech or utterance). As shown in block S21, a speech engine, which may implement any known or later developed speech recognition technology, may process the speech input (i.e., the acoustic signal) and segment the speech to obtain a textual representation. Additionally or alternatively, the speech engine may output a spectrogram of the speech input. In other examples, the spectrogram may be obtained independently of speech recognition, again using now known or later developed technology. In some embodiments, a spectrogram representation of the speech input may be generated or obtained, although a spectrogram representation of the speech input is not required for operation of the visualization engine herein. In some embodiments, alternatively or additionally, reference text may be provided along with the utterance, independent of any speech recognition performed on the utterance in block S21.

[0018] The speech engine can be fully or partially implemented by the processor 11 of the device 10. In some embodiments, at least a portion of the speech engine can be implemented by a processor located remotely from the device 10 and communicatively coupled to the device 10, e.g., by a processor of a server in wireless communication with the device 10. The speech engine can be implemented as a program (e.g., instructions stored on a computer-readable medium) that can be stored and executed locally on the device 10, stored remotely and executed locally by the device 10, or at least a portion of the program can be stored and executed on a remote computing device (e.g., a server). The device 10 can further implement a speech visualization engine (SVE), which can also be implemented as a program that can be stored locally or remotely (e.g., on a server, on a cloud) and at least a portion of the program can be executed locally by the device 10. For example, the SVE can be executed locally by the processor 11 and, when executed, can perform a visualization process according to any example herein. In some examples, speech segmentation (S22), which may be part of the speech recognition process, may be performed locally (e.g., by processor 11) or remotely (e.g., by a processor in a remote / cloud server). A visualization process for generating a visual representation of the segmented speech input can be performed locally by processor 11. In some examples, components of the SVE can be stored as program code in an external memory storage device (e.g., USB key memory, memory device of a server residing in the cloud) communicatively coupled to device 10. If any portion of process 200 (e.g., the segmentation portion) is performed remotely (e.g., in the cloud), information for generating the visual representation (e.g., segmentation characteristics, pitch information, etc.) can be communicated to the device via an external communication interface of the device (e.g., via wireless transmitter / receiver 17 or a wired connection).

[0019] To visually represent the speech (e.g., received as speech input by processor 11), the speech input is segmented. Segmentation, which involves breaking down the speech input into segments, can be performed by a speech engine that can be executed by processor 11 or another processor of device 10. For example, the speech engine can break down the speech input and segment the speech input into syllabic units (see block S22). This may be referred to as segmentation at the syllabic level. At this stage, segmentation into syllabic units can be performed by splitting the speech input such that each segment corresponds to an assumed syllable in the text representation. However, due to variability in the pronunciation of different users, especially non-native speakers who may have vowel insertions between consonants, a segmented unit that is expected to contain a single syllable when segmented at the syllabic level may actually contain multiple syllables. The segment of speech may be pronounced differently by some users (e.g., by inserting a vowel where one should not be present). Thus, the process 200 may include an accuracy check beginning at step S23. Once segmentation at the syllable level is complete (S22), the process 200 may determine the accuracy of the segmentation at the syllable level, such as by determining whether the phonemes contained in the segmented syllabic unit substantially match the expected phonemes of the syllable. For example, the process 200 may compare the syllabic unit or segment containing the relevant phonemes to a reference sequence of phonemes. The reference sequence of phonemes may be obtained based on a textual representation, using the International Phonetic Alphabet (IPA) as listed in a commonly used dictionary, by manually annotating a recording of a reference utterance by a native speaker, or by performing speech recognition on a recording of a reference utterance by a native speaker. In some embodiments, one or more modified versions of the IPA symbols may be used to more accurately represent the pronunciation of the reference utterance (e.g., to represent phonetic contractions) and / or to provide the user with additional guidance beyond that provided by the IPA symbols.For example, marks or other mechanisms for further annotating the IPA symbol may be used. In some embodiments, modified versions of the IPA symbol may include bolding the symbol, rendering smaller letters versus larger letters, etc. If the phonemes in the syllabic unit or segment are determined to correspond closely to the canonical sequence of phonemes (yes: S23), the process 200 determines that the syllabic segmentation is sufficiently accurate and proceeds to steps associated with generating a pictorial representation (also referred to as a visual representation) of the syllabic segment (at S24). If the syllabic segmentation is less accurate (no: S23), such as because the syllabic segment is determined to correspond less closely to the canonical sequence of phonemes, segmentation at the phoneme level may continue (S25). Here, the assumed syllabic unit or segment from the syllabic level segmentation in step S22 (e.g., a unit with low correspondence to the reference sequence) is reconsidered at the phoneme level, and if it is determined that the syllabic segment assumed to correspond to a single syllable actually contains more than one syllable, such as by identifying multiple vowels in a segment (YES: S26), the segment can be split into two segments, each containing one vowel (S27). After it is ensured that each segment contains one syllable, the device (e.g., processor 11) can generate a visual representation of the speech input based on the syllabic / phoneme segmentation (S24). When displaying the visual representation, the complete visualization (e.g., all objects generated for the speech input) may be displayed at once, or the display of the objects may be realized in the form of animation (e.g., successive objects are displayed sequentially after the preceding object is displayed).

[0020] FIG. 2B is a flow diagram of a process 240 for generating a visual representation or visual expression of an utterance segment according to some embodiments of the present disclosure. The process 240 can be used to at least partially implement step S24 of the process of FIG. 2A. The process 240 may be performed on segments extracted via the process of FIG. 2A or on segments extracted by another different process such as the prior art. The process 240 can be performed by an SVE according to the present disclosure, for example executed locally by the processor 11 of the device 10. The process of FIG. 2B can be used to generate a visual representation of the utterance such that one figurative object is created for each syllabic segment in the speech input (see block S241). Step S241 can include selecting objects from any suitable shaped objects such as regular shaped objects (e.g., oval, rectangular, egg-shaped, or other) for the segment, and setting parameters such as length, width, and optionally tilt angle, vertical position, color, etc. of each of the figurative objects. This step S241 can be performed for each segment of the speech input such that each segment (e.g., each segmented voiced unit such as a syllable or phoneme) is visually represented by one object. Preferably, for aesthetic purposes, the same shaped objects (e.g., all oval or all rectangular) can be used for all segments in a given visualized speech input. However, it is contemplated that different shaped objects may be used for any given visualization (e.g., when visualizing a given phrase) or set of visualizations. In some embodiments, the type of object (e.g., rectangular, oval, etc.) used for the visualizations can be configurable by the user. In other examples, the type of object used for the visualizations can be pre-programmed into the SVE.

[0021] Referring again to FIG. 2B, and also to FIG. 2C, which illustrates an exemplary visualization unit 204, the length (L) of any given object 201 may be set to represent or correspond to the duration of a given segment, whereby the duration of each of the segments of the speech input is obtained (at step S2411). For example, the start and end times, and thus the duration of any syllable / phoneme segments of the speech input may be obtained from a waveform and / or spectrogram corresponding to the speech input. The strength of each of the syllables / phoneme segments may also be obtained (e.g., from the waveform and / or spectrogram, possibly during the speech recognition process), and the width (W) of each pictorial object may be set according to the strength of the respective segment (at S2412). Steps S2411 and S2412 may be performed in any order. With this basic prosodic information captured for each object, a visualization 204 of the speech input can be generated and displayed by displaying a graphical representation of the object on a display screen (S242). In some embodiments, the process can include an additional optional step (S243) for further adjusting the visual representation of the speech input. As will be further described, other aspects of the graphical objects and their relative placement can be optionally adjusted to convey additional prosodic information about the speech input. For example, the objects can be spaced apart based on unvoiced periods between segments (e.g., periods that were not determined to correspond to detectable syllables or phonemes). In some examples, the slant or tilt angle of the objects can be set to reflect the pitch contour of the speech input. In yet another example, the individual graphical objects may not be vertically aligned, yet can be offset (e.g., relative to each other and / or to a reference frame) to convey additional prosodic information, such as the pitch height or offset of the fundamental frequency of a given segment. In yet another example, the color of an object may be selected based on the place and / or manner of articulation of the sound associated with the segment.

[0022] Returning to FIGS. 2B and 2C, a graphical representation (or visualization portion) 204 of the speech input is displayed (at S242) on a screen (e.g., display screen 13 of device 10), the graphical representation 204 including a plurality of graphical objects representing each of the segments of the speech input. In some embodiments, the visualization portion 204 is displayed after all segments of a given speech input have been analyzed and corresponding objects 201 have been created. In other embodiments, the graphical representation (e.g., individual objects 201) may be displayed sequentially while the speech input is being processed to build the visualization portion 204 of a given utterance (e.g., a spoken phrase). That is, one or more graphical objects 201 may be displayed as soon as the relevant segments have been processed and parameters of the objects (e.g., length, width, color, tilt, vertical position, spacing, etc.) have been determined. FIG. 2C illustrates an example of a visual representation 204 of an utterance (also referred to as a visual representation or visualization portion 204) according to the present disclosure. In the example of FIG. 2C, each iconographic object 201 corresponding to each identified segment in the speech input is a two-dimensional object 201 having a regular geometric shape, in this case an oval. The iconographic objects 201 are each defined by a boundary and are shown relative to a frame of reference defined in this example by time and frequency axes on a screen. Although the axes of the frame of reference are shown in FIG. 2C to facilitate understanding of the example, it will be understood that the frame of reference need not be displayed when the visualization unit 204 is provided to the user (e.g., on the display screen 13 of the device 10). The iconographic objects can have any suitable shape. For example, for an intuitive and easy-to-see visualization, the shape of the iconographic object can be selected from a rectangle, an oval, an egg, or any other regular geometric shape. Substantially any geometric shape with at least one line of symmetry (e.g., a teardrop, a trapezoid, or other) can be used. In some embodiments, the longitudinal direction (and therefore the length) of a given object 201 can be substantially in a straight line, as in the present example.However, in other examples, the longitudinal direction may follow a curve, and thus the tilt or inclination of the object may vary along the length of the object. This can be used to represent pitch variations within a single segment. Successive objects in the visualization unit 204 are associated with successive segments of the speech input such that all segments of the utterance are visually represented on the screen. In some embodiments, such as this example, the objects may be spaced apart by a distance corresponding to an unvoiced period of the speech input. For example, in FIG. 2C, multiple iconographic objects are positioned horizontally on the screen by aligning the starting ends of each of the iconographic objects at positions offset along the time axis, where the offset is based on the start time of the respective segment. As discussed above, the objects may be spaced apart by a predetermined distance, which may provide a clear visual representation of the segment and / or may convey additional prosodic information (e.g., the duration of the pause between the voiced periods). In other words, the borders of two adjacent objects may be spaced apart, in some examples, by a distance based on the duration of the unvoiced period between the two segments associated with these two adjacent objects. In the case of Fig. 2C, a visualization example of a speech input of the phrase "What if something goes wrong" spoken by a native speaker is shown, which in the example of Fig. 2C is the IPA string,

number

[0023] 2D-2G, different variations of visual manifestations or representations of speech are shown. Each of the visual representations in FIG. 2D-2G visualizes the same speech input (e.g., the same utterance of the phrase "What if something goes wrong"). As previously mentioned, different aspects of the iconic objects 201 and their relative alignment with respect to each other and / or to a frame of reference (not shown) can be modified to provide visualizations of speech with different levels of richness (e.g., conveying different amounts or types of prosodic information) while still maintaining the intuitive and user-friendly nature of the visualization. In FIG. 2D, the visual representation (or visualization) 204-1 of the speech input not only conveys the duration and intensity of each segment (e.g., each syllable or phoneme unit segmented as described above) via the length (L) and width (W) of each object 201, but also conveys pitch information by varying the tilt or slant of the objects, speech pause information via the spacing between the objects, and phoneme information by appropriately selecting the color of each object. In FIG. 2E, a simplified representation 204-2 of the same speech input is shown, where certain segmental information such as duration and intensity is conveyed via the size of each object, and pause and phoneme information is conveyed via the spacing and color of the objects. The example of FIG. 2E does not include pitch contour information, but in other examples similar to FIG. 2E, the tilt of the objects can be further varied to convey some pitch contour information (e.g., fundamental frequency) without varying the vertical offset of the objects as in FIG. 2D, thereby omitting some other pitch contour information (e.g., offset of the fundamental frequency of the segments). FIG. 2F illustrates another example 204-3 of a visual representation of speech input, similar to the example of FIG. 2C, but utilizing an egg shape that differs from the oval shape used in FIG. 2C.In FIG. 2F, basic segmental information such as duration and intensity is conveyed via the size of each object, and the duration of unvoiced periods (e.g., pauses in a speech utterance) is conveyed via the corresponding spacing between the objects. Pitch contour information may be omitted from the visualization, or at least some pitch contour information may be omitted. As noted above, the tilt of the objects in FIG. 2F may be varied without changing the vertical offset of the objects to convey at least some information about pitch. All objects in a given visualization may be displayed in the same color, and while a grayscale color (e.g., black) is shown here, it will be understood that a monochromatic visualization may utilize any color (e.g., any RGB or CMYK color). In yet another variation, as shown in FIG. 2G, the figurative objects may be displayed in various sizes (e.g., to convey duration and intensity information) and various colors (e.g., to convey phonemic information), but the pitch and pause information may be omitted. As shown in Figure 2G, here the objects are positioned to be substantially adjacent to one another (e.g., the borders of adjacent objects may abut or touch one another even if unvoiced periods are present, regardless of the duration of the unvoiced periods between adjacent segments (e.g., syllabic units). As will be appreciated, other variations combining features of the visualization techniques described herein can be used to provide simplified, user-friendly visualizations of speech that convey at least some prosodic information.

[0024] As mentioned above, each iconographic object of the visual representation of the utterance may be optionally assigned a color, and in some embodiments, the color assignment may be based on the place of articulation and / or manner of articulation of the sound associated with the segment. For example, the color may be based on the particular syllable or phoneme represented by a given segment. In instances where a segment (e.g., a syllabic unit) has multiple phonemes, the color of the object may be selected based on the first phoneme of the segment. In some embodiments, the commonality of the place of articulation and / or manner of articulation may be reflected by the commonality of the colors used for the objects. For example, segments with sounds (e.g., bilabial, labiodental, etc.) that have one common place of articulation may each be assigned a color from the same color family (e.g., different shades or nuances of pink or violet, or different shades of orange, as shown in FIG. 3B).

[0025] FIG. 3A shows a flow diagram of a process 300 for generating a visual representation of a segment according to the present disclosure, including assigning colors to objects 301 of a visual representation of an utterance 304. The process 300 for assigning colors to objects may be included as an additional optional process / step in the process of creating objects for each segment (e.g., in step S241 of process 240). As shown in step S30, the SVE (e.g., processor 11) may assign colors to objects based on the phonemes of the segment associated with the object. If a segment contains multiple phonemes, the object may be assigned a color based on the first phoneme of the associated segment (S32). To this end, the SVE (e.g., processor 11) may identify the first phoneme in each segment (S31). The actual detection of the phonemes may be performed in the segmentation process. Alternatively, phoneme segmentation may be performed for each syllable segment to identify whether the segment has multiple phonemes and / or to identify the first phoneme in the segment. The SVE (e.g., processor 11) may refer to a look-up table when selecting a color to assign to an object. In some embodiments, the look-up table may specify a unique color for each phoneme such that an appropriate color can be assigned to an object once the first phoneme of a phoneme or segment is identified. In this example, the phoneme is used to select the color of the object, but in other examples, different parameters tied to the place of articulation and / or manner of articulation may be used for color selection. For example, instead of assigning a unique color to each phoneme, one and the same color may be assigned to all sounds associated with the same place of articulation (e.g., labial, labiodental, etc.). Thus, in such examples, the look-up table may alternatively or additionally identify one corresponding color for different places of articulation and / or manner of articulation of a sound.

[0026] An example of such a color table is at least partially visually represented in FIG. 3B. The diagram in FIG. 3B illustrates the relationship between colors, phonemes including consonants, and the places (positions) of articulation in the vocal tract associated with the consonants according to an embodiment of the present disclosure. A color gradient can be associated and assigned to the associated consonants, for example, based on the places and manners of articulation in the vocal tract. For example, labial sounds [p][b][m] and [w], which are produced by the lips, can be grouped together and associated with the same color group (e.g., a pink-purple color group), and each of these phonemes can be associated with a different shade or gradient in this color group because each of these phonemes has a different manner of articulation, such as voiceless plosives, voiced plosives, nasals, and approximants, i.e., in this example, the labial sounds can be assigned different gradient color shapes of pink-purple. Similarly, there may be a gradual shift in the colors assigned to corresponding vowels, which may be based on gradual shifts in the position and opening of the speaker's vocal tract that affect the resonances specific to vowels that are typically extracted as relatively low formant frequencies (e.g., F1 and F2). It will be appreciated that the particular colors and associations are provided merely as examples, and that in other embodiments, other, different associations between colors and phonemes / sounds may be used. After the respective objects have been assigned colors (S32), the objects in the visualization portion 304 may be displayed in the appropriate colors to provide a rich visual representation of the speech.

[0027] 3C is a timing diagram of a generated visual representation 304 of an utterance, according to an embodiment of the present disclosure. The visual representation 304 in FIG. 3C is of the same utterance of the phrase "What if something goes wrong" shown in FIG. 2C, and thus the size and placement of the figurative objects 301 are the same as the size and placement of the objects 201 in FIG. 2C, with the difference being that the objects are additionally assigned colors based on the phonemes found within the segment. In this example, the first phonemes of segments #1-6 are:

number

[0028] FIG. 3D is a schematic diagram of a screen 313 including generated visual representations 317-1 and 317-2 of an utterance and facial representations 318-1 and 318-2 associated with the utterance according to further embodiments of the disclosure. In some embodiments, the screen 313 may be the display screen 13 of the device 10. For example, the screen 313 may be a touch screen. The screen 313 may display display windows 314 and 315. The window 314 may display the generated visual representations 317-1 and 317-2 of the utterance according to embodiments of the disclosure. In some embodiments, the generated visual representations 317-1 and 317-2 of the utterance may be timing diagrams such as waveforms of the utterance. In some embodiments, the utterance may be an excerpt of the same phrase (e.g., “take care” in FIG. 3D ) generated by two speakers (e.g., Tutor and User 1 in FIG. 3D ). In some embodiments, the first generated visual representation 317-1 can show a reference utterance provided by a native speaker of the language or a language teacher, and the second generated visual representation 317-2 can show a user's utterance (e.g., a learner's utterance). In some embodiments, the generated visual representations 317-1 and 317-2 can include objects 319-11 and 319-12 and objects 319-21 and 319-22, respectively. A color can be assigned to one or more of these objects, and possibly to each of objects 319-11, 319-12, 319-21, and 319-22. Different phonemes in the utterance (e.g., [t] or [k]) can be associated with different colors (e.g., light blue or gray), and thus different objects in the visual representation can be assigned different colors that correspond to the phonemes of the given utterance. Screen 313 can be configured to provide articulation instruction iconography (e.g., in the form of animation or still iconography) that provides user guidance regarding the place and / or method of articulation of one or more phoneme sounds represented in a given utterance.For example, screen 313 may display icon 316 which, when selected by a user, displays articulatory instruction iconography, for example, in secondary window 315. The content shown in the two display windows 314 and 315 of FIG. 3D (e.g., visual representations 317-1 and 317-2 and facial representations 318-1 and 318-2) may be presented in a single window, or may be provided in other suitable numbers of display windows in other embodiments herein.

[0029] Referring to the specific and non-limiting example of FIG. 3D, the system may display a respective iconographic or facial representation 318-1, 318-2 for each phoneme of the utterance, or for a subset of phonemes (e.g., for the initial phonemes of each syllable), when articulation instructions are activated. Each iconographic or facial representation 318-1 and 318-2 may reflect the place and / or manner of articulation of one or more sounds in the utterance (e.g., the [t] and [k] sounds in the phrase or utterance "take care" in FIG. 3D), optionally together with an associated waveform. In some embodiments, the articulation instructions are adapted to the reference utterance (e.g., invoked by selecting a visualization element of the reference utterance or placed in proximity to the reference utterance) to provide guidance on how the utterance should be properly pronounced to mimic the reference utterance. Articulation instructions (e.g., facial representations 318-1 and 318-2) may be presented in response to selection of icon 316 that is not part of the speech visualization, or may be presented by selection of an element of the speech visualization, such as one or more of objects 319-11 and 319-12. In some embodiments, selection of any of the objects in visual representation 317-1 of the speech may cause only the facial representation associated with that object to be displayed, while selection of icon 316 may cause the facial representations associated with each of the objects in visual representation 317-1 to be displayed, for example, as a sequence of facial representations. The facial representations associated with a given object may be visually associated with the given object, for example, by displaying a color that corresponds to the color of the given object. In some embodiments, individual facial representations 318-1 and 318-2 may be static or may be displayed as animations or videos reflecting the articulation site and / or manner of articulation of a representative sound, such as the manner in which a user should move their lips, tongue, mouth, etc. to properly pronounce a given sound.

[0030] The pitch contour of the speech input can be graphically represented according to the principles of the present disclosure. FIG. 4A is a flow diagram of a process 400 for generating a visual representation 404 of the speech input according to a further embodiment of the present disclosure. The process 400 can be used to partially implement additional steps or processes (e.g., S243) of the process 240 of FIG. 2B. In the example of FIG. 4A, the process 400 includes arranging objects to convey pitch information of the utterance and can thus be used to visually represent the pitch contour of the speech input. In other examples, the relative arrangement of objects created in step S241 of the process 240 to provide a visualization (e.g., 204, 304, etc.) can include various combinations (e.g., components of combinations of steps of the process 400 or additional steps). The process 400 can include detecting pitch parameters (e.g., fundamental frequency or other parameters representative of the perception of pitch by a listener) for each segment (S41). A pitch curve can be developed that represents the behavior of one or more physical parameters (pitch parameters) related to the perceived pitch of speech, such as the conventional fundamental frequency. The pitch parameter is not necessarily limited to the fundamental frequency, and other physical or physiological parameters that may affect the perception of the pitch of speech by a listener may be used as the pitch parameter. Based on the detected pitch parameter and the pitch curve of the speech input, such as the slope of the rise or fall of the pitch detected as an increase or decrease of the pitch parameter, a tilt (or tilt angle) can be assigned to each object (S42). The tilt of the object can be seen as the angle between the longitudinal direction of the object and a reference horizontal axis (e.g., the time axis). In some embodiments, the process 400 can then end, and the objects 401 in the visualization can be displayed with their respective tilts, but aligned substantially vertically.

[0031] Additionally or optionally, the process 400 may include vertically positioning the objects (e.g., by vertically offsetting the objects relative to each other and / or to a reference frame) to convey additional pitch information such as segmental pitch parameter offsets (e.g., segmental fundamental frequency offsets). This may be visually represented by the relative vertical positions of the objects (e.g., relative to each other and / or to a reference frame), as shown in steps S43 and S44. In some examples, the reference frame, and relative to which the vertical offsets can be determined, may be based on a predefined reference line or may be based on the minimum pitch parameter detected for a given speech input. FIG. 4B shows a timing diagram of the waveform 405 and spectrogram 407 of the same speech input as visualized in FIGS. 2C and 3C, but now showing the visualization of additional prosodic information related to pitch. The generated visual representation 404 of the speech input is shown superimposed on the spectrogram 407. As can be observed, the information conveyed by spectrogram 407 may be difficult, if not impossible, to read by a non-expert user, whereas visualization 404, which conveys at least a portion of the prosodic information contained in spectrogram 407, may be more easily understood by a non-expert user. In the superposition of visualization 404 and spectrogram 407 shown here for illustrative purposes only, to illustrate how visualization 404 can convey useful information about the prosody of the speech input to a non-expert user, objects are visually aligned to the actual fundamental frequency curve, indicated by the set of blue points, and to annotations that may typically be extracted or added to the spectrogram by a skilled / expert user.

[0032] 5A and 5B show waveforms 505a and 505b and spectrograms 507a and 507b of a first and second utterance of the same phrase. The first utterance represented by waveform 505a and spectrogram 507a may be a reference utterance (e.g., speech input by, e.g., a native speaker in the context of a language learning application). The second utterance represented by waveform 505b and spectrogram 507b may be a user utterance (e.g., speech input by a learner following a language learning exemplar). FIGS. 5A and 5B also show corresponding visual representations 504a and 504b of the first and second speech inputs, respectively, generated in accordance with the present disclosure and superimposed on the corresponding spectrograms. Also shown are the durations of each of the identified voiced segments (e.g., segment durations 506a and 506b) and the start and / or end times of at least a portion of the segments. Specific timing information is also shown, including: Also shown are segmentation details (e.g., symbolic representations 509a and 509b of segments for the first speech input and the second speech input, respectively). Compared to the first speech input (e.g., a native speaker) in FIG. 5A, the second speech input (e.g., a language learner) in FIG. 5B has:

number

number

[0033] FIG. 6A shows a waveform 605 and a spectrogram 607 plotted as a function of time, with an associated visualization 604-1 of a speech input by a user (e.g., language learner-student A) generated in accordance with the present disclosure superimposed on the spectrogram. The visualization 604-1 in FIG. 6A is from a speech input obtained from the user at a relatively early time (e.g., day 1) during the learning process, and this visualization 604-1 is also shown in FIG. 6B separated (from the spectrogram) as it may be displayed, for example, on a screen of a device (e.g., device 10) implementing the visualization techniques herein. FIG. 6C shows a visual representation 604-2 of a speech input from the same user (e.g., language learner-student A) uttering the same phrase as FIG. 6B, but obtained at a relatively later time (e.g., day 4) during the learning process. As can be seen by visual comparison of visual representation 604-1 of FIG. 6B with visual representation 604-2 of FIG. 6C, the variation in how users enunciate the same phrase can be readily observed from the different iconographic representations of objects, even though the words spoken in both examples are exactly the same. FIG. 7A shows a waveform 705 and spectrogram 707 plotted as a function of time of a speech input by a native speaker enunciating the same phrase as in FIG. 6A-6C, and an associated visual representation 704, which in FIG. 7A is superimposed on its corresponding spectrogram. FIG. 7B shows the same visual representation shown in FIG. 7A in isolation, as it may be displayed, for example, on the screen of a device (e.g., device 10) implementing the visualization techniques herein. As can be seen from the visual comparison of the visual representation 704 of the native speaker's speech input with the visual representations 604-1 and 604-2 of the user's (e.g., language learner-Student A) speech input, the two speakers' utterances have different prosody. Thus, the user can use the visual representation 704 of the reference utterance (e.g., the native speaker's utterance as shown in FIG. 7B) as a reference or comparison to improve his / her foreign language utterance (or to mimic the utterance of a particular dialect or accent of his / her native language).As also shown in FIG. 6B, the total duration of the utterances (e.g., speech input by Student A) on the first day is significantly longer due to vowel ejaculations (e.g., “fu”, “m+u”, “zu”, “u”, and “g+u”) in the user’s initial, poorly practiced utterances compared to visualizations 604-2 and 704 in FIG. 6C and FIG. 7B, and is segmented into more segments as visually seen by the increased number of objects. Also, the colors of some of the objects depicted in FIG. 6A and FIG. 6B are not seen in FIG. 6C and FIG. 7A and FIG. 7B, demonstrating that the user’s utterances (e.g., articulation methods and articulation places corresponding to syllables in a phrase) are changing over time and ideally becoming closer to the target utterances (e.g., native speaker utterances). On the other hand, the visual representation of the utterances by Student A on the fourth day in FIG. 6C appears to be more similar to the visual representation of the native speaker utterance in FIG. 7B, at least in terms of rhythm. The pitch contours, indicated by the vertical position (or height) of the objects, in comparing the visualizations of Figures 6A and 7A demonstrate the difference in the pitch characteristics of the learner's speech input compared to the native speaker's speech input. Although some of the vowel insertions have been eliminated as seen between the utterances of Figures 6B and 6C, it is still evident from the comparison of the visualizations that even at later times (e.g., after some practice), the pronunciation of consonants in some segments, such as "s" instead of "θ", still differs from the native speaker's reference utterance. Using this visualization technique, for example, by displaying the user's speech visualization adjacent to (e.g., above or below) the reference utterance, a user (e.g., a language learner) can easily perceive the difference between his or her own speech and the native speaker's speech, and thus practice and improve towards the target utterance.

[0034] In some embodiments, such as when implementing a language learning application or other speaking practice application according to examples herein, the device may display a visualization of a user (e.g., a learner) and a visualization of a reference utterance (e.g., a native speaker) with the beginnings (first ends) of the visualizations substantially vertically aligned. FIGS. 8A-8C are schematic diagrams of generated visual representations 804-1-804-3 of utterances according to embodiments of the present disclosure. In some embodiments, the visualization of the reference utterance (e.g., generated visual representation 804-1) may be displayed adjacent (e.g., substantially vertically aligned) to the visualization of the user's utterance (e.g., generated visual representation 804-2 or 804-3). In the example of FIGS. 8A-8C, generated visual representations 804-1-804-2 include visualizations of the same utterance, i.e., the same phrase 802 or excerpts thereof (e.g., “No problem, I'll take care of him.” in FIG. 8A ), produced by three different speakers (e.g., Tutor in FIG. 8A , User 1 in FIG. 8B , and User 2 in FIG. 8C ). In some embodiments, generated visual representation 804-1 can include objects that are visual representations of segments in a reference utterance provided by a tutor (e.g., a native speaker or language teacher), and generated visual representations 804-2 and 804-3 can show objects that are visual representations of segments in utterances produced by language learners (e.g., User 1 and User 2). In some cases, the objects can be displayed together with (e.g., superimposed on) a timing diagram and / or waveform of the recorded utterance from which the visualizations were generated. In some embodiments, to facilitate the language practice, a screen of a computing device associated with User 1 may display the generated visual representation of the tutor 804-1 and the generated visual representation of User 1 804-2, e.g., substantially vertically aligned. In other embodiments, the two speech visualizations may be suitably positioned in close proximity on the display in other manners, such as side-by-side.The visual representation 804-2 of user 1 in this example includes, among other things, objects 806-11, 806-12, and 806-13 that may correspond to vowel insertions (e.g., “b+u”, “vu”, and “m+u”) that may not be present in visual representation 804-1 of the reference utterance (e.g., of the tutor). Similarly, a screen of a computing device associated with user 2 may display generated visual representation 804-1 of the tutor and generated visual representation 804-3 of user 2, respectively. The visual representation 804-3 of user 2 may include, among other things, objects 806-21, 806-22, 806-23, and 806-24 that may correspond to vowel insertions (e.g., “b+u”, “m+u”, “vu”, and “m+u”) that may not be present in visual representation 804-1 of the reference utterance. By presenting the user's visualized utterances in close proximity to the visualization of the reference utterances, the system can further help the user (e.g., a learner) identify differences and understand his or her progress toward imitating the "proper" pronunciation of words, phrases, etc.

[0035] In some embodiments, such as when implementing a language learning application or other speaking practice application according to the examples herein, the device can be configured to edit the visualization of the user's utterance. Such editing may be performed in response to user input (e.g., the user specifying edits to be made to the spoken utterance) or may be performed automatically by the device to assist the user in viewing possible improvement trajectories for the user's speaking practice. As discussed herein, the visualization of the user's utterance and the visualization of the reference utterance can be displayed simultaneously (e.g., vertically or side-by-side on the screen), thereby allowing the user to review differences between the visualization of the user's utterance and the visualization of the reference utterance (e.g., native speaker). The user's utterance may then be edited by, for example, changing (e.g., increasing or decreasing) the rate of selected syllables or other segments of the utterance, reducing or amplifying the sound level, shortening or lengthening pauses between voiced and unvoiced segments, deleting or reducing one or more sounds (e.g., removing vowel ejaculations typical of native Japanese speakers), and / or applying other modifications. FIG. 9 is a schematic diagram of a flow for modifying a visual representation of an utterance, in accordance with the present disclosure. FIG. 9 illustrates a visual representation 902-1 of a reference utterance (e.g., of a tutor), which may be displayed simultaneously with one or more visual representations of the user's utterance (e.g., visual representations 902-2 through 902-4), each of which may visually represent different characteristics of the uttered utterance and its segments using objects. Visual representations 902-1 through 902-4 correspond to different utterances of the same utterance, ie, different utterances of the same word or phrase, produced by different speakers (eg, the tutor and the user in FIG. 9).

[0036] In the example of FIG. 9, the generated visual representation 902-1 includes four objects 904-11 to 904-14 that are visual representations of segments of a reference utterance (e.g., a native speaker or a language teacher), labeled in FIG. 9 as tutor. The generated visual representation 902-2 of the same utterance uttered by a user includes eight objects 904-21 to 904-28 that are visual representations of segments of an utterance of the same utterance, but generated by a user (e.g., a language learner). As can be seen, the user's utterance includes additional objects that are not present in the utterance of the reference utterance, and the properties (e.g., length, slant, etc.) and / or spacing of one or more of the objects differ between the two visualizations. For example, the objects 904-21 to 904-24 of the visualization associated with the user correspond to the objects 904-11 to 904-14 of the reference utterance that represent syllables contained in the reference utterance. On the other hand, the objects 904-25 to 904-28 in the user visualization may represent syllables that are not present in the reference utterance and are not part of the reference utterance. For example, the syllables that are not included in the reference utterance may be due to vowel ejaculation or incorrect pronunciation. The visual representation may facilitate editing the user's utterance, such as by the user selecting and specifying changes to be applied to one or more of the objects in the user's utterance, or by the system (e.g., SVE) automatically determining the differences between the user's utterance and the reference utterance, and gradually presenting the edited user utterance as feedback to assist the user in gradually improving his or her utterance. In one example, the user may edit the generated visual representation 902-2 using one or more editing steps. For example, in a first editing step, objects 904-21, 904-23, and 904-24 may be edited to reduce the speed of pronunciation of the corresponding syllables, which visually corresponds to enlarging these objects.Object 904-25 may be scaled down in response to such direct user editing of the object or as a result of the preceding object 904-21 being scaled up. Thus, when the edited visualization 902-3 of the user's utterance is played back, the syllables represented by objects 904-21, 904-23, 904-24, and 904-25 will be pronounced slower and faster, respectively. Further edits may be made, such as deleting or removing one or more objects not present in the reference utterance, such as objects 904-26 and 904-27 between objects 904-23 and 904-24, and the final object 904-28, thereby reducing the total number of sounds / syllables in the edited user's utterance. Visual representations (e.g., 902-3 and 902-4) representing the user's modified utterance of the same utterance may be generated for display. The editing process may be performed in a single step (e.g., from the "user original" utterance to the "user edited after second time" utterance) or in multiple steps as shown in the illustrated example, providing guidance for targeted gradual improvement as the user continues to practice. In the example of FIG. 9, a second editing step is shown, in which object 904-35 from the first edited user utterance is removed, and the speed of objects 904-33 and / or 904-34 may be further adjusted (e.g., increased) to arrive at the utterance shown by visual representation 902-4, which contains the same number of objects (904-41 to 904-44) as in the reference utterance. Thus, the final edited utterance, containing objects 904-31 to 904-34 corresponding to objects 904-11 to 904-14 contained in the reference utterance, will contain the same number of syllables that are pronounced substantially similarly to those captured in the reference utterance, even if by another different user.

[0037] 9 illustrates modifications including changing the speed, removing / reducing syllables, changing the timing of the start or end of syllables, but the modifications provided by the system according to the present disclosure may not be limited to those specifically illustrated herein. For example, the device may allow for a variety of other modifications or any suitable combination thereof, such as reducing or amplifying the level of each sound, shortening or lengthening the pauses between sounds, etc.

[0038] An embodiment of the present invention can be implemented by a device (e.g., a computing device) that provides a language learning system or a language learning application. An exemplary embodiment is further described with reference to FIGS. 10A-10D, which show screen captures of a display screen of a computing device configured to generate and / or provide a visual representation of an utterance according to the present disclosure. The computing device may be a portable computing device (such as a tablet or smartphone) and may include a touch screen. A visual representation of an utterance according to any example herein can be displayed on a touch screen of the computing device. For example, screen shots of the user interface shown in FIGS. 10A-10D can be displayed on a touch screen of the device 10 of FIG. 1. In other embodiments, the visualization portion may be provided on a non-touch sensitive display screen and receive user input via an input device different from the touch screen. The device can execute a program of the language learning system, one component of which may be to generate a visual representation of an utterance. Different types of utterances can be visualized as part of a language learning program. For example, as shown in FIG. 10A, a processor of a device (e.g., a smartphone) can use the visualization process described herein, which can be embodied in the form of computer-readable instructions (e.g., stored in memory 12 as an application ("app")), to generate a simplified visualization 1004a of the reference utterance, which can also be stored in memory 12. In screen shot 1002-1 shown in FIG. 10A, the device displays a visualization of the reference utterance, e.g., an utterance provided by a native speaker, on a touch screen. An audio representation (e.g., an audio playback) of the reference utterance can also be optionally provided to the user in addition to the simplified visualization 1004a before the visualization 1004a is displayed, together with the visualization 1004a, or after the visualization 1004a is displayed.The audio playback can be provided in response to a user command (e.g., in response to tapping a user control or tapping a visualization of the reference utterance on a touch screen). The audio representation of the reference utterance can also be pre-stored in memory 12 as an audio file. The audio representation (e.g., playback) can be provided to the user from audio output 15, which can be coupled to an internal speaker of the computing device or an external speaker (e.g., a headset connected wired or wirelessly to the computing device). In some embodiments, the playback of the reference utterance can be performed automatically, such as after a predetermined period following or preceding the display of the simplified visualization, or possibly simultaneously with the simplified visualization. In some embodiments, the initial playback of the reference utterance can be performed automatically. In some embodiments, the user control that allows the user to command the audio playback can be the visualization of the reference utterance 1004a, or a separate user control configured to play the reference utterance can be provided. The app can be configured to allow the user 1001 (e.g., a language learner) to tap the visualization of the reference utterance as many times as desired before moving to the next step, and the device can play the reference utterance as many times as commanded by the user, for example. In some embodiments, the text string 1006 of the reference utterance can also be displayed. As described, the text string 1006 may not have any prosodic information about the spoken utterance, whereas the visualization 1004a can convey prosodic information to aid the user in the language learning experience. In some embodiments, displaying the visualization 1004a can include displaying an animation of the objects in the visualization, which can accompany the playback of the utterance (the utterance spoken by the learner and / or the playback of the reference utterance) in real time.For example, as each segment (e.g., syllable) of the speech input is played, the corresponding object in the visualization may be animated (e.g., appearing anew, highlighting, vibrating if already displayed, blinking, changing size, moving along a trajectory, or otherwise animated) in substantial synchronization with the segment being played. As one specific but non-limiting example, the animation may include enlarging, brightening, or highlighting an object corresponding to a segment (e.g., syllable) that is of greater intensity compared to a preceding segment (e.g., a stressed syllable). In another specific but non-limiting example, an object in the visualization may be moved along a trajectory that corresponds to a drop or rise in the pitch parameter of the associated segment, such as due to an accent or at the end of a phrase (e.g., at the end of a voiced question). Any of the animation examples herein may be used in combination to provide a richer visualization that may be considered a truer representation of the prosody in speech in real time. Real-time animation of prosodic expressions as described herein can provide an improved learning tool that enhances the user experience as learners practice speaking and listening to speech in a new language (or a particular dialect of a given language).

[0039] The device may further display a user control (e.g., a recording icon 1008) configured to enable a user (e.g., a language learner) to record his / her speech on the device. As shown in screenshot 1002-2 of FIG. 10B, the user (e.g., a language learner) may select this user control (e.g., tap an icon on a touch screen), in response to which the device enters a recording mode and activates a recording function of the device to record the user's speech using a microphone (e.g., embedded in or communicatively coupled to the device). For example, in device 10, processor 11 may activate microphone input 14, such that an internal or external microphone coupled to microphone input 14 detects the sound pressure of the voice generated as speech by the language learner, thereby performing a speech recording. The speech recording may be stored temporarily (e.g., for the duration of a language training session or a portion thereof) or permanently (e.g., until explicitly deleted by the user) in a memory of the device, such as memory 12 of FIG. 1. In one embodiment, the device can then perform the segmentation process of FIG. 2A to process the recorded speech of the user 1001 (e.g., a language learner). In another embodiment, the device can transmit the user's recorded speech to a remote server. The remote server can perform the segmentation process of FIG. 2A on the language learner's recorded speech and transmit the recorded speech segmentation results back to the device. After the user's recorded speech is segmented, the device can perform a process (e.g., the process of FIG. 2B) to generate a visual representation 1004b of the recorded speech, such as by creating a pictorial representation including objects 1003-1, 1003-2, . . . 1003-n that represent segments of the user's recorded speech.As can be seen from the screen shot 1002-3 of FIG. 10C, there are differences between the visualization portion 1004a of the reference utterance and the visualization portion 1004b of the user's recorded utterance, both of which are generated using the same visualization process, but this difference may be mainly due to the difference in prosodic information of the two different utterances of the utterance (one reference, the other user) rather than the content (e.g., text string) of the uttered utterance. In this way, the simplified visualization portion can allow the user to easily perceive the difference between the native speech and the user's (e.g., learner's) own speech, aiding the user's speech learning process. As further shown in FIG. 10C, the device can display additional user controls (e.g., a recording icon) configured to allow the user to save the visualization portion 1004b of this example (e.g., visualization portions of objects 1003-1, 1003-2, etc.) on the touch screen, or any subsequent utterances of the user as further shown in FIG. 10D. In response to a user command (e.g., tapping the record icon 1010), or in some embodiments automatically once the visualization 1004b is generated, the device can store the visualization 1004b in memory 12 persistently (e.g., until explicitly deleted by the user). The visualization 1004b of the user's utterance can be time-stamped and / or tagged to allow classification, searching, report generation, and other subsequent processing of the saved visualizations of the user's utterance. Tagging and storing these visualizations allows for observing the progress of a language learner, such as by displaying saved visualizations acquired over time together. Although described here in the context of language learning, for example, when a non-native speaker wishes to learn a foreign language, embodiments of the invention, such as the embodiment described with reference to FIGS. 10A-10D, can be used for other purposes, such as for practicing narration for acting, for learning different accents or dialects of the same language, or for any other type of speech practice or training.Another use of the speech visualization tools described herein may be self-development exercises through the pronunciation of phrases. For example, the visualization techniques herein can be utilized to build habit formation exercises or tools, where word phrasing can be used as part of the habit formation process.

[0040] In addition to language learning apps, other use cases for the visualization techniques described herein are also possible. For example, a communication app can be built around the visualization techniques described herein. In some embodiments, the visualizations generated by the processes described herein can be user-generated content that can be shared with others. In one such example, a messaging application, such as a smartphone text or video messaging app, can be integrated with a visualization feature, where a speech visualization generated according to any example herein is provided in place of or in combination with any other (e.g., text, image, video) messages shared via the messaging app. This allows, particularly in the case of text messaging, to convey information that cannot be conveyed by text alone (e.g., prosodic information), such as emotional nuances, details, etc. of the spoken message.

[0041] Furthermore, text-only communication (e.g., text messaging) may sometimes be too direct, matter-of-fact, or straightforward and may not promote effective communication. The visualization techniques described herein may be used to infuse such direct and matter-of-fact communication with emotional nuance, thereby providing more effective communication. This may be applied not only to text messaging, but also to the fields of (remote) education, coaching, mentoring, counseling, therapy, and caring. In the context of education, the visualization techniques herein may convey measurable data about a speaker's speech performance, which may be tracked over time, and data associated with visualizations created according to the techniques herein may also be used to track practice and progress. Furthermore, measurable data may be collected over time, and the collected data may be used for a variety of different purposes. In particular, a learner's practice data may be useful to the learner himself, educators or staff supporting the learner, or other users associated with the learner. For example, the system may quantitatively analyze the quality of speech by counting the number of objects in the visualizations created from the learner's speech. Each object representing a segment (e.g., syllable, phoneme, etc.) can be considered as a unit of physical muscle exercise. In a language (e.g., English) learning exercise, the learner can achieve a certain number (e.g., 1 million) of segment productions as represented by objects in the visualization, and the number is counted. In one specific but non-limiting example of a language (e.g., English) learning course implementing the visualization techniques described herein, a user (e.g., a language learner) may, for example, practice listening and speaking daily (or at a different frequency, e.g., 3 times a week, 5 times a week, etc.). A given such (e.g., daily) practice session may take a certain period of time (e.g., 15-30 minutes depending on the user), and thus the user may spend about 15-30 minutes (or different durations) per day practicing the language.During a practice session, the user may be asked to practice a certain number of phrases, e.g., 20-25 phrases, each having a certain number of syllables, e.g., 8-9 syllables per phrase. Continuing with this example, if the user repeats this number of phrases a certain number of times, e.g., 14 times, in a given practice session, the user will have generated over 3000 segmental utterances, which equates to over 1 million utterances if the user practices daily; although such over 1 million utterances may seem like a significant challenge on a macro scale (e.g., in a year), breaking it down into per practice session or per utterance unit may seem more accessible to a user beginning to learn a language and thus may be useful for motivating the user in language practice. Also, communicating to the user the total number of segments generated at a macro level (e.g., over a year) can be motivating as it shows the user how daily, step-by-step practice can accumulate over time to achieve greater vocal / muscular practice results. Thus, measurable data that can be obtained from the visualization, such as the count of objects in the visualization, and that would not have been available if the user had simply vocalized / practiced without any visual feedback, can be useful for analyzing both qualitative and quantitative aspects of vocal practice. Furthermore, the objects in the visualization can be useful for other purposes, such as user behavior analysis. Different techniques in the field of data science techniques can be applied individually and collectively to the data collected over time (e.g., in the different repeated vocalizations and the associated visualizations) to extract additional qualitative and / or quantitative information.

[0042] In various other applications, a person's speech can be further characterized based on the prosodic information contained and conveyed via the current visualization methods, and this information can be used by other devices, systems, or processes, for example, to create an avatar or other representative of the user, or for use by AI speakers (such as Google Home, Alexa, Siri devices, etc.), which can utilize the prosodic information of a given user to mimic or better understand the user's communication. Also, while the visualization techniques are described herein as generating and displaying figurative objects (e.g., oval, rectangular, or other differently shaped objects) on a display, in other examples, discrete light emitting elements (or discrete groups of light emitting elements) of a suitable electronic device can be sequentially illuminated instead of a visualization portion containing discrete figurative objects. In some examples, empathetic computing devices such as those described in U.S. Patent No. 9,218,055 (Sakaguchi et al.), U.S. Patent No. 9,946,351 (Sakaguchi et al.), and U.S. Patent No. 10,222,875 (Sakaguchi et al.) may be used to render visual representations of speech as described herein. The aforementioned patents are incorporated by reference in their entireties for all purposes.

[0043] 11A-11E are screen captures of a device providing a speech visualization portion in combination with a textual representation of an utterance according to further embodiments of the present disclosure. In some embodiments, a user interface such as that shown in the screen captures of FIG. 11A-11E may be generated by and provided on a display of a portable computing device (e.g., a smartphone). Thus, in some examples, a device according to the present disclosure may be a smartphone that implements the device 10 of FIG. 1 and has a touch screen that implements the display screen 13 of the device of FIG. 1. The device (e.g., the smartphone) may be configured to execute a program (e.g., a text messaging app) that provides a text messaging service to a user. The text messaging app may be extended with a speech visualization portion according to the present disclosure. In some embodiments, the visualization may be performed on a recorded speech in real time (e.g., while a user is using the text messaging app), and the recorded speech may be converted to text and sent via the text messaging app together with the visualization portion 1104, or the visualization portion 1104 may be sent in place of the textual representation (e.g., the user's text message). In other embodiments, the device may use a model that models the user's speech utterances, thereby visualizing text messages typed on the device and sharing the visualizations with others as user-generated content. An enhanced text messaging application may be implemented by an application ("app") equipped with an SVE (or components thereof) that may be retrieved from the cloud and / or optionally stored in memory 12 of device 10.

[0044] In FIG. 11A, a device (e.g., a smartphone) is configured to display a message interface screen 1102 when, for example, an enhanced text messaging app is running on the device. The message interface screen 1102 can include standard graphical user interface (GUI) control elements (also referred to as soft controls), such as one or more soft controls 1103 (e.g., a keyboard or a record button for recording a voice message, which is then converted to text by the device) that allow a user to compose a text message. The message interface screen 1102 can display a message window 1105 that displays a draft of the message before it is sent to a recipient. The message interface screen 1102 can include the soft control 1103 that represents a keyboard including keys for typing a message, and can additionally optionally include one or more soft controls (e.g., icons 1107) for accessing other applications (apps) or data associated therewith. In some examples, the message interface screen can display one or more icons 1107 configured to allow a user to attach different user-generated content, such as images, videos, music, personal biometric data, etc., and / or activate an app or its functionality associated with a particular icon. In an enhanced text messaging app, the message interface screen 1102 may additionally include a speech visualization app (SVA) icon 1107-1 that can analyze and generate visual representations of speech in accordance with examples herein.When the speech visualization app icon 1107-1 as shown in FIG. 11A is selected (e.g., tapped), the speech visualization app is activated and provides its own SVA interface window 1109 as shown in FIG. 11B inside the text messaging app to allow the user to generate speech visualization 1104 according to this example. As part of the SVA interface window 1109, the device (e.g., smartphone) can display an icon 1109-1 that allows the user to record their own speech on the device (e.g., smartphone) and / or generate visualizations of previously recorded speech or visualizations of text messages generated or received by the user. In this example, when the user taps icon 1109-1 on the touch screen as shown in FIG. 11C, the device can enter a recording mode and activate a recording function using the device's microphone to record the user's speech. For example, with reference to device 10, processor 11 can activate audio input 14, which causes an internal or external microphone coupled to audio input 14 to detect sound waves generated by the user's speech and record and provide the detected sound waves as speech input (i.e., speech waveform or speech signal) to processor 11. The recording of the detected speech can be temporarily or permanently stored in a memory communicatively coupled to device 10, such as local memory 12 of the device of FIG. 1. In some embodiments, a user can record their own speech outside of the speech visualization application (SVA), such as via a text messaging app's functionality for recording and converting speech. In such a case, once SVA is activated, the user can tap another icon to search for previously recorded speech and generate a visualization 1104 of the previously recorded speech via SVA. The device (e.g., a smartphone) can execute one or more processes for generating a visual representation of the speech, such as by creating an object for the speech segment as described above according to any example herein.As shown in Fig. 11C, the device displays the pictorial representation of the object on the touch screen together with a message confirmation icon that prompts the user to confirm the pictorial representation of this object as a message draft. Thus, if the user is satisfied with the visualization portion displayed in the SVA interface window 1109, the user can tap an icon (e.g., icon 1109-2) to transfer the user-generated content (e.g., visualization portion 1104) to a text messaging app (e.g., message window 1105 as shown in Fig. 11D) so that the message, now in the form of visualization portion 1104, can be sent to a recipient via a soft control (e.g., send icon 1103-s) of the text messaging app, as shown in interface screen 1102e of Fig. 11E. The transmission to the recipient can be performed via a wireless transmission network, for example via wireless transmitter / receiver 17 of device 10 of Fig. 1. As further shown in interface screen 1102e of FIG. 11E, the recipient can interact with (e.g., like, reply, etc.) the message 1111 received, here in the form of a visualization 1104, in the same manner as with a traditional text message received in the form of text.

[0045] 12A-12D are screen captures of an apparatus 1200 providing a communication system including a generated visual representation of an utterance on its touch screen according to an embodiment of the present disclosure. In some embodiments, a user interface such as that shown in the screen captures of FIGS. 12A-12D may be generated by and provided on a display of a portable computing device (e.g., a smartphone). Thus, in some examples, the apparatus 1200 according to the present disclosure may be a smartphone that implements the apparatus 10 of FIG. 1 and has a touch screen that implements the display screen 13 of the apparatus of FIG. 1. The apparatus 1200 (e.g., the smartphone) may be configured to execute a program (e.g., a messaging app) that provides visual and / or text messaging services to a user. According to the present disclosure, the messaging app may be configured to enable a user to share (e.g., send and receive) speech visualizations generated according to any example herein (e.g., by an SVE) and / or content that incorporates or is based at least in part on the speech visualizations. In some embodiments, the messaging app interacts with the SVE (or components thereof) residing in the cloud or stored locally (e.g., in memory 12 of device 10) to retrieve the speech visualization and generate associated content incorporating or based in part on the speech visualization. In some embodiments, the visualization can be performed on a recorded speech in real time (e.g., as the user uses the messaging app), and the recorded speech can be optionally displayed to the user along with its associated content (e.g., icons 1207A, 1208B, or 1208D) and / or transmitted to a receiving user.

[0046] 12A-12D, a device (e.g., a smartphone) 1200 is configured to display a message interface screen 1202, for example, when a messaging app is running on the device 1200. FIGS. 12A-12D show examples of different graphical user interface elements of the messaging interface screen 1202 when a user is interacting with the messaging app to send or receive content. In FIG. 12A, the messaging interface screen 1202 displays a message window 1206A that includes an icon 1207A received from a sender. The icon 1207A can include one or more different types of content elements, such as a text element, a pictorial element, a speech visualization element, or any combination thereof. The icon 1207A of FIG. 12A includes a text message 1208A, in this example the text string "Sorry", and a speech visualization portion 1209A corresponding to the sender's speech, which can be recorded, for example, by the sender saying the word "Sorry" on his / her device, and then the content (icon 1207A) is generated and sent to the receiving user. The icon 1207A in this example further includes an iconography 1210A selected by the sender's device based on the recorded and visualized spoken message. The messaging app can communicate with a memory (e.g., local memory 12 or a cloud memory device) that stores multiple iconography, each of which is associated (e.g., via a lookup table) with a different message, for example, a common message such as "Sorry", "No problem", "No worries", "Got it", "Thanks", "Talk soon". In some examples, the same or similar icon (e.g., a graphic including a thumbs up) may be associated with multiple different text strings (e.g., "Got it" or "No problem") and thus the icon may be selected to be incorporated into content related to any of those multiple different text messages.The iconography (e.g., 1208A) of the content (e.g., icon 1207A) can visually convey information (e.g., emotion) typically associated with a particular text message (e.g., 1208A), and thus messaging apps can enrich the user experience by conveying content rather than just through text messages. In some examples, icon 1207A can additionally convey information about the pronunciation of the user of text message 1208A (e.g., pitch, speed at which the message was spoken, etc.), thereby conveying additional information to the sender about the content creator's state of mind (e.g., the content creator's emotion). In this way, messaging services can be improved, for example, by conveying information about the user's speech that could not be captured or utilized by traditional messaging apps.

[0047] As the user interacts with the messaging app, the message interface screen 1202 is updated to display additional GUI elements created through interaction with the app. For example, as shown in FIG. 12B, a second message window 1206B including an icon 1207B is displayed in the message interface screen 1202. The icon 1207B in this example represents content generated by the user of the device 1200. In some examples, the message interface screen 1202 can include different user controls (e.g., any one or more of the icons 1107 of FIG. 11A) to allow the user to attach different other user-generated content and / or interact with other applications, such as activating other apps or features thereof that reside on or are communicatively coupled to the user's device 1200. For example, referring now also to FIG. 12C, one of the icons 1107 can allow the user to activate a voice recording feature of the device 1200.

[0048] As further shown in FIG. 12C, the message interface screen 1202 may display an icon that allows the user to visualize their recorded speech, for example in response to activation of a voice recording feature, and may optionally display this icon in a separate message window 1206C (e.g., automatically created when the voice recording feature is activated). The messaging app may display an icon 1211, which may be displayed within the message window 1206C or in another suitable location on the message interface screen 1202, that, when selected by the user, is configured to generate a visual representation 1209C of the recorded speech (e.g., using a speech visualization engine (SVE)) in accordance with examples herein. In some embodiments, the speech visualization feature may also be automatically activated by activating a recording feature within the messaging app in response to selection of a single icon, such as icon 1211. In yet another example, an icon (e.g., icon 1107-1) may be selected to activate a speech visualization feature within the messaging app, which then allows the user to record and generate a visualization of the speech that the user recorded. Regardless of the mechanism by which the recording mode is activated, the device 1200 may enter a recording mode (e.g., in response to the user tapping icon 1211) to activate the recording feature using the microphone 1201 of the device 1200 to record the user's speech. For example, with reference to the device 10, the processor 11 may activate the audio input unit 14, which causes an internal or external microphone coupled to the audio input unit 14 to detect sound waves generated by the user's speech and record and provide the detected sound waves as speech input (i.e., speech waveform or speech signal) to the processor 11. The recording of the detected speech may be temporarily or permanently stored in a memory communicatively coupled to the device 10, such as the local memory 12 of the device of FIG. 1.In some embodiments, the user may record their speech outside of the messaging app, such as via other standard voice recording features of device 1200. In such a case, upon selection of icon 1211 by the user, the messaging app may present the user with a GUI for selecting or searching for a previously recorded speech, after which the messaging app subsequently generates a visualization 1209C of this previously recorded speech. In the screen capture of FIG. 12C, the messaging app has generated visualization 1209C (e.g., multiple objects 1212-1-1212-3 that may be colored and positioned to convey different characteristics of the speech, such as amplitude, pitch, etc.) according to examples herein. In some embodiments, the speech visualization 1209C may be displayed within messaging interface 1202 (e.g., temporarily before a corresponding icon is created).

[0049] Following the visualization of the speech, the messaging app can generate content (e.g., icon 1207D) related to the visualized speech, for example as shown in FIG. 12D. The content (e.g., icon 1207D) can be displayed in the message interface screen 1202, for example in yet another message window 1206D, or within the same window 1206C as the visualized speech, if the visualized speech is displayed. In some embodiments, the message window 1206D can be a confirmation window for displaying the user-generated content (e.g., icon 1207D) before the content is sent to another user. As with other icons (e.g., icons 1207A and 1207B), the icon 1207D can also include a text message 1208D, a graphic image 1210D, and / or a visualization portion 1209D of the user's speech. In this example, the icon 1207D incorporates (or includes) the visualization portion 1209D within the user-generated content to be shared. The visualization portion 1209D may be appropriately positioned in relation to the figure 1210D, which may be an illustration of a person, such as in proximity to the mouth of the person depicted in the figure. Once the user is satisfied with the user-generated content, the user may tap an icon 1213 configured to send a message to another user, and the device 1200 may respond by sending the user-generated content (e.g., icon 1207D) to the intended recipient, and may then display a copy of the content-enhanced message sent to the recipient in a message window of the messaging app. In the example of FIG. 12D, the user-generated content may be a reply to a message from the sender, and thus the user-generated content may be provided to the sender of the message 1207A. If the user is not satisfied with the user-generated content (e.g., icon 1207D), the user may re-record the speech, which may create another sequence of different visualization portions and thus the icon 1207D including another different visualization portion 1209D.

[0050] The present invention is not limited to the specific embodiments and examples described above. It is contemplated that the present invention may be embodied in different combinations other than the specific combinations described. It is also contemplated that different combinations or combinations of the specific features and aspects of the embodiments may be made and still fall within the scope of the present invention. It should be understood that different features and aspects of the disclosed embodiments may be combined with or substituted for each other to form various modes of the disclosed invention. Therefore, it is intended that at least some of the scope of the invention disclosed herein should not be limited by the specific disclosed embodiments described above.

Claims

1. 1. A method for computer-generated visualization of an utterance comprising at least one segment, the method comprising: generating a pictorial representation of an object corresponding to the speech segment, wherein generating the pictorial representation comprises: expressing the duration of the segment in terms of the length of the object; expressing the strength of the segmentation by a width of the object; expressing the segmental pitch curve in terms of the inclination angle of the object relative to a reference frame; and Displaying a graphical representation of the object on a screen of a computing device. A method comprising:

2. the pitch contour is associated with a fundamental frequency movement; generating the pictorial representation further comprises expressing the offset of the fundamental frequency of the segment by a vertical position of the object relative to the reference frame; The method of claim 1.

3. the segment is a first segment, The method comprises: displaying a first object corresponding to the first segment; displaying a second object corresponding to a second segment of the utterance that follows the first segment such that the first object and the second object are separated by an interval corresponding to an unvoiced period between the first segment and the second segment; The method of claim 1 , comprising:

4. 1. A method for computer-generated visualization of an utterance comprising at least one segment, the method comprising: generating a pictorial representation including a plurality of objects, each of the plurality of objects corresponding to a respective segment of the utterance, the segments of the utterance including a consonant, a vowel, or a combination thereof, the pictorial representation of each object being based on a pronunciation of the respective segment of the utterance, generating the pictorial representation comprising: expressing the duration of each of the segments by a length of the object, and expressing the strength of each of the segments by a width of the object; Providing spacing between adjacent objects in said iconographic representation; and displaying said graphical representation on a screen of a computing device; generating and presenting on the display a first visualization of a first set of objects of a plurality of segments of an utterance spoken by a first speaker; generating and representing on the screen a second visualization of a second set of objects of the plurality of segments of the utterance spoken by a second speaker, wherein the first set of objects or the second set of objects includes the object; method.

5. each of the plurality of objects is defined by a boundary; the spacing between borders of two adjacent objects in the pictorial representation is based on a duration of an unvoiced period. The method of claim 4.

6. A method for computer-generated visualization of an utterance comprising at least one segment, the method comprising: generating a pictorial representation including a plurality of objects, each of the plurality of objects corresponding to a respective segment of the utterance, and generating the pictorial representation includes, for each of the plurality of objects: expressing the duration of each of the segments by a length of the object, and expressing the strength of each of the segments by a width of the object; Providing spacing between adjacent objects in said iconographic representation; and displaying said graphical representation on a screen of a computing device; displaying said object in a color selected based on the place and / or manner of articulation of the sound corresponding to said segment; A method comprising:

7. The segment comprises at least one phoneme. The method of claim 4.

8. The segment includes at least one vowel in the at least one phoneme. The method of claim 7.

9. A method for computer-generated visualization of an utterance comprising at least one segment, the method comprising: generating a pictorial representation including a plurality of objects, each of the plurality of objects corresponding to a respective segment of the utterance, and generating the pictorial representation includes, for each of the plurality of objects: expressing the duration of each of the segments by a length of the object, and expressing the strength of each of the segments by a width of the object; Providing spacing between adjacent objects in said iconographic representation; and displaying said graphical representation on a screen of a computing device; displaying said object in a color selected based on a first phoneme in said segment; wherein the segment comprises at least one phoneme. method.

10. The method comprises: breaking down the utterance into segments containing at least one phoneme; representing said at least one phoneme as at least one symbol associated with said object; The method of claim 7, further comprising:

11. The first visualization unit comprises: representing a first utterance spoken by a first speaker, the first visualization unit including a first set of objects on the screen corresponding to the first utterance; The second visualization unit includes: representing a second utterance spoken by a second speaker, the second visualization portion including a second set of objects corresponding to the second utterance, the first set or the second set including the objects; 5. The method of claim 4, wherein the second visualization portion is displayed on the screen such that a first end of the first collection of objects and a first end of the second collection of objects are substantially vertically aligned on the screen.

12. the computing device further includes a microphone input; The method comprises: subsequent to displaying the first visualization, recording the second utterance via the microphone input; generating and displaying the second visualization in response to the second recorded utterance; and The method of claim 11 , further comprising:

13. the object has a shape selected from a rectangle, an oval, and an egg; The method of claim 4.

14. A method for computer-generated visualization of an utterance comprising at least one segment, the method comprising: generating a pictorial representation including a plurality of objects, each of the plurality of objects corresponding to a respective segment of the utterance, and generating the pictorial representation includes, for each of the plurality of objects: expressing the duration of each of the segments by a length of the object, and expressing the strength of each of the segments by a width of the object; Providing spacing between adjacent objects in said iconographic representation; and Displaying said graphical representation on a screen of a computing device. Including, the tilt angle of the object varies along the length of the object; method.

15. A non-transitory computer-readable medium, comprising: A non-transitory computer readable medium having stored thereon instructions executable by a computing device to perform the method of any one of claims 1 to 14.

16. 20. A system comprising a computing device and the non-transitory computer-readable medium of claim 15.

17. the computing device includes a memory including the non-transitory computer-readable medium; 17. The system of claim 16.

18. A non-transitory computer-readable medium, comprising: Stored on the non-transitory computer-readable medium are instructions executable by a computing device to perform the following operations: The operation comprises: generating a visualization of the utterance, the visualization including objects corresponding to segments of the utterance, generating the visualization comprising: expressing the duration of the segment in terms of the length of the object; expressing the strength of the segmentation by a width of the object; expressing the segmental pitch curve in terms of the inclination angle of the object relative to a reference frame; and displaying said visualization on a screen coupled to said computing device; A non-transitory computer readable medium comprising:

19. the object is a two-dimensional object having a regular geometric shape; 20. The non-transitory computer-readable medium of claim 18.

20. the object has a shape selected from an egg, an oval, and a rectangle; 20. The non-transitory computer-readable medium of claim 18.

21. the pitch contour is associated with a fundamental frequency movement; and generating the visualization further comprises representing the fundamental frequency offset of the segment by a vertical position of the object relative to the frame of reference.

20. The non-transitory computer-readable medium of claim 18.

22. the segment is a first segment of the utterance, The operation comprises: displaying a first object corresponding to the first segment; displaying a second object corresponding to a second segment of the utterance subsequent to the first segment; Including, the first object and the second object are separated by an interval corresponding to an unvoiced period between the first segment and the second segment.

20. The non-transitory computer-readable medium of claim 18.

23. The segment comprises at least one phoneme.

20. The non-transitory computer-readable medium of claim 18.

24. The segment includes at least one vowel in the at least one phoneme.

24. The non-transitory computer readable medium of claim 23.

25. the operations further include displaying the object in a color selected based on a first phoneme in the segment.

24. The non-transitory computer readable medium of claim 23.

26. The color is selected based on the place and / or manner of articulation of the sound corresponding to the segment.

26. The non-transitory computer readable medium of claim 25.

27. The operation comprises: breaking down the utterance into at least one segment containing at least one phoneme; representing said at least one phoneme together with said object as a corresponding number symbol in said visualization portion; 24. The non-transitory computer readable medium of claim 23, further comprising:

28. The operation comprises: generating and displaying on the screen a first visualization of a first utterance spoken by a first speaker, the first visualization including a first collection of objects on the screen corresponding to the first utterance; generating a second visualization of a second utterance spoken by a second speaker, the second visualization including a second set of objects corresponding to the second utterance; displaying the second visualization on the screen such that a first end of the first collection of objects and a first end of the second collection of objects are substantially vertically aligned on the screen; 20. The non-transitory computer readable medium of claim 18, comprising:

29. the computing device is coupled to a microphone input; The operation comprises: subsequent to displaying the first visualization, recording the second utterance via the microphone input; generating and displaying the second visualization in response to the second recorded utterance; and 30. The non-transitory computer readable medium of claim 28, further comprising:

30. the computing device is coupled to an audio output; The operation comprises: providing an acoustic reproduction of the first utterance via the acoustic output; and providing a user control configured to enable a user to play the audio playback of the first utterance subsequent to displaying the second visualization; 30. The non-transitory computer readable medium of claim 28, further comprising:

31. 1. A system comprising: A processor; A display and a memory containing instructions that, when executed by said processor, cause said processor to perform the following operations: said operations comprising: generating a pictorial representation of an object corresponding to a segment of speech, said pictorial representation comprising: expressing the duration of the segment in terms of the length of the object; expressing the strength of the segmentation by a width of the object; expressing the segmental pitch curve in terms of the inclination angle of the object relative to a reference frame; and Displaying a graphical representation of the object on a screen of a computing device. Including, the system.

32. The segment is a first segment, The operation comprises: displaying a first object corresponding to the first segment; displaying a second object corresponding to a second segment of the utterance that follows the first segment; placing an interval between the first object and the second object corresponding to an unvoiced period between the first segment and the second segment; 32. The system of claim 31, further comprising:

33. the operations further include displaying the object in a color selected based on the place and / or manner of articulation of the sound corresponding to the segment.

32. The system of claim 31.

34. The operation comprises: generating and displaying on the screen a first visualization of a first utterance spoken by a first speaker, the first visualization including a first collection of objects on the screen corresponding to the first utterance; generating a second visualization of a second utterance spoken by a second speaker, the second visualization including a second set of objects corresponding to the second utterance; displaying the second visualization on the screen such that a first end of the first collection of objects and a first end of the second collection of objects are substantially vertically aligned on the screen; 32. The system of claim 31, further comprising:

Images