Algorithm determination of story reader reading interruptions

Abstract

The invention discloses an algorithm determination of story reader reading interruptions. Techniques are provided for enhancing the ability of a computing device to detect when a user has interrupted reading a text source. An example method includes receiving audio data including a spoken word associated with a text source, where the audio data includes a first duration and a second duration; comparing the audio data with the data of the text source, wherein the first duration of the audio data corresponds to the data of the text source; calculating, by the processing device, a correspondence metric between the second duration of the audio data and the data of the text source; and in response to determining that the corresponding metric satisfies the threshold, transmitting a signal to cease comparing the audio data with the data of the text source.

Description

[0001] Description of the case

[0002] This application is a divisional application of Chinese invention patent application No. 201880096938.4, filed on August 27, 2018. Technical Field

[0003] The present disclosure relates to the field of computer-based human speech recognition, and more particularly to enhancing the ability of a computer device to determine that a user is no longer reading aloud text source content. Background Art

[0004] The capabilities and uses of virtual assistants are rapidly expanding. Traditional virtual assistants include some form of computer human-machine interface that enables humans to interact with the virtual assistant and have the virtual assistant perform tasks or services. Virtual assistants typically record and understand human speech and can respond by synthesizing replies. Virtual assistants can be initiated in response to touch- or gesture-based commands, or can continuously analyze their environment to detect verbal commands. When the command is detected, the virtual assistant can respond or perform one or more actions. Summary of the invention

[0005] The following is a simplified summary of the present disclosure in order to provide a basic understanding of some aspects of the present disclosure. This summary is not an extensive summary of the present disclosure. It is neither intended to identify the key or decisive elements of the present disclosure, nor to delimit any scope of the specific embodiments of the present disclosure or any scope of the claims. Its sole purpose is to present some concepts of the present disclosure in a simplified form as a preface to a more detailed description presented later.

[0006] According to a first aspect of the present disclosure, a method is provided, comprising: receiving audio data comprising spoken words associated with a text source, wherein the audio data comprises a first duration and a second duration; comparing the audio data with data of the text source, wherein the first duration of the audio data corresponds to the data of the text source; calculating, by a processing device, a correspondence metric between the second duration of the audio data and the data of the text source; and transmitting a signal to stop comparing the audio data with the data of the text source in response to determining that the correspondence metric satisfies a threshold.

[0007] The text source may include a book, and wherein the first duration of the audio data includes spoken words of the book. The method may further include: in response to determining that the content of the text source is not present for the second duration of the audio data, prompting the user to exit the story time mode. Transmitting the signal may further include transmitting a signal to deactivate one or more microphones that capture audio data. The data of the text source may include phoneme data. Comparing the audio data may include calculating a phoneme edit distance between the phoneme data of the text source and the phoneme data of the audio data. Calculating the correspondence metric between the second duration of the audio data and the data of the text source may include calculating the correspondence metric based on a plurality of phoneme edit distances. Determining that the correspondence metric satisfies a threshold may include determining that the correspondence metric is below or above a threshold for a threshold duration. Determining that the correspondence metric satisfies a threshold may indicate that the second duration of the audio data includes content that is different from the content of the text source.

[0008] According to a second aspect of the present disclosure, a system includes a processing device configured to: receive audio data including spoken words associated with a text source, wherein the audio data includes a first duration and a second duration; compare the audio data with data of the text source, wherein the first duration of the audio data corresponds to the data of the text source; calculate a correspondence metric between the second duration of the audio data and the data of the text source; and in response to determining that the correspondence metric satisfies a threshold, transmit a signal to stop comparing the audio data with the data of the text source.

[0009] The system may include data storage. The system may include a communication system for communicating over a network such as a local area network and / or a wide area network. The system may be, may be included in a virtual assistant or may be configured to implement a virtual assistant. The system may be configured to implement the method of the first aspect.

[0010] According to a third aspect of the present disclosure, a computer program product is configured such that when processed by a processing device, the computer program product causes the processing device to execute the method of the first aspect.

[0011] Individual features and / or combinations of features defined above or below with respect to any specific embodiment according to any aspect of the present disclosure may be used individually, alone, or in combination with any other defined features in any other aspect or embodiment. In addition, the present disclosure is intended to cover devices configured to perform any of the features described herein with respect to methods and / or methods of using or producing, using or making any of the device features described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present disclosure is illustrated by way of example and not limitation in the figures of the accompanying drawings.

[0013] Figure 1 An example environment with one or more computing devices is illustrated in accordance with an implementation of the present disclosure.

[0014] Figure 2 is a block diagram illustrating an example computing device having components and modules for comparing phoneme data derived from user input and phoneme data derived from a text source according to an embodiment of the present disclosure.

[0015] Figure 3 is a block diagram illustrating an example computing device having components and modules for identifying a location in a text source based on a user's audio input according to an embodiment of the invention.

[0016] Figure 4 is a block diagram illustrating an example computing device with components and modules for providing physical effects to enhance the experience of one or more users in accordance with an embodiment of the invention.

[0017] Figure 5 is a flow chart illustrating an example of a method according to an embodiment of the present disclosure.

[0018] Figure 6 is a flow chart illustrating an example of a method according to an embodiment of the present disclosure.

[0019] Figure 7 is a flow chart illustrating an example of a method according to an embodiment of the present disclosure.

[0020] Figure 8 is a flow chart illustrating an example of a method according to an embodiment of the present disclosure.

[0021] Fig. 9 is a block diagram illustrating another example of a computing device according to an embodiment of the present disclosure.

[0022] These drawings can be better understood when viewed in conjunction with the following detailed description. DETAILED DESCRIPTION

[0023] Modern computing devices generally provide features for detecting and understanding human speech. These features can be associated with a virtual assistant, which can be accessed via a resource-constrained computing device (such as a smart speaker, mobile phone, smart watch or other user device). A computing device can be associated with a microphone that can record human speech and can use a combination of local and remote computing resources to analyze human speech. Analyzing speech is generally a resource-intensive operation, and a computing device can be configured to perform some processing local to the computing device and have some processing performed remotely at a server or via a cloud service. Many virtual assistants use some form of remote speech recognition service, which treats audio data as input and converts the audio data into text, which is returned to the computing device.

[0024] When a computing device attempts to follow along using a traditional virtual assistant feature while a user is reading a text source aloud, a number of technical problems can arise. Some problems arise because a traditional virtual assistant may not be able to detect when a user has finished providing audio input (e.g., when the user continues to talk about something else). This may potentially result in unnecessary utilization of the virtual assistant's computing resources, such as processing capacitor memory, data storage, and / or network bandwidth, which may otherwise be consumed by the virtual assistant continuing to follow the user's reading of the text after the user has finished reading the text. Additionally or alternatively, this may cause the computing device to continue recording and / or processing the user's audio, which may be problematic if the user transitions to discussing private matters. Detecting when a user has stopped reading from a text source may be more challenging when the user does not follow along with the text and skips, repeats, or adds new content while the text source is being read aloud.

[0025] Various aspects and embodiments of the present technology address the above and other deficiencies by enhancing the ability of a computing device to detect when a user has interrupted reading a text source. In one example, the technology can enable a virtual assistant to more accurately detect that a user has left reading a text source to take a break, and can deactivate a microphone to avoid capturing private audio content. This may involve receiving audio data including spoken words associated with a text source, and comparing the audio data with data from the text source. The technology can calculate a correspondence metric between the content of the audio data and the content of the text source. The correspondence metric can be a probability value based on a comparison of phoneme data, text data, or other data, and can involve the use of fuzzy matching logic. When the correspondence metric meets a threshold (e.g., below a minimum correspondence threshold), the technology can cause a signal to be transmitted that stops the analysis of subsequent audio data.

[0026] The systems and methods described herein include technologies in the field of technology for enhancing computer-based human speech recognition. In particular, the technology may address technical issues such as avoiding inadvertent recording of a user's private conversations by using a comparison method that better compensates for non-linear reading of a text source (e.g., skipping, repeating, adding content). The technology may also enable a computing device to reduce power and / or other computing resource consumption by deactivating an audio sensor (e.g., a microphone) and associated data processing when the computing device detects that a user has stopped reading text.

[0027] The technology discussed below includes multiple enhancements to computing devices with or without virtual assistant features. The enhancements can be used alone or together to optimize the ability of a computing device to follow along when a text source is read aloud, and to provide special effects to supplement the environment of the listening user. In one example, the environment can include a parent reading a book aloud to one or more children. In another example, the environment can include one or more users providing a presentation, voice, or other performance to an audience. In either example, the technology can be used to enhance the environment with special effects based on analysis of data associated with the text source. Special effects can be synchronized with specific portions of the text source, such as specific spoken words or page turns.

[0028] Figure 1 An exemplary environment 100 is illustrated according to one or more aspects of the present disclosure, the exemplary environment 100 including a text source being read aloud and one or more devices that supplement the environment to enhance the user's listening experience. The environment 100 can be a physical environment, such as an indoor setting (e.g., a bedroom, a conference room), an outdoor setting (a park, the wild), or other location. The environment 100 can be referred to as a pervasive computing environment or ubiquitous computing environment, and can include embedded computing functionality. The embedded computing functionality can provide ambient intelligence that is sensitive and responsive to human presence. In one example, the environment 100 can include one or more users 110A and 110B, a text source 120, one or more computing devices 130A and 130B, and one or more physical effect devices 140A-C.

[0029] Users 110A and 110B may include human users who are able to perceive the content of a text source. User 110A may be an individual user who is reading the content of a text source, or may be multiple users who are all reading a portion of one or more text sources. User 110A may be referred to as a reader, a presenter, a broadcaster, an actor, other terms, or a combination thereof. User 110B may listen to the content of a text source read aloud. User 110B may or may not read with user 110A. In one example, user 110A may be a parent who is reading to a child user 110B. In another example, user 110A may include one or more presenters who speak to one or more users 110B as audience members. In either example, the content of text source 120 may be announced for one or more other users to listen to.

[0030] Text source 120 may be any source of content that can be interpreted and read aloud. Text source 120 may include content containing numbers, characters, words, symbols, images, or a combination thereof. The content may be arranged into a sequence that can be spoken by a user while reading or after storage. Text source 120 may be a physical or electronic book, a magazine, a presentation, a voice, a script, a play, a memo, a bulletin, an article, a blog, a post, a message, other text arrangement, or a combination thereof. Figure 1 In the example of FIG. 1 , text source 120 may be a children's book that includes a series of words and images that may be read aloud to a child.

[0031] Audible actions 112A-C may be any action or combination of actions that produces a sound that can be detected by a user or computing device. Audible actions 112A-C may be heard, sensed, or observed by the user's ears or an audio sensor (e.g., a microphone) associated with a computing device. Figure 1 As shown in, there may be multiple types of audible actions, and they may depend on where the sound originates from. Audible action 112A can be an audible action of the first type, which includes a voice (e.g., speech) that can be derived from human speech or computer-synthesized speech. The voice can be a language voice (e.g., a spoken word), a non-language voice (e.g., a laugh, a cry, a cough), other sounds, or a combination thereof. Audible action 112B can be an audible action of the second type, which includes a non-language sound (non-vocal sound) derived from a user or another source, and can include clapping, finger bouncing, other sounds, or a combination thereof. Audible action 112C can be an audible action of the third type, which includes a non-language sound caused by the user interacting with an object, and can include turning pages, closing a book, opening / closing a door, falling an object, knocking on the floor, other sounds, or a combination thereof. One or more audible actions 112A-C can be detected by one or more sensors 131A-C.

[0032] Sensors 131A-C may be coupled to computing device 130A and may enable the computing device to sense aspects of environment 100. Sensors 131A-C may include one or more audio sensors (e.g., microphones), optical sensors (e.g., ambient light sensors, cameras), atmospheric sensors (e.g., thermometers, barometers, hydrometers), motion sensors (e.g., accelerometers, gyroscopes, etc.), position sensors (e.g., global positioning system sensors (GPS)), proximity sensors, other sensing devices, or combinations thereof. Figure 1 In the example shown in , sensor 131A may be an audio sensor, sensor 131B may be an optical sensor, and sensor 131C may be a temperature sensor. One or more of sensors 131A-C may be internal to computing device 130A, external to computing device 130A, or a combination thereof, and may be connected via a wired or wireless connection (e.g., WiFi) is coupled to computing device 130A.

[0033] Computing device 130A can be any computing device capable of receiving and processing data derived from sensors 131A-C. Computing device 130A can be used as a voice command device and provide access to an integrated virtual assistant. In one example, computing device 130A can include a smart speaker, a mobile device (e.g., a phone, a tablet), a wearable device (e.g., a smart watch), a digital media player (e.g., a smart TV, a micro console, a set-top box), a personal computer (e.g., a laptop, a desktop, a workstation), a home automation device, other computing devices, or a combination thereof. In some embodiments, computing device 130A may also be referred to as a "user device," "consumer device," or "client device." The data generated by sensors 131A-C can be received by computing device 130A and can be processed locally by computing device 130A, or can be remotely transmitted from computing device 130A to another computing device (e.g., 130B).

[0034] Computing device 130A may include one or more components for processing sensor data. Figure 1 In the example shown, computing device 130A may include audio analysis component 132, text source analysis component 133, comparison component 134, non-linear reading recognition component 135, physical effect determination component 136, prediction loading component 137, and effect providing component 138. In other examples, one or more of these components or one or more features of the components may be performed by another computing device (e.g., computing device 130B). These components will be related to Figures 2 to 4 , discussed in more detail, and can be used to detect the current reading position and instruct one or more physical effect devices 140A-C to enhance the listening experience.

[0035] Physical effects devices 140A-C may be any computing device capable of causing or providing a physical effect. The physical effects may be perceived via the senses of users 110A and 110B (e.g., hearing, sight, touch, smell, and taste). Each of physical effects devices 140A-C may produce one or more physical effects, and computing device 130A may function as one or more of physical effects devices 130A-C. Physical effects devices 140A-C may provide physical effect 145, or may instruct another device to provide physical effect 145. In one example, one or more physical effects devices 140A-C may be part of or integrated with a home automation system, or may be separate from the home automation system. Figure 1 As shown in , physical effect device 130A may include a speaker or other device capable of causing or emitting an acoustic effect. Physical effect device 130B may include one or more light sources (e.g., light bulbs, pixels) or other devices capable of changing the amount of light present in environment 100 (e.g., electric curtains or blinds). Physical effect device 130C may include one or more devices that can cause tactile effects, and may include a vibration source (e.g., a massage chair), a fan that generates wind (e.g., a ceiling fan or air conditioner), a heating or cooling source (e.g., a thermostat), other devices, or a combination thereof.

[0036] Physical effect 145 can be any modification to environment 100 that can be perceived by a user or computing device, and can include acoustic effects, tactile effects, optical effects, other effects, or combinations thereof. Acoustic effects can be physical effects related to sound and can be propagated via sound waves. Sound effects can include human or animal sounds (e.g., speech or noise), atmospheric sounds (e.g., thunder, rain, wind, or other weather sounds), musical sounds (e.g., instruments, background music, theme music), object sounds (e.g., knocking, opening, closing windows, glass breaking, object collision, car driving), other sound effects, or combinations thereof. Haptic effects can be physical effects related to a user's sense of touch. Haptic effects can include breeze, vibration, temperature change, other sense of touch, or combinations thereof. Optical effects can be physical effects related to light and can be propagated via visible electromagnetic radiation. Optical effects can include an increase or decrease in ambient lighting, flashing lights, animations, other changes in the amount of light, or combinations thereof. The optical effects may come from lights (e.g., ceiling lights, table lamps), flashing lights (e.g., phone lights), curtains (e.g., blinds or curtains), projectors, electronic displays, holographic displays, lasers, other light sources, or combinations thereof. Other effects may include effects related to smell or taste (e.g., olfactory effects).

[0037] Computing device 130B may be a server coupled to computing device 130A and may be located locally or remotely from environment 100. Computing device 130B may include one or more computing devices (such as rack-mounted servers, server computers, personal computers, mainframe computers, laptop computers, tablet computers, desktop computers, routers, etc.), data storage (e.g., hard disks, memories, databases), networks, software components, and / or hardware components. In one example, computing device 130B may be used to provide remote processing and may be used as a voice processing service, such as with respect to Figure 2 In another example, computing device 130B may provide computing device 130A with access to a media item.

[0038] The media items may correspond to physical effects, text sources, profile information, voice models, instructions, other data, or a combination thereof. Examples of media items include, but are not limited to, digital sound effects, digital music, digital animations, social media information, e-books, electronic magazines, digital newspapers, digital audiobooks, digital videos, digital photos, website content, electronic journals, web blogs, real simple syndication (RSS) feeds, electronic comic books, software applications, and the like. In some embodiments, media items may be referred to as content items and may be provided over the Internet and / or via computing device 130A (e.g., smart speakers). As used herein, "media," "media items," "digital media," "digital media items," "content," and "content items" may include electronic files or records that may be loaded or executed using software, firmware, or hardware configured to present content to one or more users in environment 100. In one embodiment, computing device 130B may use one or more data stores to store media items and provide the media items to computing device 130A via network 150.

[0039] The network 150 may include one or more of a private network (e.g., a local area network (LAN), a public network (e.g., the Internet), a wide area network (WAN)), a wired network (e.g., Ethernet), a wireless network (e.g., a Wi-Fi or Bluetooth connection), a cellular network (e.g., a Long Term Evolution (LTE) network), routers, hubs, switches, server computers, and / or combinations thereof.

[0040] Typically, functions described in one embodiment as being performed by computing device 130A, computing device 130B, or physical effect devices 140A-C may be performed by one or more other devices in other embodiments. In addition, functionality attributed to a particular component may be performed by different or multiple components operating together. Computing devices 130A and 130B may also be accessed as services provided to other systems or devices through appropriate application programming interfaces. Although embodiments of the present disclosure are discussed in terms of smart speakers, these embodiments may also incorporate one or more features of a cloud service or content sharing platform.

[0041] In situations where the systems discussed herein collect personal information about client devices or users or can utilize personal information, the user may be provided with an opportunity to control whether the computing device is able to collect user information (e.g., information about the user's audio input, the user's preferences, the user's current location, social network, social behavior, activities, or profession) or to control whether and / or how content more relevant to the user is received from the computing device. In addition, certain data may be processed in one or more ways before it is stored or used so that personally identifiable information is removed. For example, the identity of the user may be processed so that personally identifiable information cannot be determined for the user, or the user's geographic location may be generalized (such as to a city, zip code, or state level) in the case where location information is obtained so that the user's specific location cannot be determined. Thus, the user may control how information about the user is collected and how it is used by the computing device.

[0042] Figure 2-4 A block diagram illustrating an exemplary computing device 130 that can detect reading position within a text source and supplement the environment with physical effects to enhance the listening experience in accordance with one or more aspects of the present disclosure. Computing device 130 can be the same or similar to computing device 130A, computing device 130B, or a combination thereof. Figure 2 Features that enable computing device 130 to receive a user's audio data and compare it to data from a textual source are discussed. Figure 3 Features that enable computing device 130 to analyze data based on audio data and text source data to detect reading positions are discussed. Figure 4 Features that enable computing device 130 to provide physical effects to modify the environment of one or more listeners are discussed. Figure 2-Figure 4 The components and modules provided in the are exemplary, and may include more or less components or modules without loss of generality. For example, two or more components may be combined into a single component, or the features of a component may be divided into two or more components. In one embodiment, one or more components may reside on different computing devices (e.g., a client device and a server device).

[0043] refer to Figure 2 , the computing device 130 may include an audio analysis component 132, a text source analysis component 133, a comparison component 134, and a data store 240. The audio analysis component 132 may receive and access audio data extracted from the environment when the user reads the text source aloud. In one example, the audio analysis component 132 may include an audio data receiving module 212 and an acoustic modeling module 214.

[0044] The audio data receiving module 212 can receive audio data 241 including one or more audible actions of the user. The audio data can include spoken words, page turning, or other audible actions captured from the user environment. The audio data 241 can be received directly from one or more sensors in the form of audio signals, or can be received indirectly from the data storage 240 or other computing device after the sensor stores the audio data 241. The audio data 241 can be in any digital or analog format, and can be accessed or received from one or more storage objects (e.g., files, database records), data streams (e.g., audio streams, video streams), data signals, other data transmission or storage protocols, or a combination thereof. The audio data 241 can be an audio recording and can be segmented into one or more durations (e.g., parts, blocks, or other units) before, during, or after being analyzed by the acoustic modeling module 214.

[0045] The acoustic modeling module 214 can use an acoustic model to analyze the audio data 241 to identify phoneme data 243A. The acoustic model can represent a known relationship between an audible action and a phoneme. A phoneme can be a unit of sound and can correspond to a sound pattern of an audible action (e.g., a spoken word). A phoneme can be a language unit, a non-language unit, other units, or a combination thereof. The acoustic modeling module 214 can convert the audio data into phonemes, which are stored in the data storage device 240 as phoneme data 243A.

[0046] The phoneme data 243A may include values ​​representing one or more phonemes extracted from the audio data 241. The phoneme data 243A may use standard or proprietary notation to represent a series of phonemes. The notation may include a specific arrangement of one or more bits, bytes, symbols, or characters representing the phonemes. In one example, a specific arrangement may include symbols placed next to or between one or more delimiters. Delimiters may include slashes, square brackets, vertical bars, brackets, commas, tabs, spaces, line breaks, other delimiters, or combinations thereof. Phonemes may be arranged into a series of phonemes representing a portion of one or more audible actions.

[0047] The text source analysis component 133 can receive and analyze data related to the text source 120. The text source 120 can be determined in view of user input, which is text-based, voice-based, touch-based, gesture-based, or other user input. For example, a user can identify the text source 120 by speaking the name of the text source 120 (e.g., the title of a book or the author), by typing and searching for the text source, by selecting a displayed text source, other selection mechanisms, or a combination thereof. Figure 2 In the example shown, the text source analysis component 133 may include a data access module 222 and a phoneme determination module 224 .

[0048] Data access module 222 can access data associated with text source 120, and can store the accessed data as text source data 242. Data access module 222 can access data from one or more sources, which can include local sources, remote sources, or a combination thereof. The local source can be the storage of computing device 130, and the remote source can be the storage of a computing device that can be accessed through a network connection. In one example, the remote source can be the same or similar to computing device 130B (e.g., a server or cloud service). The local or remote source can store data of one or media items discussed above, and the computing source can access the data. The data can then be analyzed, filtered, combined, or modified, and then stored as text source data 242.

[0049] The text source data 242 may be any data associated with the text source 120 and may be provided by or accessed from an author, publisher, distributor, partner, remote server, third-party service, other source, or a combination thereof. The text source data 242 may include descriptive data, text data, phonemic data, other data, or a combination thereof. The descriptive data may indicate a title, abstract, source (e.g., author, publisher, distributor), table of contents (e.g., chapter, section, page), index (e.g., phrases, page indicators), other data, or a combination thereof.

[0050] The text data may include one or more words of a text source 120. In one example, the words may be organized into a word sequence 122 with or without one or more images 124. The text data may be a data structure that arranges words in the same or similar manner as a user would read them (e.g., a series of consecutive words). The word sequence may be limited to words that appear in the text source 120, or may be supplemented with words or data that indicate the presence or content of non-text information (e.g., illustrations, images, tables, formatting, paragraphs, pages). In another example, the words may also or alternatively be arranged in an index data structure that indicates unique words that exist in the text source 120 but are not arranged consecutively in the manner spoken by the user. Either data structure may be supplemented with other information that may include word positions within the text source (e.g., page, line, slide), number of occurrences, variations of the words (e.g., tense, plural), other data, or combinations thereof. In one example, text source 120 may be a physical book, and text source data 242 may include words from a corresponding electronic book (eg, e-book), a third-party service, other sources, or a combination thereof.

[0051] The phoneme data of the text source 120 may be the same as or similar to the phoneme data 243B and may be a phonemic encoding of the text source 120 in a format that is the same as or similar to the phoneme data derived from the audio (e.g., the phoneme data 243A). In the example discussed above, the phoneme data 243B of the text source 120 may be included as part of the text source data 242 and accessed by the phoneme determination module 224. In another example, the phoneme data 243B may not be present in the text source data 242 and may be generated by the phoneme determination module 224.

[0052] The phoneme determination module 224 can determine the phoneme data for a particular text source 120. This can involve the phoneme determination module 224 accessing existing phoneme data 243B from a remote source, generating phoneme data 243B based on text data, or a combination thereof. When generating phoneme data 243B, the phoneme determination module 224 can access and analyze the text data of the text source data 242, and convert (e.g., derive, convert, transform, encode) the text data into phoneme data 243B. The generated phoneme data can then be associated with the text source 120 for future use by the computing device 130 or one or more other computing devices. In one example, the text data can include a sequence of words, and the generated phoneme data can include a phonetic encoding that includes a sequence of phonetic values ​​representing the sequence of words. The same sequence of phonetic values ​​can correspond to two words that sound the same but are spelled differently (e.g., homophones). Similarly, different sequences of phonetic values ​​may correspond to words that sound different even if the spelling is the same (e.g., homographs).

[0053] As described above, the phoneme data 243A and 243B can both include a phoneme sequence represented by a standard or proprietary notation. The notation can be referred to as speech transcription or phoneme transcription, and can include a specific arrangement of phoneme values ​​representing a language segment. A language segment can be any discrete unit that can be physically or auditorily identified in a speech stream. A phoneme value can include one or more symbols, characters, bytes, bits, other values, or a combination thereof. In one example, a phoneme value can be represented by one or more Unicode characters, the American Standard Code for Information Interchange (ASCII) characters, other characters, or a combination thereof. A phoneme value sequence can represent a single word, and each individual phoneme value can represent a portion of the word. For example, a first phoneme sequence can be / θΛm / , and represents the spoken word "thumb", while a second phoneme sequence can be / dΛm / , and represents the spoken word "dumb". In the example discussed below, the phoneme data 243A and 243B can include a value sequence, and each value can represent a phoneme of a phoneme vocabulary.

[0054] A phoneme vocabulary may include a set of possible phoneme values ​​for one or more languages. A phoneme vocabulary may be a phonetic alphabetic system and may represent the speech qualities of a portion of a spoken language: the sounds, phonemes, intonation, and separation of words and syllables. A phoneme vocabulary may or may not represent additional qualities of speech and variations in speech cues (e.g., tongue-biting, paraphrasing, stress, dialects). A phoneme vocabulary may be the same as or similar to a phoneme alphabet, character set, vocabulary, dictionary, other variations, or a combination thereof. In one example, a phoneme vocabulary may be based on the International Phonetic Alphabet (IPA). IPA symbols may consist of one or more elements associated with letters and diacritical marks. For example, the English letter <t>The sound of t can be represented in IPA by a single letter [t] or by a letter plus a diacritical mark. Transcription. Delimiters (e.g., slashes) can be used to signal broad or phonemic transcription; thus, depending on the context and language, / t / may not be as common as or [t] specific and can refer to or [t]. In other examples, the phoneme vocabulary can be the same as or similar to Extended Speech Assessment Method Phonetic Alphabet (X-SAMPA), Kirshenbaum (eg, ASCII-IPA, erkIPA), other phoneme vocabularies, or combinations thereof.

[0055] Comparison component 134 can compare the audio of user 110A with the content of text source 120. The examples discussed below use the phoneme data corresponding to the audio and text sources, and compare them without using speech recognition to convert the audio into text. Other examples may also or alternatively use text data, descriptive data, audio data, other data or a combination thereof. Comparison can be performed by computing device 130, by a remote computing device (e.g., cloud service) or a combination thereof. In one example, comparison component 134 can select a phoneme sequence derived from the audio and compare it with a plurality of phoneme sequences derived from the text source. In another example, comparison component 134 can compare the phoneme sequence of the text source with a plurality of phoneme sequences derived from the audio. In either example, the calculation of similarity measurement data can be based on the phoneme edit distance.

[0056] The phoneme edit distance module 232 can quantify the similarity between two phoneme sequences by determining the minimum number of operations required to convert one phoneme sequence into an exact match of another phoneme sequence. The operation may include any modification of the phoneme value (e.g., symbol) within one of the phoneme sequences. Example operations may include primitive operations, such as phoneme removal, insertion, replacement, transposition, other operations, or combinations thereof. In the example discussed above, the first phoneme sequence may be / θΛm / and represent "thumb", while the second phoneme sequence may be / dΛm / and represent "dumb". Although the two words differ by two letters, their phoneme edit distance is a value of 1, because converting the sequence to an exact match involves the replacement of a single phoneme (e.g., replacing θ with d). In one example, the phoneme edit distance may be a linear edit distance that is the same or similar to the Levenshtein distance. The Levenshtein distance may be based on the minimum number of removal, insertion, or replacement operations required to make two phoneme sequences equal. In other examples, the phoneme edit distance may also or alternatively include transposition or other operations. In either example, the phoneme edit distance may be a numerical value used to determine the similarity measurement data 244 .

[0057] The similarity measurement module 234 can access the data of the phoneme edit distance module to determine the similarity or dissimilarity between the audio and text sources. The similarity measurement module 234 can analyze the data of the phoneme edit distance module to calculate the similarity measurement data 244. The similarity measurement data 244 can represent the similarity between two or more phoneme sequences (e.g., the phonetic representation of a word or word set), and can include digital data, non-digital data, other data or a combination thereof. The similarity measurement data 244 can be based on the edit distance of one or more phoneme sequences. In one example, the similarity measurement data 244 can include the numerical value of the phoneme edit distance. In another example, the similarity measurement data 244 can include a probability value derived from the numerical value of the phoneme edit distance. For example, the similarity measurement data can be a percentage, ratio or other value based on one or more phoneme edit distances and one or more other values. Other values ​​can be the number of phonemes in one or more phoneme sequences or parts of the text source.

[0058] Data storage 240 may be a memory (e.g., random access memory), a cache, a drive (e.g., a solid state drive, a hard drive, a flash drive), a database system, or another type of component or device capable of storing data. Data storage 240 may also include multiple storage components (e.g., multiple drives or multiple databases) that may span one or more computing devices (e.g., multiple server computers).

[0059] Figure 3 A block diagram illustrating exemplary components that enable computing device 130 to analyze the data discussed above to determine a reading position or the absence of a reading position within a text source. As discussed above, portions of the audio may not completely match the text source because the user may add, skip, repeat, or reorder content of the text source when reading aloud. As a result, phoneme data derived from the audio and phoneme data derived from the text source may be challenging to compare and align. Figure 1 In the example shown, the computing device 130 may include a non-linear reading recognition component 135 that enables the computing device to determine the location within the text source that best aligns with the audio data. In one example, the non-linear reading recognition component 135 may include a fuzzy matching module 352, a location identification module 354, a reading speed module 356, and a reading interruption module 358.

[0060] The fuzzy matching module 352 can enable the computing device 130 to determine whether there is a match between the audio and text sources. The match can be the same or similar to a probabilistic match, a best match, a closest match, or any match that may not be an exact match but meets a predetermined threshold. In one example, determining a match between the audio and text sources can involve detecting that the audio segment includes one or more words of the text source. Even if the audio or text source contains other words, missing words, or contains variants of words (e.g., mispronunciation, missing plural forms), a match may be detected. The match can be referred to as a fuzzy match or an approximate match, and can be detected using fuzzy matching logic. Fuzzy matching logic can be used to compare sequences of phoneme values, and can operate at syllable-level segments, word-level segments, phrase-level segments, sentence-level segments, other segments, or combinations thereof. In one example, fuzzy matching can perform fuzzy matching using an audio segment with a predetermined length. The predetermined length can be customizable and can be any duration (e.g., 3+ seconds) or any number of word tokens (e.g., 3-4 words). When nonlinear reading is taken into account, having a predetermined length that is much smaller than the length of the text source can enhance accuracy and performance.

[0061] The fuzzy matching module 352 can impose one or more constraints to determine a match. In one example, a match is detected that may involve the use of one or more global unweighted costs. The global unweighted cost may be related to the total number of primitive operations required to convert a candidate phoneme sequence (e.g., a candidate pattern from a text source) into a selected phoneme sequence (e.g., a pattern from audio). In another example, a match is detected that may involve specifying the number of operations of each type individually, while other operations set the total cost, but different weights are allowed to be assigned to different primitive operations. The fuzzy matching module 352 can also apply the individual assignment of restrictions and weights to individual phoneme values ​​in the sequence.

[0062] The location identification module 354 can access the data of the fuzzy matching module 352 to identify a location within a text source that corresponds to an audible action of an audio (e.g., a spoken word). In one example, the text source can be a children's book, and the location can be a reading location within a word sequence of the book. In other examples, the location can be in speech, a presentation, a script, a play, other text sources, or a combination thereof. In either example, the location can be a past, current, or future reading location within the text source and can be stored as location data 345. The location data can be digital or non-digital data that identifies one or more specific phonemes, words, paragraphs, pages, sections, chapters, tables, images, slides, other locations, or a combination thereof.

[0063] The position identification module 354 can determine that the audible action matches multiple different portions of the text source. This may occur when the same word or phrase (e.g., a sequence of phonemes) is repeated multiple times in the text source. The position identification module 354 can detect the spoken word by analyzing the phoneme data, and detect that the spoken word matches multiple candidate locations within the text source. The position identification module 354 can select one or more of the multiple candidate locations based on the data of the fuzzy matching module 352. The position identification module 354 can further narrow the candidate locations (e.g., extending a predetermined segment length or using adjacent segments) by selecting a specific location based on the phoneme data of the audio before, during, or after the spoken word occurs.

[0064] The reading speed module 356 can access and analyze the position data 345 to determine the user's reading speed. The reading speed data can be determined in view of the position data, the text source data, the audio data, other data or a combination thereof, and can be stored as the reading speed data 346. The reading speed can be based on a portion of the position data 345, which identifies at least two positions in the text source. The position can correspond to a specific time, and determining the reading speed can be based on the amount of words and the amount of time between two or more positions. In one example, the amount of words can be based on the content of the text source, and the content added, skipped or repeated by the user can be ignored. In another example, the amount of words can be based on the content of the text source and also based on the content of the audio. This may be advantageous because the content of the audio can indicate that words are added, skipped, repeated, other actions or a combination thereof. In either example, the reading speed module 356 can update the reading speed data to represent the reading speed of the user over one or more durations.

[0065] The reading interruption module 358 can access and analyze any of the data discussed above to detect whether the user has interrupted reading the text source or is still reading the text source. This can be challenging because the user may have stopped reading the text source but is discussing concepts related to the text source. As a result, there may be overlap in the spoken words and the content of the text source. Detecting interruptions in reading can be important because it can enable the computing device to avoid recording private discussions. The reading interruption module 358 can determine whether the user has interrupted reading the text source by calculating one or more corresponding metrics.

[0066] The corresponding metric may indicate the similarity or dissimilarity between the corresponding portion of the audio segment and the text source. The corresponding metric may be a probability value indicating the probability that the audio segment corresponds to the position of the text source. The probability value may be a numeric or non-numeric value and may be the same or similar to a percentage, ratio, decimal or other value or a combination thereof. In one example, the value may be between 0 and 1 (e.g., 0.97), 0 and 100 (e.g., 98), or other ranges of values. One end of the range may indicate that the segment of the audio absolutely corresponds to the position of the text source (e.g., 1.0 or 100), while the other range may indicate that the segment of the audio absolutely does not correspond to the position of the text source (e.g., a value of 0).

[0067] The corresponding metric can be based on or related to multiple similarity measurements. For example, both measurements can be used to compare or contrast data derived from audio (e.g., phoneme data 243A) with data derived from a text source (e.g., phoneme data 243B). Similarity measurements (e.g., phoneme edit distance) can be used to compare or contrast written words of a text source with spoken words, and the corresponding metric can be used to compare or contrast a set of written words with a set of spoken words within a duration. The duration of the audio (e.g., a segment) can be any length of time and can include a set of words and one or more other audible actions (e.g., turning pages, closing a book). In one example, the user's audio can include a first duration and a second duration, and the reading interruption module 358 can calculate one or more corresponding metrics within the first duration and one or more corresponding metrics within the second duration. The corresponding metric can be stored as corresponding metric data 347. In other examples, the corresponding metric can also or alternatively take into account one or more signals, such as the absence of voice input within the duration, the recognition of the story text, or the recognition of a specific word or phrase that may indicate a pause. These words or phrases may include "let's stop reading," "let's finish tomorrow," "OK, I'm done," "let's pause," other phrases, or a combination of these.

[0068] The reading interruption module 358 can compare the corresponding measurement data 347 for each duration to one or more predetermined thresholds. In response to the corresponding measurement data 347 for the first duration not meeting the threshold (e.g., being above or below the threshold), the reading interruption module 358 can determine that the duration of the audio corresponds to a text source and the user audio data corresponds to the user reading the text source. In response to the corresponding measurement data 347 for the second duration meeting the threshold (e.g., being below or above the threshold), the reading interruption module 358 can determine that the duration of the audio does not correspond to the text source and the user has discontinued reading the text source. In one example, determining that the corresponding metric meets the threshold can indicate that the audio data does not match the data of the text source or that the audio data is different from the content of the text source.

[0069] The reading interruption module 358 can perform one or more actions in response to determining that the user has interrupted reading the text source. In one example, the reading interruption module 358 can transmit a signal to deactivate one or more microphones associated with the computing device to avoid capturing or recording additional audio data. In another example, the reading interruption module 358 can transmit a signal to stop analyzing audio data (e.g., comparing the audio data with data from the text source). The latter example can record audio, but cannot access or analyze the audio data. In yet another example, the reading interruption module can enable the computing device 130 to interact with the user before, during, or after transmitting the signal. For example, the computing device can interact with the user by providing a prompt (e.g., audio, visual, or a combination thereof). The prompt can ask the user whether to exit the story time mode, or can notify the user that the story time mode has been exited, and may or may not enable the user to re-enable the story time mode.

[0070] Figure 4 A block diagram illustrating exemplary components that enable computing device 130 to provide physical effects to enhance a user's experience is depicted. As discussed above, physical effects can modify the environment and can include acoustic effects, tactile effects, optical effects, other effects, or combinations thereof. In the example shown, computing device 130 can include physical effect determination component 136, prediction loading component 137, and effect providing component 138.

[0071] The physical effect determination component 136 enables the computing device 130 to identify and provide physical effects corresponding to specific portions of a text source. In one example, the physical effect determination component 136 may include an audible action correlation module 462, a contextual data module 464, and an effect selection module 466.

[0072] The audible action correlation module 462 can enable the computing device to correlate a specific physical effect with a specific audible action associated with a text source. The audible action correlation module 462 can determine the correlation based on the effect data 448 of the text source. The effect data 448 can indicate which physical effects correspond to which parts of the text source. The effect data 448 can be related to a specific physical effect with a specific position in the text source, a specific audible action of the user, a specific trigger condition (discussed below), or a combination thereof. The position in the text source may be related to an audible action (e.g., a spoken word or a page turn), or it may be unrelated to an audible action (e.g., a user viewing a graphic image). In one example, the effect data 448 can identify an audible action including a specific spoken word of the text source (e.g., a dog), and the physical effect can involve initiating an acoustic effect corresponding to the spoken word (e.g., a bark). In another example, the effect data 448 can identify an audible action (e.g., a page turn), and the physical effect can involve modifying an existing physical effect (e.g., re-adjusting ambient sound, light, or temperature).

[0073] Effect data 448 may be accessed by computing device 130 or may be created by computing device 130. In one example, computing device 130 may access or receive effect data directly or indirectly from an author, publisher, distributor, partner, third-party service, other source, or a combination thereof. Effect data 448 may be included in text source data 242 or may be separated from text source data 242. In another example, computing device 130 may create effect data based on text source data 242. For example, audible action correlation module 462 may analyze text data or phoneme data and identify physical effects corresponding to specific portions of a text source. In either example, effect data 448 may be stored in data storage 240 for enhanced access by computing device 130.

[0074] The contextual data module 464 may enable the computing device 130 to collect contextual data 449 associated with the user. The contextual data 449 may be based on the user's environment and may be obtained using one or more sensors (e.g., sensors 131A-C). The contextual data 449 may also or alternatively be based on profile data about the user, which may be accessible to the computing device 130 via direct user input or via a remote source (e.g., a network connection with a content platform or social network). In one example, the contextual data 449 may include sound data (e.g., ambient sound measurements), light data (e.g., ambient light measurements), time data (e.g., morning or evening), calendar data (making an appointment in advance for tomorrow), geographic location data (e.g., zip code, address, latitude / longitude), weather data (e.g., rain, lighting, thunderstorm, high winds, cloudy), user profile information (e.g., a child's name, age, or gender), user audio feedback (e.g., a child crying or clapping), other data, or a combination thereof.

[0075] Effect selection module 466 can enable computing device 130 to select and modify physical effects based on effect data 448, context data 449, text source data 242, other data, or a combination thereof. Effect selection module 466 can be used to select a specific physical effect (e.g., an acoustic effect) or modify the properties of a physical effect. The property can be related to the intensity, timing, pitch, transition (e.g., fade in / fade out), other features, or a combination thereof of the physical effect. Intensity can be related to the magnitude of the modification to the environment, and can be related to the volume (e.g., loudness) or brightness (e.g., brightness) of the physical effect. Timing may be related to the speed or duration of the physical effect. Computing device 130 can select physical effects based on words of the text source, and can update the properties of the physical effect based on context data. In one example, context data can include sound data of the user's environment, and the physical effect can be an acoustic effect based on a certain amount of the sound data. In another example, context data can include light data of the user's environment, and the physical effect can be an optical effect that modifies the brightness of a light source based on the light data (e.g., dimming or brightening the light). In yet another example, the contextual data may include user profile data for a parent or child and indicate the age of the audience, and wherein the physical effects include acoustic effects selected based on the age of the user (e.g., funnier dog barking sounds for younger children and more serious dog barking sounds for older children).

[0076] Effect selection module 466 can use context data to identify timing aspects related to the reading of text source. For example, time data or calendar data can be used to distinguish between reading text source at night or reading in the morning. At night, effect selection module 466 can select calmer (e.g., less stimulating) physical effects to encourage listeners to prepare for bed. This may involve reducing the brightness and volume settings for sound and optical effects and / or selecting effects with lower pitches (e.g., softer collision effects or whispers as opposed to shouting). In the morning, effect selection module 466 can select more stimulating physical effects to encourage users to get ready for the day. This may involve increasing the brightness and volume settings of acoustic and optical effects. Calendar data can also indicate whether the reading time is associated with weekends or weekdays, or whether there is an appointment coming (e.g., later in the day or early the next morning). Any of these may affect the speed at which the user reads the text source and how long or how long physical effects should be provided.

[0077] The predictive loading component 137 enables the computing device 130 to predictively load the content of the physical effect before the content is needed. Predictive loading can accelerate the ability of the computing device 130 to provide physical effects by loading the content of the physical effect before the physical effect is initiated. Predictive loading can be the same or similar to pre-fetching, pre-caching, cache pre-fetching, other concepts or combinations thereof. In one example, the predictive loading component 137 may include a prediction module 472, a trigger determination module 474, and a content loading module 476.

[0078] Prediction module 472 may enable computing device 130 to predict a time when a user will reach a particular portion of a text source. For example, prediction module 472 may determine a time when a word of the text source will be spoken before the user speaks the word. The predicted time may be a future time and may be determined based on the user's reading speed, the reading position of the text source, other data, or a combination thereof. In one example, the time may be calculated based on the user's reading speed (e.g., words per minute, pages per minute) and the difference between the current reading position and a target position in the text source (e.g., number of words, paragraphs, or pages). In other examples, prediction module 472 may use predictive models, machine learning, neural networks, or other techniques to enhance predictions based on current data, historical data, or a combination thereof.

[0079] The trigger determination module 474 can enable the computing device 130 to determine a trigger condition associated with a particular physical effect. The trigger condition can be a loading trigger condition or an initiation trigger condition. The loading trigger condition indicates when to start loading the content of the physical effect. The initiation trigger condition indicates when to start providing (e.g., playing) the content of the physical effect. Either trigger condition can correspond to a specific time or a specific position within the text source, and can be based on effect data, text source data, other data, or a combination thereof. The specific time can be an absolute time (e.g., at 8:32:02pm) or a relative time (e.g., 5 seconds before the predicted time of a word or page turn). The specific position can be a position before the word within the text source with which the physical effect is to be aligned. The specific position can be an absolute position (e.g., word 397) or a relative position (e.g., 5 words before the word "barking").

[0080] The determination of the triggering condition may be based on one or more factors related to the content, computing device, user, other aspects of the environment, or a combination thereof. Factors related to the content may include the amount of content (e.g., 1MB file size), the location of the content (e.g., remote storage), the format of the content (e.g., downloadable files, streaming chunks, or a format that requires transcoding), the duration of the content (e.g., 2-second sound effects), other aspects of the content, or a combination thereof. Factors related to the computing device may correspond to the amount and / or availability of computing resources of the computing device 130 or other computing devices. Computing resources may include connection speed (e.g., networking bandwidth), storage space (e.g., available solid-state storage), processing power (e.g., CPU speed or load), other computing resources, or a combination thereof. Factors related to the user may include the user's reading speed, current reading position, voice clarity, other aspects, or a combination thereof.

[0081] The trigger determination module 474 may use one or more factors to calculate the duration of loading or providing the content of the physical effect. The duration associated with loading the content may be referred to as the predicted loading time, and may or may not include the duration of providing (e.g., playing) the content. In one example, the trigger determination module 474 may determine the duration of loading the content of the physical effect based on the size of the content and the network bandwidth of the computing device 130. The trigger determination module 474 may use the predicted loading time to identify a specific time or position of the trigger condition. In one example, the trigger condition may be set to a time greater than or equal to the predicted time of the audible action (e.g., spoken words) minus a predetermined loading time (e.g., 5 seconds). In another example, the trigger condition may be set to a position within the text source that is equal to or prior to the position where the physical effect is expected to be aligned. This may involve selecting a position in the text source based on the predicted loading time and the reading speed. For example, if the user reads at a speed of 120 words per minute (i.e., 2 words per second) and the predicted loading time is 5 seconds, the trigger position may be 10 or more words before the word with which the physical effect should be aligned.

[0082] Content loading module 476 may enable computing device 130 to load content for one or more physical effects before initiating the physical effect. Loading content may involve computing device 130 transmitting or receiving one or more requests and responses, and may involve downloading, streaming, copying, other operations, or a combination thereof. Content may include executable data (e.g., instructions), information data (e.g., audio files or chunks), other data, or a combination thereof. Content may be stored by computing device 130 in data store 240 as content data 451. Computing device 130 may load content for physical effects from a local device (e.g., data store 240), a remote device (e.g., a server or cloud service), or a combination thereof.

[0083] The effect providing component 138 can enable the computing device 130 to provide physical effects to modify the user's environment. The effect providing component 138 can be initiated after the content for the physical effect is loaded, and can be timed so that the physical effect is provided at a time that is aligned with the audible action that is intended to be aligned with it. In one example, the effect providing component 138 can include an instruction access module 482 and a physical effect initiation module 484.

[0084] The instruction access module 482 can access instruction data associated with a physical effect. The instruction data may include a set of one or more commands, operations, processes, tasks, other instructions, or a combination thereof. The instruction may indicate a physical effect and one or more properties of the physical effect.

[0085] The physical effect initiation module 484 can access the instruction data and execute the instruction data to initiate the physical effect. The physical effect initiation module 484 can initiate the instruction before, during or after detecting the initiation trigger condition (e.g., audible action) corresponding to the physical effect. In one example, the text source can include a specific word, and the initiation of the physical effect can be in response to detecting that the audio data includes the word (e.g., matching phonemes). In another example, the physical effect initiation module 484 can determine the initiation trigger condition for initiating the physical effect. The process of determining the trigger condition for initiating the physical effect can be the same or similar to the trigger condition for initiating the content of the loaded physical effect. The instruction can enable the computing device 130 to provide the physical effect, or can enable the computing device 130 to communicate with one or more physical effect devices to provide the physical effect. In either example, the computing device 130 can directly or indirectly enable the physical effect to modify the user's environment to enhance the listening user's experience.

[0086] In one example, the physical effect initiation module 484 or the effect selection module 466 can use one or more confidence thresholds to select and / or initiate a physical effect. One or more confidence thresholds can be grouped into one or more confidence intervals that classify the probability of audio matching a specific position of a text source (e.g., a spoken word matching a word of a text source). There can be any number of confidence intervals, and the first confidence interval can indicate that there is a low probability of audio matching the text source position (e.g., >50%), and each successive confidence interval can be higher (e.g., >75%, >95%, etc.). The effect data that relates the physical effect to the position can also include a specific confidence threshold (e.g., a minimum confidence interval). For example, providing a sound effect can be associated with a higher confidence interval, and then the background effect is transformed. The computing device 130 can determine whether the confidence threshold is met before selecting or initiating a physical effect. This can involve comparing corresponding metric data, similarity metric data, other data associated with fuzzy matching, or a combination thereof. In one example, a particular location in a text source may be associated with a plurality of different physical effects, and each may correspond to a different confidence interval associated with the current reading position. When the confidence interval is high (e.g., trust that the current reading position is accurate), a particular sound effect may be initiated (e.g., a sound effect of a single dog barking at a relatively high volume), whereas when the confidence interval is low (e.g., uncertainty as to whether the current reading position is accurate), a different sound effect may be initiated (e.g., background noise of multiple dogs barking at relatively low volumes).

[0087] Figure 5-8 Flowcharts of corresponding methods 500, 600, 700, and 800 are depicted for enhancing the ability of a computing device to follow and provide special effects in real time while a text source is read aloud in accordance with one or more aspects of the present disclosure. Method 500 may involve estimating reading progress using phoneme data and fuzzy matching. Method 600 may optimize the ability of a computing device to detect when a user has stopped reading from a text source and is engaging in a private discussion. Method 700 may enable a computing device to provide physical effects that take into account the context of the user and the user's environment. Method 800 may enable a computing device to pre-cache content for physical effects to reduce latency and better synchronize physical effects with audible actions associated with a text source.

[0088] Figure 5-Figure 8 The method and each of its individual functions, routines, subroutines or operations may be performed by one or more processors of a computer device executing the method. In some implementations, one or more of the methods may be performed by a single computing device. Alternatively, one or more of the methods may be performed by two or more computing devices, each computing device performing one or more individual functions, routines, subroutines or operations of the method. To simplify the description, the method of the present disclosure is depicted and described as a series of behaviors. However, the behaviors according to the present disclosure may occur in various orders and / or simultaneously, as well as with other behaviors not presented and described herein. In addition, not all illustrated behaviors may be required to implement the method according to the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the method may be alternatively represented as a series of related states via a state diagram or an event. In addition, it should be appreciated that the methods disclosed in this specification can be stored on an article of manufacture to facilitate the transport and transfer of such methods to a computing device. The term "article of manufacture" as used herein is intended to include a computer program accessible from any computer-readable device or storage medium. In one embodiment, the method may be performed by Figure 1-Figure 4 One or more of the components in the

[0089] refer to Figure 5 , method 500 can be performed by a processing device of a client device (e.g., a smart speaker), a server device (e.g., a cloud service), other devices, or a combination thereof, and can begin at box 502. At box 502, the processing device can determine phoneme data for a text source. The text source can include a word sequence, and the phoneme data can be a phonetic encoding of the word sequence, the phonetic encoding including one or more phonetic value sequences. Each phonetic value can correspond to a phoneme, and the phoneme sequence can correspond to a spoken word. The same phonetic value sequence can correspond to words that sound the same but are spelled differently (e.g., homophones), and different phonetic value sequences can correspond to words that are spelled the same but sound different (e.g., homophones).

[0090] The processing device may access the phoneme data from a source of a text source, or may generate the phoneme data for the text source. The processing device may generate the phoneme data by phonetically encoding a sequence of words. This may involve accessing text data of the text source and generating (e.g., converting, transforming, deriving) the phoneme data based on the text data. The phoneme data may then be associated with the phoneme data for future use.

[0091] At box 504, the processing device can receive audio data including spoken words associated with the text source. The audio data can include one or more audible actions of the user, and can include spoken words, page turning or other audible actions captured from the user's environment. In one example, the processing device can directly receive audio data in the form of an audio signal from one or more sensors. In another example, the processing device can receive audio data from a data storage or another computing device. The audio data can be any digital or analog format, and can be accessed or received via one or more storage objects (e.g., files, database records), data streams (e.g., audio streams, video streams), data signals, other data transmissions or storage protocols, or a combination thereof.

[0092] At box 506, the processing device can compare the phoneme data of the text source with the phoneme data of the audio data. The comparison of the audio data and the text source can occur without using speech recognition to convert the audio data into text (e.g., recognized words), and can involve comparing the phoneme data corresponding to the audio data with the phoneme data corresponding to the text source. The comparison can include calculating a numerical value representing the similarity between two or more speech value sequences. The numerical value can be the phoneme edit distance between the phoneme data of the audio data and the phoneme data of the text source. The comparison can also involve performing a fuzzy match between the phoneme data corresponding to the audio data and the phoneme data of the text source.

[0093] At block 508, the processing device may identify a position in the word sequence based on a comparison of the phoneme data of the text source and the speech data of the audio. The identification of the position may involve determining that the spoken word matches a word in the word sequence of the text source. In one example, the text source may be a book, and the position may be a current reading position in the book. In response to completing the operations described above with reference to block 508, the method may terminate.

[0094] refer to Figure 6 , method 600 may be performed by the same processing device discussed above or a different processing device, and may begin at block 602. At block 602, a processing device may receive audio data including spoken words associated with a text source. The audio data may be segmented (e.g., tokenized, sliced, split, divided) into a first duration and a second duration. In one example, the text source may be a book, and a first portion of the audio data may correspond to the content of the book (e.g., include spoken words of the book), while a second portion of the audio data may not correspond to the content of the book (e.g., may not have spoken words from the book).

[0095] At block 604, the processing device may compare the audio data with the data of the text source. The data of the text source may include phoneme data, and comparing the audio data and the data of the text source may involve phoneme comparison. In one example, comparing the phoneme data may involve calculating a phoneme edit distance between the phoneme data of the text source and the phoneme data of the audio data.

[0096] At box 606, the processing device can calculate a correspondence metric between the second duration of the audio data and the data of the text source. Calculating the correspondence metric can include calculating the correspondence metric based on multiple phoneme edit distances. In one example, the processing device can select a set of spoken words (e.g., 3, 4, 5+ words) and compare the set of spoken words to the content of the text source. The phoneme edit distance can be determined for each word in the set or a combination of one or more words. The resulting values ​​can then be weighted, aggregated, or modified to determine the correspondence metric.

[0097] At box 608, in response to determining that the corresponding metric meets the threshold, the processing device can transmit a signal to stop comparing the audio data with the data of the text source. Determining that the corresponding metric meets the threshold can involve determining that the corresponding metric is below the threshold or above the threshold. The determination can also be based on the duration of the corresponding metric meeting or not meeting the threshold. Determining that the corresponding metric meets the threshold can indicate that the second duration of the audio data includes content different from the content of the text source, and can indicate or not indicate that the audio data does not match the data of the text source. Transmitting a signal can involve transmitting a signal that deactivates one or more microphones that capture audio data. In one example, in response to determining that the second duration of the audio data does not have the content of the text source, the processing device can cause the computing device to prompt the user to exit the story time mode. The prompt can be an audio prompt, a visual prompt, other prompts, or a combination thereof. In response to completing the operation described in reference box 608 above, the method can terminate.

[0098] refer to Figure 7 , method 700 can be performed by the same processing device or different processing devices discussed above, and can start at box 702. At box 702, the processing device can receive audio data including the user's spoken words. The spoken words can be associated with the text source that the user is reading aloud, and can include one or more other audible actions, such as page turning, spoken words that are not in the text source, and other audible actions captured from the user's environment. In one example, the processing device can directly receive audio data from one or more sensors in the form of an audio signal (e.g., for real-time use or real-time perception). In another example, the processing device can receive audio data from a data storage or another computing device. The audio data can be in any digital or analog format and can be accessed or received from one or more storage objects (e.g., files, database records), data streams (e.g., audio streams, video streams), data signals, other data transmission or storage protocols, or a combination thereof.

[0099] At box 704, the processing device can analyze the context data associated with the user. The context data may include sound data, light data, time data, weather data, calendar data, user profile data, other data or a combination thereof. In some examples, the context data may be associated with a physical effect so that the processing device can provide a physical effect that takes into account the context of the user and the user environment. In one example, the context data may include sound data of the user environment, and the physical effect may include an acoustic effect at a certain volume based on the sound data. In another example, the context data may include light data of the user environment, and the physical effect may include an optical effect that modifies the brightness of the light source based on the light data. In yet another example, the context data may include user profile data indicating the age of the child, and the physical effect may include an acoustic effect selected based on the age of the child.

[0100] At block 706, the processing device may determine a match between the audio data and the data of the text source. The processing device may identify the text source based on user input (e.g., the audio data or the touch data) and retrieve the data of the text source. The data of the text source may include phoneme data, and determining the match may involve calculating a phoneme edit distance between the phoneme data of the text source and the phoneme data of the audio data. In one example, determining a match between the audio data and the data of the text source may involve using the phoneme data of the text source to detect the audio data including a word of the text source.

[0101] At box 708, the processing device may initiate a physical effect in response to determining a match. The physical effect may correspond to a text source and be based on context data. The physical effect may modify the user's environment and may include at least one of an acoustic effect, an optical effect, and a tactile effect. The text source may include words, and initiating the physical effect may be in response to detecting that the audio data includes the words. In one example, the processing device may select a physical effect based on the words of the text source, and may update the properties of the physical effect (e.g., volume or brightness) based on the context data. In response to completing the operations described above with reference to box 708, the method may terminate.

[0102] refer to Figure 8 , method 800 can be performed by a processing device of a server device or a client device, and can start at box 802. At box 802, the processing device can identify effect data of a text source, wherein the effect data relates a physical effect to an audible action of a user. The effect data can indicate the physical effect and indicate a position in the text source related to the audible action. The position can correspond to a word, paragraph, page, or other position of the text source. In one example, the audible action can be a spoken word of the text source, and the physical effect can be an acoustic effect corresponding to the spoken word. In another example, the audible action can include turning a page, and the physical effect can be a modification of an existing acoustic effect, an optical effect, or a tactile effect.

[0103] At box 804, the processing device can receive audio data including multiple audible actions. Multiple audible actions can include one or more spoken words and one or more other audible actions of the text source, such as page turning, spoken words not in the text source, and other audible actions captured from the user's environment. In one example, the processing device can directly receive audio data from one or more sensors in the form of an audio signal (e.g., for real-time use or close to / perceived real-time). In another example, the processing device can receive audio data from a data storage or another computing device. The audio data can be in any digital or analog format, and can be accessed or received from one or more storage objects (e.g., files, database records), data streams (e.g., audio streams, video streams), data signals, other data transmissions or storage protocols, or a combination thereof.

[0104] At box 806, the processing device may determine a trigger condition based on the effect data and the text source. In one example, determining the trigger condition may involve determining a physical effect associated with a first position in the text source and selecting a second position in the text source before the first position. The selection may be based on a reading speed and a loading time associated with the physical effect, and the second position may be associated with at least one of a specific instance of a word, paragraph, page, or chapter of the text source. The processing device may then set the trigger condition to correspond to the second position in the text source. In another example, determining the trigger condition may involve calculating a duration for loading content based on the amount of content of the physical effect and the amount of available computing resources. The computing resources may be related to one or more of networking bandwidth, storage space, or processing power, and the duration may be longer when the available computing resources are low. In one example, determining the time when an audible action will occur in the future may involve identifying a time to initiate loading based on the calculated duration and the determined time of the audible action, and initiating loading content at or before the identified time. In another example, determining the time includes calculating a future time based on a reading speed and a current reading position in the text source. In yet another example, determining the time includes predicting a time when a word of the text source will be spoken before the word is spoken.

[0105] At block 808, the processing device may load content for the physical effect in response to satisfying a trigger condition. The trigger condition may be satisfied before an audible action occurs.

[0106] At block 810, the processing device may provide physical effects to modify the user's environment. In response to completing the operations described above with reference to block 810, the method may terminate.

[0107] The technology discussed herein includes a variety of enhancements to computing devices with or without virtual assistant features. The following discussion includes a number of different enhancements that can be used alone or together to optimize the ability of a computing device to follow along as a text source is read aloud, and to provide special effects to supplement the user's environment. In one example, the environment can include a parent reading a book aloud to one or more children. In another example, the environment can include one or more users providing a presentation, voice, or other performance to an audience. In either example, the technology can be used to enhance the environment with special effects based on analysis of data associated with the text source. Special effects can be synchronized with specific portions of the text source, such as specific spoken words or page turns.

[0108] In a first example, enhancement can be related to reading progress estimation based on speech fuzzy matching and confidence intervals, and can be related to the field of computer-based human speech recognition, and in particular to enhancing the ability of the computer device to identify the reading position in the text source when the user reads the text source aloud. When the user reads the text source aloud, when the computing device attempts to follow using a traditional virtual assistant function, many technical problems will occur. Because the traditional virtual assistant function performs speech recognition to convert audio into text / recognized words, some problems arise. Speech recognition generally involves an acoustic step of converting audio into phonemes and a language step of converting phonemes into text / recognized words. Before converting the spoken words into text, the language step usually waits for subsequent spoken words to establish context. The language step introduces unnecessary time delays and consumes additional computing resources. In addition, using recognized text to perform a traditional text-based comparison with a text source may be more error-prone than performing a speech comparison (e.g., a phoneme comparison). This often occurs because many words that sound the same or sound similar may be spelled very differently and produce false negatives when comparing text. In addition, traditional text comparison may not correctly address situations where users may jump around when reading a text source. For example, parts of a text source may be skipped, repeated, or new content may be added. This may make it challenging to identify the current reading position within a text source and correctly detect the reading speed. Aspects and embodiments of the present technology solve the above and other defects by providing enhanced functionality to enable a computing device to detect the current reading position in a text source when reading the text source aloud. In one example, the technology can avoid the language steps of traditional voice recognition by comparing phoneme data derived from audio with phoneme data derived from a text source. The text source can be a book, magazine, presentation, voice, script, or other source containing a sequence of words. The technology can receive audio data including words spoken by a user, and the audio data can be converted to phoneme data locally or through an assistant or a remote server (e.g., a cloud service). The phoneme data of the audio and the text source can then be compared via voice comparison rather than a more traditional text comparison. Voice comparison can be accompanied by fuzzy matching to identify the position (e.g., the current reading position) within a word sequence. The systems and methods described herein include technologies in the technical field of enhancing computer-based human voice recognition. In particular, the disclosed techniques increase the latency, accuracy, and computational resources required to identify the current reading position. This may be the result of modifying a speech analysis process (e.g., speech recognition) to avoid converting audio data into text / words. The techniques may use a speech analysis process that converts audio into phoneme data using an acoustic model, but may avoid the linguistic step of converting the phoneme data into text / words using a language model. Avoiding the linguistic step reduces latency and consumption of computational resources.Performing phoneme comparisons and using fuzzy matching can enhance the accuracy of identifying the current reading position because it can better compensate for non-linear reading of the text source (e.g., skipping, repeating, or adding content).

[0109] In a second example, the enhanced functionality may be related to an algorithmic determination that a story reader has stopped reading, and may be related to the field of computer-based human speech recognition, and in particular to enhancing the ability of a computer device to determine that a user is no longer reading aloud the content of a text source. When a computing device attempts to use a traditional virtual assistant function to follow along when a user is reading a text source aloud, many technical problems may arise. Some problems may arise because a traditional virtual assistant may not be able to detect when a user has completed audio input if the user continues to talk about other things. This may cause the computing device to continue recording the user's audio, which may be problematic if the user transitions to discussing private matters. When a user does not follow the text and skips, repeats, or adds new content while reading a text source aloud, it may be more challenging to detect when the user stops reading from a text source. Aspects and embodiments of the present technology address the above and other deficiencies by enhancing the ability of a computing device to detect when a user interrupts reading a text source. In one example, the technology may enable a virtual assistant to more accurately detect that a user has left reading a text source to take a break, and may deactivate a microphone to avoid capturing private audio content. This may involve receiving audio data including spoken words associated with a text source, and comparing the audio data with data from the text source. The technology can calculate the corresponding metric between the content of the audio data and the content of the text source. The corresponding metric can be a probability value based on the comparison of phoneme data, text data or other data, and can involve the use of fuzzy matching logic. When the corresponding metric meets the threshold (e.g., below the minimum corresponding threshold), the technology can cause the signal to stop the analysis of the subsequent audio data to be transmitted. The system and method described herein include technologies in the technical field of enhancing computer-based human voice recognition. In particular, the technology can solve the technical problem by using comparison to avoid unintentional recording of user private conversations, and the comparison can better compensate for the non-linear reading of the text source (e.g., skipping, repeating, adding content). For example, the above technology can promote more accurate and / or faster automatic control of virtual assistants to record and / or process only relevant audio. The technology can also reduce power consumption of computing devices by deactivating audio sensors (e.g., microphones) and associated data processing when the computing device detects that the user has stopped reading the text. In addition, the above technology can enable computing devices to reduce the utilization of computing resources, such as processing capacity, network bandwidth, data storage, etc., which can otherwise be used to record and / or process audio data once the user has stopped reading the text.

[0110] In a third example, the enhancement may be related to the dynamic adjustment of story time special effects based on contextual data, and may be related to the field of virtual assistants, especially the ability of virtual assistants to provide special effects while reading a text source aloud. Modern computing devices can be configured to adopt traditional virtual assistant features to provide sound effects to supplement the environment when a user reads a book aloud. For example, when a user reads the word "bark" aloud, a computing device can provide a barking effect. Sound effects are usually provided by the same entity that provides the text source and can directly correspond to a portion of the text source. As a result, the special effects may be the same independent of the user or the environment, and may not be optimized for the user's specific reading environment. Various aspects and embodiments of the present technology solve the above and other deficiencies by enabling computing devices to provide a variety of special effects based on the user's environment. In one example, the technology can enable a computing device to analyze the contextual data of the user's environment and select or customize special effects. Special effects can be physical effects that change the user's environment, including acoustic effects (e.g., music, sound effect music), optical effects (e.g., flash, ambient light), tactile effects (e.g., vibration, wind, temperature changes), other effects or combinations thereof. The technology may involve receiving and analyzing contextual data associated with a user. The contextual data may be related to weather, lighting, time of day, user feedback, user profile, other information, or a combination thereof. The technology may select or modify a physical effect corresponding to a text source based on the contextual data. For example, this may result in selecting or modifying volume, brightness, speed, pitch, or other attributes of a physical effect. The systems and methods described herein include technologies in the technical field of enhanced virtual assistants and home automation. In particular, the technology may enable a computing device to optimize an environment by using contextual data about a user and adding, removing, or modifying an environment of a physical effect to enhance the user's listening experience.

[0111] In a fourth example, the enhancement may be related to the detection of the story reader's progress to pre-cache special effects, and may be related to the field of virtual assistants, particularly to the ability of enhancing virtual assistants to pre-cache special effects for text sources that are being read aloud. When trying to use traditional virtual assistant functions to provide sound effects that are synchronized with the voice content of the text source, many technical problems arise. Because traditional virtual assistants perform voice recognition to convert audio into text and then compare based on text, some problems arise. Voice recognition generally involves an acoustic step of converting audio into phonemes and a language step of converting phonemes into text. Before translating spoken words into text, the language step typically waits for subsequent spoken words to establish context. The language step introduces time delays and consumes other computing resources on computing devices that may be resource-constrained. Because the sound effects may be large audio files downloaded from remote data sources, the delay may be further aggravated. Traditional methods may involve downloading sound effects in response to detecting spoken words, but delays may result in providing special effects long after the words are spoken. Another approach may involve downloading all sound effects when the text source is initially identified, but this may be problematic when the computing device is a resource-constrained device (e.g., a smart speaker). Aspects and embodiments of the present technology address the above and other deficiencies by providing enhanced functionality to the computing device to enhance its ability to conserve computing resources and still provide special effects that are synchronized with the text source being read aloud. This can be achieved by using data from a text source (e.g., a book) to predict future audible actions and pre-fetching associated physical effects before the corresponding audible actions occur. In one example, the technology may enable a computing device to predict when a user reaches a word in a text source before speaking the word. This may involve identifying effect data for a text source that relates a physical effect to one or more audible actions of the user. The audible action may include a word spoken by the user, or may be a page turning, closing a book, or other action that generates an audible response. The technology may determine a trigger condition based on the current reading position, reading speed, other data, or a combination thereof. In response to detecting that the trigger condition is met, the technology may enable the computing device to load content for the physical effect and subsequently provide the physical effect to modify the user's environment. The systems and methods described herein include techniques in the technical field of enhancing pre-caching based on recognition of human speech. In particular, the disclosed techniques can solve technical problems related to resource consumption when analyzing speech and downloading special effects. The techniques can also reduce the delay in providing special effects, thereby making the special effects better synchronized with human speech.

[0112] Fig. 9 is a block diagram illustrating a computer system that operates according to one or more aspects of the present disclosure. In various illustrative examples, computing system 900 may correspond to Figure 2-Figure 4 The computing device 130 of the present invention can be included in the data center supporting virtualization. In some embodiments, the computer system 900 can be connected to other computer systems (for example, via a network such as a local area network (LAN), an intranet, an extranet or the Internet). The computer system 900 can operate as a server or client computer in a client-server environment, or operate as a peer computer in a peer-to-peer or distributed network environment. The computer system 900 can be provided by any device of a group of instructions (sequential or other) that can perform a specified operation to be taken by the device. In addition, the term "computer" should include any collection of computers that perform one or more of a group of (or multiple groups) of instructions to perform any method described herein, either individually or collectively.

[0113] In another aspect, the computer system 900 may include a processing device 902, a volatile memory 904 (e.g., random access memory (RAM)), a non-volatile memory 906 (e.g., read only memory (ROM)) or an electrically erasable programmable ROM (EEPROM), and a data storage device 916 that may communicate with each other via a bus 908.

[0114] The processing device 902 may be provided by one or more processors, such as a general-purpose processor (such as, for example, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor), a very long instruction word (VLIW) microprocessor, a microprocessor that implements other types of instruction sets, or a microprocessor that implements a combination of types of instruction sets) or a special-purpose processor (such as, for example, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a network processor).

[0115] The computer system 900 may further include a network interface device 922. The computer system 900 may also include a video display unit 910 (eg, LCD), an alphanumeric input device 912 (eg, keyboard), a cursor control device 914 (eg, mouse), and a signal generating device 920.

[0116] The data storage device 916 may include a non-transitory computer-readable storage medium 924 on which instructions 926 encoding any one or more of the methods or functions described herein may be stored, including instructions for implementing the methods 500, 600, 700, or 800 and Figure 1-Figure 4 A directive for any component or module in .

[0117] The instructions 926 may also reside, completely or partially, within the volatile memory 904 and / or within the processing device 902 during execution thereof by the computer system 900, and thus the volatile memory 904 and the processing device 902 may also constitute machine-readable storage media.

[0118] Although the computer-readable storage medium 924 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" shall include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of executable instructions. The term "computer-readable storage medium" shall also include tangible media that can store or encode a set of instructions to be executed by a computer, and cause the computer to perform any one or more of the methods described herein. The term "computer-readable storage medium" shall include, but is not limited to, solid-state memories, optical media, and magnetic media.

[0119] The methods, components, and features described herein may be implemented by discrete hardware components or may be integrated into the functionality of other hardware components such as ASICs, FPGAs, DSPs, or similar devices. In addition, the methods, components, and features may be implemented by firmware modules or functional circuits within hardware resources. In addition, the methods, components, and features may be implemented by any combination of hardware resources and computer program components or computer programs.

[0120] Unless specifically stated otherwise, terms such as "initiate", "send", "receive", "analyze", etc. refer to operations and processes performed or implemented by a computer system that manipulate and transform data represented as physical (electronic) quantities within computer system registers and memories into other data similarly represented as physical quantities within computer system memories or registers or other such information storage, transmission or display devices. In addition, the terms "first", "second", "third", "fourth", etc. as used herein refer to labels used to distinguish different elements and may not have ordinal meanings according to their numerical names.

[0121] The examples described herein also relate to an apparatus for performing the methods described herein. This apparatus may be specially constructed to perform the methods described herein, or it may include a general-purpose computer system selectively programmed by a computer program stored in the computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0122] The methods and illustrative examples described herein are not inherently related to any particular computer or other apparatus. Various general purpose systems may be used in accordance with the teachings described herein, or it may prove convenient to construct more specialized apparatus to perform each of the methods 500, 600, 700, 800 and / or their individual functions, routines, subroutines, or operations. Examples of the architecture of various such systems are set forth in the description above.

[0123] The above description is intended to be illustrative rather than restrictive. Although the present disclosure has been described with reference to specific illustrative examples and embodiments, it should be appreciated that the present disclosure is not limited to the described examples and embodiments. The scope of the present disclosure should be determined with reference to the attached claims and the full scope of equivalents to which the claims are entitled.< / t>

Claims

1. A method, comprising: receiving audio data including spoken words associated with a text source; comparing the audio data with data from the text source; Based on the comparison, determining, by a processing device, whether the segment of the audio data corresponds to a location of the text source; as well as In response to determining that the segment of the audio data does not correspond to a location of the text source, a signal is transmitted indicating that a user has interrupted reading the text source, the signal causing a cessation of comparing the audio data with the data of the text source.

2. The method according to claim 1, wherein: The text source comprises a book, and wherein the audio data comprises spoken words of the book.

3. The method according to claim 1, further comprising: In response to determining that the segment of the audio data does not correspond to a location of the text source, the user is prompted to exit storytime mode.

4. The method according to claim 1, wherein: Transmitting the signal further includes transmitting the signal to deactivate one or more microphones capturing the audio data.

5. The method according to claim 1, wherein: The data of the text source includes phoneme data, and wherein comparing the audio data includes calculating a phoneme edit distance between the phoneme data of the text source and the phoneme data of the audio data.

6. The method according to claim 5, wherein: Determining whether the segment of the audio data corresponds to a location of the text source includes calculating a correspondence metric based on a plurality of phoneme edit distances.

7. The method according to claim 6, wherein: Determining that the segment of the audio data does not correspond to a location of the text source includes determining that the correspondence metric is below or above a threshold for a threshold duration.

8. The method according to claim 1, wherein: Determining that the segment of the audio data does not correspond to a location of the text source indicates that the segment of the audio data includes content that is different than content of the text source.

9. A system, comprising: a memory storing instructions; as well as a processing device coupled to the memory, wherein the instructions, when executed by the processing device, cause the processing device to perform operations comprising: receiving audio data including spoken words associated with a text source; comparing the audio data with data from the text source; Based on the comparison, determining whether the segment of the audio data corresponds to the location of the text source; and In response to determining that the segment of the audio data does not correspond to a location of the text source, a signal is transmitted indicating that a user has interrupted reading the text source, the signal causing a cessation of comparing the audio data with the data of the text source.

10. The system according to claim 9, wherein: Transmitting the signal further includes transmitting the signal to deactivate one or more microphones capturing the audio data.

11. The system according to claim 9, wherein: The data of the text source includes phoneme data, and wherein comparing the audio data includes calculating a phoneme edit distance between the phoneme data of the text source and the phoneme data of the audio data.

12. The system according to claim 11, wherein: Determining whether the segment of the audio data corresponds to a location of the text source includes calculating a correspondence metric based on a plurality of phoneme edit distances.

13. The system according to claim 12, wherein: Determining that the segment of the audio data does not correspond to a location of the text source includes determining that the correspondence metric is below or above a threshold for a threshold duration.

14. The system according to claim 9, wherein: Determining that the segment of the audio data does not correspond to a location of the text source indicates that the segment of the audio data includes content that is different than content of the text source.

15. A non-transitory computer-readable medium storing instructions that, when executed by a processing device, cause the processing device to perform operations comprising: receiving audio data including spoken words associated with a text source; comparing the audio data with data from the text source; determining, based on the comparing, whether the segment of the audio data corresponds to a location of the text source; as well as In response to determining that the segment of the audio data does not correspond to a location of the text source, a signal is transmitted indicating that a user has interrupted reading the text source, the signal causing a cessation of comparing the audio data with the data of the text source.

16. The non-transitory computer readable medium of claim 15, wherein: Transmitting the signal further includes transmitting the signal to deactivate one or more microphones capturing the audio data.

17. The non-transitory computer readable medium of claim 16, wherein: The data of the text source includes phoneme data, and wherein comparing the audio data includes calculating a phoneme edit distance between the phoneme data of the text source and the phoneme data of the audio data.

18. The non-transitory computer readable medium of claim 17, wherein: Determining whether the segment of the audio data corresponds to a location of the text source includes calculating a correspondence metric based on a plurality of phoneme edit distances.

19. The non-transitory computer readable medium of claim 18, wherein: Determining that the segment of the audio data does not correspond to a location of the text source includes determining that the correspondence metric is below or above a threshold for a threshold duration.

20. The non-transitory computer readable medium of claim 15, wherein: Determining that the segment of the audio data does not correspond to a location of the text source indicates that the segment of the audio data includes content that is different than content of the text source.

Images