Detection of story reader progress for pre-caching special effects

Abstract

This disclosure relates to an algorithm for determining reader interruptions in a story. Techniques are provided to enhance the ability of computing devices to detect when a user has interrupted reading a text source. An example method includes: receiving audio data comprising spoken words associated with a text source, wherein the audio data includes a first duration and a second duration; comparing the audio data with data from a text source, wherein the first duration of the audio data corresponds to the data from the text source; calculating a correspondence metric between the second duration of the audio data and the data from the text source by a processing device; and transmitting a signal to stop comparing the audio data with the data from the text source in response to determining that the correspondence metric meets a threshold.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 201880096938.4, filed on August 27, 2018. Technical Field

[0003] This disclosure relates to the field of computer-based human speech recognition, and more particularly to enhancing the ability of computer devices to determine when a user is no longer reading the content of a text source aloud. Background Technology

[0004] The capabilities and uses of virtual assistants are rapidly expanding. Traditional virtual assistants include some form of computer human-computer interface that enables humans to interact with the virtual assistant and have it perform tasks or services. Virtual assistants typically record and understand human voice and can respond through synthesized responses. Virtual assistants can be initiated in response to touch- or gesture-based commands, or they can continuously analyze their environment to detect verbal commands. When a command is detected, the virtual assistant can respond or perform one or more actions. Summary of the Invention

[0005] The following is a simplified summary of this disclosure to provide a basic understanding of some aspects of this disclosure. This summary is not a broad overview of this disclosure. It is not intended to identify key or defining elements of this disclosure, nor is it intended to define any scope of any particular embodiment of this disclosure or any scope of the claims. Its sole purpose is to present some of the concepts of this disclosure in a simplified form as a prelude to the more detailed description that follows.

[0006] According to a first aspect of this disclosure, a method is provided, comprising: receiving audio data including spoken words associated with a text source, wherein the audio data includes a first duration and a second duration; comparing the audio data with data from the text source, wherein the first duration of the audio data corresponds to the data from the text source; calculating a correspondence metric between the second duration of the audio data and the data from the text source by a processing device; and transmitting a signal to stop comparing the audio data with the data from the text source in response to determining that the correspondence metric satisfies a threshold.

[0007] The text source may include a book, and the first duration of the audio data includes spoken words from the book. The method may further include: prompting a user to exit Story Time mode in response to determining that the second duration of the audio data does not contain content from the text source. Sending the signal may further include sending a signal to deactivate one or more microphones capturing the audio data. The data from the text source may include phoneme data. Comparing the audio data may include calculating a phoneme edit distance between the phoneme data of the text source and the phoneme data of the audio data. Calculating the correspondence metric between the second duration of the audio data and the data from the text source may include calculating the correspondence metric based on multiple phoneme edit distances. Determining that the correspondence metric meets a threshold may include determining that the correspondence metric is below or above a threshold duration. Determining that the correspondence metric meets a threshold may indicate that the second duration of the audio data includes content different from the content of the text source.

[0008] According to a second aspect of this disclosure, a system includes a processing device configured to: receive audio data including spoken words associated with a text source, wherein the audio data includes a first duration and a second duration; compare the audio data with data from the text source, wherein the first duration of the audio data corresponds to the data from the text source; calculate a correspondence metric between the second duration of the audio data and the data from the text source; and, in response to determining that the correspondence metric satisfies a threshold, transmit a signal to stop comparing the audio data with the data from the text source.

[0009] The system may include data storage. The system may include a communication system for communicating over networks such as local area networks (LANs) and / or wide area networks (WANs). The system may be included in or configured to implement a virtual assistant. The system may be configured to implement the method of the first aspect.

[0010] According to a third aspect of this disclosure, a computer program product is configured such that, when processed by a processing device, the computer program product causes the processing device to perform the method of the first aspect.

[0011] Individual features and / or combinations of features defined above or below with respect to any particular embodiment according to any aspect of this disclosure may be used individually, alone, or in combination with any other defined features in any other aspect or implementation. Furthermore, this disclosure is intended to cover apparatuses configured to perform methods relating to and / or methods of using, producing, employing, or manufacturing any of the features described herein. Attached Figure Description

[0012] The present disclosure is illustrated by way of example and not limitation in the accompanying drawings.

[0013] Figure 1 The illustration shows an example environment having one or more computing devices according to an embodiment of the present disclosure.

[0014] Figure 2 This is a block diagram illustrating an example computing device having components and modules according to an embodiment of the present disclosure, the components and modules being used to compare phoneme data derived from user input and phoneme data derived from a text source.

[0015] Figure 3 This is a block diagram illustrating an example computing device having components and modules according to an embodiment of the present invention, the components and modules being used to identify locations in a text source based on a user's audio input.

[0016] Figure 4 This is a block diagram illustrating an example computing device having components and modules according to an embodiment of the present invention, which are used to provide physical effects to enhance the experience of one or more users.

[0017] Figure 5 This is a flowchart illustrating an example of a method according to an embodiment of the present disclosure.

[0018] Figure 6 This is a flowchart illustrating an example of a method according to an embodiment of the present disclosure.

[0019] Figure 7 This is a flowchart illustrating an example of a method according to an embodiment of the present disclosure.

[0020] Figure 8 This is a flowchart illustrating an example of a method according to an embodiment of the present disclosure.

[0021] Figure 9 This is a block diagram illustrating another example of a computing device according to an embodiment of the present disclosure.

[0022] These figures can be better understood when viewed in conjunction with the following detailed description. Detailed Implementation

[0023] Modern computing devices typically offer the ability to detect and understand the characteristics of human speech. These characteristics can be associated with virtual assistants, accessible via resource-constrained computing devices such as smart speakers, mobile phones, smartwatches, or other user devices. The computing device can be associated with a microphone capable of recording human speech and analyzing it using a combination of local and remote computing resources. Speech analysis is often a resource-intensive operation, and the computing device can be configured to perform some processing locally and some remotely at a server or via a cloud service. Many virtual assistants use some form of remote speech recognition service that takes audio data as input and converts it into text, which is then returned to the computing device.

[0024] Several technical issues arise when computing devices attempt to follow traditional virtual assistant features while a user reads a text source aloud. Problems arise because traditional virtual assistants may fail to detect when a user has finished providing audio input (e.g., when the user continues talking about something else). This can potentially lead to unnecessarily utilizing the virtual assistant's computing resources, such as processing memory, data storage, and / or network bandwidth, which can be consumed in other ways if the virtual assistant continues to follow the user's text reading after the user has finished reading. Additionally or alternatively, this may cause the computing device to continue recording and / or processing the user's audio, which could be problematic if the user shifts to discussing personal matters. Detecting when a user has stopped reading from a text source may be even more challenging when the user does not follow the text and skips, repeats, or adds new content while reading a text source aloud.

[0025] Various aspects and implementations of this technology address the aforementioned and other shortcomings by enhancing the computing device's ability to detect when a user has interrupted reading a text source. In one example, the technology could enable a virtual assistant to more accurately detect when a user has left the text source to take a break and could deactivate the microphone to avoid capturing private audio content. This might involve receiving audio data including spoken words associated with the text source and comparing the audio data with the data from the text source. The technology could calculate a correspondence metric between the content of the audio data and the content of the text source. The correspondence metric could be a probability value based on the comparison of phoneme data, text data, or other data, and could involve using fuzzy matching logic. When the correspondence metric meets a threshold (e.g., below a minimum correspondence threshold), the technology could be used to transmit a signal to stop the analysis of subsequent audio data.

[0026] The systems and methods described herein include techniques for enhancing computer-based human speech recognition. In particular, these techniques may address technical problems such as avoiding unintentional recording of a user's private conversations by using comparative methods that better compensate for non-linear reading of the text source (e.g., skipping, repeating, adding content). The techniques may also enable computing devices to reduce power and / or other computational resource consumption by disabling audio sensors (e.g., microphones) and associated data processing when the computing device detects that the user has stopped reading text.

[0027] The techniques discussed below include several enhancements to computing devices, with or without virtual assistant features. These enhancements can be used individually or in combination to optimize the computing device's ability to follow a text source being read aloud and to provide special effects to complement the listening user's environment. In one example, the environment may include a parent reading a book aloud to one or more children. In another example, the environment may include one or more users providing a presentation, voice, or other performance to an audience. In either example, the technique can be used to enhance the environment with special effects based on analysis of data associated with the text source. These special effects may be synchronized with specific portions of the text source, such as specific spoken words or page turns.

[0028] Figure 1 The illustration depicts an exemplary environment 100 according to one or more aspects of this disclosure. This exemplary environment 100 includes a text source being read aloud and one or more devices that complement the environment to enhance the user's listening experience. Environment 100 can be a physical environment, such as an indoor setting (e.g., a bedroom, a conference room), an outdoor setting (a park, the wilderness), or other locations. Environment 100 may be referred to as a ubiquitous computing environment or a pervasive computing environment and may include embedded computing functionality. Embedded computing functionality can provide environmental intelligence that is sensitive to and responsive to the presence of humans. In one example, environment 100 may include one or more users 110A and 110B, a text source 120, one or more computing devices 130A and 130B, and one or more physical effects devices 140A-C.

[0029] Users 110A and 110B may include human users capable of perceiving the content of a text source. User 110A may be an individual user reading the content of a text source, or may be multiple users all reading a portion of one or more text sources. User 110A may be referred to as a reader, presenter, announcer, actor, other terms, or combinations thereof. User 110B may listen to the content of a text source being read aloud. User 110B may or may not read alongside User 110A. In one example, User 110A may be a parent reading to a child user 110B. In another example, User 110A may include one or more presenters speaking to one or more users 110B as members of the audience. In either example, the content of text source 120 may be announced for one or more other users to listen to.

[0030] Text source 120 can be any content source that can be interpreted and read aloud. Text source 120 can include content containing numbers, characters, words, symbols, images, or combinations thereof. The content can be arranged into a sequence that can be spoken by a user while reading or after storage. Text source 120 can be a physical book or ebook, magazine, presentation, voice, script, play, memo, announcement, article, blog, post, message, other text arrangement, or a combination thereof. Figure 1 In the example, text source 120 could be a children's book that includes a series of words and images that can be read aloud to a child.

[0031] The audible action 112A-C can be any action or combination of actions that produces sound that can be detected by a user or computing device. The user's ear or an audio sensor (e.g., a microphone) associated with the computing device can hear, perceive, or observe the audible action 112A-C. Figure 1 As shown, there may be multiple types of audible actions, and they may depend on the source of the sound. Audible action 112A may be a first type of audible action, which includes spoken words (e.g., speech) that may originate from human speech or computer-synthesized speech. Spoken words may be verbal spoken words (e.g., words spoken), non-verbal spoken words (e.g., laughter, crying, coughing), other sounds, or combinations thereof. Audible action 112B may be a second type of audible action, which includes non-vocal sounds originating from the user or another source, and may include clapping, finger flicking, other sounds, or combinations thereof. Audible action 112C may be a third type of audible action, which includes non-vocal sounds caused by user interaction with an object, and may include page turning, book closing, door opening / closing, object dropping, tapping the floor, other sounds, or combinations thereof. One or more audible actions 112A-C may be detected by one or more sensors 131A-C.

[0032] Sensors 131A-C can be coupled to computing device 130A and enable the computing device to sense various aspects of environment 100. Sensors 131A-C may include one or more audio sensors (e.g., microphones), optical sensors (e.g., ambient light sensors, cameras), atmospheric sensors (e.g., thermometers, barometers, hydrometers), motion sensors (e.g., accelerometers, gyroscopes, etc.), position sensors (e.g., Global Positioning System (GPS) sensors), proximity sensors, other sensing devices, or combinations thereof. Figure 1 In the example shown, sensor 131A may be an audio sensor, sensor 131B may be an optical sensor, and sensor 131C may be a temperature sensor. One or more of sensors 131A-C may be inside computing device 130A, outside computing device 130A, or a combination thereof, and may be located via wired or wireless connections (e.g., WiFi) is coupled to computing device 130A.

[0033] Computing device 130A can be any computing device capable of receiving and processing data derived from sensors 131A-C. Computing device 130A can be used as a voice command device and provide access to an integrated virtual assistant. In one example, computing device 130A may include a smart speaker, mobile device (e.g., a phone, tablet), wearable device (e.g., a smartwatch), digital media player (e.g., a smart TV, a miniature console, a set-top box), personal computer (e.g., a laptop, desktop, workstation), home automation device, other computing devices, or combinations thereof. In some implementations, computing device 130A may also be referred to as a "user device," "consumer device," or "client device." Data generated by sensors 131A-C can be received by computing device 130A and can be processed locally by computing device 130A, or can be remotely transmitted from computing device 130A to another computing device (e.g., 130B).

[0034] The computing device 130A may include one or more components for processing sensor data. Figure 1 In the example shown, computing device 130A may include an audio analysis component 132, a text source analysis component 133, a comparison component 134, a non-linear reading recognition component 135, a physical effects determination component 136, a prediction loading component 137, and an effects providing component 138. In other examples, one or more of these components, or features of one or more of these components, may be performed by another computing device (e.g., computing device 130B). These components will be related to... Figures 2 to 4 This can be discussed in more detail, and can be used to detect the current reading position and instruct one or more physical effects devices 140A-C to enhance the listening experience.

[0035] Physical effect devices 140A-C can be any computing device capable of causing or providing physical effects. Physical effects can be perceived via the senses of users 110A and 110B (e.g., hearing, sight, touch, smell, and taste). Each of the physical effect devices 140A-C can produce one or more physical effects, and computing device 130A can be used as one or more of the physical effect devices 140A-C. Physical effect devices 140A-C can provide physical effect 145, or can instruct another device to provide physical effect 145. In one example, one or more physical effect devices 140A-C can be part of or integrated with a home automation system, or can be separate from the home automation system. Figure 1 As shown, physical effects device 130A may include a loudspeaker or other device capable of causing or emitting acoustic effects. Physical effects device 130B may include one or more light sources (e.g., light bulbs, pixels) or other devices capable of altering the amount of light present in environment 100 (e.g., motorized curtains or blinds). Physical effects device 130C may include one or more devices capable of causing tactile effects and may include vibration sources (e.g., massage chairs), fans that generate wind (e.g., ceiling fans or air conditioners), heating or cooling sources (e.g., thermostats), other devices, or combinations thereof.

[0036] Physical effect 145 can be any modification to environment 100 that is perceptible to a user or computing device, and can include acoustic effects, tactile effects, optical effects, other effects, or combinations thereof. Acoustic effects can be physical effects related to sound and can propagate via sound waves. Sound effects can include human or animal sounds (e.g., speech or noise), atmospheric sounds (e.g., thunder, rain, wind, or other weather sounds), musical sounds (e.g., instrumental music, background music, theme music), object sounds (e.g., knocking, opening a door, closing a window, breaking glass, object impact, car movement), other sound effects, or combinations thereof. Tactile effects can be physical effects related to a user's sense of touch. Tactile effects can include breezes, vibrations, temperature changes, other tactile sensations, or combinations thereof. Optical effects can be physical effects related to light and can propagate via visible electromagnetic radiation. Optical effects can include ambient lighting, flashlights, animations, other changes in light intensity, or combinations thereof, resulting in an increase or decrease. Optical effects may arise from lamps (e.g., ceiling lights, table lamps), flashlights (e.g., telephone lights), curtains (e.g., blinds or drapes), projectors, electronic displays, holographic displays, lasers, other light sources, or combinations thereof. Other effects may include those related to smell or taste (e.g., olfactory effects).

[0037] Computing device 130B may be a server coupled to computing device 130A and may be located locally or remotely from environment 100. Computing device 130B may include one or more computing devices (such as rack servers, server computers, personal computers, mainframes, laptop computers, tablet computers, desktop computers, routers, etc.), data storage (e.g., hard disks, storage, databases), networks, software components, and / or hardware components. In one example, computing device 130B may be used to provide remote processing and may be used as a voice processing service, such as regarding... Figure 2 This will be discussed in more detail. In another example, computing device 130B can provide computing device 130A with access to media items.

[0038] Media items can correspond to physical effects, text sources, profile information, voice models, instructions, other data, or combinations thereof. Examples of media items include, but are not limited to, digital sound effects, digital music, digital animation, social media messages, e-books, e-magazines, digital newspapers, digital audiobooks, digital videos, digital photos, website content, e-journals, web blogs, True Simple Aggregator (RSS) feeds, e-comic books, software applications, etc. In some implementations, media items may be referred to as content items and may be provided via the Internet and / or via computing device 130A (e.g., a smart speaker). As used herein, “media,” “media item,” “digital media,” “digital media item,” “content,” and “content item” can include electronic files or records that can be loaded or executed using software, firmware, or hardware configured to present content to one or more users in environment 100. In one implementation, computing device 130B may use one or more data stores to store media items and provide media items to computing device 130A via network 150.

[0039] Network 150 may include one or more of the following: a private network (e.g., a local area network (LAN), a public network (e.g., the Internet, a wide area network (WAN)), a wired network (e.g., Ethernet), a wireless network (e.g., Wi-Fi or Bluetooth connectivity), a cellular network (e.g., a long-term evolution (LTE) network), a router, a hub, a switch, a server computer, and / or combinations thereof.

[0040] Typically, functions described in one embodiment as being performed by computing device 130A, computing device 130B, or physical effects devices 140A-C can be performed by one or more other devices in other embodiments. Furthermore, functionality attributable to specific components can be performed by different or multiple components operating together. Computing devices 130A and 130B can also be accessed as services provided to other systems or devices via appropriate application programming interfaces. While embodiments of this disclosure have been discussed in relation to smart speakers, these embodiments can also incorporate one or more features of cloud services or content sharing platforms.

[0041] In the context of systems discussed in this paper that collect or utilize personal information about client devices or users, users can be given the opportunity to control whether computing devices can collect user information (e.g., information about the user's audio input, preferences, current location, social networks, social behavior, activities, or professions) or to control whether and / or how more relevant content is received from the computing device. Furthermore, certain data can be processed in one or more ways before it is stored or used, thereby removing personally identifiable information. For example, user identity can be processed to the point that personally identifiable information cannot be determined for that user, or, if location information is available, the user's geographic location can be generalized (e.g., to the city, zip code, or state level), making it impossible to determine the user's specific location. Therefore, users can control how information about themselves is collected and how it is used by computing devices.

[0042] Figure 2-4 The diagram illustrates a block diagram of an exemplary computing device 130 according to one or more aspects of this disclosure. This exemplary computing device 130 can detect reading positions within a text source and supplement the environment with physical effects to enhance the listening experience. The computing device 130 may be the same as or similar to computing devices 130A, 130B, or combinations thereof. Figure 2 Features that enable computing device 130 to receive user audio data and compare it with data from a text source are discussed. Figure 3 The ability of computing device 130 to analyze data based on audio data and text source data to detect features of reading location was discussed. Figure 4 Features that enable computing device 130 to provide physical effects to modify the environment of one or more listeners are discussed. Figures 2-4 The components and modules provided are exemplary and may include more or fewer components or modules without loss of generality. For example, two or more components may be combined into a single component, or the features of a component may be divided into two or more components. In one implementation, one or more components may reside on different computing devices (e.g., client devices and server devices).

[0043] refer to Figure 2 The computing device 130 may include an audio analysis component 132, a text source analysis component 133, a comparison component 134, and a data storage 240. The audio analysis component 132 can receive and access audio data extracted from the environment when a user reads a text source aloud. In one example, the audio analysis component 132 may include an audio data receiving module 212 and an acoustic modeling module 214.

[0044] The audio data receiving module 212 can receive audio data 241 including one or more audible actions of the user. The audio data may include spoken words, page turning, or other audible actions captured from the user's environment. The audio data 241 may be received directly from one or more sensors in the form of an audio signal, or indirectly from a data storage 240 or other computing device after the sensor stores the audio data 241. The audio data 241 may be in any digital or analog format and may be accessed or received from one or more storage objects (e.g., files, database records), data streams (e.g., audio streams, video streams), data signals, other data transmission or storage protocols, or combinations thereof. The audio data 241 may be an audio recording and may be segmented into one or more durations (e.g., sections, blocks, or other units) before, during, or after analysis by the acoustic modeling module 214.

[0045] The acoustic modeling module 214 can use an acoustic model to analyze audio data 241 to identify phoneme data 243A. The acoustic model can represent a known relationship between audible actions and phonemes. A phoneme can be a unit of sound and can correspond to a sound pattern of an audible action (e.g., spoken words). A phoneme can be a linguistic unit, a non-linguistic unit, other units, or a combination thereof. The acoustic modeling module 214 can convert the audio data into phonemes, which are stored as phoneme data 243A in the data storage device 240.

[0046] Phoneme data 243A may include values ​​representing one or more phonemes extracted from audio data 241. Phoneme data 243A may use standard or proprietary notation to represent a series of phonemes. This notation may include a specific arrangement of one or more bits, bytes, symbols, or characters representing a phoneme. In one example, the specific arrangement may include symbols placed next to or between one or more delimiters. Delimiters may include forward slashes, square brackets, vertical bars, parentheses, commas, tabs, spaces, newlines, other delimiters, or combinations thereof. Phonemes may be arranged into a series of phonemes representing part of one or more audible actions.

[0047] The text source analysis component 133 can receive and analyze data related to the text source 120. The text source 120 can be determined based on user input, which is text-based, voice-based, touch-based, gesture-based, or other user input methods. For example, a user can identify the text source 120 by speaking its name (e.g., the title or author of a book), by typing and searching for the text source, by selecting a displayed text source, other selection mechanisms, or combinations thereof. Figure 2 In the example shown, the text source analysis component 133 may include a data access module 222 and a phoneme determination module 224.

[0048] Data access module 222 can access data associated with text source 120 and can store the accessed data as text source data 242. Data access module 222 can access data from one or more sources, which may include local sources, remote sources, or combinations thereof. A local source may be the storage of computing device 130, while a remote source may be the storage of a computing device accessible via a network connection. In one example, a remote source may be the same as or similar to computing device 130B (e.g., a server or cloud service). A local or remote source may store data for one or more media items discussed above, and the computing source can access that data. The data can then be analyzed, filtered, combined, or modified, and subsequently stored as text source data 242.

[0049] Text source data 242 can be any data associated with text source 120 and can be provided by or accessed from authors, publishers, distributors, partners, remote servers, third-party services, other sources, or combinations thereof. Text source data 242 can include descriptive data, text data, phoneme data, other data, or combinations thereof. Descriptive data can indicate titles, summaries, sources (e.g., authors, publishers, distributors), tables of contents (e.g., chapters, sections, pages), indexes (e.g., phrases, page indicators), other data, or combinations thereof.

[0050] Text data may include one or more words from text source 120. In one example, these words may be organized into a word sequence 122 with or without one or more images 124. Text data may be a data structure that arranges words in the same or similar way as a user reads them (e.g., a series of consecutive words). The word sequence may be limited to words appearing in text source 120, or it may be supplemented with words or data indicating the presence or content of non-textual information (e.g., illustrations, images, tables, formatting, paragraphs, pages). In another example, words may also be arranged, or alternatively, in an indexed data structure that indicates unique words present in text source 120 but not arranged consecutively as spoken by the user. Any data structure may be supplemented with other information, which may include word position within the text source (e.g., page, line, slide), frequency of occurrence, word variants (e.g., tense, plural), other data, or combinations thereof. In one example, text source 120 may be a physical book, and text source data 242 may include words from a corresponding e-book (e.g., an e-book), a third-party service, other sources, or a combination thereof.

[0051] The phoneme data of text source 120 may be the same as or similar to phoneme data 243B, and may be a phoneme encoding of text source 120, the format of which is the same as or similar to the phoneme data derived from the audio (e.g., phoneme data 243A). In the example discussed above, phoneme data 243B of text source 120 may be included as part of text source data 242 and accessed by phoneme determination module 224. In another example, text source data 242 may not contain phoneme data 243B, and may be generated by phoneme determination module 224.

[0052] Phoneme determination module 224 can determine phoneme data for a specific text source 120. This may involve phoneme determination module 224 accessing existing phoneme data 243B from a remote source, generating phoneme data 243B based on text data, or a combination thereof. When generating phoneme data 243B, phoneme determination module 224 can access and analyze the text data of text source data 242 and transform (e.g., derive, convert, transform, encode) the text data into phoneme data 243B. The generated phoneme data can then be associated with text source 120 for future use by computing device 130 or one or more other computing devices. In one example, text data may include a sequence of words, and the generated phoneme data may include a speech code comprising a sequence of speech values ​​representing the sequence of words. The same sequence of speech values ​​may correspond to two words that sound the same but are spelled differently (e.g., homophones). Similarly, even if spelled the same, different sequences of speech values ​​may correspond to words that sound different (e.g., homographs).

[0053] As described above, phoneme data 243A and 243B may both include phoneme sequences represented using standard or proprietary notation. This notation may be referred to as speech transcription or phoneme transcription and may include a specific arrangement of phoneme values ​​representing speech segments. Speech segments may be any discrete unit that can be physically or audibly identified in a speech stream. Phoneme values ​​may include one or more symbols, characters, bytes, bits, other values, or combinations thereof. In one example, phoneme values ​​may be represented by one or more Unicode characters, US Standard codes for Information Interchange (ASCII) characters, other characters, or combinations thereof. A sequence of phoneme values ​​may represent a single word, and each individual phoneme value may represent a portion of that word. For example, the first phoneme sequence may be / θΛm / and represent the spoken word “thumb,” while the second phoneme sequence may be / dΛm / and represent the spoken word “dumb.” In the examples discussed below, phoneme data 243A and 243B may include sequences of values, and each value may represent a phoneme of a phoneme word.

[0054] A phoneme vocabulary can include a set of possible phoneme values ​​for one or more languages. A phoneme vocabulary can be a phonetic alphabet system and can represent a portion of the speech quality of a spoken language: phonology, phonemes, intonation, and the separation of words and syllables. A phoneme vocabulary may or may not represent additional qualities of speech and variations of speech cues (e.g., lisp, mispronunciation, stress, dialect). A phoneme vocabulary can be the same as or similar to a phonemic alphabet, character set, vocabulary, dictionary, other variants, or combinations thereof. In one example, a phoneme vocabulary can be based on the International Phonetic Alphabet (IPA). IPA symbols can consist of one or more elements related to letters and diacritics. For example, the English alphabet. <t>The sound can be represented in IPA using the single letter [t] or by a letter plus an diacritic. Transcription. Delimiters (e.g., forward slashes) can be used to signal extensive or phonemic transcription; therefore, depending on the context and language, / t / may be less important. [t] is specific and can refer to. Or [t]. In other examples, phoneme words may be the same as or similar to Extended Speech Evaluation Method phonetic alphabet (X-SAMPA), Kirshenbaum (e.g., ASCII-IPA, erkIPA), other phoneme words, or combinations thereof.

[0055] Comparison component 134 can compare the audio of user 110A with the content of text source 120. The examples discussed below use phoneme data corresponding to the audio and text sources, and compare them without using speech recognition to convert the audio to text. Other examples may also use text data, descriptive data, audio data, other data, or combinations thereof. The comparison can be performed by computing device 130, by a remote computing device (e.g., a cloud service), or a combination thereof. In one example, comparison component 134 can select a phoneme sequence derived from the audio and compare it with multiple phoneme sequences derived from the text source. In another example, comparison component 134 can compare a phoneme sequence from the text source with multiple phoneme sequences derived from the audio. In either example, the similarity measurement data can be calculated based on phoneme edit distance.

[0056] The phoneme edit distance module 232 can quantify the similarity between two phoneme sequences by determining the minimum number of operations required to transform one phoneme sequence into an exact match of another. This operation can include any modification to the phoneme values ​​(e.g., symbols) within one of the phoneme sequences. Example operations can include primitive operations such as phoneme removal, insertion, substitution, transposition, other operations, or combinations thereof. In the example discussed above, the first phoneme sequence could be / θΛm / and represent "thumb," while the second phoneme sequence could be / dΛm / and represent "dumb." Although the two words differ by two letters, their phoneme edit distance is a value of 1 because transforming the sequences into an exact match involves the substitution of a single phoneme (e.g., replacing θ with d). In one example, the phoneme edit distance can be a linear edit distance that is the same as or similar to the Levenshtein distance. The Levenshtein distance can be based on the minimum number of removal, insertion, or substitution operations required to make two phoneme sequences equal. In other examples, the phoneme edit distance may also, or alternatively, include transposition or other operations. In either example, the phoneme edit distance can be a numerical value used to determine the similarity measurement data 244.

[0057] The similarity measurement module 234 can access data from the phoneme edit distance module to determine the similarity or dissimilarity between audio and text sources. The similarity measurement module 234 can analyze the data from the phoneme edit distance module to calculate similarity measurement data 244. Similarity measurement data 244 can represent the similarity between two or more phoneme sequences (e.g., phonetic representations of words or word sets) and can include numerical data, non-numerical data, other data, or combinations thereof. Similarity measurement data 244 can be based on the edit distance of one or more phoneme sequences. In one example, similarity measurement data 244 can include a numerical value of the phoneme edit distance. In another example, similarity measurement data 244 can include probability values ​​derived from the numerical value of the phoneme edit distance. For example, similarity measurement data can be a percentage, ratio, or other value based on one or more phoneme edit distances and one or more other values. Other values ​​can be the number of phonemes in one or more phoneme sequences or portions of a text source.

[0058] Data storage 240 may be a memory (e.g., random access memory), a cache, a drive (e.g., a solid-state drive, hard disk drive, flash drive), a database system, or another type of component or device capable of storing data. Data storage 240 may also include multiple storage components (e.g., multiple drives or multiple databases) that can span one or more computing devices (e.g., multiple server computers).

[0059] Figure 3 A block diagram illustrating an exemplary component is provided, which enables computing device 130 to analyze the data discussed above to determine the reading position or absence of a reading position within the text source. As discussed above, portions of the audio may not perfectly match the text source because the user may add, skip, repeat, or reorder content from the text source while reading aloud. Consequently, the phoneme data derived from the audio and the phoneme data derived from the text source may be challenging for comparison and alignment. Figure 1 In the example shown, computing device 130 may include a non-linear reading recognition component 135, which enables the computing device to determine the position within the text source that is best aligned with the audio data. In one example, the non-linear reading recognition component 135 may include a fuzzy matching module 352, a position identification module 354, a reading speed module 356, and a reading interruption module 358.

[0060] The fuzzy matching module 352 enables the computing device 130 to determine whether a match exists between the audio and text sources. This match can be the same as or similar to a probabilistic match, a best match, a closest match, or any match that is not an exact match but meets a predetermined threshold. In one example, determining a match between the audio and text sources may involve detecting that an audio segment includes one or more words from the text source. A match may be detected even if the audio or text source contains other words, missing words, or variations of words (e.g., mispronunciation, missing plural form). This match can be referred to as a fuzzy match or an approximate match and can be detected using fuzzy matching logic. Fuzzy matching logic can be used to compare sequences of phoneme values ​​and can operate at syllable-level segments, word-level segments, phrase-level segments, sentence-level segments, other segments, or combinations thereof. In one example, fuzzy matching can be performed using an audio segment of a predetermined length. The predetermined length can be customizable and can be any duration (e.g., 3+ seconds) or any number of word tokens (e.g., 3-4 words). When considering non-linear reading, a predetermined length that is much smaller than the length of the text source can enhance accuracy and performance.

[0061] The fuzzy matching module 352 can impose one or more constraints to determine a match. In one example, detecting a match might involve using one or more global unweighted costs. The global unweighted cost might relate to the total number of original operations required to transform a sequence of candidate phonemes (e.g., a candidate pattern from a text source) into a selected sequence of phonemes (e.g., a pattern from audio). In another example, detecting a match could involve specifying the number of operations for each type separately, while other operations set the total cost, but allowing different weights to be assigned to different original operations. The fuzzy matching module 352 can also apply separate assignments of constraints and weights to individual phoneme values ​​in the sequence.

[0062] The location identification module 354 can access data from the fuzzy matching module 352 to identify locations within a text source corresponding to audible actions in audio (e.g., spoken words). In one example, the text source could be a children's book, and the location could be a reading position within a word sequence of that book. In other examples, the location could be speech, a presentation, a script, a play, other text sources, or combinations thereof. In any example, the location could be a past, present, or future reading position within the text source and could be stored as location data 345. The location data could be numeric or non-numeric data identifying one or more specific phonemes, words, paragraphs, pages, sections, chapters, tables, images, slides, other locations, or combinations thereof.

[0063] The location identification module 354 can determine that an audible action matches multiple distinct parts of a text source. This may occur when the same word or phrase (e.g., a phoneme sequence) is repeated multiple times in the text source. The location identification module 354 can detect spoken words by analyzing phoneme data and detect matches between spoken words and multiple candidate locations within the text source. The location identification module 354 can select one or more of the multiple candidate locations based on data from the fuzzy matching module 352. The location identification module 354 can further narrow down the candidate locations by selecting specific locations based on phoneme data from audio before, during, or after the spoken word (e.g., by extending a predetermined segment length or using adjacent segments).

[0064] The reading speed module 356 can access and analyze location data 345 to determine the user's reading speed. Reading speed data can be determined based on location data, text source data, audio data, other data, or combinations thereof, and can be stored as reading speed data 346. Reading speed can be based on a portion of the location data 345, which identifies at least two locations in the text source. Locations can correspond to specific times, and determining reading speed can be based on the number of words and the amount of time between two or more locations. In one example, the number of words can be based on the content of the text source, without considering content added, skipped, or repeated by the user. In another example, the number of words can be based on the content of the text source and also on the content of the audio. This can be advantageous because the content of the audio can indicate whether words are added, skipped, repeated, or other actions, or combinations thereof. In either example, the reading speed module 356 can update the reading speed data to represent the user's reading speed over one or more durations.

[0065] The reading interruption module 358 can access and analyze any data discussed above to detect whether the user has interrupted reading the text source or is still reading it. This can be challenging because the user may have stopped reading the text source but is discussing concepts related to it. As a result, there may be overlap between spoken words and the content of the text source. Detecting reading interruptions can be important because it allows computing devices to avoid recording private discussions. The reading interruption module 358 can determine whether the user has interrupted reading the text source by calculating one or more corresponding metrics.

[0066] A correspondence metric can indicate the similarity or dissimilarity between corresponding parts of an audio clip and a text source. A correspondence metric can be a probability value indicating the probability that an audio clip corresponds to a location in the text source. The probability value can be numeric or non-numeric and can be the same as or similar to percentages, ratios, decimal values, or other values ​​or combinations thereof. In one example, the value could be between 0 and 1 (e.g., 0.97), 0 and 100 (e.g., 98), or other ranges. One end of the range can indicate that the audio clip absolutely corresponds to a location in the text source (e.g., 1.0 or 100), while the other end can indicate that the audio clip absolutely does not correspond to a location in the text source (e.g., a value of 0).

[0067] Correspondence metrics can be based on or related to multiple similarity measures. For example, both measures can be used to compare or contrast data derived from audio (e.g., phoneme data 243A) with data derived from a text source (e.g., phoneme data 243B). Similarity measures (e.g., phoneme edit distance) can be used to compare or contrast written words with spoken words from a text source, while correspondence metrics can be used to compare or contrast a set of written words with a set of spoken words over a duration. The duration of audio (e.g., a segment) can be of any length and can include a set of words as well as one or more other audible actions (e.g., turning a page, closing a book). In one example, the user's audio can include a first duration and a second duration, and the reading interruption module 358 can calculate one or more correspondence metrics over the first duration and one or more correspondence metrics over the second duration. Correspondence metrics can be stored as correspondence metric data 347. In other examples, correspondence metrics can also, or alternatively, consider one or more signals, such as the absence of speech input during the duration, the absence of recognition of the story text, or the absence of recognition of a specific word or phrase that may indicate termination. These words or phrases may include "let's stop reading", "let's finish tomorrow", "OK, I'm done", "let's pause", other phrases or combinations thereof.

[0068] The reading interruption module 358 can compare the corresponding measurement data 347 for each duration with one or more predetermined thresholds. In response to the corresponding measurement data 347 for a first duration not meeting a threshold (e.g., above or below a threshold), the reading interruption module 358 can determine that the audio duration corresponds to the text source and that the user's audio data corresponds to the user reading the text source. In response to the corresponding measurement data 347 for a second duration meeting a threshold (e.g., below or above a threshold), the reading interruption can determine that the audio duration does not correspond to the text source and that the user has stopped reading the text source. In one example, determining that the corresponding metric meets a threshold may indicate that the audio data does not match the data of the text source or that the audio data differs from the content of the text source.

[0069] The reading interruption module 358 can perform one or more actions in response to determining that the user has interrupted reading the text source. In one example, the reading interruption module 358 can transmit a signal to deactivate one or more microphones associated with the computing device to avoid capturing or recording additional audio data. In another example, the reading interruption module 358 can transmit a signal to stop analyzing audio data (e.g., comparing audio data with data from the text source). The latter example can record audio but cannot access or analyze the audio data. In yet another example, the reading interruption module can cause the computing device 130 to interact with the user before, during, or after transmitting the signal. For example, the computing device can interact with the user by providing a prompt (e.g., audio, visual, or a combination thereof). The prompt can ask the user whether to exit Story Time mode, or it can notify the user that Story Time mode has been exited and whether or not the user can re-enable Story Time mode.

[0070] Figure 4 A block diagram illustrating exemplary components is provided, which enable computing device 130 to provide physical effects to enhance the user experience. As discussed above, physical effects can modify the environment and may include acoustic effects, haptic effects, optical effects, other effects, or combinations thereof. In the example shown, computing device 130 may include a physical effect determination component 136, a prediction loading component 137, and an effect providing component 138.

[0071] The physics effect determination component 136 enables the computing device 130 to identify and provide physics effects corresponding to specific portions of a text source. In one example, the physics effect determination component 136 may include an audible motion-related module 462, a scene data module 464, and an effect selection module 466.

[0072] The audible action correlation module 462 enables the computing device to correlate a specific physical effect with a specific audible action associated with a text source. The audible action correlation module 462 can determine the correlation based on effect data 448 of the text source. Effect data 448 can indicate which physical effects correspond to which parts of the text source. Effect data 448 can correlate a specific physical effect with a specific location in the text source, a specific audible action of the user, a specific triggering condition (discussed below), or a combination thereof. A location in the text source may be related to an audible action (e.g., spoken words or page turning) or unrelated to an audible action (e.g., a user viewing a graphic image). In one example, effect data 448 can identify an audible action including a specific spoken word (e.g., "dog") in the text source, and the physical effect may involve initiating an acoustic effect (e.g., a bark) corresponding to that spoken word. In another example, effect data 448 can identify an audible action (e.g., page turning), and the physical effect may involve modifying an existing physical effect (e.g., readjusting ambient sound, light, or temperature).

[0073] Effect data 448 can be accessed by or created by computing device 130. In one example, computing device 130 may access or receive effect data directly or indirectly from an author, publisher, distributor, partner, third-party service, other source, or a combination thereof. Effect data 448 may be included within text source data 242 or may be separate from text source data 242. In another example, computing device 130 may create effect data based on text source data 242. For example, audible motion correlation module 462 may analyze text data or phoneme data and identify physical effects corresponding to specific parts of the text source. In either example, effect data 448 may be stored in data storage 240 for enhanced access by computing device 130.

[0074] The context data module 464 enables the computing device 130 to collect context data 449 associated with the user. Context data 449 may be based on the user's environment and may be acquired using one or more sensors (e.g., sensors 131A-C). Context data 449 may also be based on profile data about the user, which may be accessible to the computing device 130 via direct user input or via a remote source (e.g., a network connection to a content platform or social network). In one example, context data 449 may include sound data (e.g., ambient sound measurements), light data (e.g., ambient light measurements), time data (e.g., morning or evening), calendar data (advance booking for tomorrow), geographic location data (e.g., postal code, address, latitude / longitude), weather data (e.g., rain, lighting, thunderstorms, strong winds, cloudy), user profile information (e.g., a child's name, age, or gender), user audio feedback (e.g., a child crying or clapping), other data, or combinations thereof.

[0075] The effect selection module 466 enables the computing device 130 to select and modify physical effects based on effect data 448, scene data 449, text source data 242, other data, or combinations thereof. The effect selection module 466 can be used to select a specific physical effect (e.g., an acoustic effect) or modify the attributes of a physical effect. These attributes can relate to the intensity, timing, pitch, transitions (e.g., fade-in / fade-out), other features, or combinations thereof. Intensity can relate to the magnitude of the modification to the environment and can relate to the volume (e.g., loudness) or luminance (e.g., brightness) of the physical effect. Timing may relate to the speed or duration of the physical effect. The computing device 130 can select physical effects based on words in a text source and can update the attributes of physical effects based on scene data. In one example, scene data may include sound data of the user's environment, and the physical effect may be an acoustic effect based on a certain amount of sound data. In another example, scene data may include light data of the user's environment, and the physical effect may be an optical effect that modifies the luminance of a light source based on the light data (e.g., dimming or brightening the light). In yet another example, the context data may include user profile data of the parents or children and indicate the age of the audience, and the physical effects may include acoustic effects selected based on the age of the user (e.g., a more playful dog bark for younger children and a more intense dog bark for older children).

[0076] The effects selection module 466 can use context data to identify timing aspects related to reading the text source. For example, time data or calendar data can be used to differentiate between reading the text source at night and reading it in the morning. At night, the effects selection module 466 can select more soothing (e.g., less stimulating) physical effects to encourage the listener to prepare for bed. This might involve reducing the brightness and volume settings for sound and optical effects and / or selecting effects with a lower pitch (e.g., a softer bumping effect or whisper, as opposed to shouting). In the morning, the effects selection module 466 can select more stimulating physical effects to encourage the user to prepare for the day. This might involve increasing the brightness and volume settings for acoustic and optical effects. Calendar data can also indicate whether the reading time is associated with a weekend or weekday, or whether there is an appointment (e.g., later that day or early the next morning). Any of these can influence the speed at which the user reads the text source and how long or how often physical effects should be provided.

[0077] Predictive loading component 137 enables computing device 130 to predictively load content before it is required. Predictive loading accelerates computing device 130's ability to deliver physics effects by loading the content of the physics effects before they are triggered. Predictive loading may be the same as or similar to prefetching, precaching, cache prefetching, other concepts, or combinations thereof. In one example, predictive loading component 137 may include prediction module 472, trigger determination module 474, and content loading module 476.

[0078] The prediction module 472 enables the computing device 130 to predict when a user will reach a specific portion of a text source. For example, the prediction module 472 can determine when a word in the text source will be spoken before the user speaks it. The predicted time can be a future time and can be determined based on the user's reading speed, the reading position in the text source, other data, or a combination thereof. In one example, the time can be calculated based on the user's reading speed (e.g., words per minute, pages per minute) and the difference between the current reading position and the target position in the text source (e.g., the number of words, paragraphs, or pages). In other examples, the prediction module 472 can use predictive models, machine learning, neural networks, or other techniques to enhance the prediction based on current data, historical data, or a combination thereof.

[0079] The trigger determination module 474 enables the computing device 130 to determine trigger conditions associated with a specific physical effect. Trigger conditions can be loading trigger conditions or initiating trigger conditions. Loading trigger conditions indicate when to begin loading the content of the physical effect. Initiating trigger conditions indicate when to begin providing (e.g., playing) the content of the physical effect. Either trigger condition can correspond to a specific time or a specific location within the text source and can be based on effect data, text source data, other data, or a combination thereof. A specific time can be an absolute time (e.g., 8:32:02 pm) or a relative time (e.g., 5 seconds before the predicted time of a word or page turn). A specific location can be the position within the text source before the word to which the physical effect is to be aligned. A specific location can be an absolute position (e.g., word 397) or a relative position (e.g., 5 words before the word "bark").

[0080] The determination of triggering conditions may be based on one or more factors related to content, computing device, user, environment, or other aspects or combinations thereof. Content-related factors may include the amount of content (e.g., 1MB file size), the location of the content (e.g., remote storage), the format of the content (e.g., downloadable file, streaming chunk, or format requiring transcoding), the duration of the content (e.g., 2 seconds of sound), other aspects of the content, or combinations thereof. Computing device-related factors may correspond to the amount and / or availability of computing resources on computing device 130 or other computing devices. Computing resources may include connection speed (e.g., network bandwidth), storage space (e.g., available solid-state storage), processing power (e.g., CPU speed or load), other computing resources, or combinations thereof. User-related factors may include the user's reading speed, current reading location, voice clarity, other aspects, or combinations thereof.

[0081] The trigger determination module 474 may use one or more factors to calculate the duration of loading or providing physical effects. The duration associated with loading content may be referred to as the predicted loading time and may or may not include the duration of providing (e.g., playing) the content. In one example, the trigger determination module 474 may determine the duration of loading physical effects based on the size of the content and the network bandwidth of the computing device 130. The trigger determination module 474 may use the predicted loading time to identify a specific time or location of the trigger condition. In one example, the trigger condition may be set to a time greater than or equal to the predicted time of an audible action (e.g., spoken words) minus a predetermined loading time (e.g., 5 seconds). In another example, the trigger condition may be set to a location within the text source equal to or prior to the position the physical effect is expected to align with. This may involve selecting a location within the text source based on the predicted loading time and reading speed. For example, if a user reads at a rate of 120 words per minute (i.e., 2 words per second) and the predicted loading time is 5 seconds, the trigger location might be 10 or more words before the word the physical effect should align with.

[0082] Content loading module 476 enables computing device 130 to load content for one or more physical effects before initiating the physical effects. Loading content may involve computing device 130 transmitting or receiving one or more requests and responses, and may involve downloading, streaming, copying, other operations, or combinations thereof. Content may include executable data (e.g., instructions), informational data (e.g., audio files or blocks), other data, or combinations thereof. Content may be stored by computing device 130 as content data 451 in data storage 240. Computing device 130 may load content for physical effects from local devices (e.g., data storage 240), remote devices (e.g., servers or cloud services), or combinations thereof.

[0083] Effects providing component 138 enables computing device 130 to provide physical effects to modify the user's environment. Effects providing component 138 can be initiated after content for physical effects is loaded, and can be timed so that physical effects are provided at times aligned with audible actions intended to be aligned with it. In one example, effects providing component 138 may include instruction access module 482 and physical effects initiation module 484.

[0084] The instruction access module 482 can access instruction data associated with physical effects. Instruction data may include a collection of one or more commands, operations, procedures, tasks, other instructions, or combinations thereof. Instructions may indicate physical effects and one or more properties of those physical effects.

[0085] The physics effect initiation module 484 can access and execute instruction data to initiate a physics effect. The physics effect initiation module 484 can initiate the instruction before, during, or after detecting an initiation trigger condition (e.g., audible action) corresponding to the physics effect. In one example, the text source may include specific words, and initiating a physics effect may respond to detecting that audio data includes words (e.g., matching phonemes). In another example, the physics effect initiation module 484 can determine the initiation trigger condition for initiating a physics effect. The process of determining the trigger condition for initiating a physics effect can be the same as or similar to the trigger condition used to initiate the loading of content for a physics effect. The instruction may cause the computing device 130 to provide a physics effect, or it may cause the computing device 130 to communicate with one or more physics effect devices to provide a physics effect. In either example, the computing device 130 may directly or indirectly cause the physics effect to modify the user's environment to enhance the listening user's experience.

[0086] In one example, the physics effect initiation module 484 or the effect selection module 466 may use one or more confidence thresholds to select and / or initiate a physics effect. The one or more confidence thresholds may be grouped into one or more confidence intervals that categorize the probability of an audio match with a specific location in a text source (e.g., a spoken word match a word in a text source). Any number of confidence intervals may exist, and a first confidence interval may indicate a low probability of an audio match with a text source location (e.g., >50%), and each subsequent confidence interval may be higher (e.g., >75%, >95%, etc.). Effect data that correlates physics effects with location may also include specific confidence thresholds (e.g., a minimum confidence interval). For example, providing a sound effect may be associated with a higher confidence interval, followed by a background effect change. The computing device 130 may determine whether a confidence threshold is met before selecting or initiating a physics effect. This may involve comparing corresponding metric data, similarity metric data, other data associated with fuzzy matching, or combinations thereof. In one example, a specific location in the text source can be associated with multiple different physical effects, and each can correspond to a different confidence interval associated with the current reading location. When the confidence interval is high (e.g., trusting that the current reading location is accurate), a specific sound effect can be initiated (e.g., the sound effect of a single dog barking at a high volume), while when the confidence interval is low (e.g., unsure whether the current reading location is accurate), different sound effects may be initiated (e.g., background noise of multiple dogs barking at a low volume).

[0087] Figure 5-8 Flowcharts of corresponding methods 500, 600, 700, and 800 according to one or more aspects of this disclosure are depicted, said methods 500, 600, 700, and 800 for enhancing the ability of a computing device to follow and provide special effects in real time while a text source is read aloud. Method 500 may involve estimating reading progress using phoneme data and fuzzy matching. Method 600 may optimize the computing device's ability to detect when a user stops reading from a text source and is engaged in a private discussion. Method 700 may enable the computing device to provide physical effects that take into account the user's context and environment. Method 800 may enable the computing device to pre-cachate the content of physical effects to reduce latency and better synchronize physical effects with audible actions associated with the text source.

[0088] Figures 5-8 Each of the methods and their individual functions, routines, subroutines, or operations can be executed by one or more processors of a computer device that performs the method. In some implementations, one or more of the methods can be executed by a single computing device. Alternatively, one or more of the methods can be executed by two or more computing devices, each computing device performing one or more individual functions, routines, subroutines, or operations of the method. For simplicity, the methods of this disclosure are depicted and described as a series of behaviors. However, the behaviors according to this disclosure can occur in various orders and / or simultaneously, and together with other behaviors not presented and described herein. Furthermore, not all illustrated behaviors may be required to implement the methods according to the disclosed subject matter. Additionally, those skilled in the art will understand and appreciate that the methods can alternatively be represented as a series of related states via state diagrams or events. Furthermore, it should be appreciated that the methods disclosed in this specification can be stored on an article of manufacture to facilitate the delivery and transfer of such methods to a computing device. As used herein, the term "article of manufacture" is intended to include a computer program accessible from any computer-readable device or storage medium. In one embodiment, the method can be performed by Figures 1-4 It can be executed by one or more of the components in the system.

[0089] refer to Figure 5 Method 500 can be performed by a processing device, such as a client device (e.g., a smart speaker), a server device (e.g., a cloud service), other devices, or combinations thereof, and can begin at box 502. At box 502, the processing device can determine phoneme data of a text source. The text source can include sequences of words, and the phoneme data can be a speech encoding of the word sequences, which includes one or more sequences of speech values. Each speech value can correspond to a phoneme, and the phoneme sequence can correspond to spoken words. The same sequence of speech values ​​can correspond to words that sound the same but are spelled differently (e.g., homophones), while different sequences of speech values ​​can correspond to words that are spelled the same but sound different (e.g., homonyms).

[0090] The processing device can access phoneme data from a text source or generate phoneme data for the text source. The processing device can generate phoneme data by speech encoding a sequence of words. This may involve accessing the text data of the text source and generating (e.g., converting, transforming, deriving) phoneme data based on the text data. This phoneme data can then be associated with phoneme data for future use.

[0091] At box 504, the processing device may receive audio data including spoken words associated with a text source. The audio data may include one or more audible actions of the user and may include spoken words, page turning, or other audible actions captured from the user's environment. In one example, the processing device may receive audio data directly from one or more sensors as an audio signal. In another example, the processing device may receive audio data from a data storage device or another computing device. The audio data may be in any digital or analog format and may be accessed or received via one or more storage objects (e.g., files, database records), data streams (e.g., audio streams, video streams), data signals, other data transmission or storage protocols, or combinations thereof.

[0092] At box 506, the processing device may compare phoneme data of the text source with phoneme data of the audio data. The comparison of the audio data and the text source may occur without converting the audio data to text (e.g., recognized words) using speech recognition, and may involve comparing phoneme data corresponding to the audio data with phoneme data corresponding to the text source. The comparison may include calculating a numerical value representing the similarity between two or more sequences of speech values. The numerical value may be the phoneme edit distance between the phoneme data of the audio data and the phoneme data of the text source. The comparison may also involve performing fuzzy matching between the phoneme data corresponding to the audio data and the phoneme data of the text source.

[0093] At box 508, the processing device can identify a position in a word sequence based on a comparison of phoneme data from the text source and speech data from the audio. Position identification may involve determining a match between a spoken word and a word in the word sequence of the text source. In one example, the text source could be a book, and the position could be the current reading position within that book. The method can terminate in response to completion of the operation described above with reference to box 508.

[0094] refer to Figure 6 Method 600 can be performed by the same or different processing devices discussed above, and can begin at box 602. At box 602, the processing device can receive audio data including spoken words associated with a text source. The audio data can be segmented (e.g., tokenized, fragmented, split, divided) into a first duration and a second duration. In one example, the text source can be a book, and the first portion of the audio data can correspond to the content of the book (e.g., including spoken words from the book), while the second portion of the audio data may not correspond to the content of the book (e.g., may not include spoken words from the book).

[0095] At box 604, the processing device may compare audio data with data from a text source. The text source data may include phoneme data, and comparing the audio data and the text source data may involve phoneme comparison. In one example, comparing phoneme data may involve calculating the phoneme edit distance between the phoneme data of the text source and the phoneme data of the audio data.

[0096] At box 606, the processing device can calculate a correspondence metric between the second duration of the audio data and the data from the text source. Calculating the correspondence metric can include calculating it based on multiple phoneme edit distances. In one example, the processing device can select a set of spoken words (e.g., 3, 4, 5+ words) and compare the set of spoken words with the content of the text source. A phoneme edit distance can be determined for each word or combination of one or more words in the set. The resulting values ​​can then be weighted, aggregated, or modified to determine the correspondence metric.

[0097] At box 608, in response to determining that a corresponding metric meets a threshold, the processing device may transmit a signal to stop comparing the audio data with the data from the text source. Determining that the corresponding metric meets the threshold may involve determining that the corresponding metric is below or above the threshold. This determination may also be based on the duration for which the corresponding metric meets or does not meet the threshold. Determining that the corresponding metric meets the threshold may indicate that a second duration of the audio data includes content different from the content of the text source, and may indicate or not indicate that the audio data does not match the data of the text source. Transmitting the signal may involve transmitting a signal to deactivate one or more microphones that capture audio data. In one example, in response to determining that the second duration of the audio data does not contain the content of the text source, the processing device may cause the computing device to prompt the user to exit Story Time mode. This prompt may be an audio prompt, a visual prompt, another prompt, or a combination thereof. In response to completing the operations described above with reference to box 608, the method may terminate.

[0098] refer to Figure 7 Method 700 can be performed by the same or different processing devices discussed above, and can begin at box 702. At box 702, the processing device can receive audio data including spoken words from a user. Spoken words can be associated with a text source that the user is reading aloud, and can include one or more other audible actions, such as page turning, spoken words not in the text source, and other audible actions captured from the user's environment. In one example, the processing device can receive audio data directly from one or more sensors in the form of an audio signal (e.g., for real-time use or real-time perception). In another example, the processing device can receive audio data from a data storage device or another computing device. The audio data can be in any digital or analog format and can be accessed or received from one or more storage objects (e.g., files, database records), data streams (e.g., audio streams, video streams), data signals, other data transmission or storage protocols, or combinations thereof.

[0099] At box 704, the processing device can analyze context data associated with the user. Context data may include sound data, light data, time data, weather data, calendar data, user profile data, other data, or combinations thereof. In some examples, the context data may be associated with physical effects, allowing the processing device to provide physical effects that take into account the user's context and environment. In one example, the context data may include sound data of the user's environment, and the physical effects may include acoustic effects at a certain volume based on the sound data. In another example, the context data may include light data of the user's environment, and the physical effects may include optical effects that modify the luminance of a light source based on the light data. In yet another example, the context data may include user profile data indicating a child's age, and the physical effects may include acoustic effects selected based on the child's age.

[0100] At box 706, the processing device can determine a match between audio data and data from a text source. The processing device can identify the text source based on user input (e.g., audio data or touch data) and retrieve data from the text source. The text source data may include phoneme data, and determining a match may involve calculating the phoneme edit distance between the phoneme data of the text source and the phoneme data of the audio data. In one example, determining a match between audio data and data from the text source may involve using the phoneme data of the text source to detect audio data that includes words from the text source.

[0101] At box 708, the processing device may initiate a physical effect in response to determining a match. The physical effect may correspond to a text source and is based on context data. The physical effect may modify the user's environment and may include at least one of acoustic, optical, and haptic effects. The text source may include words, and initiating a physical effect may be in response to detecting that the audio data includes words. In one example, the processing device may select a physical effect based on words in the text source and may update the attributes of the physical effect (e.g., volume or brightness) based on context data. The method may terminate in response to completing the operations described above in reference box 708.

[0102] refer to Figure 8 Method 800 can be executed by a processing device of a server device or a client device, and can begin at box 802. At box 802, the processing device can identify effect data for a text source, where the effect data relates physical effects to audible actions of a user. The effect data can indicate physical effects and indicate locations within the text source that relate to the audible action. This location can correspond to words, paragraphs, pages, or other locations within the text source. In one example, the audible action can be spoken words from the text source, and the physical effect can be an acoustic effect corresponding to the spoken words. In another example, the audible action can include page turning, and the physical effect can be a modification of existing acoustic, optical, or haptic effects.

[0103] At box 804, the processing device may receive audio data including multiple audible actions. These multiple audible actions may include one or more spoken words from a text source and one or more other audible actions, such as page turning, spoken words not present in the text source, and other audible actions captured from the user's environment. In one example, the processing device may receive audio data directly from one or more sensors in the form of an audio signal (e.g., for real-time use or proximity / perception). In another example, the processing device may receive audio data from a data storage device or another computing device. The audio data may be in any digital or analog format and may be accessed or received from one or more storage objects (e.g., files, database records), data streams (e.g., audio streams, video streams), data signals, other data transmission or storage protocols, or combinations thereof.

[0104] At box 806, the processing device may determine triggering conditions based on effect data and the text source. In one example, determining triggering conditions may involve determining a physical effect associated with a first position in the text source and selecting a second position in the text source preceding the first position. This selection may be based on the reading speed and loading time associated with the physical effect, and the second position may be associated with at least one specific instance of a word, paragraph, page, or chapter in the text source. The processing device may then set the triggering conditions to correspond to the second position in the text source. In another example, determining triggering conditions may involve calculating the duration of loading content based on the amount of physical effect content and the amount of available computing resources. Computing resources may relate to one or more of network bandwidth, storage space, or processing power, and the duration may be longer when available computing resources are low. In one example, determining the time when a future audible action will occur may involve identifying the time at which loading will be initiated based on the calculated duration and the determined time of the audible action, and initiating loading content at or before the identified time. In another example, determining the time includes calculating the future time based on the reading speed in the text source and the current reading position. In yet another example, determining the timing involves predicting when the words from the text source will be spoken before the words are spoken.

[0105] At box 808, the processing device can load content for physics effects in response to the fulfillment of a trigger condition. The trigger condition can be fulfilled before the audible action occurs.

[0106] At box 810, the processing device can provide physical effects to modify the user's environment. The method can terminate upon completion of the operation described above with reference to box 810.

[0107] The techniques discussed in this article include a variety of enhancements for computing devices, with or without virtual assistant features. The following discussion includes several different enhancements that can be used individually or together to optimize the computing device's ability to follow a text source as it is read aloud and to provide special effects to complement the user's environment. In one example, the environment may include a parent reading a book aloud to one or more children. In another example, the environment may include one or more users providing a presentation, voice, or other performance to an audience. In either example, the technique can be used to enhance the environment with special effects based on analysis of data associated with the text source. These special effects can be synchronized with specific parts of the text source, such as specific spoken words or page turns.

[0108] In the first example, the enhancement could relate to reading progress estimation based on speech fuzzy matching and confidence intervals, and could relate to the field of computer-based human speech recognition, and more specifically to enhancing the computer device's ability to identify reading positions within a text source when a user reads it aloud. When a user reads a text source aloud, several technical problems arise when the computing device attempts to follow along using traditional virtual assistant functions. This is because traditional virtual assistant functions perform speech recognition to convert audio into text / recognized words, which presents several issues. Speech recognition typically involves an acoustic step of converting audio into phonemes and a linguistic step of converting phonemes into text / recognized words. The linguistic step often waits for subsequent spoken words to establish context before converting spoken words into text. The linguistic step introduces unnecessary time delays and consumes additional computational resources. Furthermore, using recognized text to perform a traditional text-based comparison with the text source can be more prone to errors than performing a speech comparison (e.g., phoneme comparison). This often occurs because many words that sound the same or similar can be spelled very differently, producing false negations during text comparisons. Furthermore, traditional text comparison may fail to properly handle situations where users might jump around while reading a text source. For example, portions of the text source may be skipped, repeated, or new content may be added. This can make identifying the current reading position within the text source and accurately detecting reading speed challenging. Aspects and implementations of this technology address the above and other shortcomings by providing enhancements that enable computing devices to detect the current reading position within a text source while it is being read aloud. In one example, the technology avoids the language steps of traditional speech recognition by comparing phoneme data derived from audio with phoneme data derived from the text source. The text source can be a book, magazine, presentation, speech, script, or other source containing word sequences. The technology can receive audio data including words spoken by the user and can convert the audio data into phoneme data locally or via an assistant or remote server (e.g., a cloud service). The phoneme data of the audio and the text source can then be compared via speech comparison rather than the more traditional text comparison. Speech comparison may be accompanied by fuzzy matching to identify positions within the word sequence (e.g., the current reading position). The systems and methods described herein include techniques for enhancing computer-based human speech recognition. Specifically, the disclosed technique enhances the latency, accuracy, and computational resources required to identify the current reading position. This may be a result of modifying the speech analysis process (e.g., speech recognition) to avoid converting audio data into text / words. The technique can use a speech analysis process that uses an acoustic model to convert audio into phoneme data, but avoids the language step of using a language model to convert speech phoneme data into text / words. Avoiding the language step reduces latency and computational resource consumption.Performing phoneme comparisons and using fuzzy matching can enhance the accuracy of identifying the current reading position because it can better compensate for non-linear reading of the text source (e.g., skipping, repeating, or adding content).

[0109] In the second example, the enhancements may relate to the algorithmic determination of when a story reader stops reading, and may relate to the field of computer-based human speech recognition, and particularly to enhancing the computer device's ability to determine when a user is no longer reading the content of a text source aloud. Several technical problems arise when a computing device attempts to follow along with traditional virtual assistant functions while a user reads a text source aloud. Some of these problems occur because traditional virtual assistants may fail to detect when a user has finished audio input if they continue talking about other things. This can cause the computing device to continue recording the user's audio, which can be problematic if the user shifts to discussing private matters. Detecting when a user stops reading from a text source can be even more challenging when the user is not following the text and skips, repeats, or adds new content while reading the text source aloud. Aspects and implementations of this technology address these and other shortcomings by enhancing the computing device's ability to detect when a user interrupts reading a text source. In one example, the technology could enable a virtual assistant to more accurately detect when a user has left the text source to take a break and could deactivate the microphone to avoid capturing private audio content. This might involve receiving audio data including spoken words associated with the text source and comparing the audio data with the data from the text source. This technology can calculate a correspondence metric between the content of audio data and the content of a text source. The correspondence metric can be a probability value based on comparisons of phoneme data, text data, or other data, and may involve the use of fuzzy matching logic. When the correspondence metric meets a threshold (e.g., below a minimum correspondence threshold), the technology can transmit a signal that will stop the analysis of subsequent audio data. The systems and methods described herein include techniques for enhancing computer-based human speech recognition. In particular, this technology addresses the technical problem of avoiding unintentional recording of a user's private conversations by using comparisons that better compensate for non-linear reading of the text source (e.g., skipping, repetition, addition of content). For example, the above technology can facilitate more accurate and / or faster automated control of virtual assistants to record and / or process only relevant audio. The technology can also reduce the power consumption of computing devices by deactivating audio sensors (e.g., microphones) and associated data processing when the computing device detects that the user has stopped reading text. Furthermore, the above technology can enable computing devices to reduce the utilization of computing resources, such as processing capacity, network bandwidth, data storage, etc., which could otherwise be used to record and / or process audio data once the user has stopped reading text.

[0110] In the third example, the enhancements may relate to the dynamic adjustment of story-time special effects based on context data, and may be related to the domain of the virtual assistant, particularly enhancing its ability to provide special effects while the text source is read aloud. Modern computing devices can be configured to employ traditional virtual assistant features to provide sound effects that complement the environment as the user reads a book aloud. For example, the computing device could provide a barking effect when the user reads the word "bark" aloud. Sound effects are often provided by the same entity providing the text source and may correspond directly to a portion of the text source. As a result, special effects may be identical, independent of the user or the environment, and may not be optimized for the user's specific reading environment. Aspects and implementations of this technology address the above and other drawbacks by enabling computing devices to provide a wide variety of special effects based on the user's environment. In one example, the technology enables computing devices to analyze context data of the user's environment and select or customize special effects. Special effects can be physical effects that alter the user's environment, including acoustic effects (e.g., music, sound effects music), optical effects (e.g., flashlights, ambient light), tactile effects (e.g., vibration, wind, temperature changes), other effects, or combinations thereof. This technology can involve receiving and analyzing contextual data associated with a user. Contextual data can be related to weather, lighting, time of day, user feedback, user profiles, other information, or combinations thereof. The technology can select or modify physical effects corresponding to a text source based on the contextual data. For example, this could lead to the selection or modification of volume, brightness, speed, pitch, or other attributes of physical effects. The systems and methods described herein include techniques in the field of enhancing virtual assistants and home automation. In particular, this technology enables computing devices to optimize the environment by using contextual data about the user and by adding, removing, or modifying physical effects to enhance the user's listening experience.

[0111] In the fourth example, the enhancements might relate to detecting the progress of the story reader to pre-cachate special effects, and could also relate to the domain of virtual assistants, particularly enhancing their ability to pre-cachate special effects for a text source being read aloud. Several technical challenges arise when attempting to use traditional virtual assistant features to provide sound effects synchronized with the spoken content of a text source. This is because traditional virtual assistants perform speech recognition to convert audio to text and then compare it against the text. Speech recognition typically involves an acoustic step of converting audio to phonemes and a linguistic step of converting phonemes to text. The linguistic step often waits for subsequent spoken words to establish context before translating spoken words into text. The linguistic step introduces time delays and consumes additional computing resources on potentially resource-constrained computing devices. This delay can be further exacerbated because sound effects might be large audio files downloaded from remote data sources. Traditional methods might involve downloading sound effects in response to the detection of spoken words, but this delay can result in special effects being delivered long after the words have been spoken. Another approach might involve downloading all sound effects when the text source is initially identified, but this can be problematic when the computing device is resource-constrained (e.g., a smart speaker). Aspects and implementations of this technique address these and other drawbacks by providing enhancements to the computing device to improve its ability to conserve computing resources while still providing special effects synchronized with the text source being read aloud. This can be achieved by using data from the text source (e.g., a book) to predict future audible actions and prefetch associated physical effects before the corresponding audible action occurs. In one example, the technique could enable the computing device to predict when a user will arrive at a word in the text source before the user speaks it. This could involve identifying effect data from the text source that associates physical effects with one or more audible actions by the user. Audible actions could include words spoken by the user, or other actions such as turning a page, closing a book, or generating an audible response. The technique could determine triggering conditions based on the current reading position, reading speed, other data, or a combination thereof. In response to detecting that a triggering condition is met, the technique could enable the computing device to load content for the physical effects and subsequently provide the physical effects to modify the user's environment. The systems and methods described herein include techniques for enhancing pre-caching based on human speech recognition. In particular, the disclosed techniques address technical problems related to resource consumption when analyzing speech and downloading special effects. These techniques can also reduce latency in delivering special effects, thereby enabling better synchronization between special effects and human speech.

[0112] Figure 9 This is a block diagram illustrating a computer system operating according to one or more aspects of this disclosure. In various illustrative examples, the computing system 900 may correspond to... Figures 2-4 The computing device 130. The computing system may be included within a virtualization-enabled data center. In some implementations, the computer system 900 may be connected to other computer systems (e.g., via a network such as a local area network (LAN), intranet, extranet, or the Internet). The computer system 900 may operate as a server or client computer in a client-server environment, or as a peer computer in a peer-to-peer or distributed network environment. The computer system 900 may be provided as a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), cellular phone, web device, server, network router, switch, or bridge, or any device capable of executing a set of instructions (sequential or otherwise) specifying the actions to be taken by the device. Additionally, the term "computer" should include any collection of computers that individually or collectively execute a set (or more) of instructions to perform any one or more of the methods described herein.

[0113] In another aspect, the computer system 900 may include a processing device 902, a volatile memory 904 (e.g., random access memory (RAM)), a non-volatile memory 906 (e.g., read-only memory (ROM)) or electrically erasable programmable ROM (EEPROM), and a data storage device 916 that can communicate with each other via a bus 908.

[0114] The processing device 902 may be provided by one or more processors, such as general-purpose processors (e.g., complex instruction set computing (CISC) microprocessors, reduced instruction set computing (RISC) microprocessors, very long instruction word (VLIW) microprocessors, microprocessors that implement other types of instruction sets, or microprocessors that implement combinations of various types of instruction sets) or special-purpose processors (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or network processors).

[0115] The computer system 900 may further include a network interface device 922. The computer system 900 may also include a video display unit 910 (e.g., LCD), an alphanumeric input device 912 (e.g., keyboard), a cursor control device 914 (e.g., mouse), and a signal generation device 920.

[0116] Data storage device 916 may include a non-transitory computer-readable storage medium 924 on which instructions 926 encoded for any one or more of the methods or functions described herein may be stored, including instructions for implementing methods 500, 600, 700, or 800. Figures 1-4 Instructions for any component or module.

[0117] Instruction 926 may also reside wholly or partially in volatile memory 904 and / or processing device 902 during execution by computer system 900, thus volatile memory 904 and processing device 902 may also constitute machine-readable storage media.

[0118] Although computer-readable storage medium 924 is shown as a single medium in the illustrative example, the term "computer-readable storage medium" should include a single medium or multiple media storing one or more sets of executable instructions (e.g., centralized or distributed databases and / or associated caches and servers). The term "computer-readable storage medium" should also include tangible media capable of storing or encoding a set of instructions executable by a computer, and causing a computer to perform any one or more of the methods described herein. The term "computer-readable storage medium" should include, but is not limited to, solid-state memory, optical media, and magnetic media.

[0119] The methods, components, and features described herein can be implemented by discrete hardware components or integrated into the functionality of other hardware components such as ASICs, FPGAs, DSPs, or similar devices. Furthermore, the methods, components, and features can be implemented using firmware modules or functional circuitry within hardware resources. Additionally, the methods, components, and features can be implemented using any combination of hardware resources and computer program components, or a computer program.

[0120] Unless otherwise specifically stated, terms such as “initiate,” “send,” “receive,” “analyze,” etc., refer to operations and processes performed or implemented by a computer system that manipulate and transform data represented as physical (electronic) quantities within computer system registers and memories into other data represented similarly as physical quantities within computer system memory or registers or other such information storage, transmission, or display devices. Furthermore, terms such as “first,” “second,” “third,” “fourth,” etc., as used herein, are labels used to distinguish different elements and may not have ordinal meaning based on their numerical names.

[0121] The examples described herein also relate to apparatus for performing the methods described herein. This apparatus may be specifically configured to perform the methods described herein, or it may comprise a general-purpose computer system selectively programmed by a computer program stored in a computer system. Such a computer program may be stored in a computer-readable tangible storage medium.

[0122] The methods and illustrative examples described herein are not inherently related to any particular computer or other device. Various general-purpose systems can be used based on the teachings described herein, or it may prove convenient to construct more specialized devices to perform each of methods 500, 600, 700, 800 and / or their individual functions, routines, subroutines, or operations. Examples of the structures of various such systems are illustrated in the foregoing description.

[0123] The above description is intended to be illustrative and not restrictive. Although this disclosure has been described with reference to specific illustrative examples and embodiments, it should be understood that this disclosure is not limited to the described examples and embodiments. The scope of this disclosure should be determined by referring to the appended claims and the full scope of their equivalents.< / t>

Claims

1. A method comprising: Effect data that identifies the source of the text, wherein the effect data correlates physical effects with the user's audible actions; Receive audio data including multiple audible actions, wherein at least one of the multiple audible actions includes spoken words corresponding to a determined reading progress of the user relative to the text source; Set the triggering conditions related to the effect data and the text source; The processing device determines that the triggering conditions related to the effect data and the text source are met; Load the content for the physical effects before the audible action occurs; and Provide the physical effects to modify the user's environment. The triggering conditions include: The physical effect is determined to be associated with a first location in the text source; Select a second position in the text source preceding the first position, wherein the selection is based on reading speed and loading time associated with the physical effects; and The trigger condition is set to correspond to the second position in the text source.

2. The method of claim 1, wherein, The second position in the text source is associated with at least one specific instance of a word, paragraph, page, or chapter of the text source.

3. The method according to claim 1, further comprising setting the triggering conditions related to the effect data and the text source, wherein, Setting the trigger conditions includes: The duration for loading the content is calculated based on the amount of the physical effects and the amount of available computing resources. Determine the future time at which the audible action will occur; The timing of the loading is identified based on the calculated duration and the determined time of the audible action; and The trigger condition is set to correspond to a position in the text source defined according to the identified time.

4. The method according to claim 3, wherein, Determining the future time when the audible action will occur includes calculating the future time based on the reading speed and the current reading position in the text source.

5. The method according to claim 1, wherein, The effect data indicates the physical effect and indicates the location in the text source that is related to the audible action, wherein the location indicates a word, paragraph, or page in the text source.

6. The method according to claim 1, wherein, The audible actions include spoken words from the text source, and the physical effects include acoustic effects corresponding to the spoken words.

7. The method according to claim 1, wherein, The audible action includes turning a page, and the physical effect includes modifications to acoustic, optical, or tactile effects.

8. A system comprising: Memory; as well as Processing device, coupled to the memory, to perform operations including: Effect data that identifies the source of the text, wherein the effect data correlates physical effects with the user's audible actions; Receive audio data including multiple audible actions, wherein at least one of the multiple audible actions includes spoken words corresponding to a determined reading progress of the user relative to the text source; Set the triggering conditions related to the effect data and the text source; The processing device determines that the triggering conditions related to the effect data and the text source are met; Load the content for the physical effects before the audible action occurs; and Provide the physical effects to modify the user's environment. The triggering conditions include: The physical effect is determined to be associated with a first location in the text source; Select a second position in the text source preceding the first position, wherein the selection is based on reading speed and loading time associated with the physical effects; and The trigger condition is set to correspond to the second position in the text source.

9. The system according to claim 8, wherein, The second position in the text source is associated with at least one specific instance of a word, paragraph, page, or chapter of the text source.

10. The system of claim 8, wherein the operation further includes setting the triggering conditions related to the effect data and the text source, wherein, Setting the trigger conditions includes: The duration for loading the content is calculated based on the amount of the physical effects and the amount of available computing resources. Determine the future time at which the audible action will occur; The timing of the loading is identified based on the calculated duration and the determined time of the audible action; and The trigger condition is set to correspond to a position in the text source defined according to the identified time.

11. The system according to claim 10, wherein, Determining the future time when the audible action will occur includes calculating the future time based on the reading speed and the current reading position in the text source.

12. The system according to claim 8, wherein, The effect data indicates the physical effect and indicates the location in the text source that is related to the audible action, wherein the location indicates a word, paragraph, or page in the text source.

13. The system according to claim 8, wherein, The audible actions include spoken words from the text source, and the physical effects include acoustic effects corresponding to the spoken words.

14. The system according to claim 8, wherein, The audible action includes turning a page, and the physical effect includes modifications to acoustic, optical, or tactile effects.

15. A non-volatile computer-readable medium storing instructions, said instructions, when executed by a processing device, causing the processing device to perform operations, said operations including: Effect data that identifies the source of the text, wherein the effect data correlates physical effects with the user's audible actions; Receive audio data including multiple audible actions, wherein at least one of the multiple audible actions includes spoken words corresponding to a determined reading progress of the user relative to the text source; Set the triggering conditions related to the effect data and the text source; The processing device determines that the triggering conditions related to the effect data and the text source are met; Load the content for the physical effects before the audible action occurs; and Provide the physical effects to modify the user's environment. The triggering conditions include: The physical effect is determined to be associated with a first location in the text source; Select a second position in the text source preceding the first position, wherein the selection is based on reading speed and loading time associated with the physical effects; and The trigger condition is set to correspond to the second position in the text source.

16. The non-volatile computer-readable medium according to claim 15, wherein, The second position in the text source is associated with at least one specific instance of a word, paragraph, page, or chapter of the text source.

17. The non-volatile computer-readable medium of claim 15, the operation further comprising setting the triggering condition relating to the effect data and the text source, wherein, Setting the trigger conditions includes: The duration for loading the content is calculated based on the amount of the physical effects and the amount of available computing resources. Determine the future time at which the audible action will occur; The timing of the loading is identified based on the calculated duration and the determined time of the audible action; and The trigger condition is set to correspond to a position in the text source defined according to the identified time.

Images