Language Teaching Machine

Abstract

A set of machines functions as a language teaching lab. When configured with appropriate hardware, software, accessories, or any suitable combination thereof, such a language teaching lab accesses multiple sources and types of data, such as video streams, audio streams, thermal imaging data, eye tracker data, breath anemometer, biosensor data, accelerometer data, depth sensor data, or any suitable combination thereof. From the accessed data, the language teaching lab detects when a user is enunciating, for example, a word, phrase, or sentence. Other devices, systems, and methods are also disclosed.

Description

[Technical field]

[0001] The subject matter disclosed herein relates to the technical field of special purpose machines that facilitate the teaching of language, including software configured and computerized variations of such special purpose machines, and to techniques that enhance such special purpose machines relative to other special purpose machines that facilitate the teaching of language. In particular, the present disclosure addresses systems and methods that facilitate teaching one or more language skills, such as the pronunciation of words and phrases, to one or more users (e.g., students, children, or any suitable combination thereof). [Background technology]

[0002] The machine may be configured to teach language skills by presenting a graphical user interface (GUI) in the course of interacting with a user, whereby language lessons are presented on a display screen and the machine prompts the user to read aloud words and phrases that appear in the GUI presenting the language lessons.

[0003] Some embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. [Brief description of the drawings]

[0004] [Figure 1] FIG. 1 is a network diagram illustrating a network environment suitable for operating a server machine (eg, a language instruction server machine) according to some exemplary embodiments. [Diagram 2] 1 is a block diagram illustrating components of a headset suitable for use with a server machine, according to some exemplary embodiments. [Diagram 3] 1 is a block diagram illustrating components of a device suitable for use with a server machine, according to some exemplary embodiments. [Figure 4] 4 is a block diagram illustrating components of a server machine according to some exemplary embodiments. [Diagram 5]4 is a flow chart illustrating operation of a server machine in performing a method for teaching language skills (e.g., pronunciation of words) according to some exemplary embodiments. [Figure 6] 4 is a flow chart illustrating operation of a server machine in performing a method for teaching language skills (e.g., pronunciation of words) according to some exemplary embodiments. [Figure 7] 4 is a flow chart illustrating operation of a server machine in performing a method for teaching language skills (e.g., pronunciation of words) according to some exemplary embodiments. [Figure 8] FIG. 1 is a block diagram illustrating components of a machine capable of reading instructions from a machine-readable medium and performing any one or more of the methods described herein, in accordance with some example embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Exemplary methods (e.g., algorithms) facilitate teaching a language, and exemplary systems (e.g., special-purpose machines configured with special-purpose software) are configured to facilitate teaching a language. The examples are merely representative of possible variations. Unless expressly stated, structures (e.g., structural components such as modules) may be arbitrarily combined or further subdivided, and operations (e.g., procedures, algorithms, or other functions) may be sequentially altered or combined or subdivided. In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of various exemplary embodiments. However, it will be apparent to one skilled in the art that the subject matter of the present invention may be practiced without these specific details.

[0006] A set of one or more machines (e.g., computers or other devices) may be configured with appropriate hardware and software to collectively function as a language teaching lab (e.g., a fully or partially wearable, portable, or mobile language teaching lab) for one or more users. Such a language teaching lab may operate based on one or more of a variety of instructional principles, including, for example, that oral comprehension precedes written comprehension, that auditory phonemes occur early (e.g., first) in learning a language, that auditory separation from environmental noise (e.g., via one or more earphones) can facilitate learning a language, that oral recitation allows a user to compare a spoken phoneme with an auditory memory of that phoneme (e.g., in a feedback loop), and that mouth movements (e.g., mechanical movements of the user's mouth) are related to oral pronunciation. Thus, one or more machines in the language teaching lab may be configured to access multiple data sources and data types (e.g., video streams, audio streams, thermal imaging data, eye tracker data, breath anemometer data, biosensor data, accelerometer data, depth sensor data, or any suitable combination thereof), detect what the user has pronounced from the accessed data, and provide a presentation of a reference (e.g., correct or standard) pronunciation of the phrase, phrase, or sentence. The presentation of the reference pronunciation may include playing an audio of the reference pronunciation, playing a video of an actor speaking the reference pronunciation, displaying an animated model of a mouth or face speaking the reference pronunciation, displaying a texture of such an animated model mapped with an image of the user's own mouth or face speaking the reference pronunciation, or any suitable combination thereof.

[0007] FIG. 1 is a network diagram illustrating a network environment 100 suitable for operating a server machine 110 (e.g., a language instruction server machine) according to some exemplary embodiments. Network environment 100 includes server machine 110, database 115, headset 120, and device 130, all communicatively coupled to one another through network 190. Server machine 110, with or without database 115, may form all or part of a cloud 118 (e.g., a geographically distributed set of machines configured to function as a single server). Cloud 118 may form all or part of a network-based system 105 (e.g., a cloud-based server system configured to provide one or more network-based services to headset 120, device 130, or both). Server machine 110, database 115, headset 120, and device 130 may each be implemented, in whole or in part, in a special-purpose (e.g., specialized) computer system, as described below with reference to FIG. 8.

[0008] Also shown in FIG. 1 is a user 132, which may be a person (e.g., a child, a student, a language learner, or any suitable combination thereof). More generally, the user 132 may be a human user (e.g., a human), a machine user (e.g., a computer configured to interact with the device 130 by a software program), or any suitable combination thereof (e.g., a human assisted by a machine or a machine supervised by a human). The user 132 may be associated with the device 130 and may be a user of the device 130. For example, the device 130 may be a desktop computer, a vehicle computer, a home media system (e.g., a home theater system or other home entertainment system), a tablet computer, a navigation device, a portable media device, a smartphone, or a wearable device (e.g., a smart watch, smart glasses, smart clothing, or smart jewelry) owned by the user 132. Similarly, the user 132 may be associated with the headset 120 and may be a wearer of the headset 120. For example, the headset 120 may be worn on and operated from the head of the user 132. In some exemplary embodiments, headset 120 and the device are communicatively coupled to each other (e.g., independent of network 190), such as through a wired local or private network, a wireless network connection, or any suitable combination thereof.

[0009] Any of the systems or machines (e.g., databases, headsets, and devices) shown in FIG. 1 may include or be implemented in a special-purpose (e.g., specialized or non-conventional and non-general) computer that has been modified to perform one or more of the functions described herein for that system or machine (e.g., configured or programmed with special-purpose software, such as as one or more software modules of a special-purpose application, operating system, firmware, middleware, or other software program). For example, a special-purpose computer system capable of implementing any one or more of the methods described herein is described below with reference to FIG. 8, and accordingly, such a special-purpose computer may be a means for performing any one or more of the methods described herein. Within the technical field of such special-purpose computers, a special-purpose computer that is specifically modified (e.g., configured with special-purpose software) with the structure described herein to perform the functions described herein is technically improved compared to other special-purpose computers that do not have the structure described herein or are unable to perform the functions described herein. Thus, a special-purpose machine configured according to the systems and methods described herein provides an improvement over the technology of similar special-purpose machines.

[0010] As used herein, a "database" is a data storage resource and may store data structured as a text file, a table, a spreadsheet, a relational database (e.g., an object-relational database), a triple store, a hierarchical data store, or any suitable combination thereof. Additionally, any two or more of the systems or machines illustrated in Figure 1 may be combined into a single system or machine. The functionality described herein for any single system or machine may be subdivided among multiple systems or machines.

[0011] Network 190 may be a network that enables communication between systems, machines, databases, and devices (e.g., between server machine 110 and device 130). Thus, network 190 may be a wired network, a wireless network (e.g., a mobile or cellular network), or any combination thereof. Network 190 may include one or more portions that constitute a private network, a public network (e.g., the Internet), or any suitable combination thereof. Thus, network 190 may include one or more portions that incorporate a local area network (LAN), a wide area network (WAN), the Internet, a mobile telephone network (e.g., a cellular network), a wired telephone network (e.g., a Plain Old Telephone Service (POTS) network), a wireless data network (e.g., a WiFi network or a WiMax network), or any suitable combination thereof. Any one or more portions of network 190 may communicate information over a transmission medium. As used herein, "transmission medium" refers to any intangible (e.g., transmissible) medium capable of communicating (e.g., transmitting) instructions for execution by a machine (e.g., by one or more processors of such a machine) and includes digital or analog communication signals or other intangible media that communicate such software.

[0012] 2 is a block diagram illustrating components of a headset 120, according to some exemplary embodiments. Headset 120 is shown to include an inward-facing camera 210 (e.g., pointed toward or oriented to view the mouth of user 132 when headset 120 is worn), an outward-facing camera 220 (e.g., pointed toward or oriented to view the area in front of user 132 when headset 120 is worn), a microphone 230 (e.g., pointed toward or positioned near the mouth of user 132), and a speaker 240 (e.g., an audio speaker such as a headphone, an earpiece, an earbud, or any suitable combination thereof). Some exemplary embodiments of the headset (e.g., for some speech therapy applications) omit the outward-facing camera 220 or ignore the video stream of the outward-facing camera 220.

[0013] Headset 120 is also shown to include a thermal imaging device 250, an eye tracker 251 (e.g., pointed at or oriented to look at one or both eyes of user 132 when headset 120 is worn), an anemometer 252 (e.g., a breath anemometer pointed at or positioned near the mouth of user 132 when headset 120 is worn), and a set of one or more biosensors 253 (e.g., positioned or configured to measure heart rate (HR), galvanic skin response (GSR), other skin conditions, or any suitable combination thereof when user 132 is wearing headset 120).

[0014] In the illustrated exemplary embodiment, headset 120 further comprises one or more accelerometers 254 (e.g., positioned or configured to measure movement of, for example, the mouth of user 132, the tongue of user 132, the throat of user 132, or any suitable combination thereof, when headset 120 is worn), muscle stimulator 255 (e.g., a set of one or more neuromuscular electrical muscle stimulators positioned or configured to simulate one or more muscles of user 132 when headset 120 is worn), laser 256 (e.g., a low power or child-safe laser pointer pointed or oriented to emit a laser beam towards an area in front of user 132 when headset 120 is worn), and depth sensor 257 (e.g., an infrared or other type of depth sensor pointed or oriented to detect depth data in an area in front of user 132 when headset 120 is worn). As shown in FIG. 2, the various above components of headset 120, or any subgroups thereof, are configured to communicate with each other (e.g., by a bus, shared memory, or switch).

[0015] 3 is a block diagram illustrating components of device 130, according to some exemplary embodiments. Device 130 is shown to include a reading instruction module 310 (e.g., software-controlled hardware configured to interact with user 132 in presenting one or more reading tutorials), a speaking instruction module 320 (e.g., software-controlled hardware configured to interact with user 132 in presenting one or more speech tutorials), an instruction game module 330 (e.g., software-controlled hardware configured to interact with user 132 in presenting one or more instruction games), and a display screen 340 (e.g., a touch screen or other display screen). As shown in FIG. 3, the various above components of device 130 or any subgroups thereof are configured to communicate with each other (e.g., by a bus, shared memory, or switch).

[0016] 3, the read instructions module 310, the speak instructions module 320, the instruction game module 330, or any combination thereof may form all or part of an app 300 (e.g., a mobile app) that is stored (e.g., installed) on the device 130 (e.g., in response to or as a result of data being received over the network 190 by the device 130). Additionally, one or more processors 399 (e.g., hardware processors, digital processors, or any suitable combination thereof) may be included (e.g., temporarily or permanently) in the app 300, the read instructions module 310, the speak instructions module 320, the instruction game module 330, or any suitable combination thereof.

[0017] 4 is a block diagram illustrating components of server machine 110, according to some exemplary embodiments. Server machine 110 is shown to include a data access module 410, a data analysis module 420, and a pronunciation correction module 430, all configured to communicate with each other (e.g., by a bus, shared memory, or switch).

[0018] 4, the data access module 410, the data analysis module 420, and the pronunciation correction module 430 may form all or part of an app 400 (e.g., a server-side app) that is stored (e.g., installed) on the server machine 110 (e.g., in response to or as a result of data being received over the network 190). Additionally, one or more processors 499 (e.g., hardware processors, digital processors, or any suitable combination thereof) may be included (e.g., temporarily or permanently) in the app 400, the read instructions module 410, the speak instructions module 420, the instruction game module 430, or any suitable combination thereof.

[0019] Any one or more of the components (e.g., modules) described herein may be implemented using only hardware (e.g., one or more of processors 399, or one or more of processors 499, as appropriate) or a combination of hardware and software. For example, any component described herein may physically comprise one or more arrays of processors 399 or 499 (e.g., of or between processors 399 or 499), as appropriate, configured such that the component performs the operations described herein. As another example, any component described herein may comprise software, hardware, or both that configure one or more arrays of processors 399 or 499, as appropriate, such that the component performs the operations described herein. Thus, the various components described herein may comprise various arrays of processors 399 or 499 at various times, or a single array of processors 399 or 499 at various times. Each component (e.g., module) described herein is an example of a means by which the component performs the operations described herein. Additionally, any two or more components described herein may be combined into a single component. Functionality described herein for a single component may be subdivided among multiple components. Further, according to various exemplary embodiments, components described herein as implemented in a single system or machine (e.g., a single device) may be distributed across multiple systems or machines (e.g., multiple devices).

[0020] According to various exemplary embodiments, headset 120, server machine 110, device 130, or any combination thereof, functions as a mobile language learning lab for user 132. Such a language learning lab provides instruction in one or more language skills, training in those language skills, or both to user 132. This language learning lab may be enhanced by providing pronunciation analysis, contextual reading, motor muscle memory recall analysis, auditory feedback, playback object identification, handwriting recognition, gesture recognition, eye tracking, biometric analysis, or any suitable combination thereof.

[0021] 5-7 are flow charts illustrating operations of server machine 110 in performing a method 500 for teaching language skills (e.g., pronunciation of a word) according to some exemplary embodiments. The operations in method 500 may be performed by server machine 110, headset 120, device 130, or any suitable combination thereof, using the components (e.g., modules) described above with respect to Figures 2-4, or any suitable combination thereof. As shown in Figure 5, method 500 includes operations 510, 520, 530, and 540.

[0022] In operation 510, data access module 410 accesses two video and audio streams. In particular, the accessed streams are or include external and internal video streams (e.g., external and internal video streams) and an audio stream, both provided by headset 120, which includes outward-facing camera 220, inward-facing camera 210, and microphone 230. Outward-facing camera 220 of headset 120 has an outer field of view that extends away from the wearer of headset 120 (e.g., user 132). Outward-facing camera 220 generates an external video stream based on (e.g., with or from) the outer field of view. Inward-facing camera 210 of headset 120 has an inner field of view that extends towards the wearer of headset 120. Inward-facing camera 210 generates an internal video stream based on (e.g., with or from) the inner field of view. The audio stream is generated by microphone 230. In exemplary embodiments in which the headset omits the outward-facing camera 220 or ignores the external video stream (e.g., for some speech therapy applications), the data access module 410 similarly omits or ignores the external video stream.

[0023] In operation 520, the data analysis module 420 detects three co-occurrences based on the streams accessed in operation 510: visual events in the outer field of view, mouse gestures in the inner field of view, and candidate pronunciations of the phrase. The visual events are represented in the accessed external video stream, the mouse gestures are represented in the accessed internal video stream, and the candidate pronunciations are represented in the accessed audio stream. In an exemplary embodiment where the headset omits the outward-facing camera 220 or ignores the external video stream (e.g., for some speech therapy applications), the data analysis module 420 detects two things: the co-occurrence of mouse gestures in the inner field of view and the candidate pronunciations of the phrase.

[0024] At operation 530, the pronunciation correction module 430 determines (e.g., by querying the database 115) that the visual event is associated with the phrase by the database 115 and is associated with a reference pronunciation of the phrase. This determination may be made optionally by recognizing the occurrence of the phrase (e.g., by optional character recognition) or an object associated with the phrase (e.g., by shape recognition) in the external field of view. In an exemplary embodiment in which the headset omits the outward-facing camera 220 or ignores the external video stream (e.g., for some speech therapy applications), the pronunciation correction module 430 determines (e.g., by querying the database 115) that the phrase is associated with a reference pronunciation of the phrase.

[0025] At operation 540, in response to the detected coincidence of a visual event with a mouse gesture and with a candidate pronunciation of the phrase, the pronunciation correction module 430 causes the headset 120 to present (e.g., triggered, controlled, commanded, for example, by a remote control signal) a reference pronunciation of the phrase to the wearer. In an example embodiment in which the headset omits the outward-facing camera 220 or ignores an external video stream (e.g., for some speech therapy applications), the pronunciation correction module 430 causes the headset 120 to present a reference pronunciation of the phrase in response to the detected coincidence of a mouse gesture with a candidate pronunciation of the phrase.

[0026] 6, in addition to any one or more of the operations previously described, method 500 may include one or more of operations 620, 621, 622, 623, 640, 641, 650, 651, and 660. One or more of operations 620, 621, 622, and 623 may be performed as part (e.g., a predecessor task, subroutine, or portion) of operation 520. In operation 520, data analysis module 420 detects a three-way coincidence of a visual event in the outer field of view accompanied by a mouse gesture in the inner field of view and accompanied by a candidate pronunciation in the audio stream.

[0027] In operation 620, as all or part of the process of detecting a visual event, the data analysis module 420 detects a hand gesture, i.e., a touch made by the hand of the user 132 and occurring on or near a visible phrase (e.g., displayed by the device 130 in the outer field of view). The associated threshold of proximity may be a distance sufficient to distinguish the visible phrase from any other phrases visible in the outer field of view. For example, the detected hand gesture may point to the visible phrase or may identify the visible phrase (e.g., to indicate a request for assistance in reading or pronouncing the visible phrase). For example, the detected hand gesture may point to the visible phrase or may identify the visible phrase (e.g., to indicate a request for assistance in reading or pronouncing the visible phrase). As a further example, the data analysis module 420 may detect the hand of the user 132 writing or tracing the visible phrase (e.g., to indicate a request for assistance in reading or pronouncing the visible phrase). As yet a further example, the data analysis module 420 may detect the hand of the user 132 underlining or highlighting a visible phrase (e.g., to indicate a request for assistance in reading or pronouncing the visible phrase). Thus, a visual event detected in the outer field of view may include the hand of the user 132 writing a phrase, tracing a phrase, pointing to a phrase, touching a phrase, underlining a phrase, highlighting a phrase, or any suitable combination thereof. In response to a detected hand gesture or hand touch, the visible phrase identified by the hand gesture or hand touch may be treated as a phrase for which a candidate pronunciation is represented in the audio stream generated by the microphone 230.

[0028] In operation 621, as all or part of detecting a visual event, the data analysis module 420 detects the hand of the user 132 touching or moving a physical object representing a word or phrase, where the physical object is visible in the outer field of view. For example, the physical object may be a model of an animal, such as a horse or a dog. Or, the physical object may be a toy or block with a word or phrase printed or displayed thereon. The movement of the physical object may be or may include a rotation in space in the outer field of view, a translation in space in the outer field of view, or both. Thus, a visual event detected in the outer field of view may include the hand of the user 132 touching a physical object (e.g., a physical model), grabbing a physical object, moving a physical object, rotating a physical object, or any suitable combination. In response to the detected touch or movement of the physical object by the hand of the user 132, the word or phrase associated with the physical object (e.g., displayed by the physical object or associated with the physical object by the database 115) may be treated as the word or phrase for which a candidate pronunciation is represented in the audio stream generated by the microphone 230.

[0029] In operation 622, as all or part of detecting a visual event, the data analysis module 420 detects a trigger gesture made by the hand of the user 132 in the outer field of view. For example, the trigger gesture may be or may include making a predetermined hand shape, a predetermined pose with one or more fingers, a predetermined movement with the hand, or any suitable combination thereof. In response to the detected trigger gesture, a phrase for which a candidate pronunciation is represented in the audio stream produced by the microphone 230 may be identified as requesting assistance in reading or pronouncing the phrase (e.g., requesting a correction of the candidate pronunciation represented in the audio stream produced by the microphone 230).

[0030] In operation 623, as all or part of detecting a visual event, data analysis module 420 detects a laser spot (e.g., a bright spot of laser light) on a surface of a physical object in the outer field of view. For example, headset 120 may include an outwardly facing laser 256 (e.g., a laser pointer or other laser emitter) configured to indicate an object in the outer field of view by causing a spot of laser to appear on the surface of the object in the outer field of view. Outwardly facing camera 220 of headset 120 may also be configured to capture the spot of laser light and the indicated object in the outer field of view. Thus, the detected visual event in the outer field of view may include a spot of laser light appearing on a surface of a physical object in the outer field of view. In response to a detected spot of laser light appearing on a surface of the physical object, a phrase associated with the physical object (e.g., associated with the physical object by database 115) may be treated as a phrase for which a candidate pronunciation for the phrase is represented in the audio stream generated by microphone 230.

[0031] One or more of operations 640 and 641 may be performed as part (e.g., a predecessor task, subroutine, or portion) of operation 540. In operation 540, pronunciation correction module 430 causes headset 120 to present a reference pronunciation of the phrase to a wearer of headset 120 (e.g., user 132).

[0032] In operation 640, the pronunciation correction module 430 accesses a set of canonical phonemes included in the canonical pronunciation of the phrase. The set of canonical phonemes may be stored in and accessed from the database 115.

[0033] In operation 641, the pronunciation correction module 430 causes the speaker 240 in the headset 120 to play the set of canonical phonemes accessed in operation 640. The speed at which the canonical phonemes are played may vary and may be determined based on a variety of factors, as described below with respect to Figure 7. With reference to Figure 6, one or more of operations 650, 651, and 660 may be performed at any time after operation 510, and may be shown throughout the illustrated exemplary embodiment to be performed after operation 540.

[0034] In operation 650, the pronunciation correction module 430 accesses a reference set of mouth shapes (images or models of mouth shapes), each configured to speak a corresponding reference phoneme included in the reference pronunciation of the phrase. The reference set of mouth shapes may be stored in and accessed from the database 115. In some exemplary embodiments, the pronunciation correction module 430 also accesses an image of the user's own mouth or face (e.g., from the database 115) to combine with (e.g., texture map to or morph with) the reference set of mouth shapes.

[0035] In operation 651, the pronunciation correction module 430 causes the display screen to display the accessed reference set of mouth shapes to the wearer of the headset 120. In some exemplary embodiments, the headset 120 and the display screen (e.g., display screen 340) are caused to simultaneously present (e.g., in audio form) the reference pronunciation of the phrase to the wearer of the headset 120 and display (e.g., in visual form on the display screen 340) the accessed reference set of mouth shapes to the wearer of the headset 120. In some exemplary embodiments, the pronunciation correction module 430 combines (e.g., texture maps or morphs) the reference set of mouth shapes with an image of the user's own mouth or face and causes the display screen to present the resulting combination (e.g., simultaneously with the reference pronunciation of the phrase).

[0036] In operation 660, the inward-facing camera 210 of the headset 120 captures the mouth of the wearer of the headset 120 in the inner field of view. The data analysis module 420 also anonymizes the mouse gestures by cropping a portion of the inner field of view. The resulting cropped portion shows the mouse gestures without showing either eye of the wearer of the headset 120. This limited description may be useful in situations where it is important that the privacy of the wearer is maintained, such as situations where it is beneficial to avoid capturing facial features (e.g., one or both eyes) that are usable by face recognition software. In such an exemplary embodiment, the mouse gestures that are anonymized in the inner field of view are detected within the cropped portion of the inner field of view.

[0037] 7, in addition to any one or more of the operations previously described, method 500 may include one or more of operations 710, 711, 712, 713, 714, 715, 716, 730, 750, 751, 760, and 761. One or more of operations 710-716 may be performed prior to operation 520. In operation 520, data analysis module 420 detects three-way co-occurrences of visual events in the outer field of view with mouse gestures in the inner field of view, and with candidate pronunciations in the audio stream. According to various exemplary embodiments, the detection of the co-occurrences may be further based on one or more factors (e.g., conditions) detectable by data accessed in one or more of operations 710-716.

[0038] In operation 710, data access module 410 accesses a thermal image of a hand of a wearer (e.g., a user) of headset 120. For example, outward-facing camera 220 may include a thermal imaging component (e.g., thermal imager 250 or the like) configured to capture a thermal imaging component of objects in the outer field of view, or the thermal imaging component (e.g., thermal imager 250) may be a separate component of headset 120 and oriented to capture thermal images of objects in the outer field of view. Thus, visual events in the outer field of view may be detected based on a thermal image of the hand of the headset wearer.

[0039] In operation 711, data access module 410 accesses a thermal image of the mouth (e.g., showing a tongue or showing the shape of the tongue, tongue parts, or both) of a wearer of headset 120 (e.g., user 132). For example, inward-facing camera 210 may include a thermal imaging component (e.g., thermal imager 250 or the like) configured to capture thermal images of objects in the inner field of view. Alternatively, the thermal imaging component (e.g., thermal imager 250) may be a separate component of headset 120 and oriented to capture thermal images of objects in the inner field of view. Thus, mouse gestures in the inner field of view may be detected based on a thermal image of the mouth of the wearer of headset 120.

[0040] In operation 712, the data access module 410 accesses eye tracker data indicative of an eye orientation of a wearer of the headset 120 (e.g., user 132). For example, the headset 120 may further comprise an eye tracking camera (e.g., eye tracker 251). The eye tracking camera may have an additional field of view and may be configured to capture the orientation of the wearer's eye or eyes in the additional field of view. Thus, the data analysis module 420 may determine the direction in which the wearer's eye or eyes are looking based on the eye orientation indicated in the eye tracker data. Also, visual events in the outer field of view may be detected based on the observation direction in which the wearer's eyes are looking. For example, the determined observation direction may be the basis for detecting visual events (e.g., disambiguation or identification of a phrase in which a candidate pronunciation is represented in the audio stream generated by the microphone 230).

[0041] In operation 713, the data access module 410 accesses anemometer data indicative of one or more breath rates of a wearer of the headset 120 (e.g., user 132). For example, the headset 120 may include an anemometer (e.g., anemometer 252) configured to detect a rate of breath entering or leaving the mouth of the wearer of the headset 120. Thus, causing the headset 120 to present a reference pronunciation of the phrase in operation 540 may be based on the detected breath rate of the wearer of the headset 120. For example, if the anemometer data indicates improper breathing of the phrase, the pronunciation correction module 430 may generate or access (e.g., from database 115) a highly intelligible reference pronunciation of the phrase, or may obtain a highly intelligible reference pronunciation of the phrase and present (e.g., play) the highly intelligible pronunciation to the wearer of the headset 120.

[0042] In operation 714, the data access module 410 accesses biosensor data indicative of one or more physiological conditions of a wearer of the headset 120 (e.g., user 132). The biosensor data may be accessed from one or more biosensors (e.g., biosensor 253) included in or communicatively coupled to the headset 120. For example, one or more of the biosensors 253 may be located within the headset 120, communicatively coupled to the headset 120, or configured to measure the wearer's heart rate, the wearer's galvanic skin response, one or more other skin conditions (e.g., temperature or elasticity) of the wearer, the wearer's brain waves, one or more brain states of the wearer, or any suitable combination thereof. Thus, the pronunciation correction module 430 may determine the speed at which the reference pronunciation of the phrase is played (e.g., relative to the wearer) based on the information indicated in the accessed biosensor data.

[0043] In operation 715, the data access module 410 accesses accelerometer data indicative of one or more muscle movements by a wearer of the headset 120 (e.g., the user 132). The accelerometer data may be accessed from one or more accelerometers (e.g., the accelerometers 254) included in or communicatively coupled to the headset 120. For example, one or more of the accelerometers 254 may be located within the headset 120, may be communicatively coupled to the headset 120 (e.g., included in a collar worn by the wearer of the headset 120), or may be configured to detect (e.g., by measurement) one or more muscle movements made by the wearer while a candidate pronunciation of a phrase is being made. Thus, the pronunciation correction module 430 may detect a pattern of muscle movements based on the accessed accelerometer data. Also, causing the headset 120 to present a reference pronunciation of the phrase during operation may be based on the detected pattern of muscle movements. For example, if the accelerometer data indicates an inappropriate pattern of muscle movement when a candidate pronunciation of the phrase is made, the pronunciation correction module 430 may generate or access (e.g., from database 115) a highly clear reference pronunciation of the phrase, or may obtain a highly clear reference pronunciation of the phrase and present (e.g., play) the highly clear pronunciation to the wearer of headset 120.

[0044] In operation 716, the data access module 410 accesses depth sensor data indicative of a distance to an object in the outer field of view. The depth sensor data may be accessed from one or more depth sensors (e.g., depth sensor 257) included in or communicatively coupled to the headset 120. For example, the depth sensor 257 may be a stereoscopic infrared depth sensor configured to detect a distance to a physical object in the outer field of view. In some example circumstances, the outward-facing camera 220 of the headset 120 is configured to capture a hand of a wearer of the headset 120 (e.g., user 132) indicating a physical object in the outer field of view by touching a physical object at a distance detected by the depth sensor. Furthermore, the indicated object may be associated with a phrase for which a candidate pronunciation is represented in the audio stream generated by the microphone 230, and may also be associated with a reference pronunciation of the phrase. Thus, a visual event in the outer field of view may be or may include a wearer's hand touching an indicated object in the outer field of view.

[0045] In operation 730, the pronunciation correction module 430 determines a speed at which the reference pronunciation of the phrase is played. For example, the pronunciation correction module 430 may determine a playback speed (e.g., 1x, 0.9x, 1.2x, or 0.5x) for the reference pronunciation. The playback speed may be determined based on results from one or more of operations 712-715. As an example, the data analysis module 420 may detect that a wearer of the headset 120 (e.g., user 132) exhibits a state of stress, fatigue, frustration, or other physiologically detectable condition when performing the candidate pronunciation of the phrase. The detection may be based on the eye tracker data accessed in operation 712, the anemometer data accessed in operation 713, the biosensor data accessed in operation 714, the accelerometer data accessed in operation 715, or any suitable combination thereof. Based on the detected condition, the pronunciation correction module 430 may vary the playback speed of the reference pronunciation. Thus, causing headset 120 to present the reference pronunciation of the phrase in act 540 may be based on the playback speed determined in act 730, and the reference pronunciation may consequently be played back at that playback speed.

[0046] In an exemplary embodiment, the pronunciation correction module 430, when performing operation 730, determines that the speed at which the reference pronunciation is played is zero or a null value for the speed. In particular, if the data analysis module 420 detects a state of sufficiently high stress, fatigue, frustration, or other physiologically detectable condition (e.g., exceeding a threshold level) in performing the candidate pronunciation of the phrase, the pronunciation correction module 430 triggers a suggestion, recommendation, or other indication for the wearer (e.g., user 132) to take a break and to resume performing the candidate pronunciation of the phrase after a period of recovery time. In such a situation, the playing of the reference pronunciation of the phrase may be omitted or replaced by the triggered suggestion, recommendation, or other indication to take a break.

[0047] In operation 750, the pronunciation correction module 430 accesses a reference pattern of muscle movements configured to speak a reference pronunciation of the phrase. For example, the reference pattern of muscle movements may be stored in or accessed from the database 115.

[0048] In operation 751, the pronunciation correction module 430 causes one or more muscle stimulators (e.g., muscle stimulator 255, which may be or include a neuromuscular electrical muscle stimulator) to stimulate a set of one or more muscles of a wearer of the headset 120 (e.g., user 132). As an example, the muscle stimulator 255 may be included in the headset 120, may be communicatively coupled to the headset 120 (e.g., included in a collar that is communicatively coupled to the headset 120), or may be configured to stimulate a set of muscles of the wearer. Thus, the set of muscles may be exercised according to a reference pattern of muscle movement (e.g., by neuromuscular electrical stimulation (NMES)). In some exemplary embodiments, this source of muscle movement is performed with one or more repetitions of operation 540. In operation 540, a reference pronunciation of the phrase is presented to the wearer of the headset 120 (e.g., to help the wearer practice how to articulate or perform the reference pronunciation of the phrase).

[0049] In operation 760, the pronunciation correction module 430 compares the candidate pronunciation of the phrase to the reference pronunciation of the phrase. This comparison may be made on a phoneme-by-phoneme basis, such that a first sequential phoneme in the candidate pronunciation is compared to a corresponding first phoneme in the reference pronunciation, a second sequential phoneme in the candidate pronunciation is compared to a corresponding second phoneme in the reference pronunciation, etc.

[0050] At operation 761, based on the comparison made at operation 760, the pronunciation correction module 430 recommends a pronunciation tutorial to the wearer of the headset 120 (e.g., the user 132). For example, the pronunciation correction module may cause an indication (e.g., a dialog box, an alert, an audio message, any suitable combination thereof) that a pronunciation tutorial has been recommended to the wearer. In some exemplary embodiments, the wearer may respond to accepting the recommendation. In response to accepting the recommendation, the pronunciation correction module 430 may also cause the reading instruction module 310 to initiate the presentation of a reading tutorial that teaches one or more reading skills used in reading a phrase, the speaking instruction module 320 to initiate the presentation of a speech tutorial that teaches one or more speech skills used in pronouncing a phrase, the instruction game module 330 to initiate an instruction game that teaches one or more of the reading or speaking skills, or any suitable combination thereof.

[0051] According to various exemplary embodiments, one or more of the methods described herein may facilitate the teaching of a language, or from another perspective, the learning of a language. Additionally, one or more of the methods described herein may facilitate instructing a user 132 in listening to, practicing, and correcting the proper pronunciation of a phoneme, phrase, sentence, or any suitable combination thereof. Thus, one or more of the methods described herein may facilitate the teaching of a language by facilitating the learning of a learner of the language as compared to the performance of pre-existing systems and methods.

[0052] When these effects are considered in aggregate, one or more of the methods described herein may eliminate the need for certain efforts or resources associated with language instruction or language learning. The efforts made by a user in learning a language skill, by a language teacher teaching such a language skill, or both, may be reduced by the use of a special-purpose machine implementing one or more of the methods described herein. The computer resources used by one or more systems or machines (e.g., in the network environment 100) may be similarly reduced (e.g., compared to a system or machine that does not have the structure described herein or is incapable of performing the functions described herein). Examples of such computer resources include processor cycles, network traffic, computation capacity, main memory usage, graphics rendering capacity, graphics memory usage, data storage capacity, power consumption, and cooling capacity.

[0053] 8 is a block diagram illustrating components of a machine 800 that can read instructions 824 from a machine-readable medium 822 (e.g., a non-transitory machine-readable medium, a machine-readable storage medium, a computer-readable storage medium, or any suitable combination thereof) and can perform, in whole or in part, any one or more of the methodologies discussed herein, according to some exemplary embodiments. In particular, FIG. 8 illustrates a machine 800 in the example form of a computer system (e.g., a computer) within which instructions 824 (e.g., software, programs, applications, applets, apps, or other executable code) can be executed, in whole or in part, to cause the machine 800 to perform any one or more of the methodologies discussed herein.

[0054] In alternative embodiments, machine 800 may operate as a stand-alone device or may be communicatively coupled (e.g., networked) to other machines. In a networked deployment, machine 800 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a distributed (e.g., peer-to-peer) network environment. Machine 800 may be a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a mobile phone, a smart phone, a set-top box (STB), a personal digital assistant (PDA), a web appliance, a network router, a network switch, a network bridge, or any machine capable of executing instructions 824, sequentially or otherwise, that specify actions to be taken by that machine. Additionally, although only a single machine is illustrated, the term "machine" shall be deemed to include any collection of machines that individually or jointly execute instructions 824 to perform all or a portion of any one or more methodologies discussed herein.

[0055] The machine 800 comprises a processor 802 (e.g., one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more digital signal processors (DSPs), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any suitable combination thereof), a main memory 804, and a static memory 806, which are configured to communicate with each other via a bus 808. The processor 802 includes solid-state digital microcircuits (e.g., electronic, optical, or both) that are temporarily or permanently configurable by some or all of the instructions 824 such that the processor 802 is configurable to, in whole or in part, execute any one or more of the methodologies described herein. For example, a set of one or more microcircuits of the processor 802 may be configurable to execute one or more modules (e.g., software modules) described herein. In some demonstrative embodiments, processor 802 is a multi-core CPU (e.g., a dual-core CPU, a quad-core CPU, an 8-core CPU, or a 128-core CPU) in which each of a plurality of cores operates as a separate processor capable of performing, in whole or in part, any one or more of the methodologies described herein. Although the beneficial effects described herein may be provided by machine 800 having at least processor 802, these same beneficial effects may be provided by another type of machine that does not include a processor (e.g., a purely mechanical system, a purely hydraulic system, or a hybrid mechanical-hydraulic system), if the processor-less machine is configured to perform one or more of the methodologies described herein.

[0056] The machine 800 may further include a graphics display 810 (e.g., a plasma display panel (PDP), a light emitting diode (LED) display, a liquid crystal display (LCD), a projector, a cathode ray tube (CRT), or any other display capable of displaying graphics or video). The machine 800 may also include an alphanumeric input device 812 (e.g., a keyboard or keypad), a pointer input device 814 (e.g., a mouse, a touchpad, a touch screen, a trackball, a joystick, a stylus, a motion sensor, an eye tracking device, a data glove, or other pointing instrument), data storage 816, an audio generation device 818 (e.g., a sound card, an amplifier, a speaker, a headphone jack, or any suitable combination thereof), and a network interface device 820.

[0057] The data storage 816 (e.g., a data storage device) includes a machine-readable medium 822 (e.g., a tangible and non-transitory machine-readable storage medium) on which are stored instructions 824 that embody any one or more of the methodologies or functions described herein. Also, the instructions 824 may reside, completely or at least partially, in the main memory 804, in the static memory 806, in the processor 802 (e.g., in a processor's cache memory), or any suitable combination thereof, prior to or during their execution by the machine 800. Thus, the main memory 804, the static memory 806, and the processor 802 may be considered machine-readable media (e.g., a tangible and non-transitory machine-readable medium). The instructions 824 may be transmitted or received over the network 190 via the network interface device 820. For example, the network interface device 820 may communicate the instructions 824 using any one or more transfer protocols (e.g., Hypertext Transfer Protocol (HTTP)).

[0058] In some exemplary embodiments, the machine 800 may be a portable computing device (e.g., a smartphone, a tablet computer, or a wearable device) and may have one or more additional input components (inputs) 830 (e.g., sensors or gauges). Examples of such input components 830 include an image input (e.g., one or more cameras), an audio input (e.g., one or more microphones), an orientation input (e.g., a compass), a location input (e.g., a Global Positioning System (GPS) receiver, etc.), a direction meter (e.g., a gyroscope, etc.), a motion detector (e.g., one or more accelerometers, etc.), an altitude detector (e.g., an altimeter, etc.), a temperature input (e.g., a thermometer, etc.), and a gas detector (e.g., a gas sensor, etc.). Input data collected by any one or more of these input components 830 may be accessible and usable by any of the modules described herein (e.g., with appropriate privacy notices and protections, such as opt-in or opt-out consents, implemented in accordance with user preferences, applicable regulations, or any suitable combination thereof).

[0059] As used herein, the term "memory" refers to a machine-readable medium capable of storing data temporarily or permanently, and may include, but is not limited to, random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. Although machine-readable medium 822 is shown to be a single medium in the exemplary embodiment, the term "machine-readable medium" should be considered to include a single medium or multiple media capable of storing instructions (e.g., a centralized or distributed database, or associated caches and servers). The term "machine-readable medium" should also be considered to include any medium, or combination of multiple media, capable of carrying (e.g., storing or communicating) instructions 824 for execution by machine 800 such that, when executed by one or more processors (e.g., processor 802) of machine 800, the instructions 824 cause machine 800 to perform, in whole or in part, any one or more of the methodologies described herein. Thus, "machine-readable medium" refers not only to a single storage device or device, but also to a cloud-based storage system or storage network including multiple storage devices or devices. The term "machine-readable medium" is accordingly intended to include, but is not limited to, one or more tangible, non-transitory data repositories (e.g., data volumes) in the exemplary form of solid-state memory chips, optical disks, magnetic disks, or any suitable combination thereof.

[0060] As used herein, "non-transient" machine-readable media specifically excludes signals that propagate themselves. According to various exemplary embodiments, the instructions 824 for execution by the machine 800 may be communicated via a carrier medium (e.g., a machine-readable carrier medium). Examples of such carrier media include non-transitory carrier media (e.g., a non-transitory machine-readable storage medium such as a solid-state memory that is physically movable from one place to another) and transitory carrier media (e.g., a carrier wave or other propagated signal that communicates the instructions 824).

[0061] Certain exemplary embodiments are described herein as including modules. A module may constitute a software module (e.g., code stored or otherwise embodied in a machine-readable medium or transmission medium), a hardware module, or any suitable combination thereof. A "hardware module" is a tangible (e.g., non-transitory) physical component (e.g., a set of one or more processors) capable of performing certain operations, and may be configured or arranged in a particular physical manner. In various exemplary embodiments, one or more computer systems or one or more hardware modules thereof may be configured as a hardware module that operates by software (e.g., an application or portion thereof) to perform the operations described herein for that module.

[0062] In some exemplary embodiments, a hardware module may be implemented mechanically, electronically, hydraulically, or any suitable combination thereof. For example, a hardware module may comprise dedicated circuitry or logic that is permanently configured to perform certain operations. A hardware module may be, or comprise, a dedicated processor, such as a field programmable gate array (FPGA) or ASIC. A hardware module may also comprise programmable logic or circuitry that is temporarily configured by software to perform certain operations. As an example, a hardware module may comprise software contained in a CPU or other programmable processor. It should be appreciated that the decision to implement a hardware module mechanically, hydraulically, in dedicated permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be made based on cost and time considerations.

[0063] Thus, the phrase "hardware module" should be understood to encompass a tangible entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a particular manner or perform particular operations as described herein. Additionally, as used herein, the phrase "hardware-implemented module" refers to a hardware module. Given an exemplary embodiment in which the hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, if a hardware module includes a CPU configured by software to be a special-purpose processor, the CPUs may be configured as different special-purpose processors (e.g., each included in a different hardware module) at different times. The software (e.g., software modules) may accordingly configure one or more processors to be or be otherwise configured to be a particular hardware module at one time and to be or be otherwise configured to be a different hardware module at a different time.

[0064] Hardware modules can provide information to and receive information from other hardware modules. Thus, the described hardware modules can be considered to be communicatively coupled. When multiple hardware modules exist contemporaneously, communication can be achieved by signal transmission (e.g., via circuits and buses) between or among two or more hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communication between such hardware modules can be achieved, for example, through the storage and retrieval of information in a memory structure accessible to the multiple hardware modules. For example, one hardware module can perform an operation and store the output of the operation in a communicatively coupled memory (e.g., a memory device). At a later date, another hardware module can access the memory and retrieve and process the stored output. Additionally, hardware modules can initiate communication with input or output devices and can operate on resources (e.g., a collection of information from a computing resource).

[0065] Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily (e.g., by software) or permanently configured to perform the associated operations. Such processors, whether temporarily or permanently configured, may constitute processor-implemented modules that operate to perform one or more operations or functions described herein. As used herein, a "processor-implemented module" refers to a hardware module in which the hardware includes one or more processors. Thus, since a processor is an example of hardware, the operations described herein may be at least in part processor-implemented, hardware-implemented, or both, and at least some operations in any one or more of the methods described herein may be performed by one or more processor-implemented modules, hardware-implemented modules, or any suitable combination thereof.

[0066] Further, such one or more processors may perform operations in a "cloud computing" environment or as a service (e.g., within a "software as a service" (SaaS) implementation). For example, at least some operations within any one or more of the methods described herein may be performed by a group of computers (e.g., as an example of multiple machines including multiple processors), and these operations are accessible over a network (e.g., the Internet) and via one or more suitable interfaces (e.g., application program interfaces (APIs)). Execution of a particular operation may be distributed to one or more processors, whether present only within a single machine or spread across multiple machines. In some exemplary embodiments, one or more processors or hardware modules (e.g., processor-implemented modules) may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other exemplary embodiments, one or more processors or hardware modules may be distributed across multiple geographic locations.

[0067] In this specification, multiple instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed simultaneously and there is no requirement that the operations be performed in the order illustrated. In the illustrated configurations, structures and functions presented as separate components and functions may be implemented as combined structures or components with combined functionality. Similarly, structures and functions presented as single components may be implemented as separate components and functions. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter of this specification.

[0068] Some of the subject matter discussed herein may be presented in terms of algorithms or symbolic representations of operations on data stored as bits or binary digital signals in a memory (e.g., a computer memory or other machine memory). Such algorithms or symbolic representations are examples of techniques used by those skilled in the art of data processing to convey the substance of their work to others skilled in the art. As used herein, an "algorithm" refers to a self-consistent sequence of operations or similar processes leading to a desired result. In this context, algorithms and operations include physical operations of physical quantities. Such quantities may take the form of, but are not necessarily, electrical, magnetic, or optical signals that can be stored, accessed, transferred, combined, compared, or otherwise operated on by a machine. It is sometimes convenient, primarily for reasons of common usage, to refer to such signals using terms such as "data," "contents," "bits," "values," "elements," "symbols," "characters," "terms," ​​"numbers," "numbering," and the like. However, these terms are merely convenient labels and should be associated with the appropriate physical quantities.

[0069] Unless otherwise indicated, discussions herein using words such as "accessing," "processing," "detecting," "computing," "calculating," "determining," "generating," "presenting," "displaying," and the like refer to actions or processes that can be performed by a machine (e.g., a computer). A machine (e.g., a computer) manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities in one or more memories (e.g., volatile memory, non-volatile memory, or a suitable combination thereof), registers, or other machine components that receive, store, transmit, or display information. Furthermore, unless otherwise indicated, the terms "a" or "an" are used herein, as is common in patent documents, to include one or more instances rather than one instance. Finally, the conjunction "or" is used herein to mean a non-exclusive "or" unless otherwise indicated.

[0070] The following recited descriptions describe various examples of the methods, machine-readable media, and systems (eg, machines, devices, or other devices) discussed herein. A first embodiment is a method comprising: one or more processors of a machine accessing external and internal video and audio streams, all provided by a headset comprising an outward-facing camera, an inward-facing camera, and a microphone, the outward-facing camera having an outer field of view extending away from a wearer of the headset and generating the external video stream from the outer field of view, and the inward-facing camera having an inner field of view extending towards the wearer and generating the internal video stream from the inner field of view; detecting, by the one or more processors of the machine, a coincidence of a visual event in the outer field of view with a mouse gesture in the inner field of view and with a candidate pronunciation of a phrase, the visual event being represented in the external video stream, the mouse gesture being represented in the internal video stream, and the candidate pronunciation being represented in the audio stream; determining, by the one or more processors of the machine, that the visual event is associated by the database with the phrase and with a reference pronunciation of the phrase; and a presenting step in which the one or more processors of the machine, in response to the detected coincidence of the visual event with the mouse gesture and with the candidate pronunciation of the phrase, causes the headset to present the reference pronunciation of the phrase to the wearer.

[0071] A second embodiment is a method according to the first embodiment, comprising: The presenting step includes: accessing a set of canonical phonemes contained in said canonical pronunciation of said phrase; and causing a speaker in the headset to play the set of reference phonemes contained in the reference pronunciation.

[0072] A third embodiment is a method according to the first embodiment or the second embodiment, the outward-facing camera of the headset captures the phrase in the outer field of view; The method provides a method in which, upon detection of the simultaneous occurrence, the visual events in the outer visual field include a hand doing one or more of: writing the phrase, tracing the phrase, pointing to the phrase, touching the phrase, underlining the phrase, and highlighting the phrase.

[0073] A fourth embodiment is a method according to any one of the first to third embodiments, the inward-facing camera of the headset captures the wearer's mouth in the inner field of view; wherein upon the detected simultaneous occurrence, the mouse gestures in the inner field of view include the wearer's mouth sequentially assuming a candidate set of mouth shapes, each configured to utter a corresponding candidate phoneme included in the candidate pronunciation of the phrase.

[0074] A fifth embodiment is a method according to any one of the first to fourth embodiments, the inward-facing camera of the headset captures the wearer's mouth in the inner field of view; The method comprises: anonymizing the mouse gestures by cropping a portion of the inner field of view, the cropped portion showing the mouse gestures without showing any of the eyes of the wearer of the headset; wherein, upon the detected coincidence, the anonymized mouse gesture in the inner field of view is detected within the cropped portion of the inner field of view.

[0075] A sixth embodiment is a method according to any one of the first to fifth embodiments, accessing a reference set of a plurality of mouth shapes, each configured to produce a corresponding reference phoneme included in the reference pronunciation of the phrase; and displaying, on a display screen, the set of accessed mouth shape criteria to the wearer of the headset.

[0076] A seventh embodiment is a method according to the sixth embodiment, The headset and the display screen simultaneously present the reference pronunciation of the phrase to the wearer of the headset and display the reference set of accessed mouth shapes to the wearer of the headset.

[0077] An eighth embodiment is a method according to any one of the first to seventh embodiments, The method further provides a method in which the displaying step includes combining the set of mouth shape criteria with an image showing the wearer's mouth, and causing the display screen to display the resulting combination of the image and the set of mouth shape criteria.

[0078] A ninth embodiment is a method according to any one of the first to eighth embodiments, the outward-facing camera of the headset captures a physical model representing the phrase in the outer field of view; The method provides a method in which, at the detected concurrency, the visual event in the outer field of view includes a hand doing one or more of: touching the physical model, grabbing the physical model, moving the physical model, and rotating the physical model.

[0079] A tenth embodiment is a method according to any one of the first to ninth embodiments, the outward-facing camera of the headset captures the wearer's hands in the outer field of view; wherein upon the detected concurrency, the visual event in the outer field of view includes the hand making a trigger gesture indicating a correction request for correction of the candidate pronunciation.

[0080] An eleventh embodiment is a method according to the tenth embodiment, wherein the presenting step satisfies the need indicated by the trigger gesture made by the hand of the wearer.

[0081] A twelfth embodiment is a method according to any one of the first to eleventh embodiments, wherein the reference pronunciation presented in response to the detected coincidence of the visual event with the mouse gesture and with the candidate pronunciation of the phrase comprises a highly intelligible pronunciation of the phrase.

[0082] A thirteenth embodiment is a method according to any one of the first to twelfth embodiments, the outward-facing camera includes a thermal imaging component; wherein upon detection of the coincidence, the visual events in the outer field of view are detected based on a thermal image of a hand of the wearer of the headset.

[0083] A fourteenth embodiment is a method according to any one of the first to thirteenth embodiments, the inward-looking camera includes a thermal imaging component; wherein upon the detected coincidence, the mouse gesture in the inner field of view is detected based on a thermal image of a tongue of the wearer of the headset.

[0084] A fifteenth embodiment is a method according to any one of the first to fourteenth embodiments, the headset further comprises an eye tracking camera having an additional field of view and configured to capture an orientation of the wearer's eyes in the additional field of view; The method comprises: determining a direction in which the wearer's eyes are looking (e.g., a viewing direction) based on an orientation of the wearer's eyes; wherein upon detection of the coincidence, the visual event in the outer field of view is detected based on the determined direction in which the wearer's eyes are looking.

[0085] A sixteenth embodiment is a method according to any one of the first to fifteenth embodiments, the headset further comprises an anemometer configured to detect a breath rate of the wearer of the headset; The method further comprises providing a method in which the presenting step is based on the detected breath rate of the wearer of the headset.

[0086] A seventeenth embodiment is a method according to any one of the first to sixteenth embodiments, the headset further comprises a biosensor configured to detect a stress level of the wearer of the headset; The method comprises: The method further comprises triggering a presentation of an indication for the wearer of the headset to take a break based on the detected stress level of the wearer.

[0087] An 18th embodiment is a method according to any one of the 1st to 17th embodiments, the headset is communicatively coupled to a biosensor configured to detect a skin condition of the wearer of the headset; The method comprises: determining a playback speed at which the reference sound is presented to the wearer based on the skin condition detected by the biosensor; The presenting step includes playing the reference pronunciation at the playback speed determined based on the skin condition.

[0088] A 19th embodiment is a method according to any one of the 1st to 18th embodiments, the headset is communicatively coupled to a biosensor configured to detect a heart rate of the wearer of the headset; The method comprises: determining a playback speed at which the reference sounds are presented to the wearer based on the heart rate detected by the biosensor; The presenting step includes playing the reference pronunciation at the playback speed determined based on the heart rate.

[0089] A twentieth embodiment is a method according to any one of the first to nineteenth embodiments, the headset is communicatively coupled to a biosensor configured to generate an electroencephalogram of the wearer of the headset; The method comprises: determining a playback speed at which the reference pronunciation is presented to the wearer based on the brain waves detected by the biosensor; The presenting step includes playing the reference pronunciation at the playback speed determined based on the electroencephalogram.

[0090] A 21st embodiment is a method according to any one of the 1st to 20th embodiments, the headset is communicatively coupled to a set of accelerometers mounted on a collar worn by the wearer of the headset; The method comprises: detecting a pattern of muscle movement based on accelerometer data generated by the set of accelerometers in the collar; The method further comprises: providing a method in which the presenting step is based on a pattern of the detected muscle movement.

[0091] A 22nd embodiment is a method according to the 21st embodiment, the headset is communicatively coupled to a pair of neuromuscular electrical muscle stimulators mounted on a collar worn by the wearer of the headset; the detected muscle movement pattern is a candidate pattern of muscle movement made by the wearer when uttering the candidate pronunciation of the phrase; The method comprises: accessing a reference pattern of muscle movements configured to produce the reference pronunciation of the phrase; The method causes the neuromuscular electrical muscle stimulator in the collar to stimulate a set of muscles of the wearer based on the accessed baseline pattern of muscle movement.

[0092] A 23rd embodiment is a method according to any one of the first to 22nd embodiments, the headset comprises an outwardly-facing laser emitter configured to indicate an object in the outer field of view by causing a spot of laser light to appear on a surface of the object in the outer field of view; the outward-facing camera of the headset is configured to capture the spot of laser light and the object presented in the outer field of view; the represented object is associated by the database with the phrase and the reference pronunciation of the phrase; wherein upon the detected coincidence, the visual event in the outer field of view includes a spot of laser light appearing on the surface of the object shown in the outer field of view.

[0093] A 24th embodiment is a method according to any one of the first to twenty-third embodiments, the headset includes a stereo depth sensor configured to detect distances to objects in the outer field of view; the outward-facing camera of the headset is configured to capture the wearer's hand indicating the object at the distance in the outer field of view by touching the object; the represented object is associated by the database with the phrase and the reference pronunciation of the phrase; wherein the visual event in the outer field of view at the detected contemporaneity includes the wearer's hand touching the object shown in the outer field of view.

[0094] A 25th embodiment is a method according to any one of the first to twenty-fourth embodiments, performing a comparison of candidate phonemes in a candidate pronunciation of the phrase with reference phonemes in the reference pronunciation of the phrase; recommending a pronunciation tutorial to the wearer of the headset based on the comparison of the candidate phonemes to the reference phonemes.

[0095] A twenty-sixth embodiment is a machine-readable medium (e.g., a non-transitory machine-readable storage medium) comprising instructions that, when executed by one or more processors of the machine, cause the machine to: accessing external and internal video and audio streams, all provided by a headset comprising an outward-facing camera, an inward-facing camera, and a microphone, the outward-facing camera having an outer field of view extending away from a wearer of the headset and generating the external video stream from the outer field of view, and the inward-facing camera having an inner field of view extending towards the wearer and generating the internal video stream from the inner field of view; detecting a coincidence of a visual event in the outer field of view with a mouse gesture in the inner field of view and with a candidate pronunciation of a phrase, the visual event being represented in the external video stream, the mouse gesture being represented in the internal video stream, and the candidate pronunciation being represented in the audio stream; determining that the visual event is associated with a database and with a reference pronunciation of the phrase; and in response to the detected coincidence of the visual event with the mouse gesture and with the candidate pronunciation of the phrase, causing the headset to present the reference pronunciation of the phrase to the wearer.

[0096] A 27th embodiment is a system (e.g., a computer system), one or more processors; and a memory storing instructions that, when executed by one or more of the one or more processors, cause the system to: accessing external and internal video and audio streams, all provided by a headset comprising an outward-facing camera, an inward-facing camera, and a microphone, the outward-facing camera having an outer field of view extending away from a wearer of the headset and generating the external video stream from the outer field of view, and the inward-facing camera having an inner field of view extending towards the wearer and generating the internal video stream from the inner field of view; detecting a coincidence of a visual event in the outer field of view with a mouse gesture in the inner field of view and with a candidate pronunciation of a phrase, the visual event being represented in the external video stream, the mouse gesture being represented in the internal video stream, and the candidate pronunciation being represented in the audio stream; determining that the visual event is associated with a database and with a reference pronunciation of the phrase; and in response to the detected coincidence of the visual event with the mouse gesture and with the candidate pronunciation of the phrase, causing the headset to present the reference pronunciation of the phrase to the wearer.

[0097] A 28th embodiment is a system (e.g., a computer system), one or more processors; and a memory storing instructions that, when executed by one or more of the one or more processors, cause the system to: accessing a video stream and an audio stream both provided by a headset, the headset comprising an inward-facing camera and a microphone, the inward-facing camera having an inner field of view extending towards a wearer of the headset and generating the video stream from the inner field of view; detecting a co-occurrence of a mouse gesture in the inner field of view with a candidate pronunciation of a phrase, the mouse gesture being represented in the video stream and the candidate pronunciation being represented in the audio stream; determining that the phrase is associated with a reference pronunciation of the phrase through a database; and in response to the detected co-occurrence of the mouse gesture with the candidate pronunciation of the phrase, causing the headset to present the reference pronunciation of the phrase to the wearer.

[0098] A twenty-ninth embodiment provides a carrier medium carrying machine-readable instructions for controlling a machine to perform the operations (eg, method operations) performed in any one of the preceding embodiments.

Claims

1. 1. A method comprising: (510) having one or more processors of a machine access external and internal video and audio streams, all provided by a headset, the headset comprising an outward-facing camera, an inward-facing camera, and a microphone, the outward-facing camera having an outer field of view extending away from a wearer of the headset and generating the external video stream from the outer field of view, the outward-facing camera capturing words or physical models representing the words in the outer field of view, and the inward-facing camera having an inner field of view extending towards the wearer and generating the internal video stream from the inner field of view; (520), the one or more processors of the machine detecting a coincidence of a visual event in the outer field of view with a mouse gesture in the inner field of view and with a candidate pronunciation of the phrase, the visual event being a hand gesture or a touch made by a hand that is represented in the external video stream, the mouse gesture being represented in the internal video stream, and the candidate pronunciation being represented in the audio stream; determining (530) by the one or more processors of the machine that the visual event is associated by a database with the phrase and with a reference pronunciation of the phrase; and a presenting step (540) in which the one or more processors of the machine, in response to the detected coincidence of the visual event with the mouse gesture and with the candidate pronunciation of the phrase, causes the headset to present the reference pronunciation of the phrase to the wearer.

2. The presenting step includes: accessing (640) a set of reference phonemes contained in the reference pronunciation of the phrase; and playing (641) the set of reference phonemes contained in the reference pronunciation on a speaker in the headset.

3. the outward-facing camera of the headset captures the phrase in the outer field of view; 2. The method of claim 1, wherein the visual events in the outer visual field during the detected concurrency include a hand doing one or more of: writing the phrase, tracing the phrase, pointing to the phrase, touching the phrase, underlining the phrase, and highlighting the phrase.

4. the inward-facing camera of the headset captures the wearer's mouth in the inner field of view; 2. The method of claim 1, wherein at the detected contemporaneity, the mouse gestures in the inner field of view include the wearer's mouth sequentially assuming a candidate set of mouth shapes, each configured to utter a corresponding candidate phoneme included in the candidate pronunciation of the phrase.

5. the inward-facing camera of the headset captures the wearer's mouth in the inner field of view; The method comprises: anonymizing the mouse gestures by cropping a portion of the inner field of view, the cropped portion showing the mouse gestures without showing any of the eyes of the wearer of the headset (660); The method of claim 1 , wherein, in the detected concurrency, the anonymized mouse gestures in the inner field of view are detected within the cropped portion of the inner field of view.

6. accessing (650) a reference set of mouth shapes, each configured to produce a corresponding reference phoneme included in the reference pronunciation of the phrase; 2. The method of claim 1, further comprising: causing a display screen to display the set of accessed mouth shape criteria to the wearer of the headset.

7. 7. The method of claim 6, wherein the headset and the display screen (340) simultaneously present the reference pronunciation of the phrase to the wearer of the headset and display the reference set of accessed mouth shapes to the wearer of the headset.

8. 7. The method of claim 6, wherein the displaying step includes combining the set of mouth shape criteria with an image showing the wearer's mouth, and causing the display screen to display the resulting combination of the image and the set of mouth shape criteria.

9. the outward-facing camera of the headset captures a physical model representing the phrase in the outer field of view; 2. The method of claim 1 , wherein the visual events in the outer field of view during the detected concurrency include a hand doing one or more of: touching the physical model, grabbing the physical model, moving the physical model, and rotating the physical model.

10. the outward-facing camera of the headset captures the wearer's hands in the outer field of view; 2. The method of claim 1, wherein at the detected concurrency, the visual event in the outer field of view includes the hand making a trigger gesture indicating a correction request for correction of the candidate pronunciation.

11. The method of claim 10 , wherein the presenting step satisfies the need indicated by the trigger gesture made by the hand of the wearer.

12. 2. The method of claim 1 , wherein the reference pronunciation presented in response to the detected co-occurrence of the visual event with the mouse gesture and with the candidate pronunciation of the phrase comprises a highly articulate pronunciation of the phrase.

13. the outward-facing camera includes a thermal imaging component; The method of claim 1 , wherein in the detected coincidences, the visual events in the outer field of view are detected based on a thermal image of a hand of the wearer of the headset.

14. the inward-looking camera includes a thermal imaging component; The method of claim 1 , wherein at the detected coincidence, the mouse gesture in the inner field of view is detected based on a thermal image of a tongue of the wearer of the headset.

15. the headset further comprises an eye tracking camera having an additional field of view and configured to capture an orientation of the wearer's eyes in the additional field of view; The method comprises: determining a direction in which the wearer's eyes are looking based on an orientation of the wearer's eyes; The method of claim 1 , wherein at the detected coincidence, the visual events in the outer field of view are detected based on the determined direction in which the wearer's eyes are looking.

16. the headset further comprises an anemometer configured to detect a breath rate of the wearer of the headset; The method of claim 1 , wherein the indicating step is based on the detected breath rate of the wearer of the headset.

17. the headset further comprises a biosensor configured to detect a stress level of the wearer of the headset; The method comprises:

10. The method of claim 1, further comprising triggering the presentation of an indication for the wearer of the headset to take a break based on the detected stress level of the wearer.

18. the headset is communicatively coupled to a biosensor configured to detect a skin condition of the wearer of the headset; The method comprises: determining a playback speed at which the reference sound is presented to the wearer based on the skin condition detected by the biosensor; The method of claim 1 , wherein the presenting step includes playing the reference pronunciation at the playback speed determined based on the skin condition.

19. the headset is communicatively coupled to a biosensor configured to detect a heart rate of the wearer of the headset; The method comprises: determining a playback speed at which the reference sounds are presented to the wearer based on the heart rate detected by the biosensor; The method of claim 1 , wherein the presenting step includes playing the reference pronunciation at the playback speed determined based on the heart rate.

20. the headset is communicatively coupled to a biosensor configured to generate an electroencephalogram of the wearer of the headset; The method comprises: determining a playback speed at which the reference sounds are presented to the wearer based on the electroencephalogram detected by the biosensor; The method of claim 1 , wherein the presenting step includes playing the reference pronunciation at the playback speed determined based on the electroencephalogram.

21. the headset is communicatively coupled to a set of accelerometers mounted on a collar worn by the wearer of the headset; The method comprises: detecting a pattern of muscle movement based on accelerometer data generated by the set of accelerometers in the collar; The method of claim 1 , wherein the indicating step is based on a pattern of the detected muscle movement.

22. the headset is communicatively coupled to a set of neuromuscular electrical muscle stimulators mounted on a collar worn by the wearer of the headset; the detected muscle movement pattern is a candidate pattern of muscle movement made by the wearer when uttering the candidate pronunciation of the phrase; The method comprises: accessing a reference pattern of muscle movements configured to produce the reference pronunciation of the phrase; 22. The method of claim 21, further comprising causing the neuromuscular electrical muscle stimulator in the collar to stimulate a set of muscles of the wearer based on the accessed baseline patterns of muscle movement.

23. the headset further comprises an outwardly-facing laser emitter configured to indicate an object in the outer field of view by causing a spot of laser light to appear on a surface of the object in the outer field of view; the outward-facing camera of the headset is further configured to capture the spot of laser light and the object presented in the outer field of view; the represented object is associated by the database with the phrase and the reference pronunciation of the phrase; The method of claim 1 , wherein the visual event in the outer field of view at the detected coincidence further comprises the appearance of a spot of the laser light on the surface of the object shown in the outer field of view.

24. the headset includes a stereo depth sensor configured to detect distances to objects in the outer field of view; the outward-facing camera of the headset is further configured to capture a hand of the wearer of the headset indicating the object at the distance in the outer field of view by touching the object; the represented object is associated by the database with the phrase and the reference pronunciation of the phrase; The method of claim 1 , wherein the visual event in the outer field of view at the detected contemporaneity further comprises the wearer's hand touching the object shown in the outer field of view.

25. performing a comparison of candidate phonemes in a candidate pronunciation of the phrase with reference phonemes in the reference pronunciation of the phrase; The method of claim 1 , further comprising: recommending a pronunciation tutorial to the wearer of the headset based on the comparison of the candidate phonemes to the reference phonemes.

26. A machine readable medium comprising instructions which, when executed by one or more processors of a machine, cause the machine to perform a method according to any one of claims 1 to 25.

27. 1. A system comprising: one or more processors; and a memory storing instructions which, when executed by one or more of said one or more processors, cause said system to perform a method according to any one of claims 1 to 25.

Images