Voice model personalization through environmental context collection

Abstract

This article describes a device for implementing a personalized speech model via environmental context acquisition. The device includes a microphone, a context acquisition module, a confidence module, and a training module. The context acquisition module is used to determine the context associated with the captured audio signal. The confidence module is used to determine the confidence that the context is applied to the audio signal. The training module is used to train a neural network in response to the confidence being higher than a predetermined threshold.

Description

Background Art

[0001] Speech recognition systems rely on various speech models to translate spoken language into text. Voice recordings and transcripts of target users using spoken interfaces can be used to improve the accuracy of speech recognition systems by training speech models. Users can read known scripts or text to train the model to recognize the user and fine-tune the recognition of the user's speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is a diagram of a personalized speech recognition model implemented through ambient context harvesting;

[0003] Figure 2 is a process flow diagram of a method for enabling a personalized speech model acquired via environmental context;

[0004] Figure 3 is a process flow diagram of a method for instantaneous identification of context;

[0005] Figure 4 is a block diagram of an electronic device that enables a personalized voice model collected through environmental context; and

[0006] Figure 5 is a block diagram illustrating a medium including logic for personalized speech models acquired via environmental context.

[0007] The same numbers are used throughout the disclosure and drawings to refer to similar parts and features. Numbers in the 100 series refer to those originally Figure 1 ; numbers in the 200 series refer to features originally found in Figure 2 ; and so on. DETAILED DESCRIPTION

[0008] The collection of reliable speech training data is often intrusive and time-consuming. As mentioned above, users must often read known scripts or texts to train the model to recognize the user and fine-tune the recognition of the user's speech. Traditional systems often prompt users to say enrollment phrases that are time-consuming and cumbersome to use. Such enrollment phrases and other training materials used by traditional text recognition systems are not collected in the target usage setting. Speech recognition systems that collect user utterances in remote networks such as the "cloud" compromise the user's privacy and are limited to cloud usage. For example, utterances stored in the cloud are often limited to short queries because the transmission of large amounts of utterances is often computationally expensive and energy-intensive. In addition, reliable transcriptions are often difficult to develop and obtain because realistic acoustic scenarios change the text corresponding to the most direct transcription.

[0009] The embodiments described herein perform voice model personalization via environmental context acquisition. As used herein, context may refer to situational information that may affect the type of speech that occurs. Context may be based on, for example, linguistics, time, location, repetitive behavior, or any combination thereof. In an embodiment, the electronic device may be a wearable device that listens to situations in which there is high confidence in the correctness of recognition, including structured interactions (e.g., games, medical programs, etc.) and high-frequency word patterns. High-frequency word patterns may be one or more words that frequently appear in a particular context. For example, when giving directions, the words "right", "left" and "turn here" may often be spoken. This high-confidence voice is used to adapt the acoustic model stored on the wearable electronic device to personalize voice recognition, thereby improving accuracy. High-confidence data may be data that is higher than a predetermined confidence threshold. The present technology enables instant training of voice models without intrusion into the user's daily behavior. In fact, the present technology can be used to train, retrain, and adapt speaker-dependent voice recognition systems without using predetermined texts read by users. In some embodiments, the technology described herein can be used to recognize voices from multiple speakers.

[0010] Some embodiments may be implemented in one or a combination of hardware, firmware, and software. In addition, some embodiments may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by a computing platform to perform the operations described herein. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a machine-readable medium may include a read-only memory (ROM); a random access memory (RAM); a disk storage medium; an optical storage medium; a flash memory device; or an electrical, optical, acoustic, or other form of propagated signal, such as a carrier wave, an infrared signal, a digital signal, or an interface for sending and / or receiving a signal, etc.

[0011] An embodiment is an implementation or example. References in the specification to "an embodiment," "an embodiment," "some embodiments," "various embodiments," or "other embodiments" refer to specific features, structures, or characteristics described in conjunction with these embodiments being included in at least some embodiments of the present technology, but not necessarily all embodiments. Various occurrences of "an embodiment," "an embodiment," or "some embodiments" do not necessarily refer to the same embodiment. Elements or aspects in an embodiment may be combined with elements or aspects of another embodiment.

[0012] Figure 1 It is a diagram of a personalized speech recognition model 100 collected via environmental context. Although the model is illustrated as having various stages, the training, retraining, and adaptation of the model can be performed by more or less blocks than the diagram. Microphone array 102 can be used to capture audio. The captured audio is sent to speech recognition block 104, speaker recognition block 106, and archive decision block 108. Speech recognition block 104 can be used to determine the text corresponding to the language in the audio captured by microphone array 102. Speaker recognition block 106 can be used to identify a specific speaker. The present technology can be used to identify multiple speakers, and can be used to train models to recognize the speech from each identified user. Archive decision block 108 can be configured to determine whether a specific interaction should be stored as a new interaction, which will be further described below.

[0013] Language model 110 and acoustic model 112 can provide data to speech recognition block 104. In an embodiment, language model 110 and acoustic model 112 can be implemented as Hidden Markov Model (HMM) respectively, which enables speech recognition system to rely on the combination of data including language, acoustics and syntactic elements. As more contexts are obtained, language model 110 can enable context to be used as a constraint for matching various sounds to word sequences by adapting language model. In an embodiment, as described below, context can be collected and used for training, retraining and adaptation. In an embodiment, language model 110 can provide the possibility that a series of sounds belong to a specific word group. Acoustic model 112 can be used to identify the relationship between the linguistic units of audio signal and speech. In an embodiment, language model 110 and acoustic model 112 can be based on typical interactions with initial model as the root, and adapt based on observed structured interactions. Using information provided from the language model 110 and the acoustic model 112 , the speech recognition block 104 may output the recognized text stream to the conversation mode recognition 116 .

[0014] Speaker identification block 106 may be used to identify a speaker from the audio captured by microphone 102. Speaker identification may be used to simplify speech recognition at block 104. In an embodiment, speaker identification block 106 may be used to determine a primary user and multiple speakers interacting with the user. In an embodiment, speaker identification block 106 may be use data (e.g., behavior data 118) to identify the user as well as other speakers. In an embodiment, behavior data may be obtained from available databases such as calendars and history. For example, if a user has an appointment on a calendar with a particular person / speaker, speaker identification may use that appointment and any other previous appointments to apply context to identify the particular person / speaker. Moreover, the language used by the user with the particular person / speaker may be repeated or cumulative. The present technology may be used to identify context based on the user's behavior.

[0015] The speaker identification block may output a plurality of identified speakers to a speaker confidence block 120. Speaker confidence may be determined based on speaker identification techniques. In an embodiment, speaker confidence refers to the likelihood that the speaker identified by the speaker identification algorithm at block 106 is the true speaker. In speaker identification, a speaker score (perhaps normalized by an anti-speaker or background model) may be compared to a threshold and a binary decision made. Confidence scores may be derived based on scores for the original speaker, background, anti-speaker, etc. Additionally, in an embodiment, speaker confidence may be determined using Bayesian inference conditioning on other information such as location, place, etc. Given the above information, the speaker confidence block may output a speaker likelihood.

[0016] The speaker confidence from block 120, together with the clock, location, and video / camera data from block 122, can be input to the conversation mode recognition block 116. In addition, as shown, the conversation mode recognition block 116 can also take the recognized speech from block 104 and the list of structured interactions from block 124 as input. The list of structured interactions can include, for example, conversation flow, grammar, dictionary, etc. At block 116, the text determined at block 104 can be analyzed using the speaker confidence from block 120, the clock, location, and camera at block 122, and the acoustic and language confidence at block 114 to determine whether a specific conversation mode corresponds to a structured interaction. When a specific conversation mode corresponds to a structured interaction, it can be stored in the list of structured interactions at block 124. A conversation mode can refer to a series of interactions. In some cases, a series of interactions can take the form of intent and arguments, such as "greeting+give_direction(right)+quit". The conversational model can be derived through automatic speech recognition (ASR) transcription plus intent and argument identification components.

[0017] At block 108, a decision is made to archive the current mode. If the current mode is not archived, it is discarded at block 126. If the current mode is archived, it is stored as user / context specific training data at block 128. At block 128, the identified interaction mode is created or updated based on the grammar. The text sequence and the intent sequence can be used to identify the interaction mode. In an embodiment, the output of block 128 can be a flowchart derived from the text / intent sequence. If the current conversation mode is archived, a similarity metric is evaluated for each mode in the archive, which may be efficiently implemented by using hashing. This training data is used at block 130 to adapt the language model 110 and the acoustic model 112. The adaptation at block 130 includes, but is not limited to, providing data to adapt each of the language model and the acoustic model using model-specific data obtained during a specific context. Specifically, the adaptation includes training a new statistical model using the new user-specific data obtained from block 128. Training the new statistical model may depend on the statistical techniques applied to obtain the acoustic and language models.

[0018] In an embodiment, the acoustic model may be adapted by adjusting the parameters of the acoustic model to account for the new data, by adding speaker-specific parameters, or by creating a new feature vector transform that maps the feature vectors to a canonical acoustic space, etc. The language model may be adapted by adjusting the n-gram frequencies to account for the new data, by interpolating new n-gram frequencies, adding new words to the dictionary, etc.

[0019] Thus, in an initial pass through the model 100, the present technology can establish an initial context that includes the user's identity and any other associated speakers. As the microphone captures more data, based on the user's usage patterns, the list of structured interactions may increase to include additional structured interactions. The model can be an always on, always listening model. Traditional techniques focus on manual data collection and require a lot of manual input to train the model. Typically, hundreds of hours of training data (audio) are used to train a speech recognition model. The present technology introduces a model that can be trained in a variety of different contexts and is not limited to a predetermined text or pattern.

[0020] The proposed system relies on a database of structured interactions. Structured interactions are patterns of conversational flows that occur in everyday life. Examples include playing games (with a specific turn-taking structure, vocabulary, object relationships, etc.), purchasing, making phone calls, social interactions, business transactions, etc. Microphones on wearable or ambient devices are used to monitor audio, which is processed using a speech recognition engine to convert it into text (with a certain confidence level), and processed using a speaker recognition engine to identify the speaker (with a certain confidence level). The text (or text lattice) from speech recognition, the speaker identity hypothesis from the speaker recognition engine, the confidence measurement results, together with other sensor inputs (time, location, object, etc.) and other related databases (calendar, interaction history, etc.) are used to identify and characterize the current interaction pattern. The text lattice can be a set of all possible interpretations of the user input processed by the ASR, which is usually in the form of an acyclic directed graph. The interaction pattern can be, for example, a greeting, followed by a question / answer, followed by a payment, followed by a compliment.

[0021] When a structured interaction pattern is identified (e.g., a user shopping at a grocery store checkout counter), speech recognition can be rerun using a grammar and dictionary specific to the pattern to determine whether the confidence has increased. If the confidence has increased, it helps to confirm that the correct pattern has been identified. If a clear pattern is identified with high acoustic and language confidence, but does not match one of the previously stored patterns, it can be stored as a new pattern. If the acoustic confidence, language confidence, and pattern confidence of the speech recognition are high, the audio and its transcription are saved for future adaptation or training of the acoustic and language models. The confidence can be a number indicating the probability that a given decision is correct given all available data. In an embodiment, the confidence can be a score indicating that a given language, acoustic, or pattern determined relative to the current context is correct. The confidence calculation can be performed by using Bayesian methods and conditioning on relevant auxiliary information. In an embodiment, the confidence score can be determined based on specific statistical techniques used in acoustic, language, and pattern confidence. For example, the score can be obtained from the softmax in the last layer of the neural network.

[0022] Figure 2 200, which enables a personalized speech model collected via environmental context. At block 202, training data is collected. In an embodiment, the training data is collected as audio, video, or any combination thereof. The audio and video data can be monitored by a device that is always on and listening. At block 204, a structured interaction is determined. For example, the monitored audio can be processed using a speech recognition engine to convert it to text (with a certain confidence level). The speaker recognition engine can also process the text to identify the speaker (with a certain confidence level). The structured interaction can be determined based on the confidence value and the recognition of the pattern in the identified conversation.

[0023] At block 206, the model may be trained. In an embodiment, the model is trained in an unsupervised manner based on labeled high confidence training data corresponding to the identified structured interactions. If the structured interaction is not identified, the new structured interaction may be saved as an identified structured interaction.

[0024] The model can be trained based on the current mode, and the resulting structured interactions are identified from the text. In an embodiment, the training includes modification of the neural network so that each layer of the network can take audio values ​​as input and produce text found during the structured interactions. In an embodiment, the training input is applied to the input layer of the neural network, and the expected output is compared at the output layer. During the learning process, a forward pass is made through the neural network, and the output of each element is calculated layer by layer. As the network is retrained with additional audio and corresponding structured interactions, the accuracy of the model will improve.

[0025] At block 208, speech recognition is performed a second time using the retrained model. When a structured interaction pattern is identified, speech recognition can be rerun using the grammar and dictionary specific to the identified pattern to see if the confidence score has increased. In an embodiment, if the confidence increases, it helps to confirm that the correct pattern has been identified. If a clear pattern is identified with high acoustic and language confidence, but does not match one of the previously stored patterns, it can be stored as a new pattern. In addition, if each of the acoustic confidence, language confidence, and pattern confidence is high, the audio and its transcription are saved for future acoustic and language model adaptation or training.

[0026] Figure 3 300 is a process flow diagram of a method 300 for instantly identifying context. At box 302, a structured interaction is defined. The structured interaction can be defined based on the usage pattern of a specific person. The usage pattern can include, but is not limited to, location data, device status, ambient noise, etc. For example, if a user enters a location known to be a restaurant around noon every day, it can be determined that a structured interaction such as ordering a meal will occur. Such an interaction includes subcomponents, such as a greeting, a food name, a price, etc. In such an example, if the structured interaction can locate a specific restaurant as the location of the structured interaction, a menu from that restaurant can be used as text to identify some possible training data.

[0027] At block 302, a confidence score is obtained. The confidence score may indicate the likelihood that a portion of the text belongs to a particular structured interaction. At block 304, when the confidence score is above a predetermined threshold, the particular portion of the text or data is marked as training data. At block 306, when the confidence score is above a predetermined threshold, the data corresponding to the structured interaction may be marked as training data.

[0028] Figure 4It is a block diagram of an electronic device that enables a personalized speech model collected via an environmental context. The electronic device 400 can be, for example, a laptop computer, a tablet computer, a mobile phone, a smart phone, or a wearable device. The electronic device 400 may include a central processing unit (CPU) 402 configured to execute stored instructions, and a memory device 404 storing instructions executable by the CPU 402. The CPU can be coupled to the memory device 404 via a bus 406. In addition, the CPU 402 can be a single-core processor, a multi-core processor, a computing cluster, or any number of other configurations. In addition, the electronic device 400 can include more than one CPU 402. The memory device 404 may include a random access memory (RAM), a read-only memory (ROM), a flash memory, or any other suitable memory system. For example, the memory device 404 may include a dynamic random access memory (DRAM).

[0029] The electronic device 400 also includes a graphics processing unit (GPU) 408. As shown, the CPU 402 can be coupled to the GPU 408 via a bus 406. The GPU 408 can be configured to perform any number of graphics operations in the electronic device 400. For example, the GPU 408 can be configured to render or manipulate graphic images, graphic frames, videos, etc. to be displayed to a user of the electronic device 400. In some embodiments, the GPU 408 includes multiple graphics engines, each of which is configured to perform specific graphics tasks or perform a specific type of workload.

[0030] The CPU 402 may be linked to a display interface 410 via a bus 406, and the display interface 410 is configured to connect the electronic device 400 to a display device 412. The display device 412 may include a display screen as a built-in component of the electronic device 400. The display device 412 may also include a computer monitor, a television, or a projector, etc., which is externally connected to the electronic device 400.

[0031] The CPU 402 may also be connected to an input / output (I / O) device interface 414 via the bus 406, which is configured to connect the electronic device 400 to one or more I / O devices 416. The I / O devices 416 may include, for example, a keyboard and a pointing device, wherein the pointing device may include a touch pad or a touch screen, etc. The I / O devices 416 may be built-in components of the electronic device 400, or may be devices connected to the electronic device 400 from the outside.

[0032] The electronic device also includes a microphone array 418. The microphone array 418 can have any number of microphones. The microphone array 418 can be used to capture audio that will be input into the speech recognition model. Similarly, the camera 420 can be used to capture video and image data, which can be used for environmental context acquisition as described above. The speech recognition module 422 can be used to recognize speech in each of a speaker-related manner and a speaker-independent manner. The context acquisition module 424 can determine the various contexts in which speech occurs by analyzing the audio and using other information to determine conversation patterns that may be components of a specific structured interaction. The training module 426 can use the audio data together with the structured interactions derived from the audio data to train the neural network to implement the speech recognition function.

[0033] The electronic device may also include a storage device 428. The storage device 428 is a physical memory such as a hard drive, an optical drive, a flash drive, a drive array, or any combination thereof. The storage device 428 may store user data such as audio files, video files, audio / video files, and picture files. The storage device 428 may also store programming codes such as device drivers, software applications, operating systems, etc. The programming codes stored to the storage device 428 may be executed by the CPU 402, the GPU 408, or any other processor that may be included in the electronic device 400.

[0034] The CPU 402 may be linked to the cellular hardware 430 via the bus 406. The cellular hardware 430 may be any cellular technology, such as the 4G standard (the advanced International Mobile Telecommunications (IMT-Advanced) standard promulgated by the International Telecommunications Union-Radio communication Sector (ITU-R)). In this way, the electronic device 400 may access any network 436, if the network 436 is a cellular network, without being tethered or paired to another device.

[0035] CPU 402 may also be linked to WiFi hardware 432 via bus 406. WiFi hardware is hardware according to the WiFi standard (a standard published as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard). In the case where network 436 is the Internet, WiFi hardware 432 enables electronic device 400 to connect to the Internet by using the Transmission Control Protocol and Internet Protocol (TCP / IP). Therefore, electronic device 400 can achieve end-to-end connection with the Internet by addressing, routing, sending, and receiving data according to the TCP / IP protocol without using another device. In addition, Bluetooth interface 434 may be coupled to CPU 402 via bus 406. Bluetooth interface 434 is an interface according to a Bluetooth network (based on the Bluetooth standard published by the Bluetooth Special Interest Group). Bluetooth interface 434 enables electronic device 400 to pair with other Bluetooth-enabled devices via a personal area network (PAN). Therefore, network 436 may be a PAN. Examples of devices with Bluetooth capabilities include laptop computers, desktop computers, ultrabooks, tablet computers, mobile devices, or servers, etc. Although one network is illustrated, electronic device 400 may be connected to multiple networks at the same time.

[0036] Figure 4 The block diagram is not intended to indicate that the electronic device 400 will include Figure 4 Rather, computing system 400 may include fewer or additional components not shown. Figure 4 Components shown in FIG. 4 (eg, sensors, power management integrated circuits, additional network interfaces, etc.). Depending on the details of the specific implementation, the electronic device 400 may include Figure 4 Any number of additional components not shown in the drawings may be included. In addition, any functionality of CPU 402 may be implemented in part or in whole in hardware and / or in a processor. For example, the functionality may be implemented using an application specific integrated circuit, with logic implemented in a processor, with logic implemented in a dedicated graphics processing unit, or in any other device.

[0037] Figure 5 5 is a block diagram illustrating a medium 500 that includes logic for a personalized speech model acquired via an environmental context. The medium 500 may be a computer-readable medium, including a non-transitory medium storing code that may be accessed by a processor 502 via a computer bus 504. For example, the computer-readable medium 500 may be a volatile or non-volatile data storage device. The medium 500 may also be a logic unit, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or an arrangement of logic gates implemented in, for example, one or more integrated circuits.

[0038] The medium 500 may include modules 506-510 configured to perform the techniques described herein. For example, the collection module 506 may be configured to collect data to be used as input to train a neural network for speech recognition. In an embodiment, the data includes audio data. The data may also include behavioral data, such as calendar information, history, location information, etc. The context acquisition module 508 may be configured to derive context from the collected information. The context may be determined based on conversation patterns and structured interactions. The training module 510 may be configured to train a neural network based on the collected context and the collected data. In some embodiments, the modules 506-510 may be modules of computer code configured to direct the operation of the processor 502.

[0039] Figure 5 The block diagram is not intended to indicate that the medium 500 will include Figure 5 In addition, the medium 500 may include Figure 5 Any number of additional components not shown may be included, depending on the details of the specific implementation.

[0040] Example 1 is a device for implementing a personalized speech model via environmental context acquisition. The device includes a microphone for capturing an audio signal; a context acquisition module for determining a context associated with the captured audio signal; a confidence module for determining a confidence score that the context is applied to the audio signal; and a training module for training a neural network in response to a confidence score being higher than a predetermined threshold.

[0041] Example 2 includes the apparatus of Example 1, including or excluding the optional features. In this example, the context is determined by deriving structured interactions based on conversational patterns within the audio signal.

[0042] Example 3 includes the apparatus of any of Examples 1 to 2, including or excluding the optional features. In this example, the context is based at least in part on the behavioral data.

[0043] Example 4 includes the apparatus of any one of Examples 1 to 3, including or excluding the optional features. In this example, the confidence includes a language confidence.

[0044] Example 5 includes the apparatus of any one of Examples 1 to 4, including or excluding the optional features. In this example, the confidence level includes an acoustic confidence level.

[0045] Example 6 includes the apparatus of any one of Examples 1 to 5, including or excluding the optional features. In this example, the confidence includes a pattern confidence.

[0046] Example 7 includes the apparatus of any one of Examples 1 to 6, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversation pattern within the audio signal, and in response to the structured interaction being a new structured interaction, the structured interaction is stored in a database.

[0047] Example 8 includes the apparatus of any one of Examples 1 to 7, including or excluding the optional features. In this example, in the apparatus of claim 1, in response to the confidence being above a predetermined threshold, the language model and the acoustic model are adapted using the context and the audio signal.

[0048] Example 9 includes the apparatus of any one of Examples 1 to 8, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversational pattern within the audio signal, and the stored structured interaction is expanded in response to an additional identified portion of the structured interaction.

[0049] Example 10 includes the apparatus of any one of Examples 1 to 9, including or excluding the optional features. In this example, the training module iteratively trains and adapts the neural network based on the additional context and associated audio data.

[0050] Example 11 is a system for implementing a personalized speech model via environmental context collection. The system includes a microphone for capturing an audio signal; a memory for storing instructions and communicatively coupled to the microphone; and a processor communicatively coupled to a camera and the memory, wherein when the processor is to execute the instructions, the processor will perform the following operations: determine a context associated with the captured audio signal; determine a confidence score that the context is applied to the audio signal; and train a neural network in response to the confidence being higher than a predetermined threshold.

[0051] Example 12 includes the system of Example 11, including or excluding the optional features. In this example, the context is determined by deriving structured interactions based on conversational patterns within the audio signal.

[0052] Example 13 includes the system of any of Examples 11 to 12, including or excluding the optional features. In this example, the context is based at least in part on the behavioral data.

[0053] Example 14 includes the system of any of Examples 11 to 13, including or excluding the optional features. In this example, the confidence includes a language confidence.

[0054] Example 15 includes the system of any of Examples 11 to 14, including or excluding the optional features. In this example, the confidence level includes an acoustic confidence level.

[0055] Example 16 includes the system of any one of Examples 11 to 15, including or excluding the optional features. In this example, the confidence includes a pattern confidence.

[0056] Example 17 includes the system of any one of Examples 11 to 16, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversation pattern within the audio signal, and in response to the structured interaction being a new structured interaction, the structured interaction is stored in a database.

[0057] Example 18 includes the system of any one of Examples 11 to 17, including or excluding the optional features. In this example, in the system of claim 11, in response to the confidence being above a predetermined threshold, the language model and the acoustic model are adapted using the context and the audio signal.

[0058] Example 19 includes the system of any one of Examples 11 to 18, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversational pattern within the audio signal, and the stored structured interaction is expanded in response to an additional identified portion of the structured interaction.

[0059] Example 20 includes the system of any one of Examples 11 to 19, including or excluding the optional features. In this example, the training module iteratively trains and adapts the neural network based on the additional context and associated audio data.

[0060] Example 21 is a method. The method includes capturing an audio signal; determining a context associated with the captured audio signal; determining a confidence score that the context is applied to the audio signal; and training a neural network in response to the confidence score being above a predetermined threshold.

[0061] Example 22 includes the method of Example 21, including or excluding the optional features. In this example, the context is determined by deriving structured interactions based on conversational patterns within the audio signal.

[0062] Example 23 includes the method of any one of Examples 21 to 22, including or excluding the optional features. In this example, the context is based at least in part on the behavioral data.

[0063] Example 24 includes the method of any one of Examples 21 to 23, including or excluding the optional features. In this example, the confidence includes a language confidence.

[0064] Example 25 includes the method of any one of Examples 21 to 24, including or excluding the optional features. In this example, the confidence level includes an acoustic confidence level.

[0065] Example 26 includes the method of any one of Examples 21 to 25, including or excluding the optional features. In this example, the confidence includes a pattern confidence.

[0066] Example 27 includes the method of any one of Examples 21 to 26, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversation pattern within the audio signal, and in response to the structured interaction being a new structured interaction, the structured interaction is stored in a database.

[0067] Example 28 includes the method of any one of Examples 21 to 27, including or excluding the optional features. In this example, in the method of claim 21, in response to the confidence being above a predetermined threshold, the language model and the acoustic model are adapted using the context and the audio signal.

[0068] Example 29 includes the method of any one of Examples 21 to 28, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversational pattern within the audio signal, and the stored structured interaction is expanded in response to an additional identified portion of the structured interaction.

[0069] Example 30 includes the method of any one of Examples 21 to 29, including or excluding the optional features. In this example, the training module iteratively trains and adapts the neural network based on the additional context and associated audio data.

[0070] Example 31 is at least one non-transitory machine-readable medium having instructions stored therein. The computer-readable medium includes instructions that direct a processor to: capture an audio signal; determine a context associated with the captured audio signal; determine a confidence score that the context is applied to the audio signal; and train a neural network in response to the confidence score being above a predetermined threshold.

[0071] Example 32 includes the computer-readable medium of Example 31, including or excluding the optional features. In this example, the context is determined by deriving structured interactions based on conversational patterns within the audio signal.

[0072] Example 33 includes the computer readable medium of any one of Examples 31 to 32, including or excluding the optional features. In this example, the context is based at least in part on the behavioral data.

[0073] Example 34 includes the computer readable medium of any one of Examples 31 to 33, including or excluding the optional features. In this example, the confidence includes a language confidence.

[0074] Example 35 includes the computer readable medium of any one of Examples 31 to 34, including or excluding the optional features. In this example, the confidence level includes an acoustic confidence level.

[0075] Example 36 includes the computer readable medium of any one of Examples 31 to 35, including or excluding the optional features. In this example, the confidence includes a pattern confidence.

[0076] Example 37 includes the computer-readable medium of any one of Examples 31 to 36, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversation pattern within the audio signal, and in response to the structured interaction being a new structured interaction, the structured interaction is stored in a database.

[0077] Example 38 includes the computer readable medium of any one of Examples 31 to 37, including or excluding the optional features. In this example, in the machine readable medium of claim 31, in response to the confidence being above a predetermined threshold, the language model and the acoustic model are adapted using the context and the audio signal.

[0078] Example 39 includes the computer readable medium of any one of Examples 31 to 38, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversational pattern within the audio signal, and the stored structured interaction is expanded in response to an additional identified portion of the structured interaction.

[0079] Example 40 includes the computer readable medium of any one of Examples 31 to 39, including or excluding the optional features. In this example, the training module iteratively trains and adapts the neural network based on the additional context and associated audio data.

[0080] Example 41 is a device for implementing a personalized speech model via environmental context acquisition. The device includes a microphone for capturing an audio signal; a component for determining a context associated with the captured audio signal; a component for determining a confidence score that the context is applied to the audio signal; and a component for training a neural network in response to the confidence being higher than a predetermined threshold.

[0081] Example 42 includes the apparatus of Example 41, including or excluding the optional features. In this example, the context is determined by deriving structured interactions based on conversational patterns within the audio signal.

[0082] Example 43 includes the apparatus of any one of Examples 41 to 42, including or excluding the optional features. In this example, the context is based at least in part on the behavioral data.

[0083] Example 44 includes the apparatus of any one of Examples 41 to 43, including or excluding the optional features. In this example, the confidence includes a language confidence.

[0084] Example 45 includes the apparatus of any one of Examples 41 to 44, including or excluding the optional features. In this example, the confidence level includes an acoustic confidence level.

[0085] Example 46 includes the apparatus of any one of Examples 41 to 45, including or excluding the optional features. In this example, the confidence includes a pattern confidence.

[0086] Example 47 includes the apparatus of any one of Examples 41 to 46, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversation pattern within the audio signal, and in response to the structured interaction being a new structured interaction, the structured interaction is stored in a database.

[0087] Example 48 includes the apparatus of any one of Examples 41 to 47, including or excluding the optional features. In this example, in the apparatus of claim 41, in response to the confidence being above a predetermined threshold, the language model and the acoustic model are adapted using the context and the audio signal.

[0088] Example 49 includes the apparatus of any one of Examples 41 to 48, including or excluding the optional features. In this example, the context is determined by deriving a structured interaction based on a conversational pattern within the audio signal, and the stored structured interaction is expanded in response to an additional identified portion of the structured interaction.

[0089] Example 50 includes the apparatus of any one of Examples 41 to 49, including or excluding the optional features. In this example, the training module iteratively trains and adapts the neural network based on the additional context and associated audio data.

[0090] Not all components, features, structures, characteristics, etc. described and illustrated herein need to be included in one or more specific aspects. If the specification, for example, indicates that a component, feature, structure, or characteristic "may", "can", "can", or "can" be included, then the specific component, feature, structure, or characteristic does not have to be included. If the specification or claims refer to "an" element, it does not mean that there is only one element. If the specification or claims refer to "additional" elements, it does not exclude the presence of more than one additional element.

[0091] It should be noted that although some aspects have been described with reference to specific implementations, according to some aspects, other implementations are possible. In addition, the arrangement and / or order of circuit elements or other features illustrated in the drawings and / or described herein need not be arranged in the specific manner illustrated and described. According to some aspects, many other arrangements are possible.

[0092] In each system shown in the accompanying drawings, in some cases, elements may each have the same reference numeral or different reference numerals to indicate that the elements represented may be different and / or similar. However, the elements may be flexible enough to have different implementations and to work with some or all of the systems shown or described herein. The various elements shown in the accompanying drawings may be the same or different. Which is referred to as the first element and which is referred to as the second element is arbitrary.

[0093] It should be understood that the details in the foregoing examples may be used anywhere in one or more aspects. For example, all optional features of the computing device described above may also be implemented relative to the methods or computer-readable media described herein. In addition, although flow charts and / or state diagrams may have been used herein to describe various aspects, the technology is not limited to these figures or the corresponding descriptions herein. For example, the process does not need to pass through each illustrated box or state or move in exactly the same order as illustrated and described herein.

[0094] The present technology is not limited to the specific details listed herein. In fact, those skilled in the art who benefit from this disclosure will understand that many other variations can be made to the foregoing description and drawings within the scope of the present technology. Therefore, the following claims include any modifications thereto, which define the scope of the present technology.

Claims

1. A device for performing speech modeling via environmental context collection, comprising: A microphone, used to capture audio signals; a context acquisition module for detecting speech based on the audio signal, identifying a location or time associated with the capture of the audio signal, and determining a context associated with the captured audio signal based on the location or the time; a confidence module for determining a confidence score indicating a likelihood that the speech is associated with the context and performing a comparison of the confidence score with a confidence threshold; as well as A training module is configured to train a neural network based on the audio signal corresponding to the speech in response to the confidence score being above the confidence threshold.

2. The device according to claim 1, wherein: The context is determined by deriving structured interactions based on conversational patterns within the audio signal.

3. The device according to claim 1, wherein: The context is based at least in part on behavioral data.

4. The device according to claim 1, wherein: The confidence module is used to determine the confidence score based on the language confidence of the speech.

5. The device according to claim 1, wherein: The confidence module is used to determine the confidence score based on the acoustic confidence of the speech.

6. The device according to claim 1, wherein: The confidence module is used to determine the confidence score based on the pattern confidence of the speech.

7. The device according to claim 1, wherein: The context is determined by deriving a structured interaction based on a conversational pattern within the audio signal, and responsive to the structured interaction being a new structured interaction, the structured interaction is stored in a database.

8. The device according to claim 1, wherein: The training module is for adapting a language model and an acoustic model by using the context and the audio signal in response to the confidence score being higher than the confidence threshold.

9. The device according to claim 1, wherein: The context is determined by deriving a structured interaction based on a conversational pattern within the audio signal, and the stored structured interaction is extended in response to additional identified portions of the structured interaction.

10. The device according to claim 1, wherein: The training module iteratively trains and adapts the neural network based on additional context and associated audio data.

11. A system for speech modeling via environmental context acquisition, comprising: A microphone, used to capture audio signals; a memory for storing instructions and communicatively coupled to the microphone; as well as A processor is communicatively coupled to the camera and the memory, wherein when the processor is to execute the instruction, the processor will perform the following operations: detecting speech based on the audio signal; identifying a location or time associated with capture of the audio signal; determining a context associated with the captured audio signal based on the location or the time; determining a confidence score indicating a likelihood that the speech is associated with the context; performing a comparison of the confidence score to a confidence threshold; and In response to the confidence score being above the confidence threshold, a neural network is trained based on the audio signal corresponding to the speech.

12. The system according to claim 11, wherein: The context is determined by deriving structured interactions based on conversational patterns within the audio signal.

13. The system according to claim 11, wherein: The context is based at least in part on behavioral data.

14. The system according to claim 11, wherein: When the processor is to execute the instructions, the processor will determine the confidence score based on the language confidence of the speech.

15. The system according to claim 11, wherein: When the processor is to execute the instructions, the processor will determine the confidence score based on the acoustic confidence of the speech.

16. The system of claim 11, wherein: When the processor is to execute the instructions, the processor will determine the confidence score based on the pattern confidence of the speech.

17. The system of claim 11, wherein: The context is determined by deriving a structured interaction based on a conversational pattern within the audio signal, and responsive to the structured interaction being a new structured interaction, the structured interaction is stored in a database.

18. The system of claim 11, wherein: When the processor is to execute the instructions, the processor will, in response to the confidence score being higher than the confidence threshold, adapt a language model and an acoustic model by using the context and the audio signal.

19. The system of claim 11, wherein: The context is determined by deriving a structured interaction based on a conversational pattern within the audio signal, and the stored structured interaction is extended in response to additional identified portions of the structured interaction.

20. The system of claim 11, wherein: The training module iteratively trains and adapts the neural network based on additional context and associated audio data.

21. A method comprising: Capture audio signals; detecting speech based on the audio signal; identifying a location or time associated with capture of the audio signal; determining a context associated with the captured audio signal based on the location or the time; determining a confidence score indicating a likelihood that the speech is associated with the context; as well as performing a comparison of the confidence score with a confidence threshold; In response to the confidence score being above the confidence threshold, a neural network is trained based on the audio signal corresponding to the speech.

22. The method according to claim 21, wherein: The context is determined by deriving structured interactions based on conversational patterns within the audio signal.

23. The method according to claim 21, wherein: The context is based at least in part on behavioral data.

24. The method according to claim 21, wherein: The confidence score is determined based on a language confidence of the speech.

25. The method according to claim 21, wherein: The confidence score is determined based on an acoustic confidence of the speech.

Images