Language-independent dictionary-trained grapheme-to-phoneme converter and text-to-speech engine for improved speech recognition

Abstract

A language-independent dictionary-trained grapheme-to-phoneme converter and text-to-speech engine for improved speech recognition is disclosed. Techniques are described for recognizing a spoken wakeup word (WW) or command using a speech recognition system that does not need to be trained with any speech data that matches the WW / command for a human machine interface. Techniques train a word segmentation device and include decomposing a word into a plurality of combinations by splitting the word from a database at a plurality of different points for each of the plurality of combinations of unique sub-words. The word includes one or more writing units and one or more corresponding acoustic units. Techniques map acoustic units constituting a word to writing units constituting the word to generate an acoustic unit-to-writing unit mapping. The technique includes assigning a subset of the acoustic units to each of the unique sub-words based on the acoustic unit-to-write unit map to generate an acoustic unit-to-sub-word assignment of the word. Techniques accumulate acoustic units to sub-word assignments for a plurality of words from a database to create a sub-word likelihood dictionary.

Description

[0001] Cross-references to related applications

[0002] This application claims the filing date interest in U.S. Provisional Application No. 63 / 706,483, filed October 11, 2024, by the applicant, Cypress Semiconductor Corporation, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This topic relates to the field of voice-controlled human-machine interfaces. More specifically, but not limited thereto, this topic discloses techniques for recognizing wake words or phrases (collectively referred to as wake words or WWs) used to trigger voice control of a device, as well as voice commands spoken by a user to control the device, without requiring training with voice data specific to a target WW or command. Background Technology

[0004] Consumer electronic devices such as smartphones, desktop computers, laptops, and home assistive devices are voice-controlled digital devices that can be controlled by users issuing voice commands to the device. For example, users can issue voice commands to the device to make calls, send messages, play media content, obtain query responses, retrieve news, set reminders, and invoke applications and services. Voice commands issued by a speaker may be interfered with by the voices of competing speakers, noise, or interruptions from the main speaker. Speech recognition systems used to detect and recognize voice commands increasingly rely on neural networks or machine learning to improve performance. To enable speech recognition systems to respond accurately and promptly to voice commands with easily confused phonemes, various speaker accents, and different pronunciations under adverse environmental conditions, neural network or machine learning models can be trained based on speech data of the target WW and the command. Offline training of a speech recognition system may require collecting speech data that matches the target WW and the command. However, training can be complex and time-consuming. The resulting system may also require a large memory footprint, and deployment may be expensive. On the other hand, speech recognition systems that are not directly trained on the target WW and the command may not be robust enough, resulting in poor performance under challenging environmental conditions. Summary of the Invention

[0005] According to one aspect of this disclosure, a method is provided for training a word segmenter to generate acoustic unit strings from input text, the method comprising: decomposing a word into multiple combinations by splitting the word from a database at multiple different points for each of multiple combinations of unique subwords, wherein the word comprises one or more written units and one or more corresponding acoustic units; mapping the acoustic units constituting the word to the written units constituting the word to generate an acoustic unit-to-written unit mapping; assigning a subset of the acoustic units to each of the unique subwords based on the acoustic unit-to-written unit mapping to generate an acoustic unit-to-subword assignment for the word; and accumulating the acoustic unit-to-subword assignments for multiple words from the database to create a subword likelihood dictionary.

[0006] According to another aspect of this disclosure, an apparatus is provided for training a word segmenter to generate acoustic unit strings from input text, comprising: a processing system configured to: decompose a word into multiple combinations by splitting the word from a database at multiple different points for each of a plurality of combinations of unique subwords, wherein the word comprises one or more written units and one or more corresponding acoustic units; map the acoustic units constituting the word to the written units constituting the word to generate an acoustic unit-to-written unit mapping; allocate a subset of the acoustic units to each of the unique subwords based on the acoustic unit-to-written unit mapping to generate an acoustic unit-to-subword allocation for the word; and accumulate the acoustic unit-to-subword allocation for a plurality of words from the database to create a subword likelihood dictionary. Attached Figure Description

[0007] The described embodiments and their advantages can be best understood by referring to the following description taken in conjunction with the accompanying drawings. These drawings do not limit any changes in form and detail that may be made to the described embodiments by those skilled in the art without departing from the spirit and scope of the described embodiments.

[0008] Figure 1 A scenario is described in which a user, according to one aspect of this disclosure, speaks a voice command to a smartphone, including a WW followed by the command, so that the smartphone can detect and recognize the voice command;

[0009] Figure 2 A high-level architecture of a dataless speech recognition system according to one aspect of this disclosure is described;

[0010] Figure 3 Operational details of offline and online training of a data-free speech recognition architecture according to one aspect of this disclosure are described.

[0011] Figure 4The processing details of the inference phase of a data-free speech recognition architecture using a trained acoustic unit 216 trained offline and a recognition model 350 trained online, according to one aspect of this disclosure, are described.

[0012] Figure 5 Another view of the inference phase of a dataless speech recognition architecture according to one aspect of this disclosure is depicted;

[0013] Figure 6 The present disclosure describes an approach to improving the decoding of recognized phrases in an automatic speech processing (ASP) application by using a phoneme analysis block based on phoneme generation statistics output by a unit matching block, according to one aspect of the present disclosure.

[0014] Figure 7 A block diagram depicting a sequence decoding block incorporating a Hidden Markov Model (HMM) according to one aspect of this disclosure is provided.

[0015] Figure 8A shows a timeline of the phonemes of WW “Okay Infineon” according to one aspect of this disclosure;

[0016] Figure 8B shows a spectrogram of the phonemes of WW “Okay Infineon” according to one aspect of this disclosure;

[0017] Figure 8C Annotations of phonemes in WW “Okay Infineon” according to one aspect of this disclosure are shown, along with the start and end times of each phoneme;

[0018] Figure 8D This illustrates the posterior probability of the phoneme "K" in WW "Okay Infineon" according to one aspect of this disclosure. ;

[0019] Figure 9 This describes the posterior probability grouped by cell boundaries according to one aspect of this disclosure. Example of calculation;

[0020] Figure 10 A matrix of 40 phonemes used in an American English plus silence class for an acoustic model is shown according to one aspect of this disclosure;

[0021] Figure 11 One aspect of this disclosure is shown. Figure 10 A magnified view of the upper left portion of the matrix;

[0022] Figure 12A unit matching block is described, which modifies the posterior probability based on statistics calculated in a similarity matrix according to one aspect of this disclosure;

[0023] Figure 13 An HMM for identifying a sequence decoding model of a WW or command based on statistics calculated in a similarity matrix, according to one aspect of this disclosure, is shown.

[0024] Figure 14 Examples of possible pronunciations of the state in the HMM for “Okay Infineon” and the highest confusing phoneme for each pronunciation are shown, according to one aspect of this disclosure;

[0025] Figure 15 The present disclosure describes the use of a decoding compensation block to improve the sequence decoding of WW or commands from an HMM of a word / command model 730, according to one aspect of the disclosure.

[0026] Figure 16 An example of one aspect of this disclosure is shown. The direct calculation, where the x-axis shows the observation sequence over time. The y-axis indicates the state;

[0027] Figure 17A The present disclosure describes the use of a forward algorithm to evaluate, according to one aspect of the disclosure. ;

[0028] Figure 17B The Viterbi algorithm is described in one aspect of this disclosure for evaluation. ;

[0029] Figure 18A The most probable state walk of positive data in an example WW is depicted according to one aspect of this disclosure;

[0030] Figure 18B The most probable state walk of negative data for example WW is depicted according to one aspect of this disclosure;

[0031] Figure 19 One aspect of this disclosure is described, such as Figure 18B The most probable state walk for negative data in the example WW However, it is highlighted to show the state transition;

[0032] Figure 20 One aspect of this disclosure is described, such as Figure 18B The most probable state walk for negative data in the example WW , to show Maintain the number of frames in each state;

[0033] Figure 21 The top-1 statistics of the phonemes of WW'Okay Infineon are depicted according to one aspect of this disclosure;

[0034] Figure 22 A sequence decoding structure for a user-defined command set is described according to one aspect of this disclosure;

[0035] Figure 23A depicts an aspect of this disclosure provided by [the party used for...]. Figure 22 The sequence decoding structure of WW identifies the constituent words of WW "Okay Infineon";

[0036] Figure 23B depicts an aspect of this disclosure provided by [the party used for...]. Figure 22 The simple command sequence decoding structure identifies the constituent words of the simple commands "Take a picture" and "Set alarm clock to snooze";

[0037] Figure 23C depicts an aspect of this disclosure provided by [the party used for...]. Figure 22 The sequence decoding structure of the compound command identifies the common sub-command and the four second-stage sub-commands of the four compound commands: 1) "Turn the light on in the living room;" 2) "Turn the light on on the porch;" 3) "Turn the light on behind the study desk;" and 4) "Turn the light on by the stove".

[0038] Figure 23D A description of one aspect of this disclosure is provided for decoding the following... Figure 22 The large number range of units in the sequence decoding structure identifies the number-based entity “two hundred and forty-seven degrees” by its constituent number range and units;

[0039] Figure 23E Depicting one aspect of this disclosure by means of... Figure 22 The sequence decoding structure of complex commands identifies the composition of the complex command "Set oven temperature to two hundred and forty-seven degrees" as a simple command, number range, and unit;

[0040] Figure 24A flowchart is shown illustrating a method for constructing a sequence decoding structure to identify WW or commands from a user-defined command set, according to one aspect of this disclosure.

[0041] Figure 25 The disclosure describes a recognition model for WW that detects “Okay Infineon” according to one aspect of the disclosure, as well as two subsequent recognition models for CMD 1 and CMD 2 that evaluate subsequent commands.

[0042] Figure 26 The disclosure describes a recognition model for WW that detects “Okay Infineon” according to one aspect of the disclosure, and two subsequent recognition models for CMD 1 and CMD 2 that evaluate subsequent commands using state compensation techniques.

[0043] Figure 27 A modification of a recognition model for WW based on one aspect of this disclosure is described to accommodate variable silence intervals between words;

[0044] Figure 28 A phoneme tokenizer is described, according to one aspect of this disclosure, for converting the morphemes in the input text “afternoon” into a string of phonemes.

[0045] Figure 29 A block diagram depicts the training and decoding phases of a word segmenter according to one aspect of this disclosure;

[0046] Figure 30 A block diagram depicts the training phase of a word segmenter using a reference speech dictionary according to one aspect of this disclosure;

[0047] Figure 31 The text describes the word "Example" at different points and the associated tags and positions of the words according to one aspect of this disclosure;

[0048] Figure 32 The phoneme-to-graphite mapping of the word “Example” according to one aspect of this disclosure and the positions associated with the graphemes are depicted;

[0049] Figure 33 The use of one aspect of this disclosure is described. Figure 32 Phoneme-to-graphite mapping, targeting Figure 31 The sub-words of “Example” are split into phonemes of the word “Example” and assigned to sub-words;

[0050] Figure 34A phonetic dictionary according to one aspect of this disclosure is described with counted triples {subword, tag, phoneme} for all words and the probability of each triple;

[0051] Figure 35 A word segmenter using a trained sub-word likelihood dictionary (e.g., according to one aspect of this disclosure) is described. Figure 29 A block diagram of the decoding stage of the word segmenter;

[0052] Figure 36 The sub-word splitting of the word "Infineon" in WW "Okay Infineon" according to one aspect of this disclosure is described;

[0053] Figure 37 One aspect of this disclosure is described. Figure 36 A list of word sub-words of the word "Infineon", including their sub-word combinations, phonemes, corresponding phoneme probabilities, and phonetic probabilities.

[0054] Figure 38 Operational details for tuning a text-to-speech engine to match the characteristics of synthesized speech generated by the text-to-speech engine with the characteristics of real speech, according to one aspect of this disclosure, are described.

[0055] Figure 39 A block diagram depicts the use of a tuned text-to-speech engine to train or tune a data-free speech recognition system to synthesize speech that would otherwise be unavailable for the speech recognition system, according to one aspect of this disclosure.

[0056] Figure 40 One aspect of this disclosure is described. Figure 39 A block diagram of the analysis block used to analyze synthesized speech to compile statistics for sequence decoding to assist the target speech;

[0057] Figure 41A depicts a timeline of the phonemes of WW “Okay Infineon” according to one aspect of this disclosure;

[0058] Figure 41B depicts a spectrogram of the phonemes of WW “Okay Infineon” according to one aspect of this disclosure;

[0059] Figure 41C depicts the time boundary determined by the aligner block 4010 for the phonemes of WW “OkayInfineon” according to one aspect of this disclosure;

[0060] Figure 42 A flowchart of a method 4200 for operating a data-free speech recognition system according to one aspect of this disclosure is shown; and

[0061] Figure 43 A block diagram of an apparatus for implementing a data-free speech recognition system according to one aspect of this disclosure is shown. Detailed Implementation

[0062] Examples of various aspects and variations of the subject matter are described and illustrated in the accompanying drawings. The following description is not intended to limit the invention to these embodiments, but rather to enable those skilled in the art to make and use the invention.

[0063] Figure 1 A scenario is depicted where a user, according to one aspect of this disclosure, speaks a voice command, including a WW followed by a command, to a smartphone, enabling the smartphone to detect and recognize the voice command. The smartphone 101 may include three microphones 102, 103, and 104 located at different positions on the smartphone 101. Microphones 102, 103, and 104 may form a compact microphone array to capture voice signals from the user 110. As an example, the user 110 may say WW or a phrase, followed by asking “What time is it?”, to request the current time from a smart assistant application. The target voice signal may be mixed with unwanted sounds from a noisy environment. The smartphone 101 may segment the voice signal captured by the microphones into frames and may transmit the audio data frames to a voice recognition algorithm executed on the smartphone 101 or a remote server.

[0064] Methods and systems are described for WW and command recognition solutions that do not require training with any user-defined WW or command matching speech data. This approach enables users / customers to quickly and inexpensively deploy speech recognition-based human-machine interfaces (HMIs). The disclosed systems and methods enable the very rapid deployment (e.g., only) of speech recognition solutions trained on text-specific WW and commands. Because no user-defined WW and command-specific speech data is required, the system is referred to as "data-free speech recognition." Advantageously, the "data-free" system can be trained online using, for example, text or glyph representations of WW or commands within approximately a few seconds before the system is ready for use. The resulting low complexity and small memory footprint can be implemented on edge processors, while the system is robust enough to handle confused phonemes, accents, different pronunciations, and adverse environmental conditions.

[0065] A system architecture for continuous speech recognition may have a feature analysis module for the speech signal, followed by a unit matching block, a lexical decoding block, a syntactic analysis block, and a semantic analysis block to generate the recognized utterance. The feature analysis block typically involves spectral and / or temporal analysis of the speech signal, resulting in an observation vector x, which is processed by the unit matching block to characterize various speech sounds. The unit matching block may include a recognition unit for recognizing language-based sub-word units, partial or complete word units, or even multiple word units, such as phonemes, diphones, or triphones. Generally, the smaller the sub-word units, the fewer they are, but the more complex their structure in speech, and therefore, the more the system performance depends on the remaining architectural blocks.

[0066] The lexical decoding block applies word-based knowledge to the output of the recognition unit, thereby restricting the possible unit decodings by considering word structure. A word dictionary may be included to further limit the possibilities to a valid word database. The lexical decoding block can be eliminated if the output of the recognition unit is a word. The syntactic analysis block applies further constraints based on lexical syntax and correct word ordering. Finally, the semantic analysis block includes additional constraints based on meaning, reference, logic, connotation, application, etc.

[0067] Continuous speech recognition architectures can potentially handle large vocabularies. Many applications exist where the effective vocabulary is very limited, often confined to the context of a single, focused scenario, such as controlling the functions of an oven or adjusting the settings of a smart thermostat. Typically, applications include a wake word (WW) for addressing the device, followed by a limited and known set of commands. In one implementation, the application may accept commands without a WW (e.g., key-click calling). In the case of a WW, the task at hand is to recognize individual words or phrases within a virtually unlimited range of possibilities for the input speech, noise, and conditions. In such deployment scenarios, the continuous speech recognition architecture can be simplified by eliminating syntactic and semantic analysis blocks. In a further simplification, the lexical decoding block can also be eliminated if the goal of the recognition unit is to recognize the WW itself.

[0068] For command-based speech recognition, if the command set contains a limited number of words and the grammar is known and limited to the command set, syntactic analysis can generate recognized commands, thus eliminating semantic analysis. If lexical decoding can construct complete commands rather than single words, syntactic analysis can also be eliminated. One or more of the feature analysis module, unit matching block, lexical decoding block, syntactic analysis block, and semantic analysis block can leverage neural networks and machine learning models to generate a broad speech recognition architecture tailored to the task and resources at hand, depending on the application, resource requirements, performance requirements, hardware capabilities, available training data, etc.

[0069] To accelerate the training and deployment of neural network-based speech recognition architectures to recognize user-defined commands and instructions, while providing robust performance, "data-free" speech recognition architectures can receive text data instead of speech data during the online training phase to build one or more command models. During the inference phase, the neural network-based speech recognition system can use the command models and acoustic models (derived from offline training) to recognize speech as commands associated with the received text.

[0070] Figure 2 A high-level architecture of a dataless speech recognition system according to one aspect of this disclosure is described. Training of the speech recognition architecture can be divided into two parts: offline training 210 and online training 230. Offline training 210 can use a publicly available annotated speech database 212 to train a neural network-based acoustic model 218 (e.g., a neural network-based unit matching block 256) to recognize acoustic units (e.g., phonemes) during inference. Offline training 210 is performed once during system development and is independent of user-defined WWs and commands (except language). In one embodiment, offline training 210 can be language-independent, enabling the recognition of acoustic units from any language by training acoustic model 218. During the offline training phase 210, feature analysis module 214 can perform spectral and / or temporal analysis on speech signals from the annotated speech database 212 to generate vectors, which are processed by acoustic model training 216 to train acoustic model 218.

[0071] On the other hand, online training 230 can be performed using user-defined WW and command text 232 to construct one or more command models, such as lexical decoding model 238 or syntactic analysis model 240, for WW and command speech recognition during inference. In one embodiment, online training 230 can train a text-to-speech (TTS) engine (not shown) to generate synthesized speech of WW and command text 232, which is then used for lexical analysis training 234 of lexical analysis model 238 and syntactic analysis training 236 of syntactic analysis model 240. In one embodiment, lexical analysis model 238 and / or syntactic analysis model 240 can be based on statistical models, such as word models based on Hidden Markov Model (HMM) phonemes.

[0072] During the inference phase 250, the data-free speech recognition architecture uses one or more command models (e.g., lexical analysis model 238 and syntactic analysis model 240) trained online and an acoustic model 218 trained offline to recognize the user-spoken speech 252 as either WW 262 or command 264 associated with the text. The inference phase 250 can use the same feature analysis module 214 as used for the offline training 210 of the acoustic model 218. The feature analysis module 214 can perform spectral and / or temporal analysis of the speech 252 for processing by the neural network-based unit matching block 256 using the acoustic model 218. A sequence decoding block including lexical decoding 258 and syntactic decoding 260 can process the output of the unit matching block 256 using the lexical model 238 and the syntactic model, respectively, to identify WW 262 or command 264 from a set of user-defined WW and command texts 232.

[0073] Figure 3 Operational details of offline and online training of a data-free speech recognition architecture according to one aspect of this disclosure are described.

[0074] The purpose of offline training is to generate a neural network-based acoustic model 218 capable of recognizing acoustic units present in the input speech. The acoustic units used by the system can be phonemes. The neural network-based acoustic model is trained "offline" because it is independent of the target phrase (except for language). In one implementation, offline training can generate a neural network-based acoustic model 218 capable of recognizing acoustic units present in any language, allowing data-free speech recognition architectures for many languages ​​to use the same acoustic model. The target phrase used here can refer to a WW or command that the data-free speech recognition system is trained to recognize. To achieve this, offline training can employ multiple speech databases. The databases can be annotated, such as a phoneme-annotated database 320, to identify the boundaries of the phonemes contained in the database. Enhancement block 322 can enhance the phoneme-annotated database 320 to further enrich the diversity of content and improve training. The data selection / preparation module 323 can select files for training based on a desired balance of factors such as gender, native speaker, non-native speaker, accent, age, background noise, and room acoustics, and can be adjusted based on the intended target application (near-field speech, near-field, far-field, geographical location, background noise environment, etc.). The feature extraction block 214 performs spectral and temporal analysis on the input speech from the selected files and generates observation vectors, also known as observation sequences, which capture important recognition aspects and discard as many other aspects as possible.

[0075] The neural network (NN) training block 324 iteratively iterates through the set of observation sequences and corresponding annotation information to learn to distinguish and classify input speech based on specified acoustic units, in this case, phonemes. The output of the neural network-based acoustic model 218 is a sequence of softmax vectors consisting of the phonemes of the language plus silence / noise classes. In one implementation, the output of the neural network-based acoustic model 218 is a frame-by-frame estimate of the likelihood or probability of the presence of the set of identified units (phonemes in this case). In another implementation, the output of the neural network-based acoustic model 218 represents some other estimate of the presence of identified units. The accuracy and precision of this estimate may be limited and vary depending on different NN designs, the training database used, the quality of the annotations, etc.

[0076] Phoneme analysis model 326 performs analysis of the final acoustic model, captures relevant information, and incorporates this information into the decoding model. In one implementation, phoneme analysis block 326 collects statistical information on the speech content and behavior of the representation database and the trained acoustic model 218, which is later used to improve the construction and performance of the decoder model.

[0077] The decoding model is trained or built "online" because it relies on the target phrase and is executed quickly so that it can be used by users almost immediately. Online training is based on user-defined WWs and commands entered as text. Figure 3 In the diagram, solid lines represent normal paths, while dashed lines represent alternative paths. User-defined text of the target phrase is input to a tokenizer / custom dictionary block 340, which includes a tokenizer that converts the text into its phoneme string equivalent (e.g., providing a phoneme sequence corresponding to each target phrase) and a custom dictionary. The conversion can be done by a tokenizer that converts any unseen text, while the custom dictionary allows for predefined phoneme specifications for non-dictionary types of words (e.g., "Infineon" as WW) or alternative pronunciations that the tokenizer might otherwise not match. The tokenizer / custom dictionary block 340 can provide not only preferred or most probable phoneme sequences but also alternative sequences that model different pronunciations or accents. In one implementation, the user can manually input desired, preferred, and / or alternative pronunciations 342 or phoneme sequences. Based on the phoneme sequences of the target WW and the command, and the results from phoneme analysis 326 trained offline, model building block 344 generates a recognition model 350 used during inference. In one implementation, to improve performance, model building block 344 may optionally utilize text-to-speech 346 to generate synthetic data containing the target WW and commands. Analysis module 348 may analyze the synthetic data to assist model building block 344 in generating recognition model 350.

[0078] Figure 4The processing details of the inference phase of a dataless speech recognition architecture using a trained acoustic unit 216 trained offline and a recognition model 350 trained online, according to one aspect of this disclosure, are described. Input speech 252 is first processed by a Speech Start Detection Algorithm (SOD) module 410. Because the user first addresses the device using WW, there is an assumption about previous pauses, just as one would naturally do when addressing another person. The SOD module 410 is designed to trigger at the start of speech, prior to which there is a minimal amount of non-speech. The inclusion of the SOD function provides several advantages to system performance: (1) subsequent operation blocks can be gated during non-speech channels, significantly reducing processing load and power consumption; and (2) during continuous speech, SOD will only trigger when a preceding non-speech gap is observed, preventing a large amount of audio from being processed and potentially causing false detections. Feature extraction block 214 performs spectral and temporal analysis on input speech 252 during active speech to produce observation vectors consumed by acoustic model (AM) 218. Acoustic model 218 utilizes a neural network-based unit matching model 256 obtained from offline training and outputs a series of softmax vectors from the time series of the input observation vectors. Decoder block 420 obtains a series of softmax vectors and applies statistical modeling based on recognition model 350 to determine whether WW exists or which command from the set of user-defined commands was spoken.

[0079] Figure 5 Another view of the inference phase of a data-free speech recognition architecture according to one aspect of this disclosure is depicted. The NN-based unit matching block 256 is a portion of trained acoustic units 216 obtained from offline training. User-defined WW and command 510 are target phrases whose phonemes are in a recognition model 350 obtained from online training. Feature extraction block 214 extracts relevant features from the input speech on a frame-by-frame basis, where a frame is a time unit used to operate the NN-based unit matching block 256. Feature extraction block 214 feeds the extracted features to the NN-based unit matching block 256. The NN-based unit matching block 256 processes the extracted features to output a sequence of unit (phoneme) likelihood vectors per frame. Sequence decoding block 420 can combine a Hidden Markov Model (HMM) based on the text-based user-defined WW and command 510 to represent the recognized phrase.

[0080] As discussed, the NN-based unit matching block 256 is trained offline using an annotated database to learn distinguishing units (phonemes). The HMM model used for sequence decoding block 420 is based on a user-defined text input description of the expected WW and commands (e.g., target phrases). Since the system is data-free, material containing target phrases is unavailable during the offline training process. The data-free speech recognition architecture utilizes... Figure 3 The training process discussed herein achieves a highly accurate and robust decoding method that adapts to the final NN acoustic model 218 and is adapted to alternative pronunciations of the target phrase. In one aspect, the HMM model used for sequence decoding block 420 can incorporate alternative pronunciation information modeled by the recognition model 350 built from online training. For example... Figure 3 As discussed in the paper, the results of phoneme analysis of the NN-based unit matching block 256 performed by the phoneme analysis block 326 as part of the offline training of the acoustic model 218 can also be used to construct the recognition model 350.

[0081] Figure 6 The present disclosure describes an aspect of using phoneme analysis block 326 to improve the decoding of recognized phrases in an automatic speech processing (ASP) application based on phoneme generation statistics output by unit matching block 256.

[0082] The annotated database 320 includes labels for the real units within the training speech segments and their respective temporal boundaries. During offline training of the acoustic model 218, the unit matching block 256 is processed through the annotated database, while the phoneme analysis block 326 simultaneously collects and compiles information about how the likelihood result 610 relates to the real labeled units within their respective temporal boundaries. For example, the phoneme analysis block 326 collects statistical information characterizing the speech content and behavior of the annotated database 320 and the unit matching block 256. The phoneme analysis block 326 compiles the phoneme analysis block 326 statistical information into the unit recognition statistical database 620. The ASP 640 then uses the unit recognition statistical database 620 to improve its results when using the unit matching block 256 on unseen speech data. In one implementation, the phoneme analysis block 326 generates a statistical matrix characterizing the unit matching block 256 of the acoustic model 218.

[0083] In one respect, identification model 350 uses statistical matrices to assist ASP 640 (e.g., combining...). Figure 5The sequence decoding block 420 of the HMM is used to present phrases based on text-based user-defined WW and commands 510. In one aspect, the unit matching block 256 can use a statistical matrix to modify the likelihood result 610 (e.g., the Softmax value). The maximum likelihood block 630 can find the maximum value of the modified likelihood result as the identified unit (phoneme).

[0084] Figure 7 A block diagram of a sequence decoding block 420 incorporating an Hidden Markov Model (HMM) according to one aspect of this disclosure is depicted. According to one aspect of this disclosure, an HMM (shown as a word / command model 730) is a technique for statistically modeling real word processes such as speech signals. When applied to phoneme-based speech detection, each state in the HMM represents a phoneme. Start and end states are modeled by silence, while internal states represent the constituent phonemes of a word or command. The model's observations—the Softmax output probability of each phoneme—are a probability function of the state. Thus, an HMM is a doubly embedded stochastic process where the underlying stochastic process is unobservable (it is hidden) but can only be observed through another set of stochastic processes that generate the observation sequence. In one implementation, the self-transition probability is calculated based on the expected length of the phoneme of state i. .

[0085] Given a Hidden Markov Model (HMM), the HMM evaluates the probability of the observed sequence at each input vector using the Softmax value. The probability can be evaluated using either a forward algorithm or a Viterbi algorithm. Assigning phonemes to each state of the HMM is challenging and can be achieved using methods such as speech dictionaries, manual definitions, word segmenters, etc. However, the resulting speech transcription may not be a good match, especially for unseen words. Furthermore, these methods typically do not produce alternative pronunciations. In one aspect of this disclosure, a decoding compensation model combines features derived from the internal behavior of the HMM with offline analysis of matching and non-matching words to improve the discriminative power of the HMM.

[0086] Given observation sequence and model HMM evaluation in a given model The probability of observing the sequence under certain conditions (That is, given the model λ of the HMM, the observed sequence) The probability of ( ).

[0087] Word / command model 730 illustrates an example of modeling the WW “Okay Infineon” with the pronunciation / OW / K / EY / IH / N / F / IH / N / IY / AA / N / using an HMM. However, alternative pronunciations may be equally valid or common and supported. Word / command model 730 can support alternative pronunciations by allowing multiple phonemes in each state definition of the HMM. The preferred or most common phonemes for each state can be listed first and used to derive the expected length (time) of the state. A WW “Okay Infineon” with multiple pronunciations is shown.

[0088] With the phoneme The corresponding Softmax output value is the posterior probability. The estimate is given by x, where x is the observed input feature vector. Ideally, an acoustic model such as unit-matched block 256 would have 100% accuracy and 100% confidence (a softmax score of 1.0 for the real phonemes). However, this is not the case. In fact, under clean conditions, the real phonemes have shown to have the highest score in the softmax vector for approximately 70% to 80% of the time, with an average of 0.5 to 0.9.

[0089] The Softmax output includes all phonemes. The posterior probability estimate (where N is the number of phonemes (plus noise)) has the following properties on a frame-by-frame basis:

[0090] (Equation 1)

[0091] Therefore, when the Softmax value of the true phoneme is less than 1.0, the difference is included in the next most probable phoneme. Thus, a matrix can be constructed to capture this information to better characterize the Softmax output of the acoustic model. As indicated, Figure 6 The phoneme analysis block 326 can compile a matrix by iteratively processing the annotated training database 320. The annotated training database 320 can indicate the real phonemes and phoneme temporal boundaries within each audio file. For each phoneme temporal boundary, the phoneme analysis block 326 identifies and stores the Softmax vector of the frame containing the maximum value of the real phoneme. The phoneme analysis block 326 repeats this process for each utterance in the database, such that a matrix of Softmax vectors is accumulated for each phoneme. When the real phoneme is at its maximum value within the identified boundaries, the Softmax vectors are averaged over time to obtain an average Softmax vector value. Figure 10An example of a matrix representing the 40 phonemes of the Softmax output of the acoustic model in the American English plus silence class is shown. This matrix is ​​called a "similarity matrix" because the posterior probability is related to acoustically similar phonemes. The word / command model 730 addresses the problem of: given an observation sequence And the model λ of the HMM, the probability of the observed sequence How much? The word / command model 730 can use a similarity matrix to identify easily confused phonemes in each state and improve the model's performance.

[0092] The word / command model 730 also includes a decoding compensation block 740 and a general background model 750. The decoding compensation block 740 combines offline analysis from the phoneme analysis block 326, intermediate model observations and probabilities from the word / command model 730, and the current model probability. As its input, it calculates a new frame-by-frame model score for detecting the presence of a user-defined WW or command. The decoding compensation block 740 not only combines... Furthermore, by combining features derived from the internal behavior of the word / command model 730, and additionally considering offline analysis of matching words (positive input) and non-matching words (negative input) from the phoneme analysis block 326, the sequence decoding block 420 is improved relative to... The ability to distinguish.

[0093] To further improve the robustness of the sequence decoding block 420 to poorly pronounced speech, noise conditions, different speaker rates, etc., the Universal Background Model (UBM) 750 attempts to normalize these conditions when estimating user-defined WWs or commands.

[0094] In one implementation, the UBM 750 is modeled as a 3-state HMM with a leading silence state (leading sil), a speech state (Sp), and a final silence state (final sil). The HMM can be modeled by state transition probabilities and state emission probabilities. The state transition probability indicates how likely the HMM is to transition to a given current state. The state emission probability indicates the probability that the HMM will generate an observation given a given current state. The 3-state HMM can obtain the emission probability of the leading silence state from the NN Softmax entry of SIL, while the emission probability of the speech state is 1 - (the emission probability of the leading silence state). The leading silence state transition probability can be based on the expected length of the leading silence from SOD triggering until the speech begins. The speech state transition probability can be based on the average length of phonemes in the WW / CMD being processed by the UBM. The final silence state self-transition probability can be 1.0. The UBM probability is likely the maximum probability of these three states. The operation of the sequence decoding block 420 is described in detail below.

[0095] As discussed, Figure 6 The phoneme analysis block 326 can compile a similarity matrix by iteratively processing an annotated training database 320, which contains the root true phoneme and the temporal boundaries of the phonemes within each audio file.

[0096] Figure 8A shows a time diagram of the phonemes of WW “Okay Infineon” according to one aspect of this disclosure. Figure 8B shows a spectrogram of the phonemes of WW “Okay Infineon” according to one aspect of this disclosure. Figure 8C Annotations of phonemes in WW "Okay Infineon" according to one aspect of this disclosure are shown, along with the start and end times of each phoneme.

[0097] Figure 6 Unit matching block 256 (or Figure 5 The feature extraction block 214 can compute features based on the input speech from the annotated training database 320. The unit matching block 256 (also known as the acoustic model or AM) then performs the feature-to-sound mapping. The output of AM is a measure of the confidence in the presence of each phoneme in the current speech analysis window. Some common confidence measures include likelihood, such as Softmax. Assuming the... phoneme The corresponding output is the posterior probability. The estimate is given by , where x is the observed input feature vector. Since AM cannot be 100% confident in the feature-to-phoneme mapping, the posterior probability of each root true phoneme is... Less than 1.

[0098] Figure 8D The following is an example of an output from AM at 0.41 seconds for WW “Okay Infineon”, according to one aspect of this disclosure. Example. At 0.41 seconds, according to Figure 8C The annotation and analysis window are centered on the actual phoneme "K". Figure 8D The AM hypothesis shows that the most likely phoneme is “K” with a posterior probability P = 0.720, while the next most likely phonemes are “D” and “T”. The sum of the posterior probabilities is shown as 1.000. Therefore, the AM hypothesis states that the root true phoneme “K” has the highest posterior probability, but the score is << 1.0, and the differences are mainly between the similarly pronounced phonemes “D” and “T”. If this behavior is consistent, this information can be used to further increase the confidence in the existence of “K”. Then, using an annotated database with labeled root true phonemes and time boundaries, data can be obtained for all instances of each phoneme (or unit) as follows: Figure 8D The data in the middle.

[0099] Figure 9 This describes the posterior probability grouped by cell boundaries according to one aspect of this disclosure. An example of the calculation.

[0100] AM can use length The posterior probability is calculated using data within a window of (920). (910) The window is shifted at each analysis interval. (930). The center of each window represents the corresponding... The time instance. For cell U_n (940), the start time and end time are marked as follows: (950) and (960). Posterior probability and Located within the time boundary of U_n and collected as part of its data, while Located outside the boundary, use length The data within the window (920), which is shifted at each analysis interval. (930). Figure 6 The phoneme analysis block 326 can collect the posterior probabilities of phonemes from AM within the time boundary of U_n (940). (910) to calculate desired statistics, such as unit recognition statistics 620, which can further directly improve unit (phoneme) recognition performance or downstream performance of ASP results.

[0101] In one respect, the phoneme analysis block 326 can construct a matrix to capture this information and better characterize the AM output. For example, the phoneme analysis block 326 can compile the matrix by iteratively processing the annotated training database 320. For each phoneme time boundary U_n (940), the frame containing the maximum value of the root true unit (phoneme) (e.g., (920)) posterior probability vector (910) is identified and stored. Phoneme analysis block 326 repeats this process for each utterance in the database, so that accumulation is performed for each unit. (910) A matrix of values. The vector is averaged over time to obtain the average when the root true cell is at its maximum value within the identified boundary. Vector value.

[0102] Figure 10 A matrix of 40 phonemes used in an American English plus silence class for an AM, according to one aspect of this disclosure, is shown. This matrix is ​​called a "similarity matrix" because the posterior probabilities are related to acoustically similar phonemes. The matrix shows the root true units on the x-axis and the AM output values ​​on the y-axis.

[0103] Figure 11 One aspect of this disclosure is shown. Figure 10 A magnified view of the upper left portion of the matrix. The first column indicates the average maximum of the AM output "AA" when the root true phoneme is "AA". The value is 0.621. Meanwhile, the average values ​​of the phonemes {"AE", "AH", "AO", "AW", "AY"} are {0.014, 0.014, 0.099, 0.031, 0.015}, respectively.

[0104] As explained above, AM classifies phonemes with neither 100% accuracy nor 100% confidence. As shown in Equation (1), the sum of the output classes is 1. Therefore, when the Softmax value of the true phoneme is less than 1.0, the difference is included in the next most probable phoneme. The similarity matrix captures the statistics of this confusion between similar phonemes. In one aspect, to improve the discriminative power of AM, the unit matching block 256 incorporates prior information from the similarity matrix into the posterior probability estimate.

[0105] Figure 12 A unit matching block 256 is described, which modifies the posterior probability based on statistical data calculated in a similarity matrix, according to one aspect of this disclosure.

[0106] Current phoneme It is the first phoneme The posterior probability is given by the following formula:

[0107] (Equation 2)

[0108] in It is the first The first in the row The softmax value of a phoneme. When and When it is at its maximum value, the similarity matrix gives each phoneme The average value. For example, refer to Figure 11 The magnified similarity matrix in the image, the first column gives... The average Softmax score at that time, with the top three scores being {"AA"=0.621, "AO"=0.099, "AW"=0.031}. Therefore, if such scores are actually observed, the unit matching block 256 will increase. The confidence level. One way to quantify this is to identify the most likely phonemes and sum them. This works for the scenario above, but may not perform well for {"AA"=0.1, "AH"=0.6, "AO"=0.2}. In this case, the posterior probability estimate does not correlate well with the estimate in the similarity matrix.

[0109] To address this, one technique for improving posterior probability estimation is to add the Softmax value of the easily confused phonemes to a constraint that considers the ratios in the similarity matrix. (Similarity matrix) The The top N highest scores in the column are represented as { , , ..., }, then the new posterior probability estimate Formulated as:

[0110] (Equation 3)

[0111] in It is a small factor (e.g.) ), plus The ratios are adjusted to allow for some variation, since these ratios are global averages. The model compensation block 1210 modifies the likelihood result 610 (e.g., the Softmax output) as in Equation 3 to improve the posterior probability estimate only if the Softmax output is well correlated with the similarity matrix, thereby improving recognition capabilities without increasing false detections.

[0112] Advantageously, the method of modifying the posterior probability based on the calculated statistics in the similarity matrix improves speech modeling for a given acoustic model, thereby improving phoneme recognition and speech recognition. Furthermore, the acoustic model training loop can integrate the model compensation block 1210 to automatically compensate for different AM characteristics, thus avoiding the need for retraining or retuning the ASP system.

[0113] In one respect, the HMM used for decoding the posterior probability sequence of speech recognition can use Equation 3 to improve the transition probabilities of state decoding.

[0114] Figure 13 An HMM for identifying a sequence decoding model of a WW or command based on statistical data calculated in a similarity matrix is ​​shown, according to one aspect of this disclosure.

[0115] The first in HMM The status indicates the first WW or command being identified. Phonemes. The model starts and ends with a silence state, and the total number of states in the model is... . Figure 13The example shown is WW “Okay Infineon” with the phoneme transcription / OW / K / EY / IH / N / F / IH / N / IY / AA / N / . The HMM uses a similarity matrix 1310 to handle easily confused phonemes. In one implementation, the HMM may not explicitly include the silence state when it is combined with another state modeled on the phonemes.

[0116] In one respect, alternative pronunciations may be equally effective or common. HMMs can support alternative pronunciations by allowing multiple phonemes in each state definition, as shown in states 1320 and 1330. The preferred or most common phonemes for each state can be listed first and used to derive the expected length (time) of the state. HMMs can limit the number of pronunciations to minimize processing complexity. Additionally, alternative pronunciations increase the chance of false detection, so this can be traded off against an increase in detection rate.

[0117] When determining which pronunciations to include in a Hidden Markov Model (HMM), the model can include probabilities. For example, if two pronunciations have equal probabilities, the HMM can include both. However, if one pronunciation has a probability of 0.99 and the other 0.01, the HMM might not include the latter pronunciation, as including it would only slightly improve the positive recognition rate while potentially increasing the false detection rate by a disproportionate amount.

[0118] In one aspect, it is used for word segmenters that convert text into its phoneme string equivalents (e.g. Figure 3 The word segmenter / custom dictionary block 340 can identify multiple pronunciations of the word to be modeled. For example, the custom dictionary block 340 can provide not only the preferred or most likely phoneme sequence, but also alternative sequences and the probability of each phoneme sequence. In addition, advanced users with phonological knowledge can provide preferred and alternative pronunciations, especially for WWs not in the dictionary (e.g., “OkayInfineon”).

[0119] In one aspect, the Hidden Markov Model (HMM) of the sequence decoding model includes the principal articulation phoneme for each state, and up to P-1 alternative articulations for each state, for a total of up to P phonemes per state. For each articulation, the most confusing phoneme according to the similarity matrix can be included along with their respective ratios according to Equation 3. In one implementation, easily confused phonemes are included until the sum of Equation 4 is greater than 0.8 and and :

[0120] (Equation 4)

[0121] These parameters are configurable.

[0122] Figure 14An example is shown of the possible pronunciations of the state in the HMM for “Okay Infineon” according to one aspect of this disclosure, and the highest confusing phoneme for each pronunciation. Figure 14 The ratio of the highest confusing phoneme for each pronunciation in the table includes Equation 4. .

[0123] like Figure 7 As discussed herein, the sequence decoding block 420 may include a decoding compensation block 740 and a general background model 750. The decoding compensation block 740 combines offline analysis from the phoneme analysis block 326, intermediate model observations and probabilities from the HMM of the word / command model 730, and the current model probability. As input, a new frame-by-frame model score is calculated to improve the discriminative power of sequence decoding block 420.

[0124] Figure 15 The present disclosure describes the use of a decoding compensation block 740 to improve the sequence decoding of WW or commands from an HMM of a word / command model 730, according to one aspect of the disclosure.

[0125] On the one hand, HMM can be determined by parameters Let A represent the state transition probability matrix, B represent the emission probability, and π represent the initial state distribution. A is a matrix whose rows represent the probability distribution indicating the likelihood of the HMM transitioning to each state given a given current state. B estimates the observations generated by the HMM given a given current state. The probability of π is the probability distribution indicating the probability of starting the HMM in each state (usually starting in the first state).

[0126] For example, in Figure 15 In the HMM of the word / command model 730, as mentioned above, the transition probability Defined by matrix A. The observation vector x in a given state s j The probability of the following is given by the output distribution based on the emission probability B:

[0127] (Equation 5)

[0128] Given observation sequence and model HMM in a given model Calculate the probability of the observed sequence under the following conditions. (i.e., in a given HMM model) Observation sequence under the circumstances (The probability). Calculate. A direct method can be applied to all possible state sequences that could produce the observation sequence X. Summation is then performed. However, this direct calculation method is very complex.

[0129] Figure 16 An example of one aspect of this disclosure is shown. The direct calculation is shown, where the x-axis represents the observation sequence over time, and the y-axis represents the state. The arrows indicate possible state transitions or state sequences that could lead to the observation sequence X. Figure 16 This indicates that the calculation is very complex.

[0130] Conversely, Hidden Markov Models (HMMs) can be computed using a recursive method (known as the forward algorithm) based on the Markov assumptions of HMMs. .

[0131] Figure 17A The present disclosure describes the use of a forward algorithm to evaluate, according to one aspect of the disclosure. The forward algorithm leverages the principle that, since there are only N states (nodes at each time slot in the lattice), all possible state sequences will be re-merged into these N nodes, regardless of the length of the observation sequence. Then, at each time t, only the value is calculated. , Each calculation involves only N previous values. For example, Figure 17A The forward algorithm is shown to be Calculated as their respective transition probabilities , , Weighted , , The sum of.

[0132] In one implementation, HMM can further simplify the recursive approach by using the Viterbi algorithm instead of summing the most probable paths.

[0133] Figure 17B The Viterbi algorithm is described in one aspect of this disclosure for evaluating The Viterbi algorithm obtains the likelihood of the most probable path through the grid. At each time t and in state s, for The calculation produces the likelihood The calculation.

[0134] Corresponding to the jth phoneme The output value of Softmax is the posterior probability. The estimated value. Using Bayes' rule, the posterior probability. Possible and Likelihood Based on the following:

[0135] (Equation 6)

[0136] By transforming the likelihood, we obtain:

[0137] (Equation 7)

[0138] The Viterbi algorithm can use the Softmax output to evaluate the grid. Equation (7) explains that the likelihood is obtained by dividing the posterior probability by the prior probability, which means that the NN Softmax output score is divided by the relative frequency of each phoneme. Equation 7 also shows scaling the division of the posterior and prior probabilities by the probability of observing x, which can be achieved from... Figure 7 The general background model 750 is estimated. If the ASP task is to choose from a set of models (e.g., one of the Y commands), then the term p(x) can be ignored because it does not depend on the state j. However, for WW applications, the HMM distinguishes WW from all spurious outputs and therefore estimates p(x). The same applies to detecting words outside the vocabulary for a set of commands.

[0139] The model probabilities evaluated using the Viterbi algorithm are described as follows:

[0140] (Equation 8)

[0141] Where a s It is a transition probability vector, for example Figure 17B of .

[0142] Replace Equation 8 with Equation 7 get:

[0143] (Equation 9)

[0144] In one implementation, HMM can be evaluated using a modified version of Equation 9, which includes a similarity matrix formula. Using Equation 3, Equation 8 can be modified as follows:

[0145] (Equation 10)

[0146] Next, the HMM may include multiple pronunciations. Each state may include multiple phonemes in its definition. The phonemes included in the definition of state s are defined as follows: :

[0147] (Equation 11)

[0148] HMM can then assess the probability of the observed sequence as follows:

[0149] (Equation 12)

[0150] At time t in each frame, the state with the highest likelihood is known to be the most probable state. And it can be represented as:

[0151] (Equation 13)

[0152] Advantageously, the described HMM automatically optimizes the decoder model to different acoustic models to improve performance and accuracy. The HMM also recognizes different accents and pronunciations.

[0153] The Viterbi algorithm is used to evaluate the mesh and obtain the likelihood of the most likely path, but the path itself is not mentioned. However, since the HMM is modeling WW or CMD, and each state represents phonemes sequentially from start to end, the most likely state is found according to Equation (13). It may also progress sequentially over time. Therefore, The path used can also be used to distinguish states within a sequence. The most likely state sequence (also known as a state walk) can be represented as:

[0154] (Equation 14)

[0155] Where t is the first frame, >1, meaning that the most likely state is not the initial silent state.

[0156] Figure 18A The most probable state path of positive data for an example WW according to one aspect of this disclosure is depicted. As shown, for a WW that matches a phoneme in the state, This process is typically carried out in an ordered manner within the HMM's states.

[0157] Figure 18B The most probable state path for negative data in an example WW, based on one aspect of this disclosure, is depicted. As shown, The irregular behavior is significant because the definition of state phonemes does not match the phonemes in speech.

[0158] Return to reference Figure 15 The decoding compensation block 740 includes a sub-word ratio analysis block 1510, a state transition analysis block 1520, a self-transition analysis block 1530, and a top-1 statistics analysis block 1540 to improve the decoding of sequences of WW or commands from HMM.

[0159] On one hand, the sub-word ratio analysis block 1510 can analyze the time length of words in the decoded sequence of a WW or command. WW / CMD / phrases are typically composed of single words. For these words, the expected time (number of frames) for each word can be estimated. Each speaker may speak slowly or quickly, but the length ratio between words should remain roughly consistent. If more time is spent on a word relative to the expected time on another word, then a WW is unlikely to exist.

[0160] Sub-word ratio analysis block 1510 can use the most probable state sequence decoded by the Viterbi algorithm of HMM. The sub-word ratio penalty is calculated based on the states representing different words in the HMM and the number of frames spent for each word. The sub-word ratio analysis block 1510 can calculate the penalty for each word according to the following... Total length of WW The ratio, and with the expected length and Comparison:

[0161] (Equation 15)

[0162] Sub-word ratio analysis block 1510 can calculate the log-likelihood penalty according to the following: :

[0163] (Equation 16)

[0164] In one implementation method and The default value is .

[0165] Sub-word ratio analysis block 1510 can calculate the total sub-word ratio penalty as the sum of all sub-word penalties. :

[0166] (Equation 17)

[0167] in It is the number of subwords in WW or the command.

[0168] On the one hand, state transition analysis block 1520 can analyze The state transitions or jumps in the HMM. Since each state in the HMM represents a phoneme in the pronunciation of the desired word, therefore Each state should be included. When there are jumps or transitions in the input speech, it means that there are no phonemes in the input speech.

[0169] Figure 19 One aspect of this disclosure is described, such as Figure 18BThe most likely state path for negative data in the example WW However, it is highlighted to indicate the state transition. Trajectory 1910 The display shows a jump from state 2 at frame 10 to state 9 at frame 11, thus changing 6 states in the model. In the same graph, trajectory 1920... The jump from state 6 in frame 9 to state 4 in frame 10 indicates a disorder of phonemes in the input signal. Both scenarios suggest that the target WW modeled by the HMM may not exist in the speech.

[0170] State transition analysis block 1520 can apply the following to state transition penalties: Calculated as The weighted sum of all state transitions found in the process:

[0171] (Equation 18)

[0172] in, and They are The states at frames t and t-1, and It is a weighting factor. In one implementation, The default value is .

[0173] On the one hand, the self-transformation analysis block 1530 can analyze The self-transition probability of each state. The average length of each phoneme. It can be found offline from a speech database of the modeling language. Since each state in an HMM represents a phoneme, the self-transformation probability... and Based on the following:

[0174] (Equation 19)

[0175] in It is the time of each frame.

[0176] therefore, It is the expected number of frames in the corresponding state, for which... It is the preferred pronunciation. The self-transformation analysis block 1530 can compare the expected number of frames with the observed... Compare the consecutive frames spent in a certain state. If the observed number of frames in a certain state significantly exceeds the expected number of frames, then WW (Wide Frames) is unlikely to exist.

[0177] Figure 20 One aspect of this disclosure is described, such as Figure 18B The most likely state path for negative data in the example WW , to show Maintain the number of frames in each state. For example, for trajectory 2010... For the 11 frames from frame 4 to frame 15, the state remains at 4.

[0178] Self-transformation analysis block 1530 can quantify Maintain the difference between the observed frame count and the expected frame count in each state. Let... From The number of consecutive self-transformations at the beginning, and Given the expected length of the preferred articulated phoneme of state s, then the self-transformation penalty... As given below:

[0179] (Equation 20)

[0180] in It is a self-transformation factor. In one implementation, It can have a default value of 0.5.

[0181] Equation 20 indicates that if the difference between the observed self-transformation length and the expected length of a phoneme is greater than zero, then this difference is multiplied by the self-transformation factor. This is then multiplied by the logarithm of the self-transition probability to calculate the self-transition penalty corresponding to the state of that phoneme. Equation 20 calculates the sum of all such formulas used for self-transitions during the current state walk. This formula is proportional not only to the number of frames exceeding the expected threshold but also to how quickly the expected state transitions to the next state. For example, if the number of self-transitions exceeds the expected threshold by 2 frames, then if... If the value is 0.5, then the penalty is If the state is highly likely to remain in its current state, then Close to 1, and This makes sense because in this case, it's more likely to exceed two frames. However, if the phoneme length is short, the probability of self-transformation is low. For example, if ,but In this case, when the transition probability is 0.9, the possibility of exceeding the expected length by 2 frames is very small, and therefore, the self-transition penalty is large.

[0182] On the one hand, top-1 statistical data analysis block 1540 can be based on The degree of matching with the expected phonemes of WW or commands to determine L. Weighting is applied. The motivation for the analysis is that, although HMMs capture the likelihood of the input well given the model,... However, it does not inherently assign weights to the absolute ranking of phonemes in the Softmax output.

[0183] Figure 21 This table depicts the top-1 statistics of the phonemes for WW 'Okay Infineon' according to one aspect of this disclosure. It indicates which phonemes were predicted correctly with what accuracy. For example, during the time interval labeled "K", "K" is the top-ranked phoneme in the Softmax vector for 78% of the time (91% if spurious noise classes are removed). Therefore, during decoding, it is highly plausible that the top phoneme in the Softmax vector is "K" when the most likely state corresponds to the "K" state. Similarly, if "K" is not found to be the top-ranked phoneme in the Softmax vector during that time period, then WW seems unlikely to exist, regardless of the overall model's likelihood. On the other hand, the penultimate state labeled "EH" has several phonemes that are typically ranked first. In this case, too much weight should not be applied to the phonemes ranked first during those frames in that state.

[0184] Will Defined as the score of the phoneme 'phone' that is in HMM state 'state'. Therefore, from Table 1, .Will Defined as the phoneme at time t where the Softmax score is highest:

[0185] (Equation 21)

[0186] but

[0187] (Equation 22)

[0188] in It is the prior probability.

[0189] Top-1 statistical data analysis block 1540 can be based on the status The score at each frame t is counted, and then the score for each state is averaged. The top-1 statistics analysis block 1540 can average the average score for each state across states to obtain the final top-1 (ranked first) score. If a state scores no points, then that state may receive an average score of -1. The final penalty is then calculated using the following formula. :

[0190] (Equation 23)

[0191] in It is the number one scoring factor. In one implementation, It can have a default value of 3.0.

[0192] The formula rewards the top-1 sequences that match the expected sequence, assigns extra weight to phonemes that are highly expected to rank in the top-1, and penalizes the top-1 sequences that do not match, especially those phonemes that do not match the phonemes that are highly expected to rank in the top-1. Note the final penalty. It can be positive or negative.

[0193] Total compensation It is the sum of compensation from all individuals:

[0194] (Equation 24)

[0195] in

[0196] This is the total sub-word ratio penalty in Formula 17;

[0197] This is the state transition penalty in Equation 18;

[0198] It is the self-transformation penalty of Equation 20; and

[0199] This is the top-1 score penalty for Equation 23.

[0200] Decoding compensation block 740 can be modified by adjusting the model likelihood score at time t according to the following formula. To improve sequence decoding of WW or commands from HMM:

[0201] (Equation 25)

[0202] Advantageously, using the Decode Compensation Block 740 on different WW models shows a 50-90% reduction in false alarm (FA) rate. The Decode Compensation Block 740 is integrated within the decoder and operates frame-by-frame, thus working seamlessly with the Viterbi algorithm and introducing virtually no additional algorithmic or processing latency.

[0203] On the one hand, the sequence decoding structure set customized for user-defined command sets can support any combination of WW, simple commands, compound commands, and number-based commands.

[0204] Figure 22A sequence decoding structure for a user-defined command set is described according to one aspect of this disclosure. The user-defined command set may include WW (wildcard), simple commands, compound commands, number-based commands with units, and complex commands including combinations of commands, numbers, and units. The sequence decoding structure may include different combinations of structure blocks such as lexical decoding blocks, syntactic analysis blocks, and semantic analysis blocks for different command types within the command set. The structure blocks may include Hidden Markov Models (HMMs) for performing decoding or analysis. In one embodiment, the sequence decoding structure may combine the internal HMMs of cascaded structure blocks into a single HMM.

[0205] The sequence decoding structure used for WW is similar to Figure 2 The structure is depicted in the diagram. This structure may include a feature analysis module 214 that performs spectral and / or temporal analysis on the speech for processing by a unit matching block 256 based on a unit database 2210, such as an acoustic model. A lexical decoding block 2228 may process a sequence of phoneme likelihood vectors from the unit matching block 256 to present the identified WW based on features of the WW, such as limitations on possible phoneme sequences due to the structure of the WW. In one implementation, the WW is a phrase with two words, such as "Okay Infineon". The lexical decoding block 2228 may decode the WW phrase using a single structural block combining the phonemes of the two words.

[0206] Simple commands may include constituent words that have little or no commonality with other commands. For example, a simple command could be "take a picture" or "set alarm clock to snooze". Similar to the decoding structure used for WW, a single structural block for decoding simple commands may include a lexical decoding block 2238 trained to recognize one or more simple commands 2230. The lexical decoding block 2238 can process the sequence of phoneme likelihood vectors from the unit matching block 256 by modeling the combined phonemes of the constituent words of the simple command to present the recognized simple command.

[0207] Compound commands can include a mixture of common subcommands and subcommands that are unique to each compound command. For example, four compound commands 1) "Turn the light on in the living room;" 2) "Turn the light on on the porch;" 3) "Turn the light on behind the study desk;" and 4) "Turn the light on by the stove" can be broken down into a common subcommand "Turn the light on" followed by four unique second-stage subcommands.

[0208] The sequence decoding structure used to decode compound commands may include a lexical decoding block 2248 trained to identify common and unique subcommands based on a finite word dictionary 2240. A syntactic parsing block 2244 trained to identify one or more compound commands 2242 may apply constraints based on word grammar and appropriate ordering to evaluate common and unique subcommands. For example, if the syntactic parsing block 2244 identifies “Turn the light on”, it may evaluate a set of four second-stage subcommands to present the identified compound command.

[0209] If a command consists of only a few numbers, such as "Set the dial to {1,2}", it can be expanded into two separate simple commands or combined into a compound command. However, this becomes impractical when the range of numbers is large, such as setting the oven temperature to "two hundred and forty-seven degrees". A sequence decoding structure for decoding large ranges of numbers followed by numerical units (referred to as number-based entities), such as temperature, volume, currency, and time, can include a lexical decoding block 2258 trained to recognize numbers and units based on a number / unit dictionary 2250. A syntactic analysis block 2254 trained to recognize numbers followed by units can apply rule 2252 to evaluate the sequence of numbers and units. Number decoding may need to consider the past and present to determine the future. For example, if the expected range is in digits, the number "two" might be the endpoint of recognition, or if a larger range is defined, it might be followed by "hundred" or something else. Therefore, digital decoding may include a semantic analysis block 2255, which is trained to evaluate commands based on constraints such as meaning, reference, logic, implication, application (collectively referred to as application 2255) to present recognizable number-based entities.

[0210] Complex commands can include compound commands followed by a wide range of numbers and units, or simple commands. For example, a complex command could be "set oven temperature to 200 forty-seven degrees". The sequence decoding structure used to decode complex commands can combine structural blocks from decoding structures used for simple commands, compound commands, and number-based entities. For example, the sequence decoding structure for complex commands could include a lexical decoding block 2238 trained to recognize one or more simple commands 2230, a lexical decoding structure 2258 trained to recognize numbers and units based on a number / unit dictionary 2250, a syntactic analysis block 2254 trained to recognize numbers followed by units based on rule 2252, and a semantic analysis block 2255 trained to evaluate number-based entities based on constraints in application 2255. The sequence decoding structure can present a recognized complex command composed of simple commands followed by number-based entities.

[0211] Figure 23A depicts one aspect of the disclosure by Figure 22 The constituent words of WW “Okay Infineon” used for sequence decoding structure recognition in WW.

[0212] Figure 23B depicts one aspect of this disclosure by Figure 22 The sequence decoding structure used for simple commands identifies the constituent words of the simple commands "Take a picture" and "Set alarm clock to snooze".

[0213] Figure 23C depicts one aspect of the disclosure by Figure 22 The sequence decoding structure for recognizing compound commands consists of the common sub-commands and four second-stage sub-commands of the four compound commands: 1) "Turn the light on in the living room;" 2) "Turn the light on on the porch;" 3) "Turn the light on behind the study desk;" and 4) "Turn the light on by the stove".

[0214] Figure 23D Depicting one aspect of this disclosure by Figure 22 The structure used to decode sequences of large numbers with units is based on the number range and units of the entity “two hundred and forty-seven degrees”.

[0215] Figure 23E The composition of one aspect of this disclosure is described by Figure 22 The sequence decoding structure used for complex commands identifies the complex command "Set oven temperature to two hundred and forty-seven degrees", as well as the simple command, number range, and unit.

[0216] Users can define WWs and commands in the command set and can invoke design flows to map the user-defined command set to the desired sequence decoding structure as part of the training process.

[0217] Figure 24 A flowchart of a method 2400 for constructing a sequence decoding structure to identify a WW or command from a user-defined command set, according to one aspect of this disclosure, is shown. In one aspect, method 2400 can be performed by a dataless speech recognition system utilizing hardware, software, or a combination of hardware and software.

[0218] In Operation 2401, the data-free speech recognition system can select user-defined WW and commands.

[0219] In operation 2403, the dataless speech recognition system can analyze the content and inherent structure of the WW and commands. In one implementation, the WW can include multiple constituent words, and commands can be classified as simple commands, compound commands, number-based entities, and complex commands that include a combination of simple / complex commands and number-based entities.

[0220] In operation 2405, the data-free speech recognition system can build a recognition model based on this analysis for both WW and commands. In one implementation, the recognition model may include a sequence decoding structure such as an Hidden Markov Model (HMM), which is based on a given HMM model. Evaluate the probability of the observed sequence under the following circumstances. .

[0221] In operation 2407, the data-free speech recognition system can train a recognition model to recognize WW and commands (e.g., target phrases). In one implementation, as... Figure 3 The “online training” process described herein can be based on a word segmenter to train the recognition model, which converts the text of a target phrase from a user-defined command set into its phoneme string equivalent. The recognition model can be trained to achieve a highly accurate and robust decoding method that adapts to both the acoustic model and alternative pronunciations of the target phrase.

[0222] Data-free speech recognition systems can deploy recognition models to... Figure 4During the inference phase discussed herein, WW and commands are detected in the speech. After the recognition model for WW detects WW, the recognition model for commands can evaluate subsequent commands.

[0223] Figure 25 The present disclosure describes a recognition model for detecting “Okay Infineon” for WW 2510 and two subsequent recognition models for evaluating subsequent commands for CMD 1 (2520) and CMD 2 (2530), according to one aspect of the disclosure. Line 2540 marks an approximate transition from one phoneme to the next in the audio of “eon”. The recognition model for WW declares WW at frame 2550, and the recognition model for commands evaluates subsequent commands in subsequent frames. Figure 25 The diagram shows that the moment when phoneme / N / (2560) declares a WW at frame 2550 has not yet ended (e.g., the transition 2540 from phoneme / N / (2560) occurs after frame 2550). This could lead to potential decoding errors in the evaluation of subsequent commands because the recognition model used for the command is not constructed to model the end of the WW. For example, the first state of the command model is / SIL / 2570, but the currently processed frame has not yet reached the silence region, which could potentially cause the command model to miss recognizing the command.

[0224] Another potential problem is the random similarity between the last non-silent WW state and the first non-silent (S2) state in the command model. For example, if CMD 1 (2520) begins with the word "Next", then (2580) will be modeled by the phoneme / N / and happens to match well with / N / (2560) from the end of "Infineon". CMD 2 (2530) may not match / N / (2560). Therefore, CMD 1 (2520) may have a higher initial likelihood than CMD 2 (2530) and is completely unrelated to the command being spoken. This leads to a bias towards CMD 1 (2520) and a decrease in the performance of the command model. In one respect, the command model can compensate for the WW to command transition.

[0225] Figure 26 A recognition model for WW 2510 for detecting “Okay Infineon” according to one aspect of this disclosure is described, along with two subsequent recognition models for CMD 1 (2520) and CMD 2 (2530) that utilize state compensation techniques to evaluate subsequent commands. The command model may include the last non-silent state of WW as an alternative pronunciation of the preceding / SIL / state for each command. Figure 26The diagram shows / N / (2560) added to the / SIL / state 2690 of CMD 1 (2520) and CMD 2 (2530). The resulting state perfectly models the end of WW, thus matching both the trailing / N / (2560) and any silence between WW and CMD. The modified state also eliminates any random bias from commands that precisely match their leading phonemes to the trailing phonemes of WW.

[0226] Command recognition models cascade each word of the command into a single model, with each word separated by silence states. However, the amount, or even the presence, of silence gaps between words is quite variable, depending on the word and the speaker. In one respect, to better handle command-to-command transitions, command models can include the last non-silent state from the preceding word and the first non-silent state from the following word in the silence state modeled for the gaps.

[0227] Figure 27 A modification of a WW recognition model according to one aspect of this disclosure is depicted to address variable silence gaps between words. An initial CMD model 2710 shows a silence state 2720 between the last non-silent phoneme S4 ​​(2730) of word 1 (2740) and the first non-silent phoneme (2750) of word 2 (2760). A final CMD model 2770 shows a modified silence state 2780 that now includes the last non-silent phoneme S4 ​​(2730) of word 1 (2740) and the first non-silent phoneme (2750) of word 2 (2760). Including a non-silent phoneme is equivalent to an alternative pronunciation of the included state. The modified state can handle both cases where a silence gap exists or does not exist.

[0228] In one implementation, a composite command consisting of multiple stage subcommands can have a modified state for subcommand transitions, similar to a modified state for a simple command. However, a first stage subcommand may only include the previous mute state (decoding only the mute gap) and not the trailing mute state. Intermediate subcommands may not include either the previous mute state or the trailing mute state. A final stage subcommand may only include the trailing mute state and not the previous mute state.

[0229] As in Figure 24As discussed in Operation 2407, the online training process can train a recognition model to recognize WW and commands (e.g., target phrases) based on a tokenizer that converts the text of a target phrase from a user-defined set of commands into its phoneme string equivalent or other basic units called lexical units. The tokenizer analyzes the input text to identify its constituent morphemes and produces sequences of lexical units based on predefined rules, statistics, or learned patterns. For example, the tokenizer converts morphemes (the smallest units that represent language in written form) into phonemes (the smallest audio building blocks of language).

[0230] Figure 28 This disclosure describes a phoneme segmenter that converts the morphemes in the input text "afternoon" into a phoneme string, according to one aspect of the disclosure. Several methods are available for segmenters, including rule-based, statistical models, and machine learning-based methods. Rule-based methods rely on a predefined set of language rules and exceptions. Such methods require substantial domain knowledge from experts in the field and are limited in performance due to irregularities and exceptions common in many languages, including English. Statistical models use large annotated datasets containing transcribed text-to-phoneme pairs to learn the pronunciation of new or unseen words. Machine learning-based methods use deep learning models such as Long Short-Term Memory (LSTM) networks to learn the morpheme-to-phoneme (G2P) task. The recursive nature of LSTM models incorporates the context and order of morphemes into the learning process to achieve high accuracy.

[0231] Tokenizers based on machine learning or deep learning methods can achieve high transcription accuracy when performing G2P tasks. However, their performance is heavily dependent on both the quality and volume of the training data based on real speech. Training databases may be limited in availability, expensive to obtain, or may not be available in sufficient quantities (especially for different languages) to properly train the model. It is also desirable to train the tokenizer to support different accents, dialects, and languages.

[0232] A statistics-based word segmenter solution, comprising a training phase and a decoder phase, is described. During the training phase, the segmenter processes words from a reference speech dictionary containing word-lexical transcriptions. The segmenter can then break down the words in the dictionary into subwords and compile statistics to generate a custom dictionary containing the subwords and their estimated likelihoods.

[0233] During the decoding phase, the word segmenter analyzes the text input, performs sub-word search, and iteratively solves for sub-words and their likelihoods from a custom dictionary to maximize the probability of the word stream. In one aspect, during the decoding phase, the word segmenter can analyze the text input of the target phrase from a user-defined command set using a dictionary of sub-words and their estimated likelihoods to list the most probable phoneme string equivalents of the text input and their likelihoods. The data-free speech recognition system can use the first N phoneme string equivalents to train the target phrase recognition model online, allowing the sequential decoding of the target phrase's speech to adapt to the offline-trained acoustic model.

[0234] Figure 29 A block diagram depicts a training phase 2910 and a decoding phase 2950 of a word segmenter 2960 according to one aspect of this disclosure. The training phase 2910 uses a reference speech dictionary 2920 containing word-lexical transcriptions as input. For a sub-word training task 2930, the word segmenter 2960 can decompose each training word in the speech dictionary 2920 into multiple unique sub-words by splitting the word at different points. The sub-word training task 2930 can assign phonemes to sub-word splits based on the mapping between lexicals and phonemes and the positional information of the sub-words within the corresponding training words. The sub-word training task 2930 can accumulate the results of phoneme sub-word assignments for all training words in the speech dictionary 2920 to compile a sub-word likelihood dictionary 2940 containing the likelihood of each unique combination of phonemes, sub-words, and the positional information of the sub-words within the word.

[0235] In the decoding phase 2950, ​​the word segmenter 2960 can split the target text input (e.g., text input of a target phrase from a user-defined command set) into distinct and unique combinations of subwords. The word segmenter 2960 can perform a search in the subword likelihood dictionary 2940 for each subword in the combination to find the phoneme corresponding to the subword and the position of the subword within the target text input. Each combination of phoneme, subword, and subword position has a corresponding probability. The word segmenter 2960 can multiply the corresponding probabilities of all subwords in each unique combination of subword splitting to obtain the probability of the combination. The word segmenter 2960 can solve for the most probable combination among all combinations to maximize the probability of the phoneme string equivalent of the target text.

[0236] Figure 30A block diagram depicts a training phase 2910 of a word segmenter using a reference speech dictionary 2920 according to one aspect of this disclosure. The training phase 2910 can sort the words of the speech dictionary 2920 one by one to feed each word to a sub-word splitting block 3030 and a phoneme mapping block 3070. The sub-word splitting block 3030 can analyze the text of the training words from the speech dictionary 2920 and can create multiple unique “sub-words” by splitting the text at different points. The splitting of sub-words can be associated with labels and positions to indicate how the training words were split. The phoneme mapping block 3070 can map the input phoneme string of the current training word to different phonemes. The phoneme mapping block 3070 can use the labels and positions of each split to assign phonemes to sub-words. A counting block 3095 can count the sub-words of all training words in the speech dictionary 2920, their labels / positions, and associated phonemes to create a sub-word-based likelihood-based dictionary 2940.

[0237] In one embodiment of the subword splitting block 3030, the splitting block 3040 splits the current dictionary word into different subwords. For example, the splitting can be performed one character at a time in two directions, starting from the beginning of the word, and / or from the end of the word, and / or from the middle or other starting points. In one embodiment, the subword can have a minimum length of 2 characters.

[0238] When splitting, the splitting block 3040 may consider certain exceptions, conditions, rules for common beginnings / ends, etc. 3050. For example, in English, there are certain pairs of morphemes that constitute a single phoneme (e.g., ['ph', 'sh', 'ch', 'th', 'ck', 'ng', 'll', 'ss', 'tt', 'aw']). If the splitting block 3040 observes these pairs, it will not split them, but will treat each pair as a single morpheme unit. The splitting block 3040 may employ other exceptions or rules to improve the splitting, such as common endings ['ing', 'ion', etc.].

[0239] The tag / position block 3060 can categorize subwords into tags based on their different positions within the original word. For example, the tag / position block 3060 can assign tags <beginning>, <middle>, and <end> to categorize subwords located at the beginning, middle, or end of a word. The tag / position block 3060 can also assign the tag <complete> to subwords that constitute the complete original word. Additionally, the tag / position block 3060 can assign character start position numbers to track the original position of a subword within the word.

[0240] Figure 31The text describes the word "Example" at different points according to one aspect of this disclosure, along with the associated tags and positions of the subwords. As shown, the subwords have a minimum length of 2 morphemes.

[0241] Return to reference Figure 30 The speech dictionary 2920 contains character-phoneme pairs of words in the target language. In one embodiment of the phoneme mapping block 3070, the phoneme-to-character mapping block 3080 can map characters to their corresponding phonemes in each given word in the dictionary being processed. The mapping can also take into account special exceptions, conditions, character pairs, common beginnings / endings of characters, etc. 3050. The mapping can be associated with position to indicate the position number of a character within a word.

[0242] Figure 32 A phoneme-to-graphite mapping of the word “Example” according to one aspect of this disclosure is depicted, along with the positions associated with the graphemes. This mapping illustrates both one-to-many and many-to-one mappings between phonemes and graphemes.

[0243] Return to reference Figure 30 The phoneme-to-subword splitting assignment block 3090 can use phoneme-to-graph mapping to assign phonemes to subwords based on the graphemes contained in the subword split. The phoneme-to-subword assignment can take into account the tags and / or positions associated with the subwords to create {subword, phoneme, tag} triples.

[0244] Figure 33 The use of one aspect of this disclosure is described. Figure 32 Phoneme-to-graphite mapping, targeting Figure 31 The subword split of “Example” is the phoneme-to-subword assignment of the word “Example”. This assignment shows the tag associated with the subword in the triple {subword, phoneme, tag}.

[0245] Return to reference Figure 30 The counting block 3095 can accumulate the results of the phoneme-to-subword assignment generated for each word, and can calculate the likelihood (probability) of each unique {subword, phoneme, tag} triple to create a subword likelihood dictionary 2940.

[0246] Figure 34The counting triples {subword, tag, phoneme} for all words in a speech dictionary 2920 according to one aspect of this disclosure are depicted, along with the likelihood of each triple. Each subword lists all possible tag categories for that subword. For each {subword, tag} pair, the sum of the probabilities of all possible phonemes is 1.0. For example, for the subword “Ex” associated with the tag <start>, there are two possible phoneme assignments {IH / G / Z} and {EH / K / S}, with corresponding probabilities of 0.620 and 0.380, respectively, summing to 1.

[0247] The sub-word likelihood dictionary 2940 contains the likelihood of each unique {sub-word, phoneme, label} triple after processing through the phonetic dictionary 2920 of the complete input. The final sub-word likelihood dictionary 2940 may include a complete counting table as shown in 34, or it may be reduced to contain only the first N possible pronunciations to reduce table storage requirements. In one implementation, if only the most probable final word pronunciation is needed, the sub-word likelihood dictionary 2940 may be reduced to contain only the first 1 possible pronunciation for each {sub-word, label} pair.

[0248] Figure 35 A word segmenter using a trained sub-word likelihood dictionary 2940 (e.g., according to one aspect of this disclosure) is described. Figure 29 The block diagram of the decoding stage 2950 of the word segmenter (2960) is shown below. The word segmenter can analyze the input word, perform sub-word search, and iteratively solve using a trained sub-word likelihood dictionary 2940 to maximize the probability of the word stream. In one aspect, the word segmenter can use the trained sub-word likelihood dictionary 2940 to analyze the text input of WW and commands in a user-defined command set to list the most likely phoneme string equivalents of WW / command and their likelihoods.

[0249] In one implementation, splitting block 3540 can split the graphemes of an unseen input word into different combinations of subwords. Splitting block 3540 can be exhaustive, thus covering every combination of different lengths and numbers of subwords. In one implementation, splitting block 3540 used in the decoding phase can be the same as splitting block 3040 used during the training phase.

[0250] Figure 36 This describes the word splitting of the word "Infineon" in WW "Okay Infineon" according to one aspect of this disclosure.

[0251] Return to reference Figure 35When splitting, the splitting block 3540 may consider certain exceptions, conditions, rules for common beginnings / ends, etc. 3550. In one implementation, the splitting block 3540 may consider restrictions imposed by special combinations of morphemes that should not be split and common beginnings and endings. For example, the ending "eon" in the word "Infineon" should not be split as... Figure 36 The terms seen are split. The splitting block 3540 can compile all unique subword splitting combinations 3510 into a dictionary for use by other blocks. In one implementation, the exceptions, conditions, rules for common beginnings / common endings, etc., 3550 used in the decoding phase can be the same as the exceptions, conditions, rules for common beginnings / common endings, etc., 3050 used during the training phase.

[0252] The search and solver block 3520 can traverse the trained sub-word likelihood dictionary 2940 to search for sub-words contained in each unique sub-word split combination of the input word to find the phonemes corresponding to the sub-words. In one implementation, the phonemes can be based on the positions associated with the sub-words within the input word. If the search and solver block 3520 finds phonemes in the sub-word likelihood dictionary 2940 that correspond to all sub-words in the sub-word split combination, then the combination is solved, and the search and solver block 3520 can combine the corresponding phonemes of each sub-word into the corresponding solution.

[0253] The phoneme corresponding to each sub-word in the sub-word splitting combination has a likelihood (probability) found from the sub-word likelihood dictionary 2940. The search and solver block 3540 can multiply the probabilities of the phonemes of all sub-words in each unique combination corresponding to the sub-word splitting to obtain the combined speech probability. The search and solver block 3540 can compile the speech probabilities of all unique sub-word splitting combinations of the input word into the speech solution and likelihood list 3530. In one embodiment, the speech solution and likelihood list 3530 can list the sub-word splitting combinations with the most probable speech probabilities among all combinations to maximize the probability of the phoneme string equivalent of the input word for use in online training of the recognition model for a data-free speech recognition system. In one embodiment, the speech solution and likelihood list 3530 can list the sub-word splitting combinations with N most probable speech probabilities among all combinations.

[0254] Figure 37 One aspect of this disclosure is described. Figure 36 This list contains sub-word splitting combinations of the word "Infineon," the phonemes of the combinations, the corresponding probabilities of the phonemes, and the phonetic probabilities of the sub-word splitting combinations. The list shows that the most likely phoneme string is "IH / N / F / IH / N / IY / AH / N," obtained from the sub-word splitting combinations of "In / fin / eon."

[0255] Advantageously, the method described for training and decoding a statistically based word segmenter produces high transcription accuracy. This method can support lexical units in addition to phonemes. The training phase with different speech libraries can support different accents, dialects, and languages ​​without requiring any additional training data. The training phase can also support small speech dictionaries with limited speech dictionary support. The decoding phase can identify the most probable pronunciation or N most probable pronunciations, making this method attractive for data-free speech recognition systems.

[0256] In one aspect, following the training and decoding phases of the word segmenter, a data-free speech recognition system can use the word segmenter to generate phoneme strings from user-defined WW or commands used in online training of the recognition model during inference. Online training of the recognition model can compile statistics using phonemes from the word segmenter and SoftMax vectors from the acoustic model. Sequence decoding of WW or commands can use statistics to achieve highly accurate, robust phoneme string equivalents of WW or commands, adapted to the acoustic model and alternative pronunciations of WW or commands.

[0257] In one aspect, text-to-speech (TTS) engines can generate synthetic speech to modify / enhance the speech recognition model (e.g., HMM model) of a data-free speech recognition system. Figure 3 As discussed herein, to improve the performance of recognition model 350, model building block 344 can utilize TTS 346 to generate synthesized speech containing the target WW and command during online training of recognition model 350. Analysis module 348 can analyze the synthesized speech during the generation of recognition model 350 to assist model building block 344. In one embodiment, during online training of recognition model 350 using synthesized speech from TTS 346 and SoftMax vectors from acoustic model 218, model building block 344 can compile statistics. In one embodiment, statistics may include... Figure 21 The top N statistics of the phonemes of the target phrase (e.g., alternative pronunciations) and the expected length of the phonemes of the target phrase used for the self-transformation probability. Figure 15 The decoding compensation analysis block 740 can use compiled statistics to improve the sequence decoding of the target phrase.

[0258] TTS engines based on machine learning or deep learning can produce synthesized speech of such high quality that it is virtually indistinguishable from real speech to untrained listeners. TTS engines are typically capable of synthesizing hundreds of different speakers, either cloning a real target speaker or generating a purely fictional one. They can also handle different emotions, accents, and prosody. While these features increase the variability of the output speech, such variability may still not approach that of real speech. To further increase the statistical variation in synthesized speech, TTS engines can apply enhancement techniques such as time-scale modification, channel normalization, and level scaling. ASP systems can use TTS engines to tune or train speech recognition models that have already been trained on real speech. This approach can be useful when limited real speech data is available for training purposes, such as on uncommon languages ​​or for new words in evolving languages. However, when training a speech recognition model solely on synthesized speech generated from TTS, the training data may be insufficient because the synthesized speech may not accurately represent the expected statistics, spectral content, variability, etc., of real speech.

[0259] This paper describes a method for using a TTS engine to synthesize speech that is otherwise unavailable to train or tune a data-free speech recognition system so that it can recognize the target phrase using only the text or morpheme representation of the target phrase from a user-defined set of commands. The data-free speech recognition system does not rely on real speech matching the target phrase and can still achieve good performance. In one implementation, the method iteratively tunes the settings and enhancement blocks of the TTS engine to match the target characteristics of real speech and can utilize compensation blocks to further compensate / adjust the synthesized speech to real speech.

[0260] In one aspect, during online training of the recognition model of a data-free speech recognition system, the system can use an annotated database to tune the enhancement blocks and TTS settings of the TTS engine to derive compensation blocks that minimize the difference between synthesized and real speech. After tuning the TTS settings and enhancement blocks, the TTS engine can synthesize target speech based on user-defined WW and commands to aid in the online training of the target phrase recognition model.

[0261] Figure 38Operational details for tuning a TTS engine for a data-free speech recognition system, according to one aspect of this disclosure, to match the characteristics of synthesized speech generated by the TTS engine with the characteristics of real speech, are illustrated. The tuning phase tunes components of the TTS engine 3810 (including the TTS settings of selection block 3820 and parameters of enhancement block 3830) to generate compensation information 3840 to improve the similarity between the synthesized speech and real speech. The output of the tuning phase is the complete set of the TTS settings of selection block 3820, the enhancement parameters of enhancement block 3830, and the compensation information 3840.

[0262] The tuning phase uses one or more annotated databases 3850, believed to contain target or desired characteristics of speech, to tune the components. Because data-free speech recognition systems lack speech data specific to user-defined WWs and commands, the annotated database 3850 does not contain speech data for the target phrase. Instead, the annotated database 3850 may contain the entire set of speakers representing a particular language, or a set of speakers from a specific region with the desired target accent. For example, the annotated database 3850 may contain word-to-word transcriptions (e.g., text-to-speech) for all speakers.

[0263] The TTS engine 3810 takes the text of each training segment from the annotated speech database 3850 as its input to produce equivalent synthesized speech based on settings and speakers from the selection block 3820. The TTS engine 3810 may have the ability to synthesize multiple speakers and / or model different prosody (rhythm, melody, stress, duration, level, etc.).

[0264] Enhancement block 3830 can process synthesized speech using enhancement features selected by selection block 3820 to generate enhanced synthesized speech. In one implementation, enhancement features may include time scale modification (speeding up, slowing down), duct length compensation (or other spectral distortion), gain scaling, etc.

[0265] Acoustic model 3860 (e.g.) Figure 2 The acoustic model 218 can process both enhanced synthesized speech and corresponding real speech from the annotated speech database 3850. Example outputs of the acoustic model 3860 are softmax vectors of phonemes on a frame-by-frame basis.

[0266] Analysis block 3870 compares the output of synthesized speech from acoustic model 3860 with real speech. Analysis block 3870 can provide the results of this analysis to selection block 3820 to adjust TTS settings and enhancement features. The tuning phase can iterate the TTS settings and enhancement features until the synthesized speech and real speech, as analyzed by analysis block 3870, converge. After convergence, compensation block 3840 can derive compensation or mapping information for use by the data-free speech recognition system to further minimize the difference between synthesized speech and real speech usage during the online training phase of the target phrase recognition model.

[0267] Figure 39 A block diagram depicts the training or tuning of a speech recognition system that uses a tuned TTS engine to synthesize speech that would otherwise be unavailable for a data-less speech recognition system, according to one aspect of this disclosure. In one embodiment, training the data-less speech recognition system includes, as... Figure 3 The online training of the recognition model shown, or as... Figure 7 or Figure 15 The generation of the decoding compensation model shown is used to improve the sequence decoding of WW or commands.

[0268] In one implementation, the tuned TTS engine 3810 can use the TTS settings and all speakers determined during the tuning phase of the word segmenter from selection block 2920 to synthesize the target speech of a WW or command (e.g., target text). For example, the TTS engine 3810 could be Figure 3 The TTS 346 generates synthesized speech containing the target WW and commands used for online training of the recognition model 350.

[0269] Enhancement block 3830 can again utilize the enhancement features determined during the tuning phase from selection block 3820 to process the synthesized speech to generate enhanced synthesized speech. In one implementation, enhancement features may include time scale modification (speeding up, decelerating), duct length compensation (or other spectral distortion), gain scaling, etc. Analysis block 3910 can analyze the enhanced synthesized speech to tune or train the data-free speech recognition system 3920. For example, analysis block 3910 may analyze synthesized speech generated by TTS 346 to assist in, for example... Figure 3The generation of the analysis module 348 of the recognition model 350 shown herein. Analysis block 3910 can use compensation or mapping information from compensation block 3840, which is determined again during the tuning phase and is used by the data-free speech recognition system 3920, to reduce the difference between synthesized speech usage and real speech usage during online training of the recognition model. In one embodiment, analysis block 3910 can generate phoneme statistics for the synthesized speech of WW or commands. The data-free speech recognition system 3940 can use the phoneme statistics in combination with the phoneme string generated by the word segmenter 2960 from the target text to construct a recognition model for the target text. Statistics can also be used during sequence decoding of WW or commands during inference to improve the discriminative power of sequence decoding.

[0270] Figure 40 One aspect of this disclosure is described. Figure 39 The diagram shows the block diagram of analysis block 3910, which is used to analyze synthesized speech to compile statistics for sequence decoding to assist the target speech. TTS engine 3810 can synthesize synthesized speech of the target phrase based on text input, and enhancement block 3830 can utilize enhancement features to process the synthesized speech to generate, for example... Figure 39 The enhanced synthesized speech shown.

[0271] Aligner block 4010 can determine the phonemes and their temporal boundaries of the enhanced synthesized speech. In one implementation, alignment block 4010 can use a Montreal forced alignment (MFA) to determine the temporal boundaries of the phonemes. Aligner block 4010 can output the phoneme temporal boundaries to statistics collection block 4020.

[0272] Figure 41A depicts a time diagram of the phonemes of WW “Okay Infineon” according to one aspect of the present disclosure. Figure 41B depicts a spectrogram of the phonemes of WW “Okay Infineon” according to one aspect of the present disclosure. Figure 41C depicts the time boundaries determined by the alignment block 4010 for the phonemes of WW “Okay Infineon” according to one aspect of the present disclosure.

[0273] Return to reference Figure 40Feature extraction block 214 can perform spectral and temporal analysis on the enhanced synthesized speech to generate observation vectors consumed by acoustic model 218. Statistics collection block 4020 can capture the output frame of the acoustic model that is closest to the center of the phoneme boundary from aligner block 4010. In one implementation, statistics collection block 4020 can capture the output frame of the acoustic model based on majority decision, report all, etc. Statistics collection block 4020 can record the most probable phoneme of the output frame. Statistics collection block 4020 can compile statistics for each phoneme on the entire set of user-defined WWs and commands to generate top-1 statistics.

[0274] Based on one aspect of this disclosure, refer to Figure 21 To view the top-1 statistics for the phonemes of WW “Okay Infineon”, the table indicates which phonemes were correctly predicted with what accuracy. For example, the top-1 column shows the percentage of time the listed phonemes were reported as the most likely based on the Softmax output of the acoustic model, which considers the theoretical phoneme center in the first column. The noise column specifies the percentage of time the top phoneme is in the noise class. This column is ideally zero, where non-zero values ​​may be due to misalignment of phoneme boundaries. Therefore, the table scales the top-1 values ​​by the noise percentage to obtain a better estimate of the true top-1 values. The last column shows the percentage of the total noise scaling for the listed phonemes. For example, for the first “N”, the statistics collection block 4020 captured the three most likely phonemes {“N”, “NG”, “M”} of the noise scaling value in 78% of cases, thus giving a relatively high confidence level for the top phoneme reported by acoustic model 218 when dealing with the first phoneme “N”. Conversely, for the penultimate phoneme “EH”, the statistical data collection block 4020 captures the eleven most likely phonemes with noise scaling values ​​distributed in 86% of cases, thus giving a relatively low confidence level to the top phoneme reported by the acoustic model 218 when processing the last phoneme “EH”.

[0275] In one implementation, the HMM for the target phrase can use top-1 statistics to achieve the desired result, as in the HMM for the target phrase. Figure 13 Each state definition shown in states 1320 and 1330 allows for multiple phonemes to support alternative pronunciations. In one implementation, as discussed, Figure 15 The decoding compensation analysis block 740 can use top-1 statistics to weigh the degree of the most likely state matching the expected phoneme of the target phrase to improve the sequence decoding of the target phrase.

[0276] In one respect, offline analysis can utilize annotated databases used for training acoustic models (e.g., Figure 3The average length of each phoneme (in time) is calculated using a database of phoneme annotations (320).

[0277] In one implementation, if phonemes The start time of the i-th occurrence is And the end time is Then phonemes average length It is given by the following formula:

[0278] (Equation 26)

[0279] in, The phonemes used during averaging The number of times it appears.

[0280] Figure 15 The self-transition analysis block 1530 of the decoding compensation analysis block 740 can calculate the self-transition probability of the HMM state for the target phrase based on the expected length of the phoneme in time, according to the reproduced Equation 19 below:

[0281] (Equation 19)

[0282] in, It is the time of each frame (in seconds), and It is HMM. The average length of the phonemes in the state (in seconds). As discussed, the decoding compensation analysis block 740 can use the self-transition probabilities of the HMM states to improve the sequence decoding of the target phrase.

[0283] In one implementation, offline analysis can use synthesized speech as an alternative to calculating the average length of each phoneme in an annotated database used to train the acoustic model. (For example, for...) Figure 40 As described by the compiled top-1 statistics, offline analysis can incorporate phoneme boundaries using synthesized speech. The statistics collection block 4020 can calculate the length of each phoneme to find an average. This method has the advantage of finding phoneme lengths specific to those found in a given target phrase, rather than a global average. Furthermore, when using the decoding compensation analysis block 740, any differences or deviations in the duration of each phoneme between the synthesized speech and the real speech can be compensated.

[0284] Figure 42 A flowchart of a method 4200 for operating a data-free speech recognition system according to one aspect of this disclosure is shown. In one aspect, it can be... Figures 1 to 7 , Figure 12 , Figure 22 , Figures 28 to 31 , Figure 36 , Figures 39 to 40 or Figure 43 The system or device uses hardware, software, or a combination of hardware and software to perform method 1100.

[0285] In operation 4201, the system decomposes a word into multiple combinations from a database by splitting the word at multiple different points for each of the multiple combinations of unique subwords. The word comprises one or more written units and one or more corresponding acoustic units.

[0286] In operation 4203, the system maps the acoustic units that constitute the word to the written units that constitute the word to generate an acoustic unit-to-written unit mapping.

[0287] In operation 4205, the system assigns a subset of acoustic units to each of the unique subwords based on the acoustic unit to writing unit mapping to generate the acoustic unit to subword assignment for that word.

[0288] In operation 4207, the system accumulates acoustic units to sub-word assignments for multiple words from the database to create a sub-word likelihood dictionary.

[0289] Figure 43 A data processing system 4300 implementing a dataless speech recognition system according to one aspect of this disclosure is shown. For example, the data processing system 4300 can implement any of the operations described herein, including offline training of the acoustic model, online training of the decoding model, tuning of the word segmenter, and other operations for... Figures 2 to 7 , Figures 12 to 13 , Figure 15 , Figure 22 , Figures 24 to 31 , Figure 36 , Figures 39 to 40 and Figure 42 The inference operation of the data-free speech recognition system is shown in the figure. In one embodiment, the data processing system 4300 can operate on a smartphone, desktop computer, laptop computer, home auxiliary device, other voice control device, server, etc.

[0290] The microphone 4301 of the data processing system 4300 can capture audio signals to store the input signal, which includes noise and target speech, into the buffer 4303. In one embodiment, the input of the processing system (not shown) can receive audio signals captured by one or more external microphones to be stored in the buffer 4303.

[0291] Processor 4320 can read captured audio signals from the buffer for processing. Processor 4320 can retrieve computer-readable instructions from memory 4330 to execute instructions for performing the operations described above. Processor 4320 may include one or more processing cores. Memory 4330 may include one or more ROMs (Read-Only Memory), volatile random access memory (RAM), and / or other types of memory. Communication between buffer 4310, processor 4320, and memory 4330 can be performed via communication bus 4380.

[0292] In one aspect, during offline training of a neural network-based acoustic model, processor 4320 can perform feature extraction of input speech from a database with phoneme annotations to generate observation vectors, iterate the acoustic model through the observation vectors to learn to distinguish input speech based on phonemes, and analyze the phoneme vectors from the acoustic model to generate a similarity matrix.

[0293] In one aspect, during online training of the decoding model, the processor 4320 can implement a word segmenter to convert the text of a user-defined WW / command into a phoneme sequence, and can train the decoding model based on the phoneme sequence in the WW / command and a similarity matrix from offline training.

[0294] In one aspect, during the inference phase of the dataless speech recognition system, the processor 4320 can implement the SOD algorithm to detect active speech, perform feature extraction on the active speech to generate observation vectors, invoke an acoustic model based on the observation vectors to generate Softmax vectors, and apply statistical modeling to the Softmax vectors according to the decoding model to determine whether a user-defined WW or command has been spoken.

[0295] In one respect, the processor 4320 can use the training and decoding phases to tune the TTS engine to match the characteristics of real speech. For example, during the tuning of the TTS engine, the processor 4320 can use an annotated speech database to tune the TTS settings, enhancement parameters, and compensation blocks of the TTS engine to minimize the difference between the synthesized speech generated by the TTS engine and real speech.

[0296] In one aspect, processor 4320 can train the word segmenter to generate a custom dictionary containing subwords and their estimated likelihoods by applying words from a reference speech dictionary to the segmenter. During the decoding phase of the word segmenter, processor 4320 can invoke the word segmenter to analyze the text input of a user-defined WW / command using the custom dictionary of subwords and their estimated likelihoods to list the most likely phoneme string equivalents of the text input and their likelihoods, which can be used for online training of the decoding model.

[0297] The various implementations of the dataless speech recognition system described herein may include a variety of operations. These operations may be performed and / or controlled by hardware components, digital hardware and / or firmware / programmable registers (e.g., implemented in a computer-readable medium) and / or combinations thereof. The methods and illustrative examples described herein are not inherently related to any particular device or other apparatus. For example, in the inference phase of a dataless speech recognition system, processor 4320 may invoke SOD block 4340 to detect active speech, invoke feature extraction block 4350 to perform feature extraction of the active speech to generate an observation vector, invoke phoneme unit matching block 4360 to generate a Softmax vector based on the observation vector, and invoke WW / command sequence decoding block 4370 to apply statistical modeling to the Softmax vector to determine whether a user-defined WW or command has been spoken. The necessary structures for various such systems will appear as set forth in the description above.

[0298] Computer-readable media used to implement the operations of various aspects of this disclosure may be non-transitory computer-readable storage media, which may include, but are not limited to, electromagnetic storage media, magneto-optical storage media, ROM, RAM, erasable programmable memory (e.g., EPROM and EEPROM), flash memory, or other non-transitory types of media now known or hereafter developed suitable for storing configuration information.

[0299] The above description is intended to be illustrative and not restrictive. Although this disclosure has been described with reference to specific illustrative examples, it will be appreciated that this disclosure is not limited to the described examples. The scope of this disclosure should be determined by referring to the appended claims and the full scope of their equivalents.

[0300] Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein are intended to include the plural forms as well. It will also be understood that the terms “comprises,” “comprising,” “may include,” and / or “including” as used herein specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Therefore, the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting.

[0301] It should also be noted that in some alternative implementations, the functions / actions mentioned may not occur in the order shown in the figures. For example, depending on the functions / actions involved, the two figures shown consecutively may actually be executed substantially simultaneously or sometimes in reverse order.

[0302] Although the method operations are described in a specific order, it should be understood that other operations may be performed between the described operations, the described operations may be adjusted so that they occur at slightly different times, or the described operations may be distributed across a system that allows processing operations to occur at various intervals associated with the processing. For example, some operations may be performed at least partially in the reverse order, simultaneously with and / or in parallel with other operations.

[0303] Various units, circuits, or other components may be described or claimed to be "configured to" or "configurable to" perform one or more tasks. In such a context, the phrase "configured to" or "configurable to" is used to imply a structure by indicating that the unit / circuit / component includes a structure (e.g., a circuit system) that performs one or more tasks during operation. Thus, even when a specified unit / circuit / component is currently inoperable (e.g., not switched on), it can be said that the unit / circuit / component is configured to perform a task, or is configurable to perform a task. Units / circuit / components used with the language "configured to" or "configurable to" include hardware such as circuits, memory storing program instructions that can be executed to perform operations, etc. The description of a unit / circuit / component being "configured to" or "configurable to" perform one or more tasks is explicitly intended not to invoke paragraph 6 of 35 USC 112 for that unit / circuit / component.

[0304] Additionally, "configured to" or "configurable to" can include a general-purpose structure (e.g., a general-purpose circuit system) manipulated by firmware (e.g., an FPGA) to operate in a manner capable of performing the tasks (one or more) discussed. "Configured to" can also include adapting manufacturing processes (e.g., a semiconductor manufacturing facility) to manufacture devices (e.g., integrated circuits) suitable for implementing or performing one or more tasks. "Configurable to" is explicitly intended not to apply to blank media, unprogrammed processors or unprogrammed programmable logic devices, unprogrammed programmable gate arrays, or other unprogrammed devices, unless accompanied by a programmed medium that endows the unprogrammed device with the ability to be configured to perform the disclosed functions (one or more).

[0305] For illustrative purposes, the foregoing description has been described with reference to specific embodiments. However, the illustrative discussion above is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to best illustrate the principles of the embodiments and their practical application, thereby enabling those skilled in the art to best utilize the embodiments and various modifications that may be suitable for the particular intended use. Therefore, these embodiments should be considered illustrative rather than restrictive, and the invention is not limited to the details given herein, but can be modified within the scope and equivalents of the appended claims.

Claims

1. A method for training a word segmenter to generate acoustic unit strings from input text, the method comprising: The word is decomposed into the multiple combinations by splitting the word from the database at multiple different points for each of the multiple combinations of unique subwords, wherein the word includes one or more written units and one or more corresponding acoustic units; The acoustic units constituting the word are mapped to the writing units constituting the word to generate an acoustic unit-to-writing unit mapping; Based on the acoustic unit-to-writing unit mapping, a subset of the acoustic units is assigned to each of the unique sub-words to generate the acoustic unit-to-sub-word assignment of the word; and The acoustic units are accumulated to sub-word assignments for multiple words from the database to create a sub-word likelihood dictionary.

2. The method according to claim 1, wherein, The acoustic unit to writing unit mapping includes at least one of the following: Mapping from acoustic units to writing units; Mapping multiple acoustic units to writing units; or Mapping of multiple writing units to acoustic units.

3. The method according to claim 1, wherein, The acoustic unit to sub-word assignment for the word is based on the position of the point used to split the word into a combination of unique sub-words.

4. The method according to claim 1, wherein, Each of the sub-words comprises a plurality of acoustic units, and wherein the sub-word likelihood dictionary compiles the probability of a unique combination of acoustic units for each of the sub-words.

5. The method according to claim 4, wherein, The probability of a unique combination of acoustic units for each subword is a function of the subword's position within the word.

6. The method according to claim 4, further comprising: The word segmenter is invoked to synthesize the input text of the target phrase into an acoustic unit string representing the target phrase based on the sub-word likelihood dictionary.

7. The method according to claim 6, wherein, Invoking the word segmenter includes: The input text of the target phrase is decomposed into multiple unique sub-words. The sub-word likelihood dictionary is traversed to search for each sub-word in the unique sub-word segment to determine the probability of the unique sub-word segment, wherein the unique sub-word segment has a phonetic solution corresponding to a combination of acoustic units of each sub-word in the unique sub-word segment; and Analyze the probability of the unique word segment in the plurality of unique word segments to generate one or more most probable speech solutions for the target phrase.

8. The method according to claim 7, wherein, The probability of the unique sub-word split includes the product of the probabilities of each sub-word in the unique sub-word split obtained from the sub-word likelihood dictionary.

9. The method according to claim 7, wherein, The acoustic unit string representing the target phrase comprises a combination of the acoustic units of each word in a unique sub-word segmentation corresponding to the most probable speech solution among the one or more most probable speech solutions of the target phrase.

10. The method of claim 7, further comprising: Speech is processed based on one or more of the most likely speech solutions of the target phrase to detect the presence of the target phrase in the speech.

11. The method according to claim 1, further comprising: The text-to-speech (TTS) model is tuned to generate synthesized speech from input text based on real speech data and an annotated database of corresponding words, wherein the annotated database lacks real speech data corresponding to the target phrase. Based on the TTS model, synthesized speech of the target phrase is generated; Analyze the synthesized speech of the target phrase to compile statistics about the acoustic unit; and Speech is processed with the aid of the statistical data to detect the presence of the target phrase in the speech.

12. The method according to claim 11, wherein, The TTS model includes parameters for controlling the generation of synthesized speech from input text, and wherein adjusting the TTS model includes: Synthetic speech data of multiple words is generated from the annotated database based on the parameters; Process the synthesized speech data to generate a vector representing the synthesized speech data; Process the real speech data corresponding to the plurality of words to generate a vector representing the real speech data; Compare the vectors representing the synthesized speech data and the vectors representing the real speech data to generate a comparison result; and The parameters of the TTS model are iteratively adjusted based on the comparison results to generate a compensation model.

13. The method according to claim 12, wherein, The parameter includes at least one of the following: The settings of the TTS model; The speaker characteristics of the synthesized speech data; or Parameters used to enhance the synthesized speech data.

14. The method according to claim 13, wherein, The parameters used to enhance the synthesized speech data include at least one of the following: Parameters used to modify the time scale of the synthesized speech data; Parameters used to modify the spectral content of the synthesized speech data; or Gain parameters used for the synthesized speech data.

15. The method according to claim 12, wherein, The synthesized speech generated based on the TTS model for the target phrase includes: The synthesized speech of the target phrase is generated based on the compensation model.

16. The method according to claim 11, wherein, Analyzing the synthesized speech of the target phrase to compile statistics about the acoustic unit includes: The temporal boundary of each acoustic unit of the target phrase is determined based on the input text of the target phrase; Analyze the spectral and temporal characteristics of each acoustic unit to generate observation vectors; Based on the observation vector, the most probable acoustic unit is generated within the time boundary of each acoustic unit; and Analyze the most probable acoustic unit within the time boundary of each of the multiple acoustic units in the synthesized speech of multiple target phrases to generate the probability of predicting one of the multiple most probable acoustic units when processing a given acoustic unit of the speech.

17. The method according to claim 16, wherein, Processing speech with the aid of the aforementioned statistical data includes: Alternative pronunciations of the target phrase are detected by a speech recognition model based on the probability of predicting one of the plurality of most likely acoustic units when the target phrase is spoken.

18. The method according to claim 17, wherein, The speech recognition model includes a sequence of decoded states, wherein each decoded state in the sequence models each acoustic unit of the target phrase, and wherein one decoded state in the sequence models multiple acoustic units to support alternative pronunciations of the target phrase based on the probabilities.

19. The method according to claim 11, wherein, Analyzing the synthesized speech of the target phrase to compile statistics about the acoustic unit includes: The temporal boundary of each acoustic unit of the target phrase is determined based on the input text of the target phrase; Analyze the spectral and temporal characteristics of each acoustic unit to generate observation vectors; Based on the observation vector, the most probable acoustic unit is generated within the time boundary of each acoustic unit; and Analyze the most probable acoustic unit within the time boundary of each of the multiple acoustic units in the synthesized speech of multiple target phrases to generate the average time length of each acoustic unit.

20. An apparatus for training a word segmenter to generate acoustic unit strings from input text, comprising: The processing system is configured as follows: The word is decomposed into the multiple combinations by splitting the word from the database at multiple different points for each of the multiple combinations of unique subwords, wherein the word includes one or more written units and one or more corresponding acoustic units; The acoustic units constituting the word are mapped to the written units constituting the word to generate an acoustic unit to written unit mapping; Based on the acoustic unit-to-writing unit mapping, a subset of the acoustic units is assigned to each of the unique sub-words to generate the acoustic unit-to-sub-word assignment of the word; and The acoustic units are accumulated to sub-word assignments for multiple words from the database to create a sub-word likelihood dictionary.

Images