International phonetic symbol-based pronunciation dictionary automatic construction method and device, equipment and medium

Abstract

The invention provides an international phonetic symbol-based pronunciation dictionary automatic construction method, apparatus and device, and a medium. The method comprises the steps of establishing a speech recognition model with phonemes as hidden variables; and training and optimizing the speech recognition model by using a joint stochastic approximation method to realize automatic construction of a pronunciation dictionary. According to the method and the device, the correspondence between phonemes and texts can be obtained by inputting the audio feature sequence, and compared with the prior art, tasks can be performed on a new language by utilizing the pre-trained model, and the pre-trained model can be conveniently migrated to another new language for training to obtain the pronunciation dictionary of the language, so that the pronunciation dictionary of the language can be obtained. And meanwhile, the system is convenient to maintain and relatively low in cost.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular to a method, device, equipment and medium for automatically constructing a pronunciation dictionary based on international phonetic symbols. Background Art

[0002] The modeling unit of speech recognition can be a pronunciation symbol, namely a phoneme, or a written symbol of different granularity, such as a character, a subword, or a word. Phoneme-based speech recognition systems have obvious performance advantages over purely written symbol-based systems in multilingual and cross-lingual speech recognition and low-resource speech recognition. However, phoneme-based speech recognition systems usually require a pronunciation lexicon, which records the pronunciation of each word, namely the phoneme sequence.

[0003] At present, the methods for constructing pronunciation dictionaries mainly include manual construction by language experts and expansion using smaller pronunciation dictionaries.

[0004] The method of manual construction by language experts is mainly that language experts organize words and their corresponding phonemes into dictionaries based on long-term experience and knowledge accumulation. This type of dictionary is generally more accurate, but has a smaller dictionary capacity.

[0005] Although the method of manual construction by language experts is more accurate, it takes a long time to build, has high maintenance costs, and is difficult to promote. The method of expanding the existing pronunciation dictionary also has a low accuracy rate and can never exceed the accuracy rate of the original pronunciation dictionary. Summary of the invention

[0006] In view of this, the present invention aims to overcome the deficiencies of the above-mentioned problems in the prior art and proposes a method, device, equipment and medium for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet.

[0007] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0008] The first aspect of the present invention provides a method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet, comprising:

[0009] Establish a speech recognition model with phonemes as latent variables;

[0010] The speech recognition model is trained and optimized using a joint random approximation optimization method to automatically build a pronunciation dictionary.

[0011] Furthermore, the establishment of a speech recognition model with phonemes as latent variables specifically includes:

[0012] Suppose the audio feature is x, x is a feature vector sequence, its corresponding word sequence is y, and the phoneme sequence is z. The goal is to construct a conditional probability distribution P θ，ω (y, z|x), and complete the learning of the model, assuming that when z is given, x and y are independent, we get

[0013]

[0014] The parameters θ and φ represent the probability distribution p respectively. θ and p φ Two neural network parameters of ;

[0015] Construct two conditional distributions P θ (z|x) and P φ (y|z), where P θ (z|x) is the audio to phoneme sequence modeling model, P φ (y|z) is a model for the sequence from phonemes to words, and z is used as a latent variable.

[0016] Furthermore, it also includes P θ (z|x) is pre-trained to initialize the model of speech features to phonemes, and then the speech recognition model is trained and optimized using the joint random approximation optimization method.

[0017] Furthermore, the θ (z|x) is pre-trained to initialize the model of speech features to phonemes, and then the speech recognition model is trained and optimized using the joint random approximation optimization method, including:

[0018] Supervised training with phoneme annotations on multilingual datasets.

[0019] Furthermore, the method of using the joint random approximation optimization method to train and optimize the speech recognition model to automatically construct a pronunciation dictionary includes:

[0020] For the conditional distribution z~P θ (z|x) is sampled to obtain the sampling result;

[0021] Use the MIS algorithm to decide whether to accept this sampling and retain the value of this sampling;

[0022] If this sampling is received, the value of this sampling is used as P θ The true value of the encoder network in the (z|x) model is used to guide the training, and the value of this sampling is used as P φ (y|z) Input to the decoder network in the model;

[0023] The losses of the encoder network and the decoder network are added as the total loss output, and then back propagation is performed to update the parameters. After training to convergence, the target network is obtained. The input audio feature sequence is passed through P θ The (z|x) model obtains a phoneme sequence, which is then passed through the embedding layer and then through the P φ The (y|z) model obtains its corresponding word sequence to construct a pronunciation dictionary.

[0024] The second aspect of the present invention provides an automatic construction device for a pronunciation dictionary based on the International Phonetic Alphabet, comprising:

[0025] A model building unit, used to build a speech recognition model with phonemes as latent variables;

[0026] The training unit is used to train and optimize the speech recognition model using a joint random approximation optimization method to automatically build a pronunciation dictionary.

[0027] A third aspect of the present invention provides an electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, wherein the processor is used to execute the above-mentioned method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet.

[0028] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet.

[0029] Compared with the prior art, the method, device, equipment and medium for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet of the present invention have the following advantages:

[0030] The present invention can obtain the correspondence between the phonemes and text by inputting an audio feature sequence. Compared with the existing technology, it can perform tasks on a new language using a pre-trained model. Using the pre-trained model, it can be easily transferred to another new language for training to obtain a pronunciation dictionary of the language with high accuracy, easy maintenance and low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0032] Figure 1 It is a schematic flow chart of the optimization method of the combined random approximation of the present invention;

[0033] Figure 2 This is the forward and backward calculation pseudo code of the joint random approximation optimization method of the present invention. DETAILED DESCRIPTION

[0034] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0035] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.

[0036] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.

[0037] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0038] Embodiment 1:

[0039] A method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet, characterized by comprising:

[0040] Establish a speech recognition model with phonemes as latent variables;

[0041] The speech recognition model is trained and optimized using a joint random approximation optimization method to automatically build a pronunciation dictionary.

[0042] In this invention, three parts of data are needed, where the audio feature x is a feature vector sequence, its corresponding word sequence is y, and the phoneme sequence is z. The goal is to construct a conditional probability distribution p θ.ω (y, z|x), and complete the learning of the model. If we assume that x and y are independent when z is given, we can get

[0043]

[0044] The parameters θ and φ represent the probability distribution p respectively. θ and p φ The parameters of the two neural networks.

[0045] From this we can see that we need to construct two conditional distributions p θ (z|x) and p φ (y|z), that is, the sequence modeling of audio to phoneme and the sequence modeling of phoneme to word. Then z is used as a hidden variable and jointly optimized using the joint stochastic approximation optimization method (JSA).

[0046] The process of the optimization method of joint random approximation is as follows Figure 1-2 As shown:

[0047] The Monte Carlo sampling and parameter update process is repeated until convergence. Monte Carlo sampling: randomly draw κ from a uniform distribution of i=1,…,n to obtain x κ , and then obtain the phoneme sequence z through MIS sampling k When k = i, given the current factor sequence The MIS sampling process is as follows:

[0048] 1. Extract z i ~p φ (z i |x i ),

[0049] 2. The probability of accepting The importance sampling weight

[0050] In the present invention, there are

[0051]

[0052] Therefore, z is regarded as a hidden variable and p θ (z|x) can be used as an auxiliary function for joint optimization using JSA.

[0053] Notice

[0054] P θ，φ (y, z|x)=p φ (y|z)·p θ (z|x);

[0055] -log(P θ，φ (y, z|x)) = -log(P φ(y|z))-log(P θ (z|x));

[0056] Therefore, the loss value of the overall network can be obtained by adding the loss values ​​of the two networks.

[0057] Since z is a hidden variable in the model, there is no true value input during the model training process. Therefore, the present invention proposes to θ (z|x) is pre-trained to initialize the model of speech features to phonemes, and then JSA training is performed. Pre-training can be supervised training with phoneme annotations on a multi-language dataset (for example, multi-language training using the four languages ​​of German, French, Italian and Spanish in CommonVoice).

[0058] If you need to build a phoneme-based speech recognition system for a new language (for example, Polish), you can use the JSA method to initialize P with the trained model. θ (z|x), and P φ (y|z) is randomly initialized. The richer the pre-trained multilingual dataset, or the more similar the new language is to the pre-trained multilingual dataset and the more common phonemes it contains, the more effective the JSA training will be, and the better the performance of the speech recognition system for the new language will be.

[0059] In the present invention, the speech recognition model is trained and optimized using a joint random approximation optimization method to achieve automatic construction of a pronunciation dictionary. The specific process is as follows:

[0060] The input to the algorithm is the audio feature sequence x k and the word sequence y k , which are all in the form of minibatch, the phoneme sequence z k It is generated by sampling and updating in the forward process, where k is the ordinal number of the current batch in the packaged data. Speech feature sequence x k The size is batch size×num timesteps×80, where 80 is the number of features extracted, and the word sequence y k The size of is batch size × out len. In the use of JSA, we need to θ (z|x) is sampled to get z k ′, the maximum probability result obtained by beam search can be used to simulate this sampling. In the present invention, P θ(z|x) is modeled as a CTC model, and its encoder adopts a Conformer structure. For the input, it outputs a probability sequence probs. CTCBeamDecoder can be used as a sampler to obtain the sampled result beamresult according to probs. Its size is batch size×beam size×num timesteps, where beam size is the number of beams during beam search. In the present invention, beam size=1, so the size of beam result can be reduced to batch size×num timesteps, which is what we need z k ′.

[0061] Get z k ′, the MIS algorithm is used to decide whether to accept this sampling. Since there is no value at the previous moment when the algorithm starts, all data needs to be sampled once before training starts to get the value of the 0th epoch, and then it is continuously updated. The specific process is as follows:

[0062] First define the importance sampling weight function, which is defined as:

[0063]

[0064] Then decide whether to accept the sampling.

[0065]

[0066] In the above formula, z′ k is the value obtained by beam search simulation sampling, is the value obtained in the previous epoch with the same position. The algorithm decides whether to keep the value of this sampling based on the calculated ratio. If this sampling is received, the value of this sampling is used as P θ The true value of the encoder network in the (z|x) model is used to guide the training, and the value of this sampling is used as P φ (y|z) Input to the decoder network in the model.

[0067] get After that, and x k and k The loss value is obtained by inputting the network together, and the loss of the encoder network and the decoder network is added as the total loss output, and then back propagation is performed to update the parameters. After training to convergence, the target network can be obtained. The input speech feature sequence is passed through P θ(z|x) can get the phoneme sequence, which passes through the embedding layer and then passes through P φ (y|z) can get its corresponding word sequence, thus constructing a pronunciation dictionary.

[0068] Embodiment 2:

[0069] The device for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet includes:

[0070] A model building unit, used to build a speech recognition model with phonemes as latent variables;

[0071] The training unit is used to train and optimize the speech recognition model using a joint random approximation optimization method to automatically build a pronunciation dictionary.

[0072] Embodiment three:

[0073] An electronic device includes a processor and a memory connected to the processor for storing instructions executable by the processor, wherein the processor is used to execute the method for automatically constructing a pronunciation dictionary based on international phonetic symbols in the first embodiment.

[0074] Embodiment 4:

[0075] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for automatically constructing a pronunciation dictionary based on international phonetic symbols of embodiment 1 is implemented.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet, characterized in that: include: Establish a speech recognition model with phonemes as latent variables; The speech recognition model is trained and optimized using a joint random approximation optimization method to automatically build a pronunciation dictionary.

2. The method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet according to claim 1, characterized in that: The establishment of a speech recognition model with phonemes as latent variables specifically includes: Suppose the audio feature is x, x is a feature vector sequence, its corresponding word sequence is y, and the phoneme sequence is z. The goal is to construct a conditional probability distribution P θ,ω (y,z|x), and complete the model learning, assuming that when z is given, x and y are independent, we get The parameters θ and φ represent the probability distribution p respectively. θ and p φ Two neural network parameters of ; Construct two conditional distributions P θ (z|x) and P φ (y|z), where P θ (z|x) is the audio to phoneme sequence modeling model, P φ (y|z) is a model for the sequence from phonemes to words, and z is used as a latent variable.

3. The method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet according to claim 2, characterized in that: Also includes P θ (z|x) is pre-trained to initialize the model of speech features to phonemes, and then the speech recognition model is trained and optimized using the joint random approximation optimization method.

4. The method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet according to claim 3, characterized in that: The pair P θ (z|x) is pre-trained to initialize the model of speech features to phonemes, and then the speech recognition model is trained and optimized using the joint random approximation optimization method, including: Supervised training with phoneme annotations on multilingual datasets.

5. The method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet according to claim 2, characterized in that: The method of using the joint random approximation optimization method to train and optimize the speech recognition model to automatically construct a pronunciation dictionary includes: For the conditional distribution z~P θ (z|x) is sampled to obtain the sampling result; Use the MIS algorithm to decide whether to accept this sampling and retain the value of this sampling; If this sampling is received, the value of this sampling is used as P θ The true value of the encoder network in the (z|x) model is used to guide the training, and the value of this sampling is used as P φ (y|z) Input to the decoder network in the model; The losses of the encoder network and the decoder network are added as the total loss output, and then back propagation is performed to update the parameters. After training to convergence, the target network is obtained. The input audio feature sequence is passed through P θ The (z|x) model obtains a phoneme sequence, which is then passed through the embedding layer and then through the P φ The (y|z) model obtains its corresponding word sequence to construct a pronunciation dictionary.

6. An automatic construction device for a pronunciation dictionary based on the International Phonetic Alphabet, characterized in that: include: A model building unit, used to build a speech recognition model with phonemes as latent variables; The training unit is used to train and optimize the speech recognition model using a joint random approximation optimization method to automatically build a pronunciation dictionary.

7. An electronic device, comprising a processor and a memory connected to the processor for storing instructions executable by the processor, characterized in that: The processor is used to execute the method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for automatically constructing a pronunciation dictionary based on the International Phonetic Alphabet as described in any one of claims 1 to 5 is implemented.

Images