Model training method, device, phoneme recognition method, equipment and medium

Abstract

The present application discloses a training method, device, phoneme recognition method, equipment and medium of a model. The model includes an embedding network, a chimeric network and a generative network, and the method includes: obtaining a training acoustic feature vector sequence, a training text feature vector sequence and a training phoneme sequence of a language, using the embedding network to convert the training phoneme sequence to obtain a training embedded feature vector, using the chimeric network to calculate based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain multiple training latent vector sequences, using the generative network to calculate a predicted phoneme probability vector sequence and a training phoneme label vector sequence based on the training embedded feature vector and the multiple training latent vector sequences, calculating the training loss based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence, adjusting the model parameters of the phoneme recognition model based on the loss, and continuing to train the adjusted phoneme recognition model until the convergence condition is reached to obtain the phoneme recognition model.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and specifically to a model training method, device, phoneme recognition method, equipment and medium. Background Art

[0002] With the development of educational technology, automated pronunciation assessment systems have gradually become one of the important tools for foreign language learning. Such systems are designed to help learners improve their pronunciation, especially in the case of learning non-native languages ​​such as English. The pronunciation assessment system analyzes the learner's pronunciation and compares it with the standard pronunciation to identify errors or inaccuracies in pronunciation. However, in the prior art, due to the existence of polyphonic words, context-dependent pronunciation changes, weak sounds and omissions, simple speech signal analysis cannot always accurately capture the learner's actual pronunciation, resulting in low speech recognition efficiency and large errors. Summary of the invention

[0003] The present application proposes a model training method, device, phoneme recognition method, equipment and medium to address the deficiencies of the above-mentioned prior art.

[0004] In a first aspect, an embodiment of the present application provides a training method for a phoneme recognition model, wherein the phoneme recognition model includes an embedding network, a chimeric network, and a generative network, and the method includes: obtaining a training acoustic feature vector sequence, a training text feature vector sequence, and a training phoneme sequence of a training language; using the embedding network, converting the training phoneme sequence to obtain a training embedded feature vector; using the chimeric network, performing calculation processing based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain a plurality of training latent vector sequences; using the generative network, calculating a predicted phoneme probability vector sequence and a training phoneme label vector sequence based on the training embedded feature vector and the plurality of training latent vector sequences; calculating a training loss based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence, and adjusting the model parameters of the phoneme recognition model based on the training loss; continuing to train the phoneme recognition model after adjusting the model parameters until the phoneme recognition model reaches a convergence condition, thereby obtaining the trained phoneme recognition model.

[0005] In a second aspect, an embodiment of the present application provides a method, comprising: obtaining a target acoustic feature vector sequence and a target text feature vector sequence corresponding to the speech and text of a language to be recognized respectively; inputting the target acoustic feature vector sequence and the target text feature vector sequence into a phoneme recognition model to obtain a target phoneme sequence of the language to be recognized, wherein the phoneme recognition model is trained using the method described in the first aspect above.

[0006] In a third aspect, an embodiment of the present application provides a training device for a phoneme recognition model, wherein the phoneme recognition model includes an embedding network, a chimeric network, and a generative network, and the device includes: an acquisition module for acquiring a training acoustic feature vector sequence, a training text feature vector sequence, and a training phoneme sequence of a training language; a conversion module for using the embedding network to convert the training phoneme sequence to obtain a training embedded feature vector; a first calculation module for using the chimeric network to perform calculations based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain a plurality of training latent vector sequences; a second calculation module for using the generative network to calculate a predicted phoneme probability vector sequence and a training phoneme label vector sequence based on the training embedded feature vector and the plurality of training latent vector sequences; an adjustment module for calculating a training loss based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence, and adjusting the model parameters of the phoneme recognition model based on the training loss; and a training module for continuing to train the phoneme recognition model after adjusting the model parameters until the phoneme recognition model reaches a convergence condition, thereby obtaining the trained phoneme recognition model.

[0007] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect above is implemented.

[0008] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect above is implemented.

[0009] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0010] The phoneme recognition model of the embodiment of the present application includes an embedding network, a chimeric network and a generative network. The training method of the phoneme recognition model of the embodiment of the present application obtains a training acoustic feature vector sequence, a training text feature vector sequence and a training phoneme sequence of a training language, and uses the embedding network to convert the training phoneme sequence to obtain a training embedded feature vector, which can map discrete data in the training phoneme sequence to a continuous vector space, accurately capture the semantic relationship between the training phoneme sequences, and improve the performance of the model.

[0011] Furthermore, the chimeric network is used to calculate and process based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain multiple training latent vector sequences, and then the generative network is used to calculate the predicted phoneme probability vector sequence and the training phoneme label vector sequence based on the training embedded feature vector and the multiple training latent vector sequences, and the training loss is calculated based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence. Based on the training loss, the model parameters of the phoneme recognition model are adjusted, and finally the phoneme recognition model with the adjusted model parameters is continued to be trained until the phoneme recognition model reaches the convergence condition, and a trained phoneme recognition model is obtained. Through the deep training of the phoneme recognition model, the recognition accuracy and recognition efficiency of the phoneme recognition model are improved.

[0012] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0014] Figure 1 A flowchart of a method for training a phoneme recognition model provided in an embodiment of the present application is shown;

[0015] Figure 2 A flowchart of a phoneme recognition method provided by an embodiment of the present application is shown;

[0016] Figure 3 A schematic diagram of the structure of a training device for a phoneme recognition model provided in an embodiment of the present application is shown;

[0017] Figure 4 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0018] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0019] With the development of educational technology, automated pronunciation assessment systems have gradually become one of the important tools for foreign language learning. Such systems are designed to help learners improve their pronunciation, especially in the case of learning non-native languages ​​such as English. The pronunciation assessment system analyzes the learner's pronunciation and compares it with the standard pronunciation to identify errors or inaccuracies in pronunciation. However, in the prior art, due to the existence of polyphonic words, context-dependent pronunciation changes, weak sounds and omissions, simple speech signal analysis cannot always accurately capture the learner's actual pronunciation, resulting in low speech recognition efficiency and large errors.

[0020] Based on this, the embodiment of the present application provides a method for training a phoneme recognition model. The solution of the embodiment of the present application is specifically described below in conjunction with the accompanying drawings.

[0021] The phoneme recognition model in the embodiment of the present application includes an embedding network, a chimeric network and a generative network.

[0022] See also Figure 1 A flowchart of a method for training a phoneme recognition model is shown, and the method specifically comprises the following steps:

[0023] Step S101: Acquire a training acoustic feature vector sequence, a training text feature vector sequence, and a training phoneme sequence of a training language.

[0024] In the embodiments of the present application, the training language may include Chinese, English, French, Spanish, Arabic, Russian, Hindi, German, Portuguese, Japanese and the like, which are not specifically limited in the embodiments of the present application.

[0025] In one embodiment, the training speech of the training language, the training text corresponding to the training speech, and the training phoneme sequence corresponding to the training text can be collected, and then the training speech feature vector sequence corresponding to the training speech and the training text encoding sequence corresponding to the training text can be obtained. A pre-trained sound coding model is used to perform encoding processing based on the training speech feature vector sequence to obtain a training acoustic feature vector sequence. A pre-trained text coding model is used to obtain a training text feature vector sequence based on the training text encoding sequence.

[0026] Further, the training speech feature vector sequence corresponding to the training speech is obtained, specifically, the Mel-frequency cepstrum vector of the training speech is calculated using a Mel filter, and then the Mel-frequency cepstrum vector is converted and normalized to obtain the training speech feature vector sequence. Specifically, the Mel-frequency cepstrum vector of the training speech can be calculated using a Mel filter, and then the natural logarithm or common logarithm of the Mel-frequency cepstrum vector is taken, and the Mel-frequency cepstrum vector after taking the natural logarithm or common logarithm is normalized to a preset range using a feature scaling technique to obtain the training speech feature vector sequence.

[0027] For example, assume that 40 Mel filters are selected to process the training language to obtain a Mel-frequency cepstrum vector, and then the Mel-frequency cepstrum vector is logarithmically processed to obtain a first result, the first result is truncated to a lower bound so that the minimum value in the first result is greater than or equal to the maximum value in the first result minus 8 to obtain a second result, and the second result is scaled to a specific interval. The interval in the embodiment of the present application is: [max / 4-1~max / 4+1], where max represents the maximum value in the first result.

[0028] The Mel frequency cepstrum of the training speech is calculated using a Mel filter, and then the Mel frequency cepstrum is converted and standardized to obtain a feature vector sequence of the training speech, which can effectively ensure the consistency of the training speech and the effectiveness of the algorithm.

[0029] Furthermore, the training text encoding sequence corresponding to the training text is obtained, specifically, by using a pre-trained language segmentation model to segment the training text to obtain vocabulary units, and then obtaining the training text encoding sequence based on the vocabulary units and a preset encoding table.

[0030] It should be noted that the training speech and training text of the above training language can be obtained based on open source speech data, and the open source speech data can be derived from open source speech data sets such as LibriSpeech and WSJ, which is not specifically limited in this application. For data containing only speech data, the corresponding training text can be obtained through a commercial automatic speech recognition (ASR) interface; for data containing only training text, the CMU pronunciation dictionary can be used to infer the phoneme sequence of its pronunciation. For words not included in the dictionary, the rules in G2P (Grapheme-to-Phoneme, i.e., grapheme-to-phoneme conversion) are used for inference, so as to obtain the complete phoneme sequence corresponding to the training speech.

[0031] Step S102: using the embedding network, converting the training phoneme sequence to obtain a training embedding feature vector.

[0032] In one embodiment, the embedding network includes a phoneme embedding layer and a first position encoding layer. The embedding network is used to transform the training phoneme sequence to obtain a training embedded feature vector. Specifically, the phoneme embedding layer and the first position encoding layer are used to transform the input training phoneme sequence to obtain a first vector representation and a second vector representation, and the first vector representation and the second vector representation are integrated to obtain a training embedded feature vector.

[0033] It should be noted that the training embedded feature vector integrates the phoneme features and position information of the training phoneme sequence.

[0034] Step S103: using the mosaic network, performing calculations based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain a plurality of training latent vector sequences.

[0035] In one embodiment, the chimeric network includes a type encoding layer, a second position encoding layer, a conversion layer and a first self-attention layer. The chimeric network is used to perform calculations based on a training acoustic feature vector sequence and a training text feature vector sequence to obtain multiple training latent vector sequences, including: using the conversion layer to convert and splice the training acoustic feature vector sequence and the training text feature vector sequence to obtain a training input feature vector sequence; using the type encoding layer, while the conversion layer splices the converted training acoustic feature vector sequence and the training text feature vector sequence, constructs a training type feature vector sequence according to the splicing position; using the second position encoding layer, encodes the position information corresponding to the training acoustic feature vector sequence or the training text feature vector sequence to obtain a training position feature vector sequence; using the first self-attention layer, based on the training input feature vector sequence, the training type feature vector sequence and the training position feature vector sequence, calculates and processes to obtain multiple training latent vector sequences.

[0036] Based on the above implementation, in some modified implementations, the first self-attention layer may include multiple layers, all of which have a filtering self-attention mechanism, and the training latent vector sequence is obtained by using the first self-attention layer through the following formula (1):

[0037]

[0038] in, represents the hidden vector sequence of the i+1th layer, represents the hidden vector sequence of the i-th layer, MHA represents the multi-head attention mechanism, FFN represents the feedforward neural network, and f represents the filter function, which can be composed of a linear layer and a sigmoid activation function.

[0039] Through the filtering self-attention mechanism of the first self-attention layer, we can better distinguish the attention paradigms of different modalities and obtain better representation vectors, thereby improving the recognition accuracy of the phoneme recognition model and reducing recognition errors.

[0040] Step S104: using the generative network, based on the training embedded feature vector and multiple training latent vector sequences, a predicted phoneme probability vector sequence and a training phoneme label vector sequence are calculated.

[0041] In one embodiment, the generative network includes a second attention layer and a phoneme label layer. The generative network is used to calculate a predicted phoneme probability vector sequence and a training phoneme label vector sequence based on a training embedded feature vector and a plurality of training latent vector sequences, including: using the phoneme label layer to encode the training embedded feature vector to obtain a training phoneme label vector sequence, and then using the second attention layer to obtain the sequence lengths of the training acoustic feature vector sequence and the training text feature vector sequence, and based on the plurality of training latent vector sequences and the sequence lengths, a predicted phoneme probability vector sequence is calculated.

[0042] Based on the above implementation, in some modified implementations, the second attention layer includes multiple layers, and the predicted phoneme probability vector sequence is calculated based on multiple training latent vector sequences and sequence lengths using the second attention layer, including: using the second attention layer and the following formula (2) to obtain a predicted phoneme probability vector sequence based on multiple training latent vector sequences and sequence lengths.

[0043]

[0044]

[0045]

[0046] in, Represents the intermediate hidden vector in the generative network and the intermediate hidden vector in the chimeric network The attention weight between , N represents the number of layers in the second attention layer, j and m represent the subscripts of the summation function, represents the training latent vector sequence corresponding to the second attention layer of the first layer, A represents the attention value of the intermediate latent vector in the generated network and any one of the multiple training latent vector sequences, represents the intermediate hidden vector in the generative network, represents multiple training latent vector sequences of the second attention layer of the jth layer, M represents the sequence length, f represents the attention calculation function, represents the intermediate hidden vector of the second attention layer of the jth layer in the mosaic network, represents the intermediate hidden vector of the second attention layer of the kth layer in the mosaic network, Represents the intermediate latent vector in the chimeric network.

[0047] On the basis of the above implementation, in some modified implementations, the training embedded feature vector is encoded using the phoneme label layer to obtain a training phoneme label vector sequence. Specifically, it can be: based on the predefined probability and the collected and organized weak reading and skimming rules, the training embedded feature vector is transformed to obtain a training phoneme label vector sequence.

[0048] For example, assuming the English language is an example, if a word ends with a consonant and is followed by a word that starts with the same sound, there is a 60% probability that the final consonant of the previous word will be omitted. For example, the t pronunciation of what in "what to do" is connected to the t pronunciation of to, and there is a 60% probability that the t in what will be removed in the phoneme sequence; a word ending with / b / , / p / , / t / , / d / , / k / , / g / , followed by a word that starts with a consonant, has a 30% probability of omitting the final consonant of the previous word, such as "just do it"; a word starting with "h" has a 10% probability that the [h] sound will be omitted; plosives are adjacent to plosives, and the first plosive has a 60% probability of being omitted.

[0049] Step S105: Calculate the training loss based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence, and adjust the model parameters of the phoneme recognition model based on the training loss.

[0050] In one implementation, the cross entropy loss may be calculated based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence, and then the gradient may be calculated by back propagation, and then the weights of the phoneme recognition model may be adjusted based on the optimizer and the gradient.

[0051] Step S106: Continue to train the phoneme recognition model after adjusting the model parameters until the phoneme recognition model reaches a convergence condition, thereby obtaining a trained phoneme recognition model.

[0052] In the embodiment of the present application, when the model is trained before the model parameters are adjusted, the parameters of the optimizer can be configured as: AdamW, learning rate 1e-3, decay coefficient 0.0001, the learning rate strategy adopts a linear decay strategy with preheating, the preheating steps are 1000 steps, the decay rate is 0.05, and the minimum learning rate is 1e-5; the batch size is 256, 3 epochs are trained, and the weight selects the model weight of the last step. When the model parameters are adjusted to continue training the model, the parameters of the optimizer can be configured as: AdamW, learning rate 3e-5, decay coefficient 0.0001, the learning rate strategy adopts a linear decay strategy with preheating, the preheating steps are 200 steps, the decay rate is 0.1, and the minimum learning rate is 1e-6; the batch size is 64, 5 epochs are trained, and the loss value of the validation set is calculated on the validation set every 100 steps, and the lowest global validation set loss value is selected as the final model weight.

[0053] The phoneme recognition model of the embodiment of the present application includes an embedding network, a chimeric network and a generative network. The training method of the phoneme recognition model of the embodiment of the present application obtains a training acoustic feature vector sequence, a training text feature vector sequence and a training phoneme sequence of a training language, and uses the embedding network to convert the training phoneme sequence to obtain a training embedded feature vector, which can map discrete data in the training phoneme sequence to a continuous vector space, accurately capture the semantic relationship between the training phoneme sequences, and improve the performance of the model.

[0054] Furthermore, the chimeric network is used to calculate and process based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain multiple training latent vector sequences, and then the generative network is used to calculate the predicted phoneme probability vector sequence and the training phoneme label vector sequence based on the training embedded feature vector and the multiple training latent vector sequences, and the training loss is calculated based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence. Based on the training loss, the model parameters of the phoneme recognition model are adjusted, and finally the phoneme recognition model with the adjusted model parameters is continued to be trained until the phoneme recognition model reaches the convergence condition, and a trained phoneme recognition model is obtained. Through the deep training of the phoneme recognition model, the recognition accuracy and recognition efficiency of the phoneme recognition model are improved.

[0055] See also Figure 2 The present application also provides a method for identifying factors.

[0056] The embodiment of the present application also provides a phoneme recognition method. The solution of the embodiment of the present application is described in detail below in conjunction with the accompanying drawings.

[0057] See also Figure 2 A flowchart of a phoneme recognition method is shown, the method specifically comprising the following steps:

[0058] Step S201: Obtain a target acoustic feature vector sequence and a target text feature vector sequence corresponding to the speech and text of the language to be recognized, respectively.

[0059] The method for obtaining the target acoustic feature vector sequence and the target text feature vector sequence in the embodiment of the present application is the same as the method for obtaining the training acoustic feature vector sequence and the training text feature vector sequence described above, and will not be repeated here.

[0060] Step S202: inputting the target acoustic feature vector sequence and the target text feature vector sequence into a phoneme recognition model to obtain a target phoneme sequence of the language to be recognized.

[0061] In the embodiment of the present application, the phoneme recognition model is trained using the training method of the phoneme recognition model provided in the above embodiment.

[0062] In one implementation, the target acoustic feature vector sequence and the target text feature vector sequence are input into a phoneme recognition model, and the target phoneme sequence of the language to be recognized can be inferred and calculated by a beam search algorithm, wherein the beam size in the beamsearch parameter is set to 3 in this example; the maximum decoding length is 50.

[0063] See also Figure 3 The present application also provides a training device for a phoneme recognition model, which is used to execute the training method for the phoneme recognition model described in the above embodiment. The device includes:

[0064] An acquisition module 301 is used to acquire a training acoustic feature vector sequence, a training text feature vector sequence and a training phoneme sequence of a training language;

[0065] A conversion module 302 is used to convert the training phoneme sequence using the embedding network to obtain a training embedding feature vector;

[0066] A first calculation module 303 is used to use the mosaic network to perform calculation processing based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain multiple training latent vector sequences;

[0067] A second calculation module 304 is used to calculate a predicted phoneme probability vector sequence and a training phoneme label vector sequence using the generation network based on the training embedded feature vector and the plurality of training latent vector sequences;

[0068] An adjustment module 305, configured to calculate a training loss based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence, and adjust a model parameter of the phoneme recognition model based on the training loss;

[0069] The training module 306 is used to continue training the phoneme recognition model after adjusting the model parameters until the phoneme recognition model reaches a convergence condition, thereby obtaining the trained phoneme recognition model.

[0070] The training device for the phoneme recognition model provided in the embodiment of the present application and the training method for the phoneme recognition model provided in the above embodiment are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0071] The present application also provides an electronic device corresponding to the training method of the phoneme recognition model and the phoneme recognition method provided in the above embodiment. Figure 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 4As shown, the electronic device 40 may include: a processor 400, a memory 401, a bus 402 and a communication interface 403, and the processor 400, the communication interface 403 and the memory 401 are connected via the bus 402; the memory 401 stores a computer program that can be run on the processor 400, and when the processor 400 runs the computer program, the training method of the phoneme recognition model and the phoneme recognition method provided in any of the aforementioned embodiments of the present application are executed.

[0072] The memory 401 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port 403 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0073] The bus 402 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 401 is used to store programs, and the processor 400 executes the programs after receiving the execution instruction. The training method of the phoneme recognition model and the phoneme recognition method disclosed in any implementation of the embodiment of the present application may be applied to the processor 400, or implemented by the processor 400.

[0074] The processor 400 may be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 400. The above processor 400 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 401, and the processor 400 reads the information in the memory 401 and completes the steps of the above method in combination with its hardware.

[0075] The electronic device provided in the embodiment of the present application and the training method of the phoneme recognition model and the phoneme recognition method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0076] An embodiment of the present application also provides a computer-readable storage medium corresponding to the training method of the phoneme recognition model and the phoneme recognition method provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the training method of the phoneme recognition model and the phoneme recognition method provided in any of the aforementioned embodiments.

[0077] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0078] An embodiment of the present application also provides a computer program product corresponding to the training method of the phoneme recognition model and the phoneme recognition method provided in the aforementioned embodiment, including a computer program, which is executed by a processor to implement the training method of the phoneme recognition model and the phoneme recognition method provided in the aforementioned embodiments.

[0079] The computer-readable storage medium and computer program product provided in the above-mentioned embodiments of the present application are based on the same inventive concept as the training method of the phoneme recognition model and the phoneme recognition method provided in the embodiments of the present application, and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0080] It should be noted that:

[0081] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language. It should be understood that the content of the application described herein can be realized using various programming languages, and the description of the specific language above is to disclose the best mode of implementation of the application.

[0082] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0083] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0084] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0085] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0086] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the creation device of the virtual machine according to the embodiment of the present application. The present application can also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0087] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0088] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for training a phoneme recognition model, characterized in that: The phoneme recognition model includes an embedding network, a chimeric network and a generative network, and the method includes: Obtaining a training acoustic feature vector sequence, a training text feature vector sequence, and a training phoneme sequence of a training language; Using the embedding network, converting the training phoneme sequence to obtain a training embedding feature vector; Using the mosaic network, performing calculations based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain a plurality of training latent vector sequences; Using the generative network, based on the training embedded feature vector and the plurality of training latent vector sequences, a predicted phoneme probability vector sequence and a training phoneme label vector sequence are calculated; Calculating a training loss based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence, and adjusting a model parameter of the phoneme recognition model based on the training loss; Continue to train the phoneme recognition model after adjusting the model parameters until the phoneme recognition model reaches a convergence condition, thereby obtaining the trained phoneme recognition model; The mosaic network includes a type encoding layer, a second position encoding layer, a conversion layer and a first self-attention layer. The mosaic network is used to perform calculations based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain multiple training latent vector sequences, including: Using the conversion layer, the training acoustic feature vector sequence and the training text feature vector sequence are converted and concatenated to obtain a training input feature vector sequence; using the type encoding layer, while the converted training acoustic feature vector sequence and the training text feature vector sequence are concatenated at the conversion layer, a training type feature vector sequence is constructed according to the concatenation position; using the second position encoding layer, the position information corresponding to the training acoustic feature vector sequence or the training text feature vector sequence is encoded to obtain a training position feature vector sequence; using the first self-attention layer, calculations are performed based on the training input feature vector sequence, the training type feature vector sequence and the training position feature vector sequence to obtain a plurality of training latent vector sequences; The generative network includes a second attention layer and a phoneme label layer, and the use of the generative network to calculate a predicted phoneme probability vector sequence and a training phoneme label vector sequence based on the training embedded feature vector and a plurality of training latent vector sequences includes: The training embedded feature vector is encoded using the phoneme label layer to obtain the training phoneme label vector sequence; the sequence lengths of the training acoustic feature vector sequence and the training text feature vector sequence are obtained using the second attention layer, and the predicted phoneme probability vector sequence is calculated based on multiple training latent vector sequences and the sequence lengths.

2. The training method of the phoneme recognition model according to claim 1, characterized in that: The embedding network includes a phoneme embedding layer and a first position encoding layer. The embedding network is used to convert the training phoneme sequence to obtain a training embedding feature vector, including: Using the phoneme embedding layer and the first position encoding layer, respectively converting the input training phoneme sequence to obtain a first vector representation and a second vector representation; The first vector representation and the second vector representation are integrated to obtain the training embedding feature vector.

3. The training method of the phoneme recognition model according to claim 1, characterized in that: The first self-attention layer includes multiple layers, and all of them have a filtered self-attention mechanism. The first self-attention layer is used to obtain the training latent vector sequence through the following formula: in, represents the hidden vector sequence of the i+1th layer, represents the hidden vector sequence of the i-th layer, MHA represents the multi-head attention mechanism, FFN represents the feedforward neural network, and f represents the filter function.

4. The method for training a phoneme recognition model according to claim 1, characterized in that: The second attention layer includes multiple layers, and the use of the second attention layer to calculate the predicted phoneme probability vector sequence based on the multiple training latent vector sequences and the sequence lengths includes: Using the second attention layer and the following formula, based on the plurality of training latent vector sequences and the sequence lengths, the predicted phoneme probability vector sequence is obtained; in, Represents the intermediate hidden vector in the generative network The intermediate hidden vector in the embedded network The attention weight between them, N represents the number of the second attention layer, j and m represent the subscripts of the summation function, represents the training latent vector sequence corresponding to the first layer and the second attention layer, A represents the attention value of the intermediate latent vector in the generating network and any one of the multiple training latent vector sequences, represents the intermediate hidden vector in the generative network, represents a plurality of said training latent vector sequences of the j-th layer of the second attention layer, M represents said sequence length, f represents an attention calculation function, represents the intermediate hidden vector of the second attention layer of the jth layer in the embedded network, represents the intermediate hidden vector of the second attention layer of the kth layer in the embedded network, Represents the intermediate latent vector in the embedded network.

5. A phoneme recognition method, characterized in that: include: Obtaining a target acoustic feature vector sequence and a target text feature vector sequence corresponding to the speech and text of the language to be recognized, respectively; The target acoustic feature vector sequence and the target text feature vector sequence are input into a phoneme recognition model to obtain a target phoneme sequence of the language to be recognized, wherein the phoneme recognition model is trained using the training method for the phoneme recognition model described in any one of claims 1-4.

6. A training device for a phoneme recognition model, characterized in that: The phoneme recognition model includes an embedding network, a mosaic network and a generative network, and the device includes: An acquisition module, used to acquire a training acoustic feature vector sequence, a training text feature vector sequence and a training phoneme sequence of a training language; A conversion module, used to convert the training phoneme sequence using the embedding network to obtain a training embedding feature vector; A first calculation module is used to use the mosaic network to perform calculation processing based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain multiple training latent vector sequences; A second calculation module is used to calculate a predicted phoneme probability vector sequence and a training phoneme label vector sequence using the generation network based on the training embedded feature vector and the plurality of training latent vector sequences; An adjustment module, configured to calculate a training loss based on the predicted phoneme probability vector sequence and the training phoneme label vector sequence, and adjust a model parameter of the phoneme recognition model based on the training loss; A training module, used to continue training the phoneme recognition model after adjusting the model parameters until the phoneme recognition model reaches a convergence condition, thereby obtaining the trained phoneme recognition model; The mosaic network includes a type encoding layer, a second position encoding layer, a conversion layer and a first self-attention layer. The mosaic network is used to perform calculations based on the training acoustic feature vector sequence and the training text feature vector sequence to obtain multiple training latent vector sequences, including: Using the conversion layer, the training acoustic feature vector sequence and the training text feature vector sequence are converted and concatenated to obtain a training input feature vector sequence; using the type encoding layer, while the converted training acoustic feature vector sequence and the training text feature vector sequence are concatenated at the conversion layer, a training type feature vector sequence is constructed according to the concatenation position; using the second position encoding layer, the position information corresponding to the training acoustic feature vector sequence or the training text feature vector sequence is encoded to obtain a training position feature vector sequence; using the first self-attention layer, calculations are performed based on the training input feature vector sequence, the training type feature vector sequence and the training position feature vector sequence to obtain a plurality of training latent vector sequences; The generative network includes a second attention layer and a phoneme label layer, and the use of the generative network to calculate a predicted phoneme probability vector sequence and a training phoneme label vector sequence based on the training embedded feature vector and a plurality of training latent vector sequences includes: The training embedded feature vector is encoded using the phoneme label layer to obtain the training phoneme label vector sequence; the sequence lengths of the training acoustic feature vector sequence and the training text feature vector sequence are obtained using the second attention layer, and the predicted phoneme probability vector sequence is calculated based on multiple training latent vector sequences and the sequence lengths.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the training method of the phoneme recognition model described in any one of claims 1 to 4 and the phoneme recognition method described in claim 5 are implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the training method of the phoneme recognition model described in any one of claims 1 to 4 and the phoneme recognition method described in claim 5 are implemented.

Images