Method, apparatus, device, medium and program product for speech recognition

Abstract

According to the embodiment of the invention, a method, a device, equipment, a medium and a program product for voice recognition are provided. In the method, a target text sequence of a natural language is processed into a plurality of first phoneme semantic representations; performing mapping on the plurality of first phoneme semantic representations to obtain at least one statistic for duration distribution of the plurality of first phoneme semantic representations; converting the plurality of first phoneme semantic representations into a plurality of frame-level second phoneme semantic representations according to durations corresponding to the plurality of first phoneme semantic representations on the basis of a specified statistic in the at least one statistic; performing voice prediction on the plurality of second phoneme semantic representations based on a trained machine learning model to obtain a prediction result; and recognizing a voice signal corresponding to the target text sequence based on the prediction result.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and particularly to methods, apparatus, devices, computer-readable storage media, and computer program products for speech recognition. Background Technology

[0002] Text-to-Speech (TTS) is applied to speech synthesis, using neural network design to intelligently convert text into natural speech. Commonly used speech synthesis frameworks include VITS, JETS, and PITS. Among them, the VITS model can generate relatively high-quality speech.

[0003] However, the synthesized speech produced by the VITS model currently suffers from numerous pitch sandhi issues. Therefore, it is hoped that the pitch sandhi problems of the VITS model can be solved to generate higher quality speech. Summary of the Invention

[0004] In a first aspect of this disclosure, a method for speech recognition is provided. The method includes: processing a target text sequence of natural language into a plurality of first phoneme semantic representations; performing mapping on the plurality of first phoneme semantic representations to obtain at least one statistic of the duration distribution of the plurality of first phoneme semantic representations; converting the plurality of first phoneme semantic representations into a plurality of second phoneme semantic representations at the frame level based on a specified statistic from the at least one statistic, according to the duration corresponding to the plurality of first phoneme semantic representations; performing speech prediction on the plurality of second phoneme semantic representations based on a trained machine learning model to obtain a prediction result; and identifying the speech signal corresponding to the target text sequence based on the prediction result.

[0005] In a second aspect of this disclosure, an apparatus for speech recognition is provided. The apparatus includes: a processing module configured to process a target text sequence of natural language into a plurality of first phoneme semantic representations; a mapping module configured to perform mapping on the plurality of first phoneme semantic representations to obtain at least one statistic for the duration distribution of the plurality of first phoneme semantic representations; a conversion module configured to convert the plurality of first phoneme semantic representations into a plurality of second phoneme semantic representations at the frame level based on a specified statistic from the at least one statistic, according to the duration corresponding to the plurality of first phoneme semantic representations; a prediction module configured to perform speech prediction on the plurality of second phoneme semantic representations based on a trained machine learning model to obtain a prediction result; and a recognition module configured to recognize the speech signal of the target text sequence based on the prediction result.

[0006] In a third aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the method of the first aspect.

[0007] In a fourth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the method of the first aspect.

[0008] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0009] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown;

[0012] Figure 2 A schematic diagram of the network structure of the VITS model during the training phase according to the traditional approach is shown.

[0013] Figure 3 A flowchart of a process for speech recognition according to some embodiments of the present disclosure is shown;

[0014] Figure 4 A schematic diagram of the network structure of an improved VITS model according to some embodiments of the present disclosure is shown;

[0015] Figure 5 Block diagrams of apparatus for speech recognition according to some embodiments of the present disclosure are shown; and

[0016] Figure 6 A block diagram of an electronic device capable of implementing one or more embodiments of the present disclosure is shown. Detailed Implementation

[0017] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0018] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] In this document, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information, thereby enabling the user to choose whether to provide personal information to the software or hardware such as electronic devices, applications, servers or storage media that perform the operation of the technical solution disclosed herein, based on the prompt message.

[0023] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, such as a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0025] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0026] A neural network is a machine learning network based on deep learning. A neural network processes input and provides a corresponding output, typically consisting of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the input to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each node processing the input from the layer above.

[0027] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating its parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. In the application phase, the model can be used to process actual inputs based on the trained parameter values ​​to determine the corresponding output.

[0028] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In this example environment 100, a target application 120 is installed on a terminal device 110. In some embodiments, the target application 120 may be an application capable of providing voice recognition.

[0029] In some embodiments, user 140 may interact with target application 120 via terminal device 110 and / or an attachment device of terminal device 110. Further, user 140 may input text information in the interactive window of target application 120 and then receive voice messages from target application 120 corresponding to that text information.

[0030] In some embodiments, terminal device 110 communicates with server device 130 to provision services to target application 120. In environment 100, target TTS model 135 communicates with server device 130. In some examples, server device 130 can assist in providing services to target application 120 by invoking target TTS model 135. Target TTS model 135 can run on other devices / systems different from server device 130 and terminal device 110. Although Figure 1 Only a single target TTS model 135 is shown in the figure, but it is understood that more models may exist depending on the specific application needs. Different models may be configured to determine different speech recognition operations in different ways, or to provide other functions in the target application 120, and so on.

[0031] In some embodiments, the target TTS model 135 may have text-to-speech capabilities. In some embodiments, the target TTS model 135 may include speech synthesis frameworks such as VITS, JETS, and PITS. The target TTS model 135 can be pre-trained from a large amount of data, thereby enabling it to understand semantic information of text modalities and other modalities. A suitable TTS model can be selected according to actual needs.

[0032] The target TTS model 135 can run locally on the terminal device 110 or server device 130, or be deployed on a remote device, in a cloud environment, etc. When running locally, the terminal device 110 or server device 130 can directly provide the model input to the locally installed target TTS model 135 and obtain the model output generated by the target TTS model 135. When running remotely, the terminal device 110 or server device 130 provides data to other devices through communication connections. Other devices determine the model input based on the obtained data and provide this model input to the target TTS model 135. After obtaining the model output of the target TTS model 135, other electronic devices then provide the model output through communication connections with the terminal device 110 or server device 130.

[0033] In some embodiments, terminal device 110 may be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. In some embodiments, terminal device 110 may also support any type of user-facing interface (such as "wearable" circuitry). Server device 130 may be various types of computing systems / servers capable of providing computing power, including but not limited to mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0034] It should be understood that the structure and function of the various elements in environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this disclosure.

[0035] As briefly mentioned earlier, TTS (Text to Speech) technology can intelligently convert text into natural speech streams through the design of neural networks. Traditional TTS technology typically consists of two stages: an acoustic model stage and a vocoder model stage. These two stages use some empirical acoustic features as connections, such as the most commonly used Mel spectrum. However, this two-stage TTS has the following problems:

[0036] 1) The model needs to be trained in two stages, which increases the difficulty and complexity of training.

[0037] 2) This may lead to a mismatch in features between the two-stage models.

[0038] Specifically, if both the vocoder and the acoustic model are trained using Mel spectra, then during inference, the Mel spectra predicted by the acoustic model need to be fed into the vocoder. However, the vocoder has never seen the predicted Mel spectra during training, leading to inconsistencies between the training and inference phases. While training the vocoder using the predicted Mel spectra can mitigate this inconsistency to some extent, it is not as effective as end-to-end training. Furthermore, this type of training requires two phases, waiting for the acoustic model to complete its training before starting the vocoder training.

[0039] To overcome the problems of traditional two-stage training in TTS, several end-to-end speech synthesis frameworks have emerged, such as VITS, JETS, and PITS. The VITS model is used to generate high-quality speech. However, the synthesized speech from the VITS model suffers from numerous pronunciation and tone sandhi issues.

[0040] Figure 2 A schematic diagram of the network structure 200 of the VITS model during the training phase according to the traditional scheme is shown.

[0041] like Figure 2 As shown in network structure 200, the VITS model employs a VAE (Variational Equivalent). The VAE sampling process essentially solves the one-to-many problem of TTS using the VAE paradigm, reducing the common problem of prosody flattening, thus resulting in relatively rich prosody in VITS. Furthermore, the VAE sampling process transforms the one-to-many problem into a many-to-one problem, ensuring that the text samples corresponding to the same audio are different each time. However, it is precisely because of this sampling process, and Flow 210 (Flow is equivalent to a transformation function f that converts the posterior distribution to the prior distribution), that... θ The VAE sampling process leads to tone shifting issues in VITS. VAE sampling yields text representations with relatively high degrees of freedom. If the model is not powerful and robust enough, these high-degree-of-freedom representations can cause tone shifting problems. In other words, sampling results in text representations that the model has not seen before or representations that approximate pronunciation, leading to tone shifting issues during the final synthesis process.

[0042] The VITS model employs a normalizing flow, which can lead to some degree of pitch distortion. While the normalizing flow can alter the data distribution mode, it also requires a sampling process, which in turn introduces pitch distortion issues.

[0043] Furthermore, the pronunciation sandhi problem can also be reduced to the calculation process of averaging over the phoneme range in the KL loss function of the VITS model itself. The KL loss formula of the VITS model (the final loss function formula derived from the normalized flow and KL loss) is shown below:

[0044] kl = logs_p - logs_q - 0.5 (1)

[0045] kl += 0.5*((z_p-m_p)**2)*torch.exp(-2. * logs_p) (2)

[0046] In formula (2), z_p - m_p is calculated to make the output of Flow 210 as close as possible to the output of text encoder 220. However, there is a problem at this point: m_p is a phoneme-level representation that is then expanded to the frame level, meaning there is a lot of repetitive content. But z_p is a fine-grained frame-level representation learned from the Mel spectrum (linear spectrum). Letting m_p learn this fine-grained frame-level representation involves a learning average process. This leads to m_p's semantic learning possibly learning the average of the audio representation, which is not quite the same as the length adjuster of FastSpeech (Fast Speech Synthesis) 1 / 2.

[0047] In FastSpeech, the length adjuster, combined with the decoder 230 model, first extends the phoneme-level text encoder 220 representation to the frame level. Then, the decoder 230 further learns the differences between each frame and its neighboring frames, gradually predicting the Mel spectrum of the target. In VITS, this reverse process, where the loss calculation is performed directly using the text encoder 220 output and the frame-level representation, is completely different. Therefore, due to the learning averaging problem in VITS's KL loss function, VITS may learn problematic text representations, leading to numerous pitch distortions in the final synthesized speech.

[0048] Therefore, in order to generate higher quality speech, it is desirable to solve the problem of tone sandhi in the VITS model.

[0049] According to embodiments of this disclosure, an improved scheme for speech recognition is proposed. The scheme according to embodiments of this disclosure processes a target text sequence of natural language into multiple first phoneme semantic representations. Mapping is performed on the multiple first phoneme semantic representations to obtain at least one statistic for the duration distribution of the multiple first phoneme semantic representations. Based on a specified statistic from the at least one statistic, the multiple first phoneme semantic representations are converted into multiple second phoneme semantic representations at the frame level according to the duration corresponding to the multiple first phoneme semantic representations. Speech prediction is performed on the multiple second phoneme semantic representations based on a trained machine learning model to obtain a prediction result. The speech signal corresponding to the target text sequence is identified based on the prediction result.

[0050] In this way, by optimizing the network structure of VITS, the pronunciation tone sandhi problem of the VITS model can be solved, the robustness of the VITS model can be improved, and the quality of speech synthesis can be ensured without affecting it.

[0051] The following description will continue with reference to the accompanying drawings, which will provide some exemplary embodiments of this disclosure.

[0052] Figure 3A flowchart of a process 300 for speech recognition according to some embodiments of the present disclosure is shown. For ease of discussion, reference will be made to... Figure 1 These embodiments are described in the context of environment 100. These embodiments can be implemented in... Figure 1 The server-side device 130 is used for implementation. In other embodiments, these embodiments may also be implemented in the terminal device 110, or through the cooperation of the terminal device 110 and the server-side device 130. For the purposes of discussion, the following specific embodiments are exemplified by implementation in the server-side device 130.

[0053] In box 310, server device 130 processes the target text sequence of natural language into multiple first phoneme semantic representations. Combined with... Figure 1 The target text sequence may be, for example, text information input by user 140 via target application 120 on terminal device 110. This text information can be sent from terminal device 110 to server device 130.

[0054] In some embodiments, when processing a target text sequence of natural language into multiple first phoneme semantic representations, the server device 130 can decompose the target text sequence into multiple phonemes and encode the multiple phonemes into multiple first phoneme semantic representations respectively.

[0055] Figure 4 A schematic diagram of an example network structure 400 for an improved VITS model according to some embodiments of the present disclosure is shown. Network structure 400 can be implemented in... Figure 1 In environment 100. Specifically, network structure 400 can be implemented as target TTS model 135 in environment 100. Network structure 400 can be used in process 300. In some embodiments, such a VITS model can be a VITS 3 model, that is, the result of two iterations based on the VITS model.

[0056] In network structure 400, server device 130 can decompose the target text sequence into multiple phonemes 410, and then use text encoder 420 to encode the multiple phonemes 410 into multiple first phoneme semantic representations h. text .

[0057] Return to reference Figure 3 In box 320, server device 130 performs mapping on the semantic representations of multiple first phonemes to obtain at least one statistic for the duration distribution of the semantic representations of multiple first phonemes. Combined with... Figure 4 In some embodiments, the mapping layer 430 can be used to represent the semantics of multiple first phonemes h. text Perform the mapping.

[0058] In some embodiments, at least one statistic for the duration distribution of semantic representations of multiple first phonemes may include the mean and / or variance of the duration distribution of semantic representations of multiple first phonemes. It should be understood that at least one statistic for the duration distribution of semantic representations of multiple first phonemes may also include, for example, standard deviation, median, etc., without limitation.

[0059] In some embodiments, when performing mapping on a plurality of first phoneme semantic representations to obtain at least one statistic for the duration distribution of the plurality of first phoneme semantic representations, the server device 130 may perform at least one mapping on the plurality of first phoneme semantic representations to obtain at least one statistic for the duration distribution of the plurality of first phoneme semantic representations, wherein each mapping corresponds to a single statistic among the at least one statistic.

[0060] by Figure 4 For example, mapping layer 430 can be used to perform two mappings on the semantic representations of multiple first phonemes, respectively, to obtain the mean μ of the duration distribution of the semantic representations of multiple first phonemes. θ and variance σ θ .

[0061] refer to Figure 3 In box 330, based on a specified statistic from at least one statistic, the server device 130 converts multiple first phoneme semantic representations into multiple second phoneme semantic representations at the frame level according to the duration corresponding to the multiple first phoneme semantic representations. In some embodiments, the specified statistic may be the mean. In other embodiments, the specified statistic may also be other statistics, such as variance or standard deviation.

[0062] Combination Figure 4 In some embodiments, the length adjuster 440 can be used to expand the multiple first phoneme semantic representations into multiple second phoneme semantic representations at the frame level according to the pronunciation duration corresponding to the multiple first phoneme semantic representations.

[0063] refer to Figure 3 In box 340, server device 130 performs speech prediction on the semantic representations of multiple second phonemes based on a trained machine learning model to obtain the prediction result. The steps in box 340 can be performed during the inference phase (also known as the application phase) of the machine learning model.

[0064] Combination Figure 4 In some embodiments, the machine learning model may include a Transformer-based decoder 450. The process of training the machine learning model will be discussed below.

[0065] refer to Figure 3In box 350, server device 130 identifies the speech signal corresponding to the target text sequence based on the prediction result.

[0066] In some embodiments, a vocoder can be used to identify the speech signal corresponding to the target text sequence based on the prediction result. Combined with Figure 4 The vocoder could be, for example, the HiFi-GAN decoder 460.

[0067] In some embodiments, the machine learning model can be trained by the server device 130 inputting multiple sample phoneme semantic representations into the machine learning model to be trained, to obtain the predicted acoustic implicit representations output by the machine learning model, and training the machine learning model based on the predicted acoustic implicit representations. Figure 4 The Transformer decoder 450 can output an acoustic implicit representation Z based on the semantic representation of multiple sample phonemes in the input.

[0068] In some embodiments, multiple sample phoneme semantic representations can be obtained as follows: The server device 130 can process a sample text sequence of natural language into multiple initial sample phoneme semantic representations. Then, the server device 130 can perform mapping on the multiple initial sample phoneme semantic representations to obtain sample statistics corresponding to specified statistics. Based on the sample statistics, the server device 130 can convert the multiple initial sample phoneme semantic representations into multiple sample phoneme semantic representations at the frame level according to the duration corresponding to the multiple initial sample phoneme semantic representations.

[0069] In such an embodiment, the text encoder 420, mapping layer 430, and length adjuster 440 in the network structure 400 can still be used to obtain semantic representations of multiple sample phonemes.

[0070] In some embodiments, when training a machine learning model based on a predicted acoustic implicit representation, the server device 130 may utilize the mean absolute error (MAE) loss function based on the predicted acoustic implicit representation to train the machine learning model.

[0071] Based on the above embodiments, by adding a Transformer-based decoder 450 to the example network structure 400 of VITS, it can be used to predict the acoustic implicit representation Z encoded by the posterior encoder 470, and the MAE loss function is used for Z prediction. By using the Transformer decoder 450 for Z prediction, the pronunciation distortion problem caused by the sampling operation required in the Flow 480 network can be reduced. Thus, during the training phase, only the alignment module 485 in VITS needs to be learned through Flow 480, while during the inference phase, the box 490 part can be completely ignored, and only the Transformer decoder 450 is used for Z prediction, and then audio synthesis is performed through the HiFi-GAN decoder 460.

[0072] The Transformer decoder 450 can use only the mean μ obtained through the mapping layer 430. θ The corresponding phoneme semantic representation is used as input, without using variance σ. θ Furthermore, no sampling is performed. This method avoids the operations of Flow 480 and sampling in VAE, effectively alleviating the pronunciation distortion problem.

[0073] Furthermore, if the 490-bound portion is removed to avoid the learning averaging problem of the KL loss function used therein, the monotonic alignment search algorithm (MAS) cannot be used for speech and text alignment search. To verify whether the learning averaging problem of the KL loss is causing the pitch distortion problem, the alignment result obtained by forced alignment through MFA can be used to predict phoneme duration, so the 490-bound portion can be completely removed. Removing the 490-bound portion eliminates the need for KL loss, and the trained model does not have the pitch distortion problem. Therefore, this verifies that the use of KL loss, Flow 480, and VAE sampling leads to the pitch distortion problem in the VITS model. However, while this solves the pitch distortion problem, the quality of the final synthesized audio deteriorates.

[0074] Specifically, the deterioration in the quality of the final synthesized audio is due to insufficient gradient backpropagation to the text encoder 420, leading to poor convergence. Previously, using Flow 480 allowed for sufficient gradient backpropagation through its relatively simple network structure, helping the text encoder 420 learn more expressive representations. Therefore, it's still necessary to continue using Flow 480 and the KL loss, although the KL loss can be simplified to some extent.

[0075] Specifically, during training, the variance of VAE sampling can be fixed at 1, without learning, thus reducing model complexity. Simultaneously, because the variance is fixed at 1, the above formula (1) does not need to be calculated in the KL loss, thereby reducing the complexity of the loss function. The reason for fixing the variance at 1 is twofold: 1) Firstly, the original VITS simplified the Flow 480 operation, only changing the mean of the distribution without altering the variance. 2) Secondly, after VITS training, it was found that the variance values ​​of the Flow 480 input and output distributions are basically consistent, so the variance of these two distributions can be fixed at 1. This reduces the complexity of the model and the loss function, alleviating the pronunciation distortion problem.

[0076] Furthermore, after the model is trained, even if speech synthesis is performed via Flow 480 instead of Transformer decoder 450, the fixed variance helps the overall model significantly reduce tone distortion. Moreover, during synthesis via Flow 480, fixing the variance of the VAE to 0 in the output of text encoder 420 also significantly reduces tone distortion compared to a variance of 1. Therefore, if Flow 480 is used for speech synthesis, the variance of the VAE sampling can be fixed to 0. If Transformer decoder 450 is used for speech synthesis, only the phoneme semantic representations corresponding to the mean learned by text encoder 420 can be used as input to Transformer decoder 450.

[0077] Through the embodiments of this disclosure, a Transformer decoder 450 is added to the VITS model to predict the acoustic implicit representation Z, thereby alleviating the pronunciation distortion problem caused by VAE sampling and Flow.

[0078] Furthermore, regarding the use of variance in VAEs, fixing the variance of the two distributions to 1 during training can reduce the complexity of the model and the loss function. Fixing the variance to 0 during inference (i.e., completely disallowing sampling) can significantly reduce pronunciation errors in synthesized speech.

[0079] Figure 5 A schematic structural block diagram of a speech recognition apparatus 500 according to certain embodiments of the present disclosure is shown. The apparatus 500 may be implemented as or included in a server device 130. The various modules / components in the apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0080] As shown in the figure, the device 500 includes a processing module 510 configured to process a target text sequence of natural language into a plurality of first phoneme semantic representations; a mapping module 520 configured to perform mapping on the plurality of first phoneme semantic representations to obtain at least one statistic for the duration distribution of the plurality of first phoneme semantic representations; a conversion module 530 configured to convert the plurality of first phoneme semantic representations into a plurality of second phoneme semantic representations at the frame level based on a specified statistic from the at least one statistic and according to the duration corresponding to the plurality of first phoneme semantic representations; a prediction module 540 configured to perform speech prediction on the plurality of second phoneme semantic representations based on a trained machine learning model to obtain a prediction result; and a recognition module 550 configured to recognize the speech signal of the target text sequence based on the prediction result.

[0081] In some embodiments, the processing module 510 is further configured to decompose the target text sequence into multiple phonemes; and to encode the multiple phonemes into multiple first phoneme semantic representations respectively.

[0082] In some embodiments, the mapping module 520 is further configured to perform at least one mapping on a plurality of first phoneme semantic representations to obtain at least one statistic for the duration distribution of the plurality of first phoneme semantic representations, wherein each mapping corresponds to a single statistic among the at least one statistic.

[0083] In some embodiments, at least one statistic for the duration distribution of semantic representations of multiple first phonemes includes the mean and / or variance of the duration distribution of semantic representations of multiple first phonemes.

[0084] In some embodiments, the apparatus 500 further includes a training module configured to input multiple sample phoneme semantic representations into a machine learning model to be trained to obtain a predicted acoustic implicit representation output by the machine learning model; and to train the machine learning model based on the predicted acoustic implicit representation.

[0085] In some embodiments, the apparatus 500 further includes a sample acquisition module configured to process a sample text sequence of natural language into a plurality of initial sample phoneme semantic representations; perform mapping on the plurality of initial sample phoneme semantic representations to obtain sample statistics corresponding to a specified statistic; and based on the sample statistics, convert the plurality of initial sample phoneme semantic representations into a plurality of sample phoneme semantic representations at the frame level according to the duration corresponding to the plurality of initial sample phoneme semantic representations.

[0086] In some embodiments, the device 500 is also configured to train a machine learning model using a mean absolute error loss function based on the predictive acoustic implicit representation.

[0087] In some embodiments, the machine learning model includes a decoder based on the Transformer model.

[0088] The units and / or modules included in device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units and / or modules in device 500 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0089] Figure 6 A block diagram is shown illustrating an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that... Figure 6 The electronic device 600 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 6 The electronic device 600 shown can be used to achieve Figure 1 Server device 130 or Figure 5 The device 500.

[0090] like Figure 6 As shown, electronic device 600 is in the form of a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 600.

[0091] Electronic device 600 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 620 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 630 can be a removable or non-removable medium and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 600.

[0092] Electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 6 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 620 may include computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0093] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 600 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 600 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0094] Input device 650 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 660 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 600 can also communicate with one or more external devices (not shown) via communication unit 640 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 600, or with any device that enables electronic device 600 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0095] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0096] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0098] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0100] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for speech recognition, comprising: The target text sequence of natural language is processed into multiple first phoneme semantic representations; A mapping is performed on the semantic representations of the plurality of first phonemes to obtain at least one statistic for the duration distribution of the semantic representations of the plurality of first phonemes; Based on a specified statistic among the at least one statistic, the multiple first phoneme semantic representations are converted into multiple second phoneme semantic representations at the frame level according to the duration corresponding to the multiple first phoneme semantic representations. Speech prediction is performed on the semantic representations of the multiple second phonemes based on a trained machine learning model to obtain prediction results; as well as Based on the prediction results, the speech signal corresponding to the target text sequence is identified.

2. The method according to claim 1, wherein processing the target text sequence of natural language into a plurality of first phoneme semantic representations comprises: The target text sequence is decomposed into multiple phonemes; as well as The plurality of phonemes are respectively encoded into the plurality of first phoneme semantic representations.

3. The method of claim 1, wherein performing mapping on the plurality of first phoneme semantic representations to obtain at least one statistic for the duration distribution of the plurality of first phoneme semantic representations comprises: At least one mapping is performed on the plurality of first phoneme semantic representations to obtain at least one statistic for the duration distribution of the plurality of first phoneme semantic representations, wherein each mapping corresponds to a single statistic among the at least one statistic.

4. The method of claim 1, wherein at least one statistic for the duration distribution of the plurality of first phoneme semantic representations includes the mean and / or variance of the duration distribution of the plurality of first phoneme semantic representations.

5. The method of claim 1, wherein the machine learning model is trained in the following manner: Multiple sample phoneme semantic representations are input into the machine learning model to be trained to obtain the predicted acoustic implicit representation output by the machine learning model; and The machine learning model is trained based on the predicted acoustic implicit representation.

6. The method according to claim 5, further comprising obtaining the semantic representations of the plurality of sample phonemes by means of: The sample text sequence of natural language is processed into multiple initial sample phoneme semantic representations; Mapping is performed on the semantic representations of the multiple initial sample phonemes to obtain sample statistics corresponding to the specified statistics; Based on the sample statistics, the semantic representations of the multiple initial sample phonemes are converted into frame-level semantic representations of the multiple sample phonemes according to the duration corresponding to the multiple initial sample phoneme semantic representations.

7. The method of claim 5, wherein training the machine learning model based on the predicted acoustic implicit representation comprises: The machine learning model is trained using the mean absolute error loss function based on the predicted acoustic implicit representation.

8. The method of claim 1, wherein the machine learning model includes a decoder based on a Transformer model.

9. An apparatus for speech recognition, comprising: The processing module is configured to process a target text sequence of natural language into multiple first phoneme semantic representations; The mapping module is configured to perform mapping on the semantic representations of the plurality of first phonemes to obtain at least one statistic for the duration distribution of the semantic representations of the plurality of first phonemes. The conversion module is configured to convert the plurality of first phoneme semantic representations into a plurality of second phoneme semantic representations at the frame level based on a specified statistic among the at least one statistic and according to the duration corresponding to the plurality of first phoneme semantic representations. The prediction module is configured to perform speech prediction on the semantic representations of the plurality of second phonemes based on a trained machine learning model, and obtain prediction results. as well as The recognition module is configured to recognize the speech signal of the target text sequence based on the prediction result.

10. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, the computer program being executable by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 8.

Images