Voice processing model training method, voice processing method, device and apparatus

Abstract

The present disclosure provides a speech processing model training method, a speech processing method, an apparatus and a device, and belongs to the technical field of computers. The method comprises: performing speech coding on a sample speech signal to obtain semantic embedding representation and acoustic embedding representation of the sample speech signal; performing phoneme extraction and phoneme coding on a reference speech text of the sample speech signal to obtain phoneme embedding representation of the reference speech text; training a speech processing model based on the semantic embedding representation, the acoustic embedding representation and the phoneme embedding representation, the speech processing model being used for real-time speech synthesis on an input speech. The method introduces semantic information and acoustic information in the model training process, so that the model can learn cleaner semantic information, more language information can be retained when processing the speech, and the naturalness of the synthesized speech is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method for a speech processing model, a speech processing method, an apparatus, and a device. Background Technology

[0002] A voice dialogue system is a technological system that enables human-computer voice interaction. The voice processing scheme is the core component of a voice dialogue system. Traditional voice processing schemes typically employ a concatenated approach of automatic speech recognition, large language models, and text-to-speech conversion, using text as an intermediate modality to achieve voice processing. However, this approach, using text as an intermediate modality, fails to preserve paralinguistic information in the speech, such as emotion and intonation, resulting in a low degree of naturalness in the synthesized speech. Summary of the Invention

[0003] This disclosure provides a training method for a speech processing model, a speech processing method, an apparatus, and a device. This method introduces semantic and acoustic information during model training, enabling the model to learn cleaner semantic information. When processing speech, it retains more linguistic information, thus improving the naturalness of the synthesized speech.

[0004] According to one aspect of the embodiments of this disclosure, a method for training a speech processing model is provided, the method comprising:

[0005] Speech encoding is performed on the sample speech signal to obtain the semantic embedding representation and acoustic embedding representation of the sample speech signal;

[0006] Phoneme extraction and phoneme encoding are performed on the reference speech text of the sample speech signal to obtain the phoneme embedding representation of the reference speech text;

[0007] Based on the semantic embedding representation, the acoustic embedding representation, and the phoneme embedding representation, a speech processing model is trained, which is used to perform real-time speech synthesis on the input speech.

[0008] According to another aspect of the embodiments of this disclosure, a speech processing method is provided, the method comprising:

[0009] The speech processing model is used to encode the Mel spectrogram of the speech signal to be processed to obtain intermediate speech features. The speech processing model is trained based on the training method of the above speech processing model.

[0010] Based on the speech processing model, the intermediate speech features are semantically embedded to obtain the semantic embedded representation of the speech signal;

[0011] Based on the speech processing model, the intermediate speech features are acoustically embedded to obtain the acoustic embedding representation of the speech signal;

[0012] Based on the speech processing model, the semantic embedding representation and the acoustic embedding are processed to obtain the predicted Mel spectrogram;

[0013] The predicted Mel spectrogram is decoded based on the speech processing model to obtain the speech processing result.

[0014] According to another aspect of the embodiments of this disclosure, a training apparatus for a speech processing model is provided, the apparatus comprising:

[0015] The speech coding unit is configured to perform speech coding on the sample speech signal to obtain a semantic embedding representation and an acoustic embedding representation of the sample speech signal;

[0016] A phoneme coding unit is configured to extract and encode phonemes from a reference speech text of the sample speech signal to obtain a phoneme embedding representation of the reference speech text.

[0017] The training unit is configured to train a speech processing model based on the semantic embedding representation, the acoustic embedding representation, and the phoneme embedding representation, the speech processing model being used to perform real-time speech synthesis on the input speech.

[0018] In some embodiments, the speech coding unit is configured to extract a Mel spectrogram of the sample speech signal; perform speech coding on the Mel spectrogram to obtain intermediate speech features; perform semantic embedding on the intermediate speech features to obtain the semantic embedding representation; and perform acoustic embedding on the intermediate speech features to obtain the acoustic embedding representation.

[0019] In some embodiments, the phoneme encoding unit is configured to extract phonemes from a reference speech text of the sample speech signal to obtain a phoneme sequence corresponding to the reference speech text; and to encode the phoneme sequence to obtain a phoneme embedding representation of the reference speech text.

[0020] In some embodiments, the phoneme encoding unit is further configured to normalize the length of the phoneme sequence to obtain the phoneme sequence of a target length.

[0021] In some embodiments, the training unit is configured to: determine a first loss based on the semantic embedding representation and the phoneme embedding representation, wherein the first loss is a similarity loss between the semantic embedding representation and the phoneme embedding representation; determine a predicted Mel spectrogram based on the semantic embedding representation and the acoustic embedding representation; determine a second loss based on the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal, wherein the second loss is the mean square error between the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal; and train the speech processing model based on the first loss and the second loss.

[0022] In some embodiments, the training unit is configured to sum the semantic embedding representation and the acoustic embedding representation to obtain an intermediate embedding representation; and to decode the intermediate embedding representation to obtain the predicted Mel spectrogram.

[0023] In some embodiments, the phoneme encoding unit is further configured to align the sample speech signal and the reference speech text in a time dimension, so that the morphemes in the reference speech text have the same length as the corresponding speech segments in the sample speech signal.

[0024] According to another aspect of the present disclosure, a voice processing apparatus is provided, the apparatus comprising:

[0025] The speech coding unit is configured to perform speech coding on the Mel spectrogram of the speech signal to be processed based on the speech processing model to obtain intermediate speech features. The speech processing model is trained based on the training method of the above-mentioned speech processing model.

[0026] The semantic embedding unit is configured to perform semantic embedding on the intermediate speech features based on the speech processing model to obtain a semantic embedding representation of the speech signal;

[0027] An acoustic embedding unit is configured to perform acoustic embedding on the intermediate speech features based on the speech processing model to obtain an acoustic embedding representation of the speech signal;

[0028] The speech decoding unit is configured to process the semantic embedding representation and the acoustic embedding based on the speech processing model to obtain a predicted Mel spectrogram;

[0029] The speech decoding unit is further configured to decode the predicted Mel spectrogram based on the speech processing model to obtain the speech processing result.

[0030] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising:

[0031] One or more processors;

[0032] Memory used to store the executable program code of the processor;

[0033] The processor is configured to execute the program code to implement the training method of the aforementioned speech processing model, or to implement the aforementioned speech processing method.

[0034] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the training method of the above-described speech processing model, or to perform the above-described speech processing method.

[0035] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the training method of the above-described speech processing model, or implements the above-described speech processing method.

[0036] This disclosure provides a training scheme for a speech processing model. By introducing semantic and acoustic information during the model training process, the model can learn cleaner semantic information, retain more linguistic information when processing speech, and improve the naturalness of synthesized speech.

[0037] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0038] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0039] Figure 1 This is a schematic diagram illustrating the implementation environment of a training method for a speech processing model according to an exemplary embodiment.

[0040] Figure 2 This is a flowchart illustrating a training method for a speech processing model according to an exemplary embodiment.

[0041] Figure 3 This is a flowchart illustrating another method for training a speech processing model according to an exemplary embodiment.

[0042] Figure 4 This is a flowchart illustrating a method for training a speech processing model according to an exemplary embodiment.

[0043] Figure 5 This is a flowchart illustrating a speech processing method according to an exemplary embodiment.

[0044] Figure 6 This is a block diagram illustrating a training apparatus for a speech processing model according to an exemplary embodiment.

[0045] Figure 7 This is a block diagram illustrating a voice processing apparatus according to an exemplary embodiment.

[0046] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0047] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0048] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0049] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this disclosure are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample speech signals and reference speech texts involved in this disclosure were obtained with full authorization.

[0050] Figure 1 This is a schematic diagram illustrating the implementation environment of a training method for a speech processing model according to an exemplary embodiment. See also: Figure 1 The implementation environment specifically includes: terminal 101 and server 102. Terminal 101 can be connected to server 102 via wireless network or wired network.

[0051] Terminal 101 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), and laptop computer. An application can be installed and run on terminal 101 for playing streaming data, such as multimedia or live data. This application is associated with server 102, which provides background services to terminal 101.

[0052] Terminal 101 can refer to one of a plurality of terminals, and this embodiment uses terminal 101 as an example. Those skilled in the art will know that the number of terminals can be more or less. For example, there can be several terminals, or dozens or hundreds of terminals, or more. This embodiment does not limit the number of terminals or the type of devices.

[0053] Server 102 can be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of servers may be more or fewer, and this disclosure does not limit this. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, server 102 undertakes the main computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the main computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture. Server 102 can be connected to terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of servers may be more or fewer, and this disclosure does not limit this.

[0054] Figure 2 This is a flowchart illustrating a training method for a speech processing model according to an exemplary embodiment, such as... Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0055] In step S201, the sample speech signal is speech encoded to obtain the semantic embedding representation and acoustic embedding representation of the sample speech signal.

[0056] In this embodiment, the sample speech signal is a speech segment used for model training. This sample speech signal can come from various sources, such as human speech, musical clips, etc. The sample speech signal contains rich information, including the speaker's content, intonation, and timbre.

[0057] Semantic embedding is a digital representation of the semantic information contained in a sample speech signal. This semantic embedding can be used to understand the content of speech and perform natural language processing tasks such as speech recognition and semantic understanding. Optionally, the semantic embedding can be in vector form.

[0058] Acoustic embeddings focus on the acoustic features of sample speech signals. They include information such as frequency, amplitude, timbre, and prosody. Acoustic embeddings can be used for tasks such as speaker identification, distinguishing different speech styles, and speech synthesis. Optionally, acoustic embeddings can be in vector form.

[0059] In step S202, phoneme extraction and phoneme encoding are performed on the reference speech text of the sample speech signal to obtain the phoneme embedding representation of the reference speech text.

[0060] In this embodiment, the reference speech text is an accurate transcription of the speech content of the sample speech signal. A phoneme is the smallest unit of speech defined based on its natural properties. Phoneme extraction refers to identifying individual phonemes from the reference speech text. For example, in English, phonemes may include vowels (such as "a", "e", "i", "o", "u") and consonants (such as "b", "c", "d", etc.).

[0061] Alternatively, phoneme extraction can be achieved using linguistic rules and algorithms. These algorithms can analyze character sequences in text and determine the position and type of each phoneme based on specific language patterns and pronunciation rules.

[0062] After phonemes are extracted, they can be encoded, which involves converting them into numerical or vector representations. The encoded phonemes can then be used for subsequent speech processing tasks, such as speech recognition and speech synthesis. Encoding quantifies and represents the features of phonemes, enabling computers to understand and manipulate them.

[0063] Phoneme embedding representation refers to a digital and abstract form of expression obtained by extracting and encoding phonemes from a reference speech text. Phoneme embedding representation embeds the features of phonemes into a specific vector space, allowing different phonemes to be distinguished by differences in their vectors. Optionally, phoneme embedding representation is a phoneme embedding vector.

[0064] In step S203, a speech processing model is trained based on semantic embedding representation, acoustic embedding representation, and phoneme embedding representation. The speech processing model is used to perform real-time speech synthesis on the input speech.

[0065] In this embodiment, semantic embedding reflects the semantic information in the speech signal, that is, the specific meaning and content expressed by the speech. Acoustic embedding focuses on the acoustic characteristics of speech, including frequency, amplitude, timbre, and prosody. Phoneme embedding is the digital representation of phonemes in speech text. Phoneme embedding serves as supervisory information in model training; therefore, by training the speech processing model using these three representations, the model can learn clean and accurate semantic and acoustic information.

[0066] This application provides a training scheme for a speech processing model. By introducing semantic and acoustic information during the model training process, the model can learn cleaner semantic information, retain more language information when processing speech, and improve the naturalness of synthesized speech.

[0067] In some embodiments, speech encoding is performed on the sample speech signal to obtain a semantic embedding representation and an acoustic embedding representation of the sample speech signal, including:

[0068] Extract the Mel spectrogram of the sample speech signal;

[0069] Speech encoding is performed on the Mel spectrogram to obtain intermediate speech features;

[0070] Semantic embedding is performed on the intermediate speech features to obtain a semantic embedding representation;

[0071] Acoustic embedding is performed on the intermediate speech features to obtain the acoustic embedding representation.

[0072] By extracting the Mel spectrogram of the sample speech signal, speech encoding is performed on the Mel spectrogram to obtain intermediate speech features, and then semantic and acoustic embedding is performed to obtain the corresponding embedding representation. This method comprehensively utilizes multiple features of speech, which helps to understand and process speech signals more accurately and improves the performance and effect of speech processing.

[0073] In some embodiments, phoneme extraction and phoneme encoding are performed on the reference speech text of the sample speech signal to obtain a phoneme embedding representation of the reference speech text, including:

[0074] Phoneme extraction is performed on the reference speech text of the sample speech signal to obtain the phoneme sequence corresponding to the reference speech text;

[0075] Phoneme encoding is performed on the phoneme sequence to obtain the phoneme embedding representation of the reference speech text.

[0076] By extracting phonemes from the reference speech text of the sample speech signal to obtain a phoneme sequence, and then performing phoneme encoding to obtain a phoneme embedding representation, the phoneme information in the text can be effectively converted into a digital representation that is easy to process. This provides more accurate and targeted features for subsequent speech processing tasks, which helps to improve the performance and effectiveness of speech processing.

[0077] In some embodiments, the method further includes:

[0078] The length of the phoneme sequence is normalized to obtain the phoneme sequence of the target length.

[0079] Normalizing the length of phoneme sequences helps to ensure that different phoneme sequences have consistent lengths, which facilitates subsequent speech processing tasks and improves processing efficiency and accuracy.

[0080] In some embodiments, a speech processing model is trained based on semantic embedding representation, acoustic embedding representation, and phoneme embedding representation, including:

[0081] Based on semantic embedding representation and phoneme embedding representation, a first loss is determined, which is the similarity loss between semantic embedding representation and phoneme embedding representation;

[0082] Based on semantic and acoustic embedding representations, predictive Mel spectrograms are determined.

[0083] Based on the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal, a second loss is determined, which is the mean square error between the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal.

[0084] The speech processing model is trained based on the first loss and the second loss.

[0085] By determining the semantic and phoneme similarity loss and the mean squared error between the predicted and actual Mel spectrograms, the speech processing model can be trained using these two losses. This approach comprehensively considers multiple speech features, improving the model's performance in semantic understanding, acoustic processing, and other aspects, making the model more accurate and effective.

[0086] In some embodiments, determining the predicted Mel spectrogram based on semantic embedding representation and acoustic embedding representation includes:

[0087] The semantic embedding representation and the acoustic embedding representation are summed to obtain the intermediate embedding representation;

[0088] Decoding the intermediate embedding representation yields the predicted Mel spectrogram.

[0089] By determining the predicted Mel spectrogram, semantic and acoustic information can be comprehensively utilized to provide more accurate spectral feature predictions for speech processing, thereby improving the performance and effectiveness of speech processing models.

[0090] In some embodiments, the method further includes:

[0091] The sample speech signal and the reference speech text are aligned in the time dimension so that the morphemes in the reference speech text have the same length as the corresponding speech segments in the sample speech signal.

[0092] By aligning the sample speech signal and the reference speech text in the time dimension, it can be ensured that the morphemes in the reference speech text have the same length as the corresponding speech segments in the sample speech signal. This helps to improve the accuracy and efficiency of speech processing and provides a better foundation for subsequent speech analysis and processing tasks.

[0093] The above Figure 2 The diagram shown is a flowchart of a training method for a speech processing model according to the present disclosure. The training scheme for the speech processing model provided in this disclosure will be further elaborated below. Figure 3 This is a flowchart illustrating another method for training a speech processing model according to an exemplary embodiment, see [link to flowchart]. Figure 3 This method is performed by an electronic device and includes the following steps:

[0094] In step S301, the sample dataset is obtained.

[0095] In this embodiment, the sample dataset includes multiple sample data pairs. Each sample data pair consists of a sample speech signal and a corresponding reference speech text. The sample speech signal is an actual sound recording, which can be a human voice, a musical clip, or other audio segment. The reference speech text is an accurate description or transcription of the content of the speech signal. For example, if the sample speech signal is a person saying "The weather is nice today," then the corresponding reference speech text is the sentence "The weather is nice today."

[0096] Optionally, this sample dataset is obtained by collecting a large-scale, authorized text-to-speech pair dataset. This dataset covers a variety of contexts and emotional expressions. Through a large number of sample data pairs, the model can learn the correspondence between speech signals and text, as well as the association between different speech features and semantic content.

[0097] In step S302, the sample speech signal and the reference speech text are aligned in the time dimension so that the morphemes in the reference speech text have the same length as the corresponding speech segments in the sample speech signal.

[0098] In this embodiment, temporal alignment is a process of matching speech signals and text on a time scale. Since speech is a time-series data, and text also has its inherent temporal order (e.g., words in a sentence appear sequentially), aligning them on a temporal dimension allows for a better understanding and processing of the relationship between them. By ensuring that the morphemes (such as words and syllables) in the reference speech text are consistent in time length with the corresponding speech segments in the sample speech signal, a more accurate correspondence between text and speech can be established, facilitating subsequent analysis and processing.

[0099] Alternatively, computer devices can use algorithms such as Dynamic Time Warping (DTW) to align text and speech in the time dimension.

[0100] It's important to note that by aligning the sample speech signal and the reference speech text along the temporal dimension, the model can more accurately understand the correspondence between speech and text. For example, in speech recognition tasks, the model can more easily determine the position and duration of each morpheme in the speech signal, thereby improving recognition accuracy. For some speech processing tasks, such as speech synthesis and speech alignment, temporal alignment clarifies the specific position and duration of each morpheme in the text within the speech signal, providing more accurate information for these tasks and facilitating subsequent processing. Furthermore, temporal alignment helps the model better learn the features of speech and text, improving model performance and generalization ability.

[0101] In step S303, the Mel spectrogram of the sample speech signal is extracted.

[0102] In this application embodiment, Mel spectrogram is a method for feature representation of speech signals. Mel spectrograms are typically obtained by transforming the speech signal from the time domain to the frequency domain and then performing a nonlinear transformation of the frequency using the Mel scale. The Mel scale is a frequency scale designed based on human auditory characteristics. The Mel scale has higher resolution in the low-frequency range and lower resolution in the high-frequency range, which better reflects the differences in human ear sensitivity to different frequencies. Mel spectrograms can effectively extract important features from speech signals, such as pitch, timbre, and energy. These features are crucial for various speech processing tasks such as speech recognition, speech synthesis, and speaker recognition.

[0103] Optionally, 80 to 128 Mel bands are typically used when extracting Mel spectra from sample speech signals. Mel bands are used in fields such as speech processing and audio analysis to extract features from audio signals. By calculating information such as the energy distribution of the audio signal across different Mel bands, a more representative feature representation of the audio content, such as the Mel spectrum, can be obtained.

[0104] In step S304, the Mel spectrogram is speech encoded to obtain intermediate speech features.

[0105] In this embodiment, speech encoding of the Mel spectrogram is a process of further converting the information in the Mel spectrogram into an intermediate speech feature representation that is more suitable for subsequent processing. Speech encoding enables the extraction of higher-level, more abstract speech features for use in tasks such as speech recognition and speech synthesis.

[0106] Optionally, speech coding is implemented through a speech encoder. The speech encoder comprises multiple layers of one-dimensional convolutional networks. A one-dimensional convolutional network is a neural network structure suitable for processing sequential data. During speech coding, multiple layers of one-dimensional convolutional networks can progressively extract and abstract features from the Mel spectrogram. Each layer of the convolutional network can learn different levels of speech features, from low-level local frequency features to high-level global speech patterns. Through the stacking of multiple layers, more complex and abstract intermediate speech features can be progressively extracted.

[0107] In step S305, semantic embedding is performed on the intermediate speech features to obtain a semantic embedding representation.

[0108] In this embodiment, semantic embedding of intermediate speech features refers to further mapping these features into a semantic space to obtain a representation of semantic information related to the speech content. The semantic embedding representation obtained through this process can more accurately reflect the specific meaning and semantic content conveyed by the speech signal. Through semantic embedding, speech features can be closely combined with semantic information, providing more valuable input for subsequent tasks such as semantic understanding and natural language processing.

[0109] In some embodiments, the speech processing model includes a linear layer for semantic embedding. Optionally, this linear layer is a fully connected layer. The linear layer (fully connected layer) in the neural network acts as a linear transformation of the input features. During semantic embedding, the linear layer takes intermediate speech features as input and maps them to a new semantic space through linear operations such as weighted summation. Optionally, the linear layer consists of a set of weight matrices and bias vectors. When intermediate speech features are input to the linear layer, each feature value is multiplied by the corresponding weight in the weight matrix, and then the bias vector is added to obtain the output value. By adjusting the values ​​of the weight matrix and bias vector, the linear layer can learn how to convert intermediate speech features into a semantic embedding representation.

[0110] In step S306, the intermediate speech features are acoustically embedded to obtain an acoustic embedding representation.

[0111] In this embodiment, acoustic embedding of intermediate speech features refers to mapping these features into a space that reflects the acoustic characteristics of the speech, thereby obtaining an acoustic embedding representation. The acoustic embedding representation is used to represent the acoustic features of speech, such as frequency, amplitude, timbre, and prosody. Through acoustic embedding, key acoustically related information can be extracted from the speech signal, providing more targeted features for subsequent speech processing tasks.

[0112] In some embodiments, the speech processing model includes a linear layer through which acoustic embedding is performed. Optionally, the linear layer is a fully connected layer.

[0113] In step S307, phoneme extraction is performed on the reference speech text of the sample speech signal to obtain the phoneme sequence corresponding to the reference speech text.

[0114] In this embodiment, the interpretation of phonemes is described in step S202 above, and will not be repeated here. The phoneme sequence is formed by arranging the phonemes in the order they appear in the text. The phoneme sequence represents the most basic structure of the reference speech text at the speech level.

[0115] In some embodiments, the electronic device uses natural speech processing tools to convert reference speech text into a sequence of phonemes.

[0116] In step S308, the length of the phoneme sequence is normalized to obtain a phoneme sequence of the target length.

[0117] In this embodiment, normalizing the length of a phoneme sequence refers to the process of adjusting phoneme sequences of different lengths to have the same target length. Optionally, the electronic device can normalize the length of the phoneme sequence to a fixed length, i.e., the target length, using interpolation or compression techniques.

[0118] In step S309, the phoneme sequence is phoneme encoded to obtain the phoneme embedding representation of the reference speech text.

[0119] In this embodiment, phoneme encoding of a phoneme sequence refers to converting the phoneme sequence into a digital representation more suitable for computer processing and analysis. Phoneme encoding can extract key features from the phoneme sequence and represent these key features as a vector or matrix for use in subsequent speech processing tasks. Through phoneme encoding, the information in the phoneme sequence can be compressed and abstracted, enabling computers to process and understand the content of speech text more efficiently.

[0120] Phoneme embedding is a representation that maps a sequence of phonemes to a low-dimensional vector space. In this vector space, each phoneme is represented as a vector, and the dimension of the vector is typically much smaller than the length of the original phoneme sequence. Phoneme embedding can better reflect the semantic and acoustic features of phonemes, as well as the similarity and correlation between phonemes. Optionally, the dimension of phoneme embedding is typically between 256 and 512.

[0121] In some instances, electronic devices may use multi-layer bidirectional LSTM (Bi-LSTM, Long Short-Term Memory) or Transformer structures to capture contextual information.

[0122] In step S310, a first loss is determined based on the semantic embedding representation and the phoneme embedding representation. The first loss is the similarity loss between the semantic embedding representation and the phoneme embedding representation.

[0123] In this embodiment, semantic embedding representation is a digital representation of the semantic information contained in the sample speech signal. Phoneme embedding representation is a digital representation of the phonemes in the reference speech text of the sample speech signal. Since similarity loss measures the degree of similarity between two vector representations, and semantic embedding representation and phoneme embedding representation both originate from the same speech signal and its reference speech text, they are similar to some extent. Therefore, the similarity loss between semantic embedding representation and phoneme embedding representation is determined as the first loss. By determining the first loss as the similarity loss based on semantic embedding representation and phoneme embedding representation, the speech processing model can learn the relationship between semantic and phoneme features during training, thereby improving the model's performance and accuracy.

[0124] In some embodiments, electronic devices can use various methods to calculate similarity loss, such as cosine similarity, Euclidean distance, etc. Taking cosine similarity as an example, the cosine value of the angle between two vectors is first calculated. The closer the cosine value is to 1, the more similar the two vectors are; the closer the cosine value is to 0, the less similar the two vectors are. By calculating the cosine similarity between semantic embedding representations and phoneme embedding representations and using it as the value of the loss function, the model can be prompted to adjust its parameters during training to make the semantic embedding representations and phoneme embedding representations more similar.

[0125] In step S311, the predicted Mel spectrogram is determined based on the semantic embedding representation and the acoustic embedding representation.

[0126] In this embodiment, since the semantic embedding representation and the acoustic embedding representation are derived from intermediate speech features, which are extracted from the Mel spectrogram of the sample speech signal, a new Mel spectrogram can be predicted based on the semantic embedding representation and the acoustic embedding representation. For ease of description, this is referred to as the predicted Mel spectrogram. In other words, the model utilizes the semantic information in the semantic embedding representation and the acoustic features in the acoustic embedding representation to attempt to learn the relationship between the semantic and acoustic features of speech and the Mel spectrogram. This allows the model to accurately predict the corresponding Mel spectrogram based on the semantic and acoustic features of the input speech signal, thereby providing a more effective feature representation for subsequent speech processing tasks, such as speech synthesis and speech recognition.

[0127] In some embodiments, determining the predicted Mel spectrogram based on semantic and acoustic embedding representations includes: summing the semantic and acoustic embedding representations to obtain an intermediate embedding representation; then decoding the intermediate embedding representation to obtain the predicted Mel spectrogram. By summing the semantic and acoustic embedding representations to obtain an intermediate embedding representation, and then decoding it to determine the predicted Mel spectrogram, the semantic information and acoustic features of speech can be comprehensively utilized, resulting in a more accurate prediction of the Mel spectrogram. This provides a more effective feature representation for speech processing tasks and helps improve the performance and accuracy of tasks such as speech synthesis and recognition.

[0128] In step S312, a second loss is determined based on the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal. The second loss is the mean square error between the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal.

[0129] In this embodiment, the mean squared error (MSE) is used to calculate the average of the squared differences between the predicted and true values. For each frequency point and time frame, the difference between the value in the predicted Mel spectrogram and the corresponding value in the Mel spectrogram of the sample speech signal is calculated, and then these differences are squared and averaged. The MSE effectively measures the overall difference between the predicted and true Mel spectrograms at various frequency points and time frames. By minimizing the MSE, the model can continuously adjust its parameters during training, making the predicted Mel spectrogram closer to the true Mel spectrogram, thereby improving the model's accuracy and generalization ability.

[0130] In step S313, a speech processing model is trained based on the first loss and the second loss.

[0131] In this embodiment, training based on the first and second losses allows the model to simultaneously consider features from multiple aspects, including semantics, phonemes, and acoustics, thereby learning the essential features of speech signals more comprehensively. By continuously adjusting parameters to reduce the loss, the model can gradually improve its ability and accuracy in processing speech signals, better adapting to different speech processing tasks and data.

[0132] It should be noted that, to make the training scheme of the speech processing model provided in this application easier to understand, please refer to [link / reference needed]. Figure 4 As shown, Figure 4 This is a flowchart illustrating a method for training a speech processing model according to an exemplary embodiment. Figure 4 As shown, Mel spectrograms are extracted from the sample speech signal, and features are extracted from the Mel spectrograms to obtain intermediate speech features. Semantic embedding representations and acoustic embedding representations are extracted through two linear layers, respectively. The semantic and acoustic embedding representations are summed and decoded to obtain the predicted Mel spectrogram. Phoneme sequences are extracted from the reference speech text. The length of the phoneme sequence is normalized using a length adjuster. The phoneme sequence is phoneme encoded using a phoneme encoder to obtain phoneme embedding representations. A first loss is determined based on the semantic and phoneme embedding representations. A second loss is determined based on the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal. Finally, the speech processing model is trained based on the first and second losses.

[0133] This application provides a training scheme for a speech processing model. By introducing semantic and acoustic information during the model training process, the model can learn cleaner semantic information, retain more language information when processing speech, and improve the naturalness of synthesized speech.

[0134] Figure 5 This is a flowchart illustrating a speech processing method according to an exemplary embodiment, such as... Figure 5 As shown, the method is performed by an electronic device and includes the following steps:

[0135] In step S501, speech encoding is performed on the Mel spectrogram of the speech signal to be processed based on the speech processing model to obtain intermediate speech features.

[0136] In this embodiment, the speech processing model is trained based on the training method of the speech processing model in the above embodiments. This step is the same as step S304 above, and will not be repeated here.

[0137] In step S502, semantic embedding is performed on the intermediate speech features based on the speech processing model to obtain the semantic embedding representation of the speech signal.

[0138] In this embodiment of the application, this step is the same as step S305 above, and will not be repeated here.

[0139] In step S503, the intermediate speech features are acoustically embedded based on the speech processing model to obtain the acoustic embedding representation of the speech signal.

[0140] In this embodiment of the application, this step is the same as step S306 above, and will not be repeated here.

[0141] In step S504, the semantic embedding representation and acoustic embedding are processed based on the speech processing model to obtain the predicted Mel spectrogram.

[0142] In this embodiment of the application, this step is the same as step S311 above, and will not be repeated here.

[0143] In step S505, the predicted Mel spectrogram is decoded based on the speech processing model to obtain the speech processing result.

[0144] In the embodiments of this application, the speech processing results can take various forms, depending on the speech processing task. For example, in a speech synthesis task, the result might be a synthesized speech waveform; in a speech recognition task, the result might be the recognized text content. The speech processing results obtained by decoding the predicted Mel spectrogram can provide valuable outputs for various practical applications. For example, in fields such as voice assistants, speech translation, and audio processing, these results can help achieve functions such as human-computer interaction, information transmission, and audio optimization.

[0145] This application provides a speech processing scheme that, through a streaming codec structure, significantly reduces system latency and enables real-time speech synthesis. Furthermore, by combining semantic and acoustic representations during model processing, it retains more speech information and improves the naturalness of the synthesized speech.

[0146] Figure 6 This is a block diagram illustrating a training apparatus for a speech processing model according to an exemplary embodiment. Figure 6 As shown, the device includes: a speech encoding unit 601, a phoneme encoding unit 602, and a training unit 603.

[0147] The speech coding unit 601 is configured to perform speech coding on the sample speech signal to obtain a semantic embedding representation and an acoustic embedding representation of the sample speech signal;

[0148] Phoneme coding unit 602 is configured to extract and encode phonemes from a reference speech text of a sample speech signal to obtain a phoneme embedding representation of the reference speech text.

[0149] Training unit 603 is configured to train a speech processing model based on semantic embedding representation, acoustic embedding representation, and phoneme embedding representation. The speech processing model is used to perform real-time speech synthesis on the input speech.

[0150] In some embodiments, the speech coding unit 601 is configured to extract the Mel spectrogram of the sample speech signal; perform speech coding on the Mel spectrogram to obtain intermediate speech features; perform semantic embedding on the intermediate speech features to obtain a semantic embedding representation; and perform acoustic embedding on the intermediate speech features to obtain an acoustic embedding representation.

[0151] In some embodiments, the phoneme encoding unit 602 is configured to extract phonemes from a reference speech text of a sample speech signal to obtain a phoneme sequence corresponding to the reference speech text; and to encode the phoneme sequence to obtain a phoneme embedded representation of the reference speech text.

[0152] In some embodiments, the phoneme encoding unit 602 is further configured to normalize the length of the phoneme sequence to obtain a phoneme sequence of a target length.

[0153] In some embodiments, the training unit 603 is configured to: determine a first loss based on semantic embedding representation and phoneme embedding representation, the first loss being a similarity loss between the semantic embedding representation and the phoneme embedding representation; determine a predicted Mel spectrogram based on semantic embedding representation and acoustic embedding representation; determine a second loss based on the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal, the second loss being the mean square error between the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal; and train a speech processing model based on the first loss and the second loss.

[0154] In some embodiments, the training unit 603 is configured to sum the semantic embedding representation and the acoustic embedding representation to obtain an intermediate embedding representation; and to decode the intermediate embedding representation to obtain a predicted Mel spectrogram.

[0155] In some embodiments, the phoneme encoding unit 602 is further configured to align the sample speech signal and the reference speech text in the time dimension so that the lengths of the morphemes in the reference speech text are consistent with the lengths of the corresponding speech segments in the sample speech signal.

[0156] It should be noted that the speech processing model training device provided in the above embodiments is only an example of the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the speech processing model training device and the speech processing model training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0157] Regarding the training apparatus for the speech processing model in the above embodiments, the specific methods by which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0158] Figure 7 This is a block diagram illustrating a voice processing apparatus according to an exemplary embodiment. Figure 7 As shown, the device includes: a speech encoding unit 701, a semantic embedding unit 702, an acoustic embedding unit 703, and a speech decoding unit 704.

[0159] The speech coding unit 701 is configured to perform speech coding on the Mel spectrogram of the speech signal to be processed based on the speech processing model to obtain intermediate speech features. The speech processing model is trained based on the training method of the above speech processing model.

[0160] The semantic embedding unit 702 is configured to perform semantic embedding on intermediate speech features based on a speech processing model to obtain a semantic embedding representation of the speech signal;

[0161] The acoustic embedding unit 703 is configured to perform acoustic embedding on intermediate speech features based on a speech processing model to obtain an acoustic embedding representation of the speech signal.

[0162] The speech decoding unit 704 is configured to process the semantic embedding representation and acoustic embedding based on the speech processing model to obtain the predicted Mel spectrogram;

[0163] The speech decoding unit 704 is also configured to decode the predicted Mel spectrogram based on the speech processing model to obtain the speech processing result.

[0164] It should be noted that the voice signal processing apparatus provided in the above embodiments is only illustrated by the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the voice signal processing apparatus and the voice processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0165] Regarding the speech signal processing apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0166] In this application embodiment, the electronic device can be a terminal or a server. When the electronic device is a terminal, the terminal acts as the execution subject to implement the technical solution provided in this application embodiment; when the electronic device is a server, the server acts as the execution subject to implement the technical solution provided in this application embodiment; or, the technical solution provided in this application can be implemented through interaction between the terminal and the server. For example, the server trains a speech processing model, the terminal sends the speech signal to be processed to the server, the server processes the speech signal based on the speech processing model, obtains a discrete speech representation, and returns it to the terminal. The terminal obtains the discrete speech representation returned by the server. This application embodiment does not limit this.

[0167] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Typically, the electronic device 800 includes a processor 801 and a memory 802.

[0168] Processor 801 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 801 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 801 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 801 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 801 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0169] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 802 are used to store at least one program code, which is executed by the processor 801 to implement the training method of the speech processing model provided in the method embodiments of this disclosure, or the at least one program code is executed by the processor 801 to implement the speech processing method provided in the method embodiments of this disclosure.

[0170] In some embodiments, the electronic device 800 may optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, and a power supply 808.

[0171] Peripheral device interface 803 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 801 and memory 802. In some embodiments, processor 801, memory 802 and peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 801, memory 802 and peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0172] The radio frequency (RF) circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 804 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 804 can communicate with other electronic devices via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 804 may also include circuitry related to NFC (Near Field Communication), which is not limited herein.

[0173] Display screen 805 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 805 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 801 for processing. In this case, display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 805, which serves as the front panel of the electronic device 800; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of the electronic device 800 or in a folded design; in still other embodiments, display screen 805 may be a flexible display screen, disposed on a curved or folded surface of the electronic device 800. Furthermore, display screen 805 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. Display screen 805 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0174] The camera assembly 806 is used to acquire images or videos. Optionally, the camera assembly 806 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device, and the rear-facing camera is located on the back of the electronic device. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0175] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 801 for processing, or input to the radio frequency circuit 804 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the electronic device 800. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0176] Power supply 808 is used to supply power to various components in electronic device 800. Power supply 808 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 808 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0177] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the electronic device 800, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0178] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 802 including instructions, which can be executed by a processor 801 of an electronic device 800 to complete the training method of the aforementioned speech processing model, or to complete the aforementioned speech processing method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0179] A computer program product includes a computer program that, when executed by a processor, implements the training method of the aforementioned speech processing model, or implements the aforementioned speech processing method.

[0180] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0181] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a speech processing model, characterized in that, The method includes: Speech encoding is performed on the sample speech signal to obtain the semantic embedding representation and acoustic embedding representation of the sample speech signal. The sample speech signal contains the speaker's content, intonation, and timbre. The semantic embedding representation is a digital expression of the semantic information contained in the sample speech signal. The acoustic embedding representation includes information on the frequency, amplitude, timbre, and prosody of the sound. Phoneme extraction and phoneme encoding are performed on the reference speech text of the sample speech signal to obtain the phoneme embedding representation of the reference speech text; Based on the semantic embedding representation and the phoneme embedding representation, a first loss is determined, wherein the first loss is the similarity loss between the semantic embedding representation and the phoneme embedding representation; Based on the semantic embedding representation and the acoustic embedding representation, the predicted Mel spectrogram is determined; Based on the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal, a second loss is determined, wherein the second loss is the mean square error between the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal; Based on the first loss and the second loss, a speech processing model is trained, which is used to perform real-time speech synthesis on the input speech.

2. The training method for the speech processing model according to claim 1, characterized in that, The step of performing speech encoding on the sample speech signal to obtain the semantic embedding representation and acoustic embedding representation of the sample speech signal includes: Extract the Mel spectrogram of the sample speech signal; The Mel spectrogram is speech encoded to obtain intermediate speech features; Semantic embedding is performed on the intermediate speech features to obtain the semantic embedding representation; The intermediate speech features are acoustically embedded to obtain the acoustic embedding representation.

3. The training method for the speech processing model according to claim 1, characterized in that, The step of extracting and encoding phonemes from the reference speech text of the sample speech signal to obtain the phoneme embedding representation of the reference speech text includes: Phoneme extraction is performed on the reference speech text of the sample speech signal to obtain the phoneme sequence corresponding to the reference speech text; Phoneme encoding is performed on the phoneme sequence to obtain the phoneme embedding representation of the reference speech text.

4. The training method for the speech processing model according to claim 3, characterized in that, The method further includes: The length of the phoneme sequence is normalized to obtain the phoneme sequence of the target length.

5. The training method for the speech processing model according to claim 1, characterized in that, The step of determining the predicted Mel spectrogram based on the semantic embedding representation and the acoustic embedding representation includes: The semantic embedding representation and the acoustic embedding representation are summed to obtain the intermediate embedding representation; The intermediate embedding representation is decoded to obtain the predicted Mel spectrum.

6. The training method for the speech processing model according to any one of claims 1-5, characterized in that, The method further includes: The sample speech signal and the reference speech text are aligned in the time dimension so that the morphemes in the reference speech text have the same length as the corresponding speech segments in the sample speech signal.

7. A speech processing method, characterized in that, The method includes: Speech encoding is performed on the Mel spectrogram of the speech signal to be processed based on the speech processing model to obtain intermediate speech features. The speech processing model is trained based on any one of claims 1-6. Based on the speech processing model, the intermediate speech features are semantically embedded to obtain the semantic embedded representation of the speech signal; Based on the speech processing model, the intermediate speech features are acoustically embedded to obtain the acoustic embedding representation of the speech signal; Based on the speech processing model, the semantic embedding representation and the acoustic embedding are processed to obtain the predicted Mel spectrogram; The predicted Mel spectrogram is decoded based on the speech processing model to obtain the speech processing result.

8. A training device for a speech processing model, characterized in that, The device includes: The speech coding unit is configured to perform speech coding on a sample speech signal to obtain a semantic embedding representation and an acoustic embedding representation of the sample speech signal. The sample speech signal contains the speaker's content, intonation, and timbre. The semantic embedding representation is a digital representation of the semantic information contained in the sample speech signal. The acoustic embedding representation includes information on the frequency, amplitude, timbre, and prosody of the sound. A phoneme coding unit is configured to extract and encode phonemes from a reference speech text of the sample speech signal to obtain a phoneme embedding representation of the reference speech text. The training unit is configured to: determine a first loss based on the semantic embedding representation and the phoneme embedding representation, wherein the first loss is a similarity loss between the semantic embedding representation and the phoneme embedding representation; determine a predicted Mel spectrogram based on the semantic embedding representation and the acoustic embedding representation; determine a second loss based on the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal, wherein the second loss is the mean square error between the predicted Mel spectrogram and the Mel spectrogram of the sample speech signal; and train a speech processing model based on the first loss and the second loss, wherein the speech processing model is used to perform real-time speech synthesis on the input speech.

9. A voice processing device, characterized in that, The device includes: The speech coding unit is configured to perform speech coding on the Mel spectrogram of the speech signal to be processed based on a speech processing model to obtain intermediate speech features, wherein the speech processing model is trained based on any one of claims 1-6; The semantic embedding unit is configured to perform semantic embedding on the intermediate speech features based on the speech processing model to obtain a semantic embedding representation of the speech signal; An acoustic embedding unit is configured to perform acoustic embedding on the intermediate speech features based on the speech processing model to obtain an acoustic embedding representation of the speech signal; The speech decoding unit is configured to process the semantic embedding representation and the acoustic embedding based on the speech processing model to obtain a predicted Mel spectrogram; The speech decoding unit is further configured to decode the predicted Mel spectrogram based on the speech processing model to obtain the speech processing result.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the training method of the speech processing model as described in any one of claims 1 to 6, or to implement the speech processing method as described in claim 7.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the speech processing model as described in any one of claims 1 to 6, or perform the speech processing method as described in claim 7.

12. A computer program product comprising a computer program that, when executed by a processor, implements a training method for a speech processing model as described in any one of claims 1 to 6, or implements a speech processing method as described in claim 7.

Images