Voice discrete representation model training method and voice signal processing method

Abstract

The present disclosure provides a speech discrete representation model training method and a speech signal processing method, and belongs to the technical field of computers. The method introduces the reference speech text and reference fundamental frequency information corresponding to the sample speech signal as supervision information to train the speech discrete representation model, so that the trained speech discrete representation model can efficiently and accurately extract the speech discrete representation of the speech signal, and the speech discrete representation can fully reflect the semantic content and fundamental frequency information contained in the original speech signal without losing speaker information and prosody information. Compared with the scheme of clustering to extract speech discrete features, the accuracy and effectiveness of the speech discrete representation are significantly improved, so that the speech discrete features can significantly improve the performance of downstream tasks such as speech synthesis, speech recognition, etc.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a training method for a discrete speech representation model and a method for processing speech signals. Background Technology

[0002] Speech synthesis technology based on large language models can synthesize natural, fluent, and expressive speech, and has broad application prospects in fields such as intelligent voice assistants, audiobooks, and virtual broadcasters. The core of speech synthesis technology lies in its ability to accurately extract discrete representations of speech signals that reflect their characteristics. In other words, the accuracy and effectiveness of these discrete representations directly affect the naturalness and accuracy of the synthesized speech.

[0003] Currently, a common method for extracting discrete speech features is to extract speech features through self-supervised learning and then obtain discrete speech representations through clustering.

[0004] However, the discrete speech features extracted by the above methods will lose a lot of information, such as speaker information and prosodic information, which will lead to various problems in downstream speech synthesis tasks, such as unclear pronunciation and reduced naturalness. Summary of the Invention

[0005] This disclosure provides a training method for a discrete speech representation model and a speech signal processing method. Compared with clustering-based methods for extracting discrete speech features, this method significantly improves the accuracy and effectiveness of discrete speech representation, enabling discrete speech features to significantly enhance the performance of downstream tasks such as speech synthesis and speech recognition.

[0006] According to one aspect of the embodiments of this disclosure, a method for training a discrete speech representation model is provided, the method comprising:

[0007] Acquire sample speech signals, reference speech text, and reference baseband information, wherein the reference speech text is the correct text content corresponding to the sample speech signal, and the reference baseband information includes the actual baseband characteristics of the sample speech signal;

[0008] The sample speech signal is input into the speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector;

[0009] The speech discrete representation model is trained based on the semantic continuous vector, the fundamental frequency continuous vector, the reference speech text, and the reference fundamental frequency information.

[0010] According to another aspect of the embodiments of this disclosure, a method for processing speech signals is provided, the method comprising:

[0011] The speech signal to be processed is input into the speech discrete representation model, which is trained using the above-mentioned training method for speech discrete representation models.

[0012] Based on the deep neural network in the discrete speech representation model, the speech signal is converted into hidden layer features in a continuous space;

[0013] Based on the deep learning network in the discrete speech representation model, feature extraction is performed on the hidden layer features to obtain the speech features of the speech signal in continuous space.

[0014] The speech features are vector quantized and encoded to obtain a discrete speech representation of the speech signal.

[0015] According to another aspect of the embodiments of this disclosure, a training apparatus for a discrete speech representation model is provided, the apparatus comprising:

[0016] The acquisition unit is configured to acquire a sample speech signal, a reference speech text, and reference baseband information, wherein the reference speech text is the correct text content corresponding to the sample speech signal, and the reference baseband information includes the actual baseband characteristics of the sample speech signal.

[0017] The feature extraction unit is configured to input the sample speech signal into a speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector.

[0018] The training unit is configured to train the speech discrete representation model based on the semantic continuous vector, the fundamental frequency continuous vector, the speech text, and the reference fundamental frequency information.

[0019] In some embodiments, the feature extraction unit includes:

[0020] The first extraction subunit is configured to extract features from the sample speech signal input to the speech discrete representation model to obtain the speech features of the sample speech signal in continuous space.

[0021] The vectorization subunit is configured to perform vector quantization encoding on the speech features to obtain a quantized feature vector.

[0022] The second extraction subunit is configured to extract the semantic continuous vector and the fundamental frequency continuous vector from the quantized feature vector.

[0023] In some embodiments, the first extraction subunit is configured to convert the sample speech signal input to the speech discrete representation model into hidden layer features in the continuous space based on the deep neural network in the speech discrete representation model; and to extract features from the hidden layer features based on the deep learning network in the speech discrete representation model to obtain the speech features of the sample speech signal in the continuous space.

[0024] In some embodiments, the second extraction subunit is configured to perform semantic decoding on the quantized feature vector based on the semantic decoding module to obtain the semantic continuous vector; and to perform fundamental frequency decoding on the quantized feature vector based on the fundamental frequency decoding module to obtain the fundamental frequency continuous vector.

[0025] In some embodiments, the training unit is configured to: determine a first loss based on the semantic continuous vector and the reference speech text corresponding to the sample speech signal, wherein the first loss indicates the continuous temporal classification loss between the semantic continuous vector and the reference semantic text; determine a second loss based on the fundamental frequency continuous vector and the reference fundamental frequency information, wherein the second loss indicates the mean squared error loss between the fundamental frequency continuous vector and the reference fundamental frequency information; determine a third loss based on the semantic continuous vector and the hidden layer features of the sample speech signal in continuous space, wherein the third loss indicates the mean squared error loss between the semantic continuous vector and the hidden layer features; and train the speech discrete representation model based on the first loss, the second loss, and the third loss.

[0026] In some embodiments, the apparatus further includes:

[0027] The feature extraction unit is further configured to extract features from the reference speech text corresponding to the sample speech signal to obtain the first text features;

[0028] The feature alignment unit is configured to perform forced alignment of the first text feature based on the forced alignment module to obtain the second text feature;

[0029] The training unit is further configured to determine a fourth loss based on the second text features and the speech features, the fourth loss being used to represent the relative entropy loss between the second text features and the speech features;

[0030] The training unit is also configured to train the discrete speech representation model based on the four losses.

[0031] According to another aspect of the present disclosure, a speech signal processing apparatus is provided, the apparatus comprising:

[0032] The input unit is configured to input the speech signal to be processed into the speech discrete representation model, which is trained by the above-mentioned speech discrete representation model training method.

[0033] The first processing unit is configured to convert the speech signal into hidden layer features in a continuous space based on the deep neural network in the speech discrete representation model.

[0034] The second processing unit is configured to extract features from the hidden layer features based on the deep learning network in the speech discrete representation model to obtain the speech features of the speech signal in continuous space.

[0035] The vectorization unit is configured to perform vector quantization encoding on the speech features to obtain a discrete speech representation of the speech signal.

[0036] According to another aspect of the embodiments of this disclosure, an electronic device is provided, the electronic device comprising:

[0037] One or more processors;

[0038] Memory used to store the executable program code of the processor;

[0039] The processor is configured to execute the program code to implement the training method of the above-mentioned discrete speech representation model, or to implement the above-mentioned speech signal processing method.

[0040] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is able to perform the training method of the above-described discrete speech representation model or the processing method of the above-described speech signal.

[0041] According to another aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the above-described training method for a discrete speech representation model or the above-described speech signal processing method.

[0042] This disclosure provides a training scheme for a discrete speech representation model. By introducing reference speech text and reference fundamental frequency information corresponding to the sample speech signal as supervision information during training, the trained discrete speech representation model can efficiently and accurately extract the discrete speech representation of the speech signal. Furthermore, this discrete speech representation can fully reflect the semantic content and fundamental frequency information contained in the original speech signal without losing speaker information or prosodic information. Compared to clustering-based methods for extracting discrete speech features, this significantly improves the accuracy and effectiveness of discrete speech representation, enabling discrete speech features to significantly enhance the performance of downstream tasks such as speech synthesis and speech recognition.

[0043] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0044] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0045] Figure 1 This is a schematic diagram illustrating the implementation environment of a training method for a discrete speech representation model according to an exemplary embodiment.

[0046] Figure 2 This is a flowchart illustrating a training method for a discrete speech representation model according to an exemplary embodiment.

[0047] Figure 3 This is a flowchart illustrating another method for training a discrete speech representation model according to an exemplary embodiment.

[0048] Figure 4 This is a flowchart illustrating a training method for a discrete speech representation model according to an exemplary embodiment.

[0049] Figure 5 This is a flowchart illustrating a method for processing speech signals according to an exemplary embodiment.

[0050] Figure 6 This is a flowchart of a method for processing speech signals according to an exemplary embodiment.

[0051] Figure 7 This is a block diagram illustrating a training apparatus for a discrete speech representation model according to an exemplary embodiment.

[0052] Figure 8 This is a block diagram illustrating a speech signal processing apparatus according to an exemplary embodiment.

[0053] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0055] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0056] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this disclosure are authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample speech signals and reference speech texts involved in this disclosure were obtained with full authorization.

[0057] Figure 1 This is a schematic diagram illustrating the implementation environment of a training method for a discrete speech representation model according to an exemplary embodiment. See also Figure 1 The implementation environment specifically includes: terminal 101 and server 102. Terminal 101 can be connected to server 102 via wireless network or wired network.

[0058] Terminal 101 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), and laptop computer. An application can be installed and run on terminal 101 for playing streaming data, such as multimedia or live data. This application is associated with server 102, which provides background services to terminal 101.

[0059] Terminal 101 can refer to one of a plurality of terminals, and this embodiment uses terminal 101 as an example. Those skilled in the art will know that the number of terminals can be more or less. For example, there can be several terminals, or dozens or hundreds of terminals, or more. This embodiment does not limit the number of terminals or the type of devices.

[0060] Server 102 can be at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of servers may be more or fewer, and this disclosure does not limit this. Of course, server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, server 102 undertakes the main computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the main computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture. Server 102 can be connected to terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of servers may be more or fewer, and this disclosure does not limit this.

[0061] Figure 2 This is a flowchart illustrating a training method for a discrete speech representation model according to an exemplary embodiment, such as... Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0062] In step S201, the sample speech signal, the reference speech text, and the reference baseband information are acquired.

[0063] In this embodiment of the disclosure, the sample speech signal refers to the speech data used for training and testing, the reference speech text is the correct text content corresponding to the sample speech signal, and the reference baseband information includes the actual baseband characteristics of the sample speech signal.

[0064] In step S202, the sample speech signal is input into the speech discrete representation model to obtain the semantic continuous vector and the fundamental frequency continuous vector.

[0065] In this embodiment, the discrete speech representation model is a model to be trained. This discrete speech representation model is used to process the input speech signal and output a discrete speech representation. The discrete speech representation model includes a module for feature extraction. This module can extract features from the sample speech signal of the input discrete speech representation, thereby obtaining the speech features of the sample speech signal. These speech features are speech feature vectors of the sample speech signal in continuous space.

[0066] In step S203, a speech discrete representation model is trained based on semantic continuous vectors, fundamental frequency continuous vectors, reference speech text, and reference fundamental frequency information.

[0067] In this embodiment, reference speech text and reference fundamental frequency information corresponding to the sample speech signal are used as supervision information. The training loss is determined based on the difference between the semantics contained in the reference speech text and the semantic continuous vector, and the difference between the fundamental frequency contained in the reference fundamental frequency information and the fundamental frequency continuous vector. The parameters of the speech discrete representation model are then adjusted based on the training loss. The trained speech discrete representation model is obtained by repeatedly adjusting the parameters of the speech discrete representation model.

[0068] This disclosure provides a training scheme for a discrete speech representation model. By introducing reference speech text and reference fundamental frequency information corresponding to the sample speech signal as supervision information during training, the trained discrete speech representation model can efficiently and accurately extract the discrete speech representation of the speech signal. Furthermore, this discrete speech representation can fully reflect the semantic content and fundamental frequency information contained in the original speech signal without losing speaker information or prosodic information. Compared to clustering-based methods for extracting discrete speech features, this significantly improves the accuracy and effectiveness of discrete speech representation, enabling discrete speech features to significantly enhance the performance of downstream tasks such as speech synthesis and speech recognition.

[0069] In some embodiments, the sample speech signal is input into the speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector, including:

[0070] Feature extraction is performed on the sample speech signal of the input speech discrete representation model to obtain the speech features of the sample speech signal in continuous space;

[0071] The speech features are vector quantized and encoded to obtain quantized feature vectors;

[0072] Extract semantic continuous vectors and fundamental frequency continuous vectors from quantized feature vectors.

[0073] By inputting sample speech signals into a speech discrete representation model, semantic continuous vectors and fundamental frequency continuous vectors are obtained through feature extraction, vector quantization encoding, and corresponding extraction operations. This helps to effectively represent speech signals in different dimensions, making them easier to use in subsequent processing.

[0074] In some embodiments, feature extraction is performed on the sample speech signal of the input speech discrete representation model to obtain the speech features of the sample speech signal in continuous space, including:

[0075] Based on the deep neural network in the discrete speech representation model, the sample speech signal input to the discrete speech representation model is converted into hidden layer features in continuous space.

[0076] Based on the deep learning network in the discrete speech representation model, feature extraction is performed on the hidden layer features to obtain the speech features of the sample speech signal in continuous space.

[0077] By using a deep neural network in the discrete speech representation model, the sample speech signal is converted into hidden features in a continuous space, and further feature extraction is performed on the hidden features to obtain the speech features of the sample speech signal in the continuous space. This can more effectively mine the intrinsic information of the speech signal, improve the accuracy and richness of speech features, and provide better feature representation for subsequent speech processing tasks.

[0078] In some embodiments, extracting semantically continuous vectors and fundamental frequency continuous vectors from quantized feature vectors includes:

[0079] Based on the semantic decoding module, the quantized feature vector is semantically decoded to obtain a semantically continuous vector;

[0080] Based on the baseband decoding module, the quantized feature vector is decoded using the baseband to obtain a continuous baseband vector.

[0081] By performing semantic decoding on the quantized feature vector to obtain a semantically continuous vector, and by performing fundamental frequency decoding on the quantized feature vector to obtain a fundamental frequency continuous vector, more expressive semantic information and fundamental frequency information can be extracted from the quantized features, providing richer and more accurate features for speech processing, which helps to improve the performance and effectiveness of speech processing tasks.

[0082] In some embodiments, a speech discrete representation model is trained based on semantic continuous vectors, fundamental frequency continuous vectors, reference speech text, and reference fundamental frequency information, including:

[0083] Based on the semantic continuous vector and the reference speech text corresponding to the sample speech signal, a first loss is determined. The first loss is used to indicate the continuous temporal classification loss between the semantic continuous vector and the reference semantic text.

[0084] Based on the fundamental frequency continuous vector and the reference fundamental frequency information, a second loss is determined. The second loss is used to indicate the mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information.

[0085] Based on semantic continuous vectors and hidden layer features of sample speech signals in continuous space, a third loss is determined. The third loss is used to indicate the mean squared error loss between semantic continuous vectors and hidden layer features.

[0086] A discrete speech representation model is trained based on the first loss, the second loss, and the third loss.

[0087] By determining the first loss based on semantic continuous vectors and reference semantic text, the second loss based on fundamental frequency continuous vectors and reference fundamental frequency information, and the third loss based on semantic continuous vectors and hidden layer features, and training the speech discrete representation model based on these three losses, the model can be optimized from multiple perspectives such as semantics, fundamental frequency, and the relationship with hidden layer features. This improves the accuracy of the model in semantic understanding of speech signals, fundamental frequency prediction, and feature extraction, thereby enhancing the performance of the speech discrete representation model.

[0088] In some embodiments, the method further includes:

[0089] Feature extraction is performed on the reference speech text corresponding to the sample speech signal to obtain the first text feature;

[0090] The first text feature is forcibly aligned using the forced alignment module to obtain the second text feature;

[0091] Based on the second text features and speech features, a fourth loss is determined, which is used to represent the relative entropy loss between the second text features and speech features;

[0092] A discrete speech representation model is trained based on four loss methods.

[0093] The first text feature is obtained by extracting features from the reference speech text corresponding to the sample speech signal. The second text feature is obtained by processing it through the forced alignment module. The fourth loss is determined based on the second text feature and the speech feature. The speech discrete representation model is then trained based on the fourth loss. This allows the model to achieve more accurate alignment between text and speech, thereby improving the accuracy and performance of the model in speech processing.

[0094] The above Figure 2 The diagram shows a flowchart of a training method for a discrete speech representation model according to this disclosure. The training scheme for the discrete speech representation model provided in this disclosure will be further elaborated below. Figure 3 This is a flowchart illustrating another training method for a discrete speech representation model according to an exemplary embodiment, see [link to flowchart]. Figure 3 This method is performed by an electronic device and includes the following steps:

[0095] In step S301, the sample speech signal, the reference speech text, and the reference baseband information are acquired.

[0096] In this embodiment of the disclosure, the sample speech signal refers to speech data used for training and testing. The sample speech signal is continuous, meaning it is uninterrupted in time. Optionally, the sample speech signal contains rich information, such as the speaker's speech content, intonation, and emotion. The speech signal changes over time and has obvious time-series characteristics. The reference speech text is the correct text content corresponding to the sample speech signal. In other words, the reference speech text explicitly expresses the semantics of the sample speech signal in natural language. The reference fundamental frequency information is usually obtained by analyzing the sample speech signal or by labeling known correct fundamental frequencies. The reference fundamental frequency information represents the actual fundamental frequency characteristics of the sample speech signal.

[0097] In step S302, based on the deep neural network in the speech discrete representation model, the sample speech signal input to the speech discrete representation model is converted into hidden layer features in continuous space.

[0098] In this embodiment, the discrete speech representation model includes a pre-trained model, which is a deep neural network model. A deep neural network is a machine learning model with multiple hidden layers, capable of automatically learning complex feature representations of input data. A deep neural network typically consists of an input layer, multiple hidden layers, and an output layer. The input layer receives sample speech signals, the hidden layers extract features through nonlinear transformations, and the output layer produces the final output. Each hidden layer contains a large number of neurons, which are connected by weights to perform weighted summation and activation function processing on the input content.

[0099] This pre-trained model is used to convert the input speech signal into hidden layer vectors in a continuous space while preserving information from the original speech. Hidden layer features are typically more abstract and higher-level than the original speech signal, containing some key information about the speech signal, but they are not directly interpretable. This allows hidden layer features in the continuous space to provide richer information, which helps improve the performance and accuracy of speech processing systems.

[0100] It's important to note that continuous space refers to a state where the speech signal changes continuously in both time and amplitude. In continuous space, the time axis of the speech signal is continuous, without any discontinuities. This means that the speech signal has a definite value at any given moment, and the changes between adjacent moments are smooth. Furthermore, the amplitude of the speech signal also changes continuously. Amplitude represents the intensity of sound, and in continuous space, amplitude can take any real value. For example, the volume of a sound can change continuously from very faint to very loud, without obvious jumps or discrete values.

[0101] In contrast to continuous space is discrete space, which represents the state after a continuous speech signal has been converted into a discrete digital signal through processes such as sampling and quantization. Sampling discretizes the continuous speech signal along the time axis, taking a sample value at regular time intervals. This time interval is called the sampling period, and the sampling frequency is the number of sampling points per unit time. Quantization converts continuous amplitude values ​​into a finite number of discrete amplitude values. Through quantization, the amplitude of the speech signal can be represented digitally, typically using binary numbers.

[0102] In step S303, based on the deep learning network in the speech discrete representation model, the hidden layer features are extracted to obtain the speech features of the sample speech signal in continuous space.

[0103] In this embodiment, a deep learning network is a machine learning model with a multi-layered structure, capable of automatically learning complex patterns and features from large amounts of data. In speech processing, deep learning networks typically include convolutional neural networks, recurrent neural networks, long short-term memory networks, etc., which can effectively process the time-series characteristics and spectral features of speech signals. This embodiment does not limit the network structure of the deep learning network. The deep learning network extracts deep features from the hidden layer features of the sample speech signal to obtain highly compressed speech features in a continuous space.

[0104] The purpose of extracting features from hidden layer features is to further mine and refine more valuable speech features so that they can be better used for subsequent speech processing tasks, such as speech recognition, speech synthesis, and speech emotion analysis.

[0105] Optionally, the hidden layer features can be extracted by performing linear or nonlinear transformations on the hidden layer features to map them to a new feature space, making the features more compact, independent and discriminative.

[0106] In step S304, the speech features are vector quantized and encoded to obtain quantized feature vectors.

[0107] In the embodiments disclosed herein, Vector Quantization (VQ) is a widely used technique in the field of signal processing, especially playing an important role in speech and image coding. The core idea of ​​VQ is to represent the input signal vector using a finite number of representative codewords, that is, to map a high-dimensional signal space to a relatively low-dimensional codebook space.

[0108] For example, a speech signal can be viewed as a vector composed of multiple feature parameters. Using VQ (Voice-Q Test), a pre-designed codebook is found, containing several codewords. Each codeword is also a vector representing a specific signal pattern. The input speech vector is compared with each codeword in the codebook, and the closest codeword is used to represent the input speech vector.

[0109] It's important to note that the key to vector quantization coding is the codebook. A codebook is a collection of codewords, each representing a specific signal vector. Codebook design typically employs clustering algorithms to cluster a large number of training data vectors, resulting in different codebook regions. When a new input signal vector arrives, the distance between this input signal vector and each codeword in the codebook is calculated. The codeword with the smallest distance is selected as the quantized result of the input signal vector. This process maps the input signal vector to a specific codeword in the codebook space. The quantized codeword can be encoded using a smaller number of bits. For example, if there are 256 codewords in the codebook, each codeword can be represented using 8 bits. This achieves data compression.

[0110] After vector quantization encoding, the original speech feature vector is replaced by a codeword in the codebook; this codeword is the quantized feature vector. The quantized feature vector retains the main information of the original speech features but is represented in a less dimensional and more concise form.

[0111] It should be noted that when performing vector quantization encoding on speech features, we can obtain both quantized feature vectors (or simply quantized feature vectors) and discrete speech representations. The difference lies in that quantized feature vectors are obtained by vectorizing (e.g., clustering) speech features in a continuous space, while discrete speech representations are obtained by mapping quantized feature vectors onto codewords in a codebook. During the model training phase, quantized feature vectors are used for model training.

[0112] Optionally, vector quantization encoding is implemented through a vectorization module, which discretizes continuous feature vectors to obtain corresponding discrete representations. This vectorization module can be constructed based on a deep learning network, and the embodiments of this disclosure do not limit the structure of the vectorization module. Optionally, the vectorization module can also restore the discrete representations back to continuous feature vectors.

[0113] It should be noted that since the quantized feature vector retains the main information of the original speech features, semantic continuous vectors and fundamental frequency continuous vectors can be extracted from it. The semantic continuous vector represents the semantic information in speech. Semantics refers to the meaning and content expressed by language. The semantic information of speech includes multiple levels such as vocabulary, grammar, and semantic relationships. The fundamental frequency continuous vector represents the fundamental frequency information in speech. Fundamental frequency is the lowest frequency component in a sound, also known as the fundamental tone. In speech, the fundamental frequency determines the pitch. Different people have different fundamental frequencies when they speak, and even the fundamental frequency of the same person changes under different emotions and tones.

[0114] In step S305, based on the semantic decoding module, the quantized feature vector is semantically decoded to obtain a semantically continuous vector.

[0115] In this embodiment, the semantic decoding module is a module built based on a deep learning network. The semantic decoding module is used during the model training phase to convert quantized feature vectors into semantically related continuous vectors, referred to as semantically continuous vectors. Semantically continuous vectors can more directly represent the semantic content of speech signals. Compared to quantized feature vectors, semantically continuous vectors have continuity in the semantic space, meaning that adjacent vectors also have semantic similarity and coherence.

[0116] In step S306, the quantized feature vector is decoded using the baseband decoding module to obtain a continuous baseband vector.

[0117] In this embodiment, the fundamental frequency decoding module is a module built based on a deep learning network. The fundamental frequency decoding module is used during the model training phase to convert quantized feature vectors into fundamental frequency-related continuous vectors, simply referred to as fundamental frequency continuous vectors. These fundamental frequency continuous vectors can more directly represent the fundamental frequency features of the speech signal. Compared to quantized feature vectors, the fundamental frequency continuous vectors are numerically continuously changing and can reflect the dynamic changes in the fundamental frequency in the speech signal.

[0118] In step S307, a first loss is determined based on the semantic continuous vector and the reference speech text. The first loss is used to indicate the continuous temporal classification loss between the semantic continuous vector and the reference semantic text.

[0119] In this embodiment, the reference speech text corresponding to the sample speech signal is used as supervision information for model training. The continuous temporal classification loss between the semantic continuous vector and the reference semantic text is determined as the first loss. The continuous temporal classification loss is a loss function commonly used in sequence-to-sequence learning tasks and is suitable for scenarios such as speech recognition. In this embodiment, the first loss mainly considers the alignment relationship between the input sequence (the semantic continuous vector corresponding to the speech signal) and the output sequence (the reference semantic text), as well as the classification accuracy at different time steps.

[0120] Optionally, the calculation process of continuous temporal classification loss includes the following steps:

[0121] First, for an input semantically continuous vector, a series of possible output sequences are predicted, which may contain repeated characters or whitespace characters.

[0122] Then, a dynamic programming algorithm is used to find the optimal alignment between the input sequence and the reference semantic text.

[0123] Finally, the loss value is calculated based on the alignment results. The loss value reflects the degree of difference between the model's prediction and the reference semantic text.

[0124] In step S308, a second loss is determined based on the fundamental frequency continuous vector and the reference fundamental frequency information. The second loss is used to indicate the mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information.

[0125] In this embodiment, reference fundamental frequency information is used as supervisory information for model training. The mean squared error (MSE) loss between the continuous fundamental frequency vector and the reference fundamental frequency information is determined as the second loss. MSE is a commonly used loss function used to measure the difference between two numerical sequences. In this embodiment, the second loss is used to calculate the MSE loss between the continuous fundamental frequency vector and the reference fundamental frequency information. Optionally, for each time point, the square of the difference between the corresponding fundamental frequency value in the continuous fundamental frequency vector and the corresponding fundamental frequency value in the reference fundamental frequency information is first calculated, and then the squared differences over all time points are summed and averaged. This second loss serves as an indicator to guide the model in adjusting parameters during training to reduce the difference between the continuous fundamental frequency vector and the reference fundamental frequency information. By minimizing the second loss, the speech discrete representation model can gradually improve its predictive ability for the fundamental frequency features of speech signals.

[0126] In step S309, a third loss is determined based on the semantic continuous vector and the hidden layer features of the sample speech signal in the continuous space. The third loss is used to indicate the mean squared error loss between the semantic continuous vector and the hidden layer features.

[0127] In this embodiment, the semantic continuous vector is derived from the quantized feature vector, which is obtained from the speech features of the sample speech signal in continuous space. These speech features are obtained from the hidden layer features. Therefore, the purpose of determining the third loss is to measure the accuracy of the model in semantic feature extraction and transformation. By calculating the difference between the semantic continuous vector and the hidden layer features, the model's ability to learn and express speech semantics can be evaluated. This third loss, as an indicator, can guide the model to adjust its parameters during training to reduce the difference between the semantic continuous vector and the hidden layer features. By continuously optimizing the third loss, the speech discrete representation model can better learn the semantic features of the speech signal, improving its understanding and expression of semantics.

[0128] In step S310, feature extraction is performed on the reference speech text corresponding to the sample speech signal to obtain the first text feature.

[0129] In this embodiment of the disclosure, the main purpose of feature extraction from the reference speech text is to transform the text from the original character sequence into a more abstract and representative feature representation, which, for ease of description, is referred to as the first text feature. The first text feature can better reflect the semantic, grammatical, and structural information of the text, facilitating subsequent processing and analysis.

[0130] Optionally, feature extraction of the reference speech text is achieved through a text feature extraction module. This text feature extraction module is built based on a deep learning network, and the structure of the text feature extraction module is not limited in this embodiment. This text feature extraction module is used for text feature extraction.

[0131] For example, taking word embedding as an example, the reference speech text is first segmented into individual words or phrases. Then, a pre-trained word embedding model is used to convert each word into a corresponding vector representation. Finally, these word vectors are combined or further processed to obtain the feature representation of the entire reference speech text, i.e., the first text feature.

[0132] In step S311, the first text feature is forcibly aligned based on the forced alignment module to obtain the second text feature.

[0133] In this embodiment, a forced alignment module is used to precisely align text and speech signals. The forced alignment module can determine the corresponding temporal position of each element in the text (such as words, phonemes, etc.) in the speech signal. This forced alignment module is built based on a deep learning network, and this embodiment does not limit the structure of the forced alignment module. Compared to the first text feature, the second text feature more accurately reflects the alignment relationship between text and speech. The second text feature can provide more accurate text information for subsequent speech processing tasks.

[0134] Alternatively, forced alignment can be achieved using methods such as speech recognition technology and dynamic programming algorithms. First, preliminary speech recognition is performed on the input speech signal to obtain a possible text transcription result. Then, this transcription result is compared with known first text features, and the optimal alignment method is found through continuous adjustment and optimization.

[0135] In step S312, a fourth loss is determined based on the second text features and the speech features. The fourth loss is used to represent the relative entropy loss between the second text features and the speech features.

[0136] In this embodiment, the second text feature is obtained by forcibly aligning the first text feature. Speech features are a set of numerical values ​​extracted from sample speech signals that reflect the essential attributes of the speech signal. Therefore, by determining the fourth loss, the degree of difference between the second text feature and the speech feature can be measured, thereby guiding the optimization of the speech discrete representation model so that the text feature and speech feature can be better matched. By minimizing the fourth loss, the speech discrete representation model can gradually learn how to better match the text feature and speech feature, improving the performance of speech processing.

[0137] The relative entropy loss between the second text features and the speech features is determined as the fourth loss. Relative entropy, also known as KL divergence, is a metric used to measure the difference between two probability distributions. In this embodiment, the second text features and the speech features are treated as two probability distributions, and the fourth loss is determined by calculating the relative entropy between them. The calculation process typically involves estimating the probability density functions of the two feature distributions and then calculating the value of the relative entropy. If the two distributions are identical, the relative entropy is zero; if the two distributions differ significantly, the value of the relative entropy will be large.

[0138] In step S313, a discrete speech representation model is trained based on the first loss, the second loss, the third loss, and the fourth loss.

[0139] In this embodiment, the first loss is the continuous temporal classification loss between the semantically continuous vector and the reference semantic text, which measures the model's accuracy in semantic understanding and conversion. By minimizing the first loss, the speech discrete representation model can better learn the semantic information in the speech signal and accurately convert it into text form. The second loss is the mean squared error loss between the fundamental frequency continuous vector and the reference fundamental frequency information, which evaluates the model's predictive ability for the fundamental frequency features of the speech signal. Optimizing the second loss allows the speech discrete representation model to more accurately capture pitch changes in speech, improving the naturalness and intelligibility of the speech. The third loss is the mean squared error loss between the semantically continuous vector and the hidden layer features, which helps the speech discrete representation model to better extract and represent speech features at the semantic level. By reducing the third loss, the speech discrete representation model can more effectively convert the speech signal into a continuous vector with semantic meaning. The fourth loss is the relative entropy loss between the second text feature and the speech feature, which reflects the degree of matching between the text feature and the speech feature. Reducing the fourth loss allows the speech discrete representation model to better align text and speech, improving the accuracy of speech processing. By training with multiple loss mechanisms, the discrete speech representation model can be optimized simultaneously in multiple aspects, including semantic understanding, fundamental frequency prediction, feature extraction, and text-to-speech alignment, thereby improving the model's overall performance. The resulting model can more accurately represent the features of speech signals and better perform speech processing tasks such as speech recognition and speech synthesis.

[0140] It should be noted that steps S309-S311 above are optional. That is, when training the discrete speech representation model, three losses (first loss, second loss, and third loss) can be used for training, or four losses (first loss, second loss, third loss, and fourth loss) can be used for training. Optionally, the first, second, and third losses can be used for training first, followed by the fourth loss. This disclosure does not limit this approach.

[0141] To make the training scheme of the speech discrete representation model provided in the embodiments of this disclosure easier to understand, see [link to documentation]. Figure 4 As shown, Figure 4 This is a flowchart illustrating a training method for a discrete speech representation model according to an exemplary embodiment. Figure 4As shown, the sample speech signal is input into the pre-trained model. Then, the hidden layer features output by the pre-trained model are input into the feature extraction module to obtain speech features in continuous space. The speech features are input into the vector quantization module to obtain quantized feature vectors and discrete speech representations. The discrete speech representations do not participate in the model training process. Then, based on the semantic decoding module, the quantized feature vectors are semantically decoded to obtain semantically continuous vectors; based on the fundamental frequency decoding module, the quantized feature vectors are fundamentally decoded to obtain fundamental frequency continuous vectors. Based on the text feature extraction module, features are extracted from the reference speech text to obtain the first text features. Based on the forced alignment module, the first text features are forcibly aligned to obtain the second text features. Then, the first loss, second loss, third loss, and fourth loss are determined respectively.

[0142] This disclosure provides a training scheme for a discrete speech representation model. By introducing reference speech text and reference fundamental frequency information corresponding to the sample speech signal as supervision information during training, the trained discrete speech representation model can efficiently and accurately extract the discrete speech representation of the speech signal. Furthermore, this discrete speech representation can fully reflect the semantic content and fundamental frequency information contained in the original speech signal without losing speaker information or prosodic information. Compared to clustering-based methods for extracting discrete speech features, this significantly improves the accuracy and effectiveness of discrete speech representation, enabling discrete speech features to significantly enhance the performance of downstream tasks such as speech synthesis and speech recognition.

[0143] Figure 5 This is a flowchart illustrating a method for processing speech signals according to an exemplary embodiment, such as... Figure 5 As shown, the method is performed by an electronic device and includes the following steps:

[0144] In step S501, the speech signal to be processed is input into the speech discrete representation model, which is trained using the training method described above.

[0145] In step S502, the speech signal is converted into hidden layer features in continuous space based on the deep neural network in the speech discrete representation model.

[0146] In step S503, based on the deep learning network in the speech discrete representation model, feature extraction is performed on the hidden layer features to obtain the speech features of the speech signal in continuous space.

[0147] In step S504, the speech features are vector quantized and encoded to obtain a discrete speech representation of the speech signal.

[0148] To make the speech signal processing scheme provided in the embodiments of this disclosure easier to understand, see [link to relevant documentation]. Figure 6As shown, Figure 6 This is a flowchart illustrating a method for processing speech signals according to an exemplary embodiment. Figure 6 As shown, the speech signal is input into the speech discrete representation model. The pre-trained model in the speech discrete representation model converts the speech signal into hidden features in a continuous space. Based on the feature extraction module in the speech discrete representation model, features are extracted from the hidden features to obtain the speech features of the speech signal in continuous space. Based on the vector quantization module in the speech discrete representation model, the speech features are vector quantized and encoded to obtain the speech discrete representation and quantized feature vector of the speech signal. The quantized feature vector does not participate in the speech signal processing stage.

[0149] Figure 7 This is a block diagram illustrating a training apparatus for a discrete speech representation model according to an exemplary embodiment. Figure 7 As shown, the device includes: an acquisition unit 701, a feature extraction unit 702, and a training unit 703.

[0150] The acquisition unit 701 is configured to acquire a sample speech signal, a reference speech text, and reference baseband information. The reference speech text is the correct text content corresponding to the sample speech signal, and the reference baseband information includes the actual baseband characteristics of the sample speech signal.

[0151] The feature extraction unit 702 is configured to input the sample speech signal into the speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector.

[0152] Training unit 703 is configured to train a discrete speech representation model based on semantic continuous vectors, fundamental frequency continuous vectors, speech text, and reference fundamental frequency information.

[0153] In some embodiments, the feature extraction unit 702 includes:

[0154] The first extraction subunit is configured to extract features from the sample speech signal of the input speech discrete representation model to obtain the speech features of the sample speech signal in continuous space.

[0155] The vectorization subunit is configured to perform vector quantization encoding on speech features to obtain quantized feature vectors;

[0156] The second extraction subunit is configured to extract semantic continuous vectors and fundamental frequency continuous vectors from the quantized feature vectors.

[0157] In some embodiments, the first extraction subunit is configured to convert the sample speech signal of the input speech discrete representation model into hidden layer features in continuous space based on a deep neural network in the speech discrete representation model; and to extract features from the hidden layer features based on the deep learning network in the speech discrete representation model to obtain the speech features of the sample speech signal in continuous space.

[0158] In some embodiments, the second extraction subunit is configured to perform semantic decoding on the quantized feature vector based on the semantic decoding module to obtain a semantically continuous vector; and to perform fundamental frequency decoding on the quantized feature vector based on the fundamental frequency decoding module to obtain a fundamental frequency continuous vector.

[0159] In some embodiments, the training unit 703 is configured to: determine a first loss based on a semantic continuous vector and a reference speech text corresponding to a sample speech signal, wherein the first loss indicates the continuous temporal classification loss between the semantic continuous vector and the reference semantic text; determine a second loss based on a fundamental frequency continuous vector and reference fundamental frequency information, wherein the second loss indicates the mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information; determine a third loss based on a semantic continuous vector and hidden layer features of the sample speech signal in continuous space, wherein the third loss indicates the mean square error loss between the semantic continuous vector and the hidden layer features; and train a speech discrete representation model based on the first loss, the second loss, and the third loss.

[0160] In some embodiments, the apparatus further includes:

[0161] The feature extraction unit 702 is also configured to extract features from the reference speech text corresponding to the sample speech signal to obtain the first text features;

[0162] The feature alignment unit is configured to perform forced alignment of the first text features based on the forced alignment module to obtain the second text features;

[0163] Training unit 703 is also configured to determine a fourth loss based on the second text features and speech features, the fourth loss being used to represent the relative entropy loss between the second text features and speech features.

[0164] Training unit 703 is also configured to train a discrete speech representation model based on four-loss.

[0165] This disclosure provides a training apparatus for a discrete speech representation model. By introducing reference speech text and reference fundamental frequency information corresponding to the sample speech signal as supervision information during training, the trained discrete speech representation model can efficiently and accurately extract the discrete speech representation of the speech signal. Furthermore, this discrete speech representation can fully reflect the semantic content and fundamental frequency information contained in the original speech signal without losing speaker information or prosodic information. Compared to clustering-based methods for extracting discrete speech features, this significantly improves the accuracy and effectiveness of the discrete speech representation, enabling the discrete speech features to significantly enhance the performance of downstream tasks such as speech synthesis and speech recognition.

[0166] It should be noted that the speech discrete representation model training device provided in the above embodiments is only an example of the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the speech discrete representation model training device and the speech discrete representation model training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0167] Regarding the training apparatus for the discrete speech representation model in the above embodiments, the specific manner in which each module performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0168] Figure 8 This is a block diagram illustrating a speech signal processing apparatus according to an exemplary embodiment. Figure 8 As shown, the device includes: an input unit 801, a first processing unit 802, a second processing unit 803, and a vectorization unit 804.

[0169] The input unit 801 is configured to input the speech signal to be processed into the speech discrete representation model, which is trained by the above-mentioned training method for the speech discrete representation model.

[0170] The first processing unit 802 is configured to convert speech signals into hidden layer features in a continuous space based on a deep neural network in a speech discrete representation model.

[0171] The second processing unit 803 is configured to extract features from hidden layer features based on a deep learning network in the speech discrete representation model, thereby obtaining speech features of the speech signal in continuous space.

[0172] Vectorization unit 804 is configured to perform vector quantization encoding on speech features to obtain a discrete speech representation of the speech signal.

[0173] It should be noted that the voice signal processing apparatus provided in the above embodiments is only illustrated by the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device can be divided into different functional units to complete all or part of the functions described above. In addition, the voice signal processing apparatus and the voice signal processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0174] Regarding the speech signal processing apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0175] In this embodiment, the electronic device can be a terminal or a server. When the electronic device is a terminal, the terminal acts as the execution subject to implement the technical solutions provided in this embodiment; when the electronic device is a server, the server acts as the execution subject to implement the technical solutions provided in this embodiment; or, the technical solutions provided in this disclosure can be implemented through interaction between the terminal and the server. For example, the server trains a discrete speech representation model, the terminal sends the speech signal to be processed to the server, the server processes the speech signal based on the discrete speech representation model, obtains a discrete speech representation, and returns it to the terminal. The terminal obtains the discrete speech representation returned by the server. This embodiment does not limit this aspect.

[0176] Figure 9 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Typically, the electronic device 900 includes a processor 901 and a memory 902.

[0177] Processor 901 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0178] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 are used to store at least one program code, which is executed by the processor 901 to implement the training method for the discrete speech representation model provided in the method embodiments of this disclosure, or the at least one program code is executed by the processor 901 to implement the speech signal processing method provided in the method embodiments of this disclosure.

[0179] In some embodiments, the electronic device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0180] Peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 901 and memory 902. In some embodiments, processor 901, memory 902 and peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 901, memory 902 and peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0181] The radio frequency (RF) circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 904 can communicate with other electronic devices through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this disclosure.

[0182] Display screen 905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 901 for processing. In this case, display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 905, which serves as the front panel of electronic device 900; in other embodiments, there may be at least two display screens 905, respectively disposed on different surfaces of electronic device 900 or in a folded design; in still other embodiments, display screen 905 may be a flexible display screen, disposed on a curved or folded surface of electronic device 900. Furthermore, display screen 905 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 905 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0183] The camera assembly 906 is used to acquire images or videos. Optionally, the camera assembly 906 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device, and the rear-facing camera is located on the back of the electronic device. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0184] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 901 for processing, or input to the radio frequency circuit 904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located in a different part of the electronic device 900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert the electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0185] Power supply 908 is used to supply power to the various components in electronic device 900. Power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 908 includes a rechargeable battery, the rechargeable battery can support wired or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0186] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on the electronic device 900, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0187] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 902 including instructions. These instructions can be executed by a processor 901 of an electronic device 900 to complete the training method for the discrete speech representation model described above, or to complete the processing method for the speech signal described above. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0188] A computer program product includes a computer program that, when executed by a processor, implements the training method for the aforementioned discrete speech representation model, or implements the aforementioned speech signal processing method.

[0189] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0190] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A training method for a discrete speech representation model, characterized in that, The method includes: Acquire sample speech signals, reference speech text, and reference baseband information, wherein the reference speech text is the correct text content corresponding to the sample speech signal, and the reference baseband information includes the actual baseband characteristics of the sample speech signal; The sample speech signal is input into the speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector; Based on the semantic continuous vector, the fundamental frequency continuous vector, the reference speech text, and the reference fundamental frequency information, a first loss, a second loss, a third loss, and a fourth loss are determined. The first loss is used to indicate the continuous temporal classification loss between the semantic continuous vector and the reference speech text. The second loss is used to indicate the mean squared error loss between the fundamental frequency continuous vector and the reference fundamental frequency information. The third loss is used to indicate the mean squared error loss between the semantic continuous vector and the hidden layer features of the sample speech signal in continuous space. The fourth loss is used to represent the relative entropy loss between the text features of the reference speech text and the speech features of the sample speech signal in continuous space. The speech discrete representation model is trained based on the first loss, the second loss, the third loss, and the fourth loss.

2. The training method for the discrete speech representation model according to claim 1, characterized in that, The step of inputting the sample speech signal into the speech discrete representation model to obtain the semantic continuous vector and the fundamental frequency continuous vector includes: Feature extraction is performed on the sample speech signal input to the discrete speech representation model to obtain the speech features of the sample speech signal in continuous space; The speech features are vector quantized and encoded to obtain quantized feature vectors; Extract the semantic continuous vector and the fundamental frequency continuous vector from the quantized feature vector.

3. The training method for the discrete speech representation model according to claim 2, characterized in that, The step of extracting features from the sample speech signal input to the discrete speech representation model to obtain the speech features of the sample speech signal in continuous space includes: Based on the deep neural network in the discrete speech representation model, the sample speech signal input to the discrete speech representation model is converted into hidden layer features in the continuous space; Based on the deep learning network in the discrete speech representation model, feature extraction is performed on the hidden layer features to obtain the speech features of the sample speech signal in the continuous space.

4. The training method for the discrete speech representation model according to claim 2, characterized in that, The step of extracting the semantic continuous vector and the fundamental frequency continuous vector from the quantized feature vector includes: Based on the semantic decoding module, the quantized feature vector is semantically decoded to obtain the semantically continuous vector; Based on the baseband decoding module, the quantized feature vector is decoded using baseband decoding to obtain the continuous baseband vector.

5. The training method for the discrete speech representation model according to claim 2, characterized in that, The method further includes: Feature extraction is performed on the reference speech text to obtain the first text feature; The first text feature is forcibly aligned using the forced alignment module to obtain the second text feature; The fourth loss is determined based on the second text features and the speech features.

6. A method for processing speech signals, characterized in that, The method includes: The speech signal to be processed is input into the speech discrete representation model, which is obtained by training according to any one of claims 1-5; Based on the deep neural network in the discrete speech representation model, the speech signal is converted into hidden layer features in a continuous space; Based on the deep learning network in the discrete speech representation model, feature extraction is performed on the hidden layer features to obtain the speech features of the speech signal in continuous space. The speech features are vector quantized and encoded to obtain a discrete speech representation of the speech signal.

7. A training device for a discrete speech representation model, characterized in that, The device includes: The acquisition unit is configured to acquire a sample speech signal, a reference speech text, and reference baseband information, wherein the reference speech text is the correct text content corresponding to the sample speech signal, and the reference baseband information includes the actual baseband characteristics of the sample speech signal. The feature extraction unit is configured to input the sample speech signal into a speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector. The training unit is configured to determine a first loss, a second loss, a third loss, and a fourth loss based on the semantic continuous vector, the fundamental frequency continuous vector, the reference speech text, and the reference fundamental frequency information. The first loss indicates the continuous temporal classification loss between the semantic continuous vector and the reference speech text. The second loss indicates the mean squared error loss between the fundamental frequency continuous vector and the reference fundamental frequency information. The third loss indicates the mean squared error loss between the semantic continuous vector and the hidden features of the sample speech signal in the continuous space. The fourth loss indicates the relative entropy loss between the text features of the reference speech text and the speech features of the sample speech signal in the continuous space. The speech discrete representation model is trained based on the first loss, the second loss, the third loss, and the fourth loss.

8. A speech signal processing apparatus, characterized in that, The device includes: An input unit is configured to input the speech signal to be processed into a speech discrete representation model, which is obtained by training through any one of claims 1-5; The first processing unit is configured to convert the speech signal into hidden layer features in a continuous space based on the deep neural network in the speech discrete representation model. The second processing unit is configured to extract features from the hidden layer features based on the deep learning network in the speech discrete representation model to obtain the speech features of the speech signal in continuous space. The vectorization unit is configured to perform vector quantization encoding on the speech features to obtain a discrete speech representation of the speech signal.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; Memory used to store the executable program code of the processor; The processor is configured to execute the program code to implement the training method of the speech discrete representation model as described in any one of claims 1 to 5, or to implement the speech signal processing method as described in claim 6.

10. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the training method of the speech discrete representation model as described in any one of claims 1 to 5, or to implement the speech signal processing method as described in claim 6.

11. A computer program product comprising a computer program that, when executed by a processor, implements a training method for a discrete speech representation model as described in any one of claims 1 to 5, or implements a speech signal processing method as described in claim 6.

Images