Speech discrete representation model training method and speech signal processing method

Abstract

The invention provides a voice discrete representation model training method and a voice signal processing method, and belongs to the technical field of computers. According to the method, a reference voice text and reference fundamental frequency information corresponding to a sample voice signal are introduced as supervision information for training when a voice discrete representation model is trained, so that the voice discrete representation model obtained by training can efficiently and accurately extract voice discrete representation of the voice signal; and the speech discrete representation can fully reflect semantic content and fundamental frequency information contained in an original speech signal, and speaker information, rhythm information and the like cannot be lost. Compared with a scheme of extracting the speech discrete features in a clustering manner, the method has the advantages that the accuracy and effectiveness of speech discrete representation are remarkably improved, and the speech discrete features can remarkably improve the performance of downstream tasks such as speech synthesis and speech recognition.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method for a discrete speech representation model and a method for processing a speech signal. Background Art

[0002] Speech synthesis technology based on large language models can synthesize natural, fluent, and expressive speech, and has broad application prospects in the fields of intelligent voice assistants, audio books, virtual anchors, etc. Among them, the core of speech synthesis technology lies in whether it can accurately extract the discrete representation of speech that reflects the characteristics of speech signals. In other words, the accuracy and effectiveness of the discrete representation of speech directly affect the naturalness and accuracy of the synthesized speech.

[0003] Currently, a common method for extracting discrete speech features is to extract speech features through self-supervised learning, and then obtain discrete speech representations through clustering.

[0004] However, the discrete speech features extracted by the above method will lose a lot of information, such as speaker information, prosody information, etc., which will cause various problems in downstream speech synthesis tasks such as unclear pronunciation and reduced naturalness. Summary of the invention

[0005] The present disclosure provides a method for training a speech discrete representation model and a method for processing speech signals. Compared with the method of clustering to extract speech discrete features, this method significantly improves the accuracy and effectiveness of speech discrete representation, so that speech discrete features can significantly improve the performance of downstream tasks such as speech synthesis and speech recognition.

[0006] According to one aspect of an embodiment of the present disclosure, a method for training a discrete speech representation model is provided, the method comprising:

[0007] Acquire a sample speech signal, a reference speech text, and reference fundamental frequency information, wherein the reference speech text is a correct text content corresponding to the sample speech signal, and the reference fundamental frequency information includes an actual fundamental frequency feature of the sample speech signal;

[0008] Inputting the sample speech signal into a speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector;

[0009] The speech discrete representation model is trained based on the semantic continuous vector, the fundamental frequency continuous vector, the reference speech text and the reference fundamental frequency information.

[0010] According to another aspect of the present disclosure, a method for processing a speech signal is provided, the method comprising:

[0011] Inputting the speech signal to be processed into a speech discrete representation model, wherein the speech discrete representation model is trained by the training method of the speech discrete representation model;

[0012] Based on the deep neural network in the discrete speech representation model, the speech signal is converted into hidden layer features in a continuous space;

[0013] Based on the deep learning network in the speech discrete representation model, feature extraction is performed on the hidden layer features to obtain speech features of the speech signal in a continuous space;

[0014] The speech features are vector quantized and encoded to obtain a speech discrete representation of the speech signal.

[0015] According to another aspect of an embodiment of the present disclosure, a training device for a speech discrete representation model is provided, the device comprising:

[0016] An acquisition unit is configured to acquire a sample speech signal, a reference speech text, and reference fundamental frequency information, wherein the reference speech text is a correct text content corresponding to the sample speech signal, and the reference fundamental frequency information includes an actual fundamental frequency feature of the sample speech signal;

[0017] A feature extraction unit is configured to input the sample speech signal into a speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector;

[0018] The training unit is configured to train the speech discrete representation model based on the semantic continuous vector, the fundamental frequency continuous vector, the speech text and the reference fundamental frequency information.

[0019] In some embodiments, the feature extraction unit comprises:

[0020] A first extraction subunit is configured to perform feature extraction on the sample speech signal input into the speech discrete representation model to obtain speech features of the sample speech signal in a continuous space;

[0021] A vectorization subunit is configured to perform vector quantization encoding on the speech feature to obtain a quantized feature vector;

[0022] The second extraction subunit is configured to extract the semantic continuous vector and the fundamental frequency continuous vector from the quantized feature vector.

[0023] In some embodiments, the first extraction subunit is configured to convert the sample speech signal input into the discrete speech representation model into hidden layer features in the continuous space based on the deep neural network in the discrete speech representation model; and extract features from the hidden layer features based on the deep learning network in the discrete speech representation model to obtain speech features of the sample speech signal in the continuous space.

[0024] In some embodiments, the second extraction subunit is configured to perform semantic decoding on the quantized feature vector based on a semantic decoding module to obtain the semantic continuous vector; and perform baseband decoding on the quantized feature vector based on a baseband decoding module to obtain the baseband continuous vector.

[0025] In some embodiments, the training unit is configured to determine a first loss based on the semantic continuous vector and the reference speech text corresponding to the sample speech signal, the first loss being used to indicate the continuous temporal classification loss between the semantic continuous vector and the reference semantic text; determine a second loss based on the fundamental frequency continuous vector and the reference fundamental frequency information, the second loss being used to indicate the mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information; determine a third loss based on the semantic continuous vector and the hidden layer features of the sample speech signal in a continuous space, the third loss being used to indicate the mean square error loss between the semantic continuous vector and the hidden layer features; and train the speech discrete representation model based on the first loss, the second loss and the third loss.

[0026] In some embodiments, the apparatus further comprises:

[0027] The feature extraction unit is further configured to extract features from a reference speech text corresponding to the sample speech signal to obtain a first text feature;

[0028] A feature alignment unit, configured to perform forced alignment on the first text feature based on a forced alignment module to obtain a second text feature;

[0029] The training unit is further configured to determine a fourth loss based on the second text feature and the speech feature, wherein the fourth loss is used to represent a relative entropy loss between the second text feature and the speech feature;

[0030] The training unit is also configured to train the speech discrete representation model based on the four losses.

[0031] According to another aspect of an embodiment of the present disclosure, a speech signal processing device is provided, the device comprising:

[0032] An input unit is configured to input a speech signal to be processed into a speech discrete representation model, wherein the speech discrete representation model is trained by the training method of the speech discrete representation model;

[0033] A first processing unit is configured to convert the speech signal into a hidden layer feature in a continuous space based on a deep neural network in the speech discrete representation model;

[0034] A second processing unit is configured to extract the hidden layer features based on the deep learning network in the speech discrete representation model to obtain the speech features of the speech signal in a continuous space;

[0035] The vectorization unit is configured to perform vector quantization encoding on the speech feature to obtain a speech discrete representation of the speech signal.

[0036] According to another aspect of an embodiment of the present disclosure, an electronic device is provided, the electronic device comprising:

[0037] one or more processors;

[0038] A memory for storing program codes executable by the processor;

[0039] Among them, the processor is configured to execute the program code to implement the above-mentioned training method of the discrete speech representation model, or to implement the above-mentioned method of processing the speech signal.

[0040] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above-mentioned training method of the discrete speech representation model, or execute the above-mentioned method of processing the speech signal.

[0041] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the training method of the above-mentioned discrete speech representation model, or implements the above-mentioned method for processing speech signals.

[0042] The disclosed embodiment provides a training scheme for a discrete speech representation model. By introducing reference speech text and reference fundamental frequency information corresponding to a sample speech signal as supervisory information when training the discrete speech representation model, the trained discrete speech representation model can efficiently and accurately extract the discrete speech representation of the speech signal, and the discrete speech representation can fully reflect the semantic content and fundamental frequency information contained in the original speech signal without losing speaker information and prosody information, etc. Compared with the scheme of clustering to extract discrete speech features, the accuracy and effectiveness of discrete speech representation are significantly improved, so that discrete speech features can significantly improve the performance of downstream tasks such as speech synthesis and speech recognition.

[0043] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0045] Figure 1 It is a schematic diagram of an implementation environment of a method for training a discrete speech representation model according to an exemplary embodiment.

[0046] Figure 2 The present invention is a flowchart of a method for training a discrete speech representation model according to an exemplary embodiment.

[0047] Figure 3 The figure is a flowchart of another method for training a discrete speech representation model according to an exemplary embodiment.

[0048] Figure 4 It is a flowchart of a method for training a discrete speech representation model provided according to an exemplary embodiment.

[0049] Figure 5 The present invention is a flowchart of a method for processing a speech signal according to an exemplary embodiment.

[0050] Figure 6 It is a flowchart of a method for processing a speech signal provided according to an exemplary embodiment.

[0051] Figure 7 It is a block diagram of a training device for a discrete speech representation model according to an exemplary embodiment.

[0052] Figure 8 The invention is a block diagram of a device for processing a speech signal according to an exemplary embodiment.

[0053] Fig. 9 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0054] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0055] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0056] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions. For example, the sample voice signals and reference voice texts involved in this disclosure are all obtained with full authorization.

[0057] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a method for training a speech discrete representation model according to an exemplary embodiment. Figure 1 The implementation environment specifically includes: a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0058] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), and a laptop computer. An application may be installed and run on the terminal 101, and the application is used to play information stream data, such as multimedia data or live broadcast data. The application is associated with the server 102, and the server 102 provides background services to the terminal 101.

[0059] The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is described by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less. For example, the above terminals may be several, or the above terminals may be dozens or hundreds, or a greater number. The embodiment of the present disclosure does not limit the number and device type of the terminals.

[0060] The server 102 is at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure. Of course, the server 102 may also include other functional servers to provide more comprehensive and diversified services. In some embodiments, the server 102 undertakes the main computing work, and the terminal 101 undertakes the secondary computing work; or, the server 102 undertakes the secondary computing work, and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 use a distributed computing architecture for collaborative computing. The server 102 can be connected to the terminal 101 and other terminals via a wireless network or a wired network. Optionally, the number of the above servers may be more or less, which is not limited in the embodiments of the present disclosure.

[0061] Figure 2 is a flowchart of a method for training a speech discrete representation model according to an exemplary embodiment. Figure 2 As shown, the method is performed by an electronic device and includes the following steps:

[0062] In step S201, a sample speech signal, a reference speech text and reference fundamental frequency information are obtained.

[0063] In the embodiments of the present disclosure, the sample speech signal refers to speech data used for training and testing, the reference speech text is the correct text content corresponding to the sample speech signal, and the reference fundamental frequency information includes the actual fundamental frequency characteristics of the sample speech signal.

[0064] In step S202, the sample speech signal is input into a speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector.

[0065] In the disclosed embodiment, the speech discrete representation model is a model to be trained, and the speech discrete representation model is used to process the input speech signal and output the speech discrete representation. The speech discrete representation model includes a module for feature extraction, through which the sample speech signal of the input speech discrete representation can be feature extracted to obtain the speech feature of the sample speech signal. The speech feature is the speech feature vector of the sample speech signal in the continuous space.

[0066] In step S203, a speech discrete representation model is trained based on the semantic continuous vector, the fundamental frequency continuous vector, the reference speech text and the reference fundamental frequency information.

[0067] In the disclosed embodiment, the reference speech text and reference fundamental frequency information corresponding to the sample speech signal are used as supervisory information, and the training loss is determined based on the difference between the semantics contained in the reference speech text and the semantic continuous vector, and the difference between the fundamental frequency contained in the reference fundamental frequency information and the fundamental frequency continuous vector, and then the parameters of the speech discrete representation model are adjusted based on the training loss. The trained speech discrete representation model is obtained by adjusting the parameters of the speech discrete representation model multiple times.

[0068] The disclosed embodiment provides a training scheme for a discrete speech representation model. By introducing reference speech text and reference fundamental frequency information corresponding to a sample speech signal as supervisory information when training the discrete speech representation model, the trained discrete speech representation model can efficiently and accurately extract the discrete speech representation of the speech signal, and the discrete speech representation can fully reflect the semantic content and fundamental frequency information contained in the original speech signal without losing speaker information and prosody information, etc. Compared with the scheme of clustering to extract discrete speech features, the accuracy and effectiveness of discrete speech representation are significantly improved, so that discrete speech features can significantly improve the performance of downstream tasks such as speech synthesis and speech recognition.

[0069] In some embodiments, the sample speech signal is input into a speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector, including:

[0070] Extracting features of the sample speech signal input into the discrete speech representation model to obtain speech features of the sample speech signal in a continuous space;

[0071] Perform vector quantization encoding on the speech features to obtain a quantized feature vector;

[0072] The semantic continuous vector and fundamental frequency continuous vector are extracted from the quantized feature vector.

[0073] By inputting the sample speech signal into the speech discrete representation model, obtaining the semantic continuous vector and the fundamental frequency continuous vector through feature extraction, vector quantization encoding and corresponding extraction operations, it helps to effectively represent the speech signal in different dimensions and facilitate its use in subsequent processing.

[0074] In some embodiments, feature extraction is performed on a sample speech signal input into a speech discrete representation model to obtain speech features of the sample speech signal in a continuous space, including:

[0075] Based on the deep neural network in the speech discrete representation model, the sample speech signal input into the speech discrete representation model is converted into hidden layer features in the continuous space;

[0076] Based on the deep learning network in the discrete speech representation model, the hidden layer features are extracted to obtain the speech features of the sample speech signal in the continuous space.

[0077] The sample speech signal is converted into hidden features in a continuous space through the deep neural network in the discrete speech representation model, and the hidden features are further extracted to obtain the speech features of the sample speech signal in the continuous space. This can more effectively mine the intrinsic information of the speech signal, improve the accuracy and richness of the speech features, and provide better feature representation for subsequent speech processing tasks.

[0078] In some embodiments, extracting a semantic continuous vector and a fundamental frequency continuous vector from the quantized feature vector includes:

[0079] Based on the semantic decoding module, the quantized feature vector is semantically decoded to obtain a semantic continuous vector;

[0080] Based on the baseband decoding module, the baseband decoding is performed on the quantized feature vector to obtain a baseband continuous vector.

[0081] By performing semantic decoding on the quantized feature vector to obtain a semantic continuous vector through the semantic decoding module, and by performing baseband decoding on the quantized feature vector to obtain a baseband continuous vector through the baseband decoding module, more expressive semantic information and baseband information can be extracted from the quantized features respectively, providing richer and more accurate features for speech processing, which helps to improve the performance and effect of speech processing tasks.

[0082] In some embodiments, training a speech discrete representation model based on the semantic continuous vector, the fundamental frequency continuous vector, the reference speech text, and the reference fundamental frequency information includes:

[0083] Determine a first loss based on the semantic continuous vector and a reference speech text corresponding to the sample speech signal, where the first loss is used to indicate a continuous temporal classification loss between the semantic continuous vector and the reference semantic text;

[0084] Determine a second loss based on the fundamental frequency continuous vector and the reference fundamental frequency information, where the second loss is used to indicate a mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information;

[0085] Based on the semantic continuous vector and the hidden layer features of the sample speech signal in the continuous space, determining a third loss, where the third loss is used to indicate the mean square error loss between the semantic continuous vector and the hidden layer features;

[0086] Based on the first loss, the second loss and the third loss, a speech discrete representation model is trained.

[0087] By determining the first loss based on the semantic continuous vector and the reference semantic text, the second loss based on the fundamental frequency continuous vector and the reference fundamental frequency information, and the third loss based on the semantic continuous vector and the hidden layer features, and training the speech discrete representation model based on these three losses, the model can be optimized from multiple angles such as semantics, fundamental frequency, and the relationship with hidden layer features, thereby improving the model's accuracy in semantic understanding of speech signals, fundamental frequency prediction, and feature extraction, thereby improving the performance of the speech discrete representation model.

[0088] In some embodiments, the method further comprises:

[0089] Extracting features of a reference speech text corresponding to the sample speech signal to obtain a first text feature;

[0090] Performing forced alignment on the first text feature based on a forced alignment module to obtain a second text feature;

[0091] Based on the second text feature and the speech feature, determining a fourth loss, where the fourth loss is used to represent a relative entropy loss between the second text feature and the speech feature;

[0092] Based on four losses, a discrete speech representation model is trained.

[0093] The first text feature is obtained by extracting features from the reference speech text corresponding to the sample speech signal, and the second text feature is obtained through processing by the forced alignment module. The fourth loss is determined based on the second text feature and the speech feature, and the speech discrete representation model is trained based on the fourth loss. This can enable the model to better achieve precise alignment of text and speech, thereby improving the accuracy and performance of the model in speech processing.

[0094] Above Figure 2 Shown is a flow chart of a method for training a discrete speech representation model of the present disclosure. The training scheme of the discrete speech representation model provided by the present disclosure is further elaborated below. Figure 3 is a flowchart of another method for training a discrete speech representation model according to an exemplary embodiment. Figure 3 The method is performed by an electronic device and comprises the following steps:

[0095] In step S301, a sample speech signal, a reference speech text and reference fundamental frequency information are obtained.

[0096] In the disclosed embodiments, the sample speech signal refers to speech data used for training and testing. The sample speech signal is continuous, that is, it is uninterrupted in time. Optionally, the sample speech signal contains rich information, such as the speech signal contains a variety of information such as the speaker's speech content, intonation, emotion, etc. The speech signal changes over time and has obvious time series characteristics. Among them, the reference speech text is the correct text content corresponding to the sample speech signal. In other words, the reference speech text clearly expresses the semantics of the sample speech signal in the form of natural language. The reference fundamental frequency information is usually obtained by analyzing the sample speech signal or marking the known correct fundamental frequency. The reference fundamental frequency information represents the actual fundamental frequency characteristics of the sample speech signal.

[0097] In step S302, based on the deep neural network in the speech discrete representation model, the sample speech signal input into the speech discrete representation model is converted into hidden layer features in a continuous space.

[0098] In the disclosed embodiment, the speech discrete representation model includes a pre-trained model, which is a deep neural network model. A deep neural network is a machine learning model with multiple hidden layers, and a deep neural network can automatically learn complex feature representations of input data. A deep neural network is usually composed of an input layer, multiple hidden layers, and an output layer. The input layer receives a sample speech signal, the hidden layer extracts features through nonlinear transformation, and the output layer generates the final output result. Each hidden layer contains a large number of neurons, which are connected by weights, and the input content is weighted summed and activated.

[0099] This pre-trained model is used to convert the input speech signal into a hidden layer vector in a continuous space and retain the information in the original speech. The hidden layer features are usually more abstract and advanced than the original speech signal, and contain some key information of the speech signal, but are not directly interpretable, so that the hidden layer features in the continuous space can provide richer information and help improve the performance and accuracy of the speech processing system.

[0100] It should be noted that continuous space refers to the state in which the speech signal changes continuously in both time and amplitude. In continuous space, the time axis of the speech signal is continuous without discontinuities. This means that the speech signal has a clear value at any time, and the changes between adjacent moments are smooth. In addition, the amplitude of the speech signal also changes continuously. The amplitude represents the strength of the sound, and in continuous space, the amplitude can take any real value. For example, the volume of the sound can change continuously from very weak to very loud, without obvious jumps or discrete values.

[0101] Corresponding to the continuous space is the discrete space, which is the state after the continuous voice signal is converted into a discrete digital signal through sampling and quantization. Among them, sampling is to discretize the continuous voice signal on the time axis, and take a sample value at a certain time interval. This time interval is called the sampling period, and the sampling frequency is the number of sampling points per unit time. Quantization is to convert the continuous amplitude value into a finite number of discrete amplitude values. Through quantization, the amplitude of the voice signal can be represented by a number, usually using a binary number.

[0102] In step S303, based on the deep learning network in the speech discrete representation model, feature extraction is performed on the hidden layer features to obtain the speech features of the sample speech signal in the continuous space.

[0103] In the disclosed embodiments, a deep learning network is a machine learning model with multiple hierarchical structures that can automatically learn complex patterns and features from large amounts of data. In speech processing, deep learning networks generally include convolutional neural networks, recurrent neural networks, long short-term memory networks, and other types, which can effectively process the time series characteristics and spectral characteristics of speech signals. The disclosed embodiments do not limit the network structure of the deep learning network. The deep learning network extracts deep features of the sample speech signal from the hidden features to obtain highly compressed speech features in a continuous space.

[0104] The purpose of feature extraction of latent features is to further explore and refine more valuable speech features so that they can be better used in subsequent speech processing tasks such as speech recognition, speech synthesis, speech sentiment analysis, etc.

[0105] Optionally, the method of extracting the hidden layer features may be to perform a linear or nonlinear transformation on the hidden layer features to map the hidden layer features to a new feature space, so that the features are more compact, independent and distinguishable.

[0106] In step S304, vector quantization encoding is performed on the speech features to obtain a quantized feature vector.

[0107] In the disclosed embodiments, vector quantization (VQ) is a technology widely used in the field of signal processing, especially in speech and image coding. The core idea of ​​VQ is to represent the input signal vector with a finite number of representative codewords, that is, to map the high-dimensional signal space to a relatively low-dimensional codebook space.

[0108] For example, for a speech signal, a speech can be regarded as a vector composed of multiple feature parameters. Through VQ, a pre-designed codebook is found, which contains several codewords. Each codeword is also a vector, representing a specific signal pattern. The input speech vector is compared with each codeword in the codebook, and the closest codeword is found, and this codeword is used to represent the input speech vector.

[0109] It should be noted that the key to vector quantization coding is the codebook. The codebook is a set of codewords, each of which represents a specific signal vector. Codebook design usually uses a clustering algorithm to cluster a large number of training data vectors to obtain different codebook regions. When there is a new input signal vector, the distance between the input signal vector and each codeword in the codebook is calculated. The codeword with the smallest distance is selected as the quantization result of the input signal vector. This process is to map the input signal vector to a specific codeword in the codebook space. The quantized codeword can be encoded with a smaller number of bits. For example, if there are 256 codewords in the codebook, 8 bits can be used to represent a codeword. In this way, data compression is achieved.

[0110] After vector quantization encoding, the original speech feature vector is replaced by a codeword in the codebook, which is the quantized feature vector. The quantized feature vector retains the main information of the original speech features, but is expressed in a less dimensional and more concise form.

[0111] It should be noted that when the speech features are vector quantized and encoded, a quantized feature vector can be obtained, referred to as a quantized feature vector, and a discrete representation of speech can also be obtained. The difference between the quantized feature vector and the discrete representation of speech is that the quantized feature vector is obtained by vectorizing the speech features in the continuous space (such as clustering), while the discrete representation of speech is the result of mapping the quantized feature vector to the codeword in the codebook. In the model training stage, the quantized feature vector is used for model training.

[0112] Optionally, the vector quantization encoding is implemented by a vectorization module, which is used to discretize the continuous feature vector to obtain a corresponding discretized representation. The vectorization module can be constructed based on a deep learning network, and the embodiment of the present disclosure does not limit the structure of the vectorization module. Optionally, the vectorization module can also restore the discrete representation back to a continuous feature vector.

[0113] It should be noted that, since the quantized feature vector retains the main information of the original speech features, the semantic continuous vector and the fundamental frequency continuous vector can be extracted from the quantized feature vector. Among them, the semantic continuous vector is used to represent the semantic information in the speech. Semantics refers to the meaning and content expressed by the language. The semantic information of speech includes content at multiple levels such as vocabulary, grammar, and semantic relations. The fundamental frequency continuous vector is used to represent the fundamental frequency information in the speech. The fundamental frequency refers to the lowest frequency component in the sound, also known as the fundamental frequency. In speech, the fundamental frequency determines the pitch of the sound. The fundamental frequency will be different when different people speak, and the fundamental frequency of the same person will also change under different emotions and tones.

[0114] In step S305, semantic decoding is performed on the quantized feature vector based on the semantic decoding module to obtain a semantic continuous vector.

[0115] In the disclosed embodiment, the semantic decoding module is a module built based on a deep learning network. The semantic decoding module is used to convert the quantized feature vector into a continuous vector related to the semantic content during the model training phase, referred to as a semantic continuous vector. Among them, the semantic continuous vector can more directly represent the semantic content of the speech signal. Compared with the quantized feature vector, the semantic continuous vector has continuity in the semantic space, that is, adjacent vectors are also semantically similar and coherent.

[0116] In step S306, based on the baseband decoding module, baseband decoding is performed on the quantized feature vector to obtain a baseband continuous vector.

[0117] In the disclosed embodiment, the baseband decoding module is a module constructed based on a deep learning network. The baseband decoding module is used to convert the quantized feature vector into a continuous vector related to the baseband in the model training stage, referred to as the baseband continuous vector. Among them, the baseband continuous vector can more directly represent the baseband characteristics of the speech signal. Compared with the quantized feature vector, the baseband continuous vector changes continuously in value and can reflect the dynamic changes of the baseband in the speech signal.

[0118] In step S307, a first loss is determined based on the semantic continuous vector and the reference speech text, where the first loss is used to indicate a continuous temporal classification loss between the semantic continuous vector and the reference semantic text.

[0119] In the embodiment of the present disclosure, the reference speech text corresponding to the sample speech signal is used as the supervisory information for model training. The continuous time series classification loss between the semantic continuous vector and the reference semantic text is determined as the first loss. Among them, the continuous time series classification loss is a loss function commonly used in sequence-to-sequence learning tasks, and is suitable for scenarios such as speech recognition. In the embodiment of the present disclosure, the first loss mainly considers the alignment relationship between the input sequence (the semantic continuous vector corresponding to the speech signal) and the output sequence (reference semantic text), as well as the classification accuracy at different time steps.

[0120] Optionally, the calculation process of continuous temporal classification loss includes the following steps:

[0121] First, for the input semantic continuous vector, a series of possible output sequences are predicted, which may contain repeated characters or blank characters.

[0122] Then, a dynamic programming algorithm is used to find the best alignment between the input sequence and the reference semantic text.

[0123] Finally, the loss value is calculated based on the alignment result, which reflects the degree of difference between the model prediction result and the reference semantic text.

[0124] In step S308, a second loss is determined based on the fundamental frequency continuous vector and the reference fundamental frequency information, where the second loss is used to indicate a mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information.

[0125] In the embodiment of the present disclosure, the reference fundamental frequency information is used as the supervisory information for model training. The mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information is determined as the second loss. Among them, the mean square error (MSE) is a commonly used loss function used to measure the difference between two numerical sequences. In the embodiment of the present disclosure, the second loss is used to calculate the mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information. Optionally, for each time point, the square of the difference between the corresponding fundamental frequency value in the fundamental frequency continuous vector and the corresponding fundamental frequency value in the reference fundamental frequency information is first calculated, and then the square differences of all time points are summed and averaged. The second loss can be used as an indicator to guide the model to adjust parameters during the model training process to reduce the difference between the fundamental frequency continuous vector and the reference fundamental frequency information. By minimizing the second loss, the discrete representation model of speech can gradually improve the ability to predict the fundamental frequency characteristics of the speech signal.

[0126] In step S309, a third loss is determined based on the semantic continuous vector and the hidden layer features of the sample speech signal in the continuous space, where the third loss is used to indicate the mean square error loss between the semantic continuous vector and the hidden layer features.

[0127] In the disclosed embodiment, the semantic continuous vector comes from the quantized feature vector, and the quantized feature vector is obtained from the speech features of the sample speech signal in the continuous space, and the speech features of the sample speech signal in the continuous space are obtained from the hidden layer features. Therefore, the purpose of determining the third loss is to measure the accuracy of the model in semantic feature extraction and conversion. By calculating the difference between the semantic continuous vector and the hidden layer features, the model's ability to learn and express speech semantics can be evaluated. The third loss can be used as an indicator to guide the model to adjust parameters during the model training process to reduce the difference between the semantic continuous vector and the hidden layer features. By continuously optimizing the third loss, the speech discrete representation model can better learn the semantic features of the speech signal and improve the understanding and expression of semantics.

[0128] In step S310, feature extraction is performed on a reference speech text corresponding to the sample speech signal to obtain a first text feature.

[0129] In the disclosed embodiment, the main purpose of extracting features from the reference speech text is to convert the text from the original character sequence into a more abstract and representative feature representation form, which is called the first text feature for ease of description. The first text feature can better reflect the semantics, grammar, structure and other aspects of the text, which is convenient for subsequent processing and analysis.

[0130] Optionally, feature extraction of the reference speech text is implemented by a text feature extraction module. The text feature extraction module is constructed based on a deep learning network, and the embodiment of the present disclosure does not limit the structure of the text feature extraction module. The text feature extraction module is used for text feature extraction.

[0131] For example, taking the word embedding method as an example, the reference speech text is first segmented and split into individual words or phrases. Then, each word is converted into a corresponding vector representation using a pre-trained word embedding model. Finally, these word vectors are combined or further processed to obtain the feature representation of the entire reference speech text, i.e., the first text feature.

[0132] In step S311, the first text feature is forced to be aligned based on the forced alignment module to obtain the second text feature.

[0133] In an embodiment of the present disclosure, a forced alignment module is used to accurately align text with a speech signal. The forced alignment module can determine the corresponding time position of each element in the text (such as a word, phoneme, etc.) in the speech signal. The forced alignment module is constructed based on a deep learning network, and the embodiment of the present disclosure does not limit the structure of the forced alignment module. Compared with the first text feature, the second text feature more accurately reflects the alignment relationship between text and speech. The second text feature can provide more accurate text information for subsequent speech processing tasks.

[0134] Optionally, forced alignment can be implemented using speech recognition technology and dynamic programming algorithms. First, perform preliminary speech recognition on the input speech signal to obtain a possible text transcription result. Then, compare this transcription result with the known first text feature, and find the best alignment through continuous adjustment and optimization.

[0135] In step S312, a fourth loss is determined based on the second text feature and the speech feature, where the fourth loss is used to represent the relative entropy loss between the second text feature and the speech feature.

[0136] In the disclosed embodiment, since the second text feature is obtained by forcibly aligning the first text feature. The speech feature is a set of values ​​extracted from the sample speech signal that can reflect the essential attributes of the speech signal. Therefore, by determining the fourth loss, the degree of difference between the second text feature and the speech feature can be measured, thereby guiding the optimization of the speech discrete representation model so that the text feature and the speech feature can be better matched. By minimizing the fourth loss, the speech discrete representation model can gradually learn how to better match the text feature and the speech feature, thereby improving the performance of speech processing.

[0137] The relative entropy loss between the second text feature and the speech feature is determined as the fourth loss. Relative entropy, also known as KL divergence, is an indicator used to measure the difference between two probability distributions. In the embodiment of the present disclosure, the second text feature and the speech feature are regarded as two probability distributions, and the fourth loss is determined by calculating the relative entropy between them. The calculation process usually involves estimating the probability density function of the two feature distributions, and then calculating the value of the relative entropy. If the two distributions are exactly the same, the relative entropy is zero; if the two distributions are very different, the value of the relative entropy will be larger.

[0138] In step S313, a speech discrete representation model is trained based on the first loss, the second loss, the third loss and the fourth loss.

[0139] In the disclosed embodiment, the first loss is the continuous temporal classification loss between the semantic continuous vector and the reference semantic text, which can measure the accuracy of the model in semantic understanding and conversion. By minimizing the first loss, the speech discrete representation model can better learn the semantic information in the speech signal and accurately convert it into text form. The second loss is the mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information, which can evaluate the model's prediction ability for the fundamental frequency characteristics of the speech signal. By optimizing the second loss, the speech discrete representation model can more accurately capture the pitch changes of the speech and improve the naturalness and intelligibility of the speech. The third loss is the mean square error loss between the semantic continuous vector and the hidden layer feature, which can help the speech discrete representation model to better extract and represent speech features at the semantic level. By reducing the third loss, the speech discrete representation model can more effectively convert the speech signal into a continuous vector with semantic meaning. The fourth loss is the relative entropy loss between the second text feature and the speech feature, which can reflect the degree of matching between the text feature and the speech feature. By reducing the fourth loss, the speech discrete representation model can better achieve text and speech alignment and improve the accuracy of speech processing. By training based on multiple losses, the speech discrete representation model can simultaneously optimize semantic understanding, fundamental frequency prediction, feature extraction, and text-speech alignment, thereby improving the overall performance of the model. The model trained in this way can more accurately represent the characteristics of speech signals and better implement speech processing tasks such as speech recognition and speech synthesis.

[0140] It should be noted that the above steps S309-S311 are optional steps. That is, when training the discrete speech representation model, three losses, namely the first loss, the second loss and the third loss, can be used for training, or four losses, namely the first loss, the second loss, the third loss and the fourth loss, can be used for training. Optionally, three losses, namely the first loss, the second loss and the third loss, can be used for training first, and then the fourth loss can be used for training. The disclosed embodiments do not limit this.

[0141] In order to make the training scheme of the speech discrete representation model provided in the embodiment of the present disclosure easier to understand, see Figure 4 As shown, Figure 4 FIG. 1 is a flowchart of a method for training a speech discrete representation model according to an exemplary embodiment. Figure 4As shown, the sample speech signal is input into the pre-training model, and then the hidden features output by the pre-training model are input into the feature extraction module to obtain the speech features in the continuous space. The speech features are input into the vector quantization module to obtain the quantized feature vector and the discrete representation of speech. Among them, the discrete representation of speech does not participate in the model training process. Then, based on the semantic decoding module, the quantized feature vector is semantically decoded to obtain a semantic continuous vector; based on the baseband decoding module, the quantized feature vector is baseband decoded to obtain a baseband continuous vector. Based on the text feature extraction module, the reference speech text is feature extracted to obtain the first text feature. Based on the forced alignment module, the first text feature is forced to align to obtain the second text feature. Then the first loss, the second loss, the third loss and the fourth loss are determined respectively.

[0142] The disclosed embodiment provides a training scheme for a discrete speech representation model. By introducing reference speech text and reference fundamental frequency information corresponding to a sample speech signal as supervisory information when training the discrete speech representation model, the trained discrete speech representation model can efficiently and accurately extract the discrete speech representation of the speech signal, and the discrete speech representation can fully reflect the semantic content and fundamental frequency information contained in the original speech signal without losing speaker information and prosody information, etc. Compared with the scheme of clustering to extract discrete speech features, the accuracy and effectiveness of discrete speech representation are significantly improved, so that discrete speech features can significantly improve the performance of downstream tasks such as speech synthesis and speech recognition.

[0143] Figure 5 is a flow chart of a method for processing a speech signal according to an exemplary embodiment. Figure 5 As shown, the method is performed by an electronic device and includes the following steps:

[0144] In step S501, the speech signal to be processed is input into a speech discrete representation model, and the speech discrete representation model is trained by the above-mentioned speech discrete representation model training method.

[0145] In step S502, based on the deep neural network in the discrete speech representation model, the speech signal is converted into hidden layer features in a continuous space.

[0146] In step S503, based on the deep learning network in the speech discrete representation model, feature extraction is performed on the hidden layer features to obtain the speech features of the speech signal in the continuous space.

[0147] In step S504, the speech features are vector quantized and encoded to obtain a speech discrete representation of the speech signal.

[0148] In order to make the speech signal processing solution provided by the embodiment of the present disclosure easier to understand, see Figure 6As shown, Figure 6 FIG. 1 is a flowchart of a method for processing a speech signal according to an exemplary embodiment. Figure 6 As shown, the speech signal is input into the speech discrete representation model. The speech signal is converted into hidden features in a continuous space based on the pre-trained model in the speech discrete representation model. Based on the feature extraction module in the speech discrete representation model, the hidden features are extracted to obtain the speech features of the speech signal in the continuous space. Based on the vector quantization module in the speech discrete representation model, the speech features are vector quantized and encoded to obtain the speech discrete representation and quantized feature vector of the speech signal. Among them, the quantized feature vector does not participate in the processing of the speech signal.

[0149] Figure 7 is a block diagram of a training device for a speech discrete representation model according to an exemplary embodiment. Figure 7 As shown, the device includes: an acquisition unit 701, a feature extraction unit 702 and a training unit 703.

[0150] The acquisition unit 701 is configured to acquire a sample speech signal, a reference speech text and reference fundamental frequency information, wherein the reference speech text is a correct text content corresponding to the sample speech signal, and the reference fundamental frequency information includes an actual fundamental frequency feature of the sample speech signal;

[0151] The feature extraction unit 702 is configured to input the sample speech signal into the speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector;

[0152] The training unit 703 is configured to train a speech discrete representation model based on the semantic continuous vector, the fundamental frequency continuous vector, the speech text and the reference fundamental frequency information.

[0153] In some embodiments, the feature extraction unit 702 includes:

[0154] A first extraction subunit is configured to perform feature extraction on a sample speech signal input into the speech discrete representation model to obtain speech features of the sample speech signal in a continuous space;

[0155] A vectorization subunit is configured to perform vector quantization encoding on the speech feature to obtain a quantized feature vector;

[0156] The second extraction subunit is configured to extract a semantic continuous vector and a fundamental frequency continuous vector from the quantized feature vector.

[0157] In some embodiments, the first extraction subunit is configured to convert the sample speech signal input into the discrete speech representation model into hidden layer features in a continuous space based on the deep neural network in the discrete speech representation model; and perform feature extraction on the hidden layer features based on the deep learning network in the discrete speech representation model to obtain the speech features of the sample speech signal in the continuous space.

[0158] In some embodiments, the second extraction subunit is configured to perform semantic decoding on the quantized feature vector based on the semantic decoding module to obtain a semantic continuous vector; and perform baseband decoding on the quantized feature vector based on the baseband decoding module to obtain a baseband continuous vector.

[0159] In some embodiments, the training unit 703 is configured to determine a first loss based on the semantic continuous vector and the reference speech text corresponding to the sample speech signal, the first loss being used to indicate the continuous temporal classification loss between the semantic continuous vector and the reference semantic text; determine a second loss based on the fundamental frequency continuous vector and the reference fundamental frequency information, the second loss being used to indicate the mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information; determine a third loss based on the semantic continuous vector and the hidden layer features of the sample speech signal in a continuous space, the third loss being used to indicate the mean square error loss between the semantic continuous vector and the hidden layer features; and train a speech discrete representation model based on the first loss, the second loss, and the third loss.

[0160] In some embodiments, the apparatus further comprises:

[0161] The feature extraction unit 702 is further configured to extract features from a reference speech text corresponding to the sample speech signal to obtain a first text feature;

[0162] A feature alignment unit, configured to perform forced alignment on the first text feature based on the forced alignment module to obtain a second text feature;

[0163] The training unit 703 is further configured to determine a fourth loss based on the second text feature and the speech feature, where the fourth loss is used to represent a relative entropy loss between the second text feature and the speech feature;

[0164] The training unit 703 is further configured to train a speech discrete representation model based on four losses.

[0165] The disclosed embodiment provides a training device for a speech discrete representation model. By introducing reference speech text and reference fundamental frequency information corresponding to a sample speech signal as supervisory information when training the speech discrete representation model, the trained speech discrete representation model can efficiently and accurately extract the speech discrete representation of the speech signal, and the speech discrete representation can fully reflect the semantic content and fundamental frequency information contained in the original speech signal without losing speaker information and prosody information, etc. Compared with the solution of clustering to extract speech discrete features, the accuracy and effectiveness of speech discrete representation are significantly improved, so that speech discrete features can significantly improve the performance of downstream tasks such as speech synthesis and speech recognition.

[0166] It should be noted that the training device for the discrete representation model of speech provided in the above embodiment is only illustrated by the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the training device for the discrete representation model of speech provided in the above embodiment and the training method embodiment of the discrete representation model of speech belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0167] Regarding the training device for the discrete speech representation model in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0168] Figure 8 FIG. 1 is a block diagram of a speech signal processing device according to an exemplary embodiment. Figure 8 As shown, the device includes: an input unit 801, a first processing unit 802, a second processing unit 803 and a vectorization unit 804.

[0169] An input unit 801 is configured to input a speech signal to be processed into a speech discrete representation model, wherein the speech discrete representation model is trained by the training method of the speech discrete representation model;

[0170] The first processing unit 802 is configured to convert the speech signal into a hidden layer feature in a continuous space based on a deep neural network in a discrete speech representation model;

[0171] The second processing unit 803 is configured to extract the hidden layer features based on the deep learning network in the speech discrete representation model to obtain the speech features of the speech signal in the continuous space;

[0172] The vectorization unit 804 is configured to perform vector quantization encoding on the speech features to obtain a speech discrete representation of the speech signal.

[0173] It should be noted that the speech signal processing device provided in the above embodiment is only illustrated by the division of the above functional units. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the electronic device is divided into different functional units to complete all or part of the functions described above. In addition, the speech signal processing device provided in the above embodiment and the speech signal processing method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0174] Regarding the speech signal processing device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0175] In the embodiments of the present disclosure, the electronic device may be a terminal or a server. When the electronic device is a terminal, the terminal is used as the execution subject to implement the technical solution provided by the embodiments of the present disclosure; when the electronic device is a server, the server is used as the execution subject to implement the technical solution provided by the embodiments of the present disclosure; or, the technical solution provided by the present disclosure is implemented through interaction between the terminal and the server, such as the server training a discrete speech representation model, the terminal sends the speech signal to be processed to the server, and the server processes the speech signal based on the discrete speech representation model to obtain a discrete speech representation and returns it to the terminal. The terminal obtains the discrete speech representation returned by the server. The embodiments of the present disclosure are not limited to this.

[0176] Fig. 9 9 is a block diagram of an electronic device according to an exemplary embodiment. Generally, the electronic device 900 includes: a processor 901 and a memory 902 .

[0177] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0178] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one program code, which is used to be executed by the processor 901 to implement the training method of the speech discrete representation model provided in the method embodiment of the present disclosure, or the at least one program code is used to be executed by the processor 901 to implement the speech signal processing method provided in the method embodiment of the present disclosure.

[0179] In some embodiments, the electronic device 900 may further optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902 and the peripheral device interface 903 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 903 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907 and a power supply 908.

[0180] The peripheral device interface 903 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0181] The radio frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 904 can communicate with other electronic devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 904 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0182] The display screen 905 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 905 is a touch display screen, the display screen 905 also has the ability to collect touch signals on the surface or above the surface of the display screen 905. The touch signal can be input to the processor 901 as a control signal for processing. At this time, the display screen 905 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 905 can be one, and the front panel of the electronic device 900 is set; in other embodiments, the display screen 905 can be at least two, which are respectively set on different surfaces of the electronic device 900 or are folded; in some other embodiments, the display screen 905 can be a flexible display screen, which is set on the curved surface or folded surface of the electronic device 900. Even, the display screen 905 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode, organic light-emitting diode).

[0183] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the electronic device, and the rear camera is arranged on the back of the electronic device. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0184] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 901 for processing, or input them into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 900. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0185] The power supply 908 is used to power various components in the electronic device 900. The power supply 908 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 908 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0186] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the electronic device 900, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0187] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 902 including instructions, and the instructions can be executed by a processor 901 of an electronic device 900 to complete the training method of the discrete representation model of speech, or to complete the processing method of the speech signal. Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0188] A computer program product includes a computer program, which, when executed by a processor, implements the training method of the above-mentioned discrete speech representation model, or implements the above-mentioned method for processing speech signals.

[0189] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0190] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training a discrete speech representation model, characterized in that: The method comprises: Acquire a sample speech signal, a reference speech text, and reference fundamental frequency information, wherein the reference speech text is a correct text content corresponding to the sample speech signal, and the reference fundamental frequency information includes an actual fundamental frequency feature of the sample speech signal; Inputting the sample speech signal into a speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector; The speech discrete representation model is trained based on the semantic continuous vector, the fundamental frequency continuous vector, the reference speech text and the reference fundamental frequency information.

2. The method for training a discrete speech representation model according to claim 1, characterized in that: The step of inputting the sample speech signal into a speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector comprises: Extracting features of the sample speech signal input into the speech discrete representation model to obtain speech features of the sample speech signal in a continuous space; Performing vector quantization encoding on the speech feature to obtain a quantized feature vector; The semantic continuous vector and the fundamental frequency continuous vector are extracted from the quantized feature vector.

3. The method for training a discrete speech representation model according to claim 2, characterized in that: The step of extracting features from the sample speech signal input into the speech discrete representation model to obtain speech features of the sample speech signal in a continuous space includes: Based on the deep neural network in the speech discrete representation model, converting the sample speech signal input into the speech discrete representation model into the hidden layer features in the continuous space; Based on the deep learning network in the discrete speech representation model, feature extraction is performed on the hidden layer features to obtain the speech features of the sample speech signal in the continuous space.

4. The method for training a discrete speech representation model according to claim 2, characterized in that: The step of extracting the semantic continuous vector and the fundamental frequency continuous vector from the quantized feature vector comprises: Based on a semantic decoding module, semantic decoding is performed on the quantized feature vector to obtain the semantic continuous vector; Based on the baseband decoding module, baseband decoding is performed on the quantized feature vector to obtain the baseband continuous vector.

5. The method for training a discrete speech representation model according to claim 1, characterized in that: The step of training the speech discrete representation model based on the semantic continuous vector, the fundamental frequency continuous vector, the reference speech text, and the reference fundamental frequency information includes: Determine a first loss based on the semantic continuous vector and the reference speech text, wherein the first loss is used to indicate a continuous temporal classification loss between the semantic continuous vector and the reference semantic text; Determine a second loss based on the fundamental frequency continuous vector and the reference fundamental frequency information, where the second loss is used to indicate a mean square error loss between the fundamental frequency continuous vector and the reference fundamental frequency information; Determine a third loss based on the semantic continuous vector and the hidden layer features of the sample speech signal in a continuous space, wherein the third loss is used to indicate a mean square error loss between the semantic continuous vector and the hidden layer features; The speech discrete representation model is trained based on the first loss, the second loss and the third loss.

6. The method for training a discrete speech representation model according to claim 5, characterized in that: The method further comprises: Performing feature extraction on the reference speech text to obtain a first text feature; Performing forced alignment on the first text feature based on a forced alignment module to obtain a second text feature; Based on the second text feature and the speech feature, determining a fourth loss, wherein the fourth loss is used to represent a relative entropy loss between the second text feature and the speech feature; Based on the four losses, the speech discrete representation model is trained.

7. A method for processing a speech signal, characterized in that: The method comprises: Inputting the speech signal to be processed into a speech discrete representation model, wherein the speech discrete representation model is obtained by training according to any one of claims 1 to 6; Based on the deep neural network in the discrete speech representation model, the speech signal is converted into hidden layer features in a continuous space; Based on the deep learning network in the speech discrete representation model, feature extraction is performed on the hidden layer features to obtain speech features of the speech signal in a continuous space; The speech features are vector quantized and encoded to obtain a speech discrete representation of the speech signal.

8. A training device for a discrete speech representation model, characterized in that: The device comprises: An acquisition unit is configured to acquire a sample speech signal, a reference speech text, and reference fundamental frequency information, wherein the reference speech text is a correct text content corresponding to the sample speech signal, and the reference fundamental frequency information includes an actual fundamental frequency feature of the sample speech signal; A feature extraction unit is configured to input the sample speech signal into a speech discrete representation model to obtain a semantic continuous vector and a fundamental frequency continuous vector; The training unit is configured to train the speech discrete representation model based on the semantic continuous vector, the fundamental frequency continuous vector, the speech text and the reference fundamental frequency information.

9. A speech signal processing device, characterized in that: The device comprises: An input unit, configured to input a speech signal to be processed into a speech discrete representation model, wherein the speech discrete representation model is obtained by training according to any one of claims 1 to 6; A first processing unit is configured to convert the speech signal into a hidden layer feature in a continuous space based on a deep neural network in the speech discrete representation model; A second processing unit is configured to extract the hidden layer features based on the deep learning network in the speech discrete representation model to obtain the speech features of the speech signal in a continuous space; The vectorization unit is configured to perform vector quantization encoding on the speech feature to obtain a speech discrete representation of the speech signal.

10. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the training method of the discrete speech representation model as described in any one of claims 1 to 6, or to implement the method for processing the speech signal as described in claim 7.

11. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the training method of the discrete speech representation model as described in any one of claims 1 to 6, or implement the speech signal processing method as described in claim 7.

12. A computer program product, comprising a computer program, which, when executed by a processor, implements the method for training a discrete speech representation model as described in any one of claims 1 to 6, or implements the method for processing a speech signal as described in claim 7.

Images