Voice generation method and apparatus, product, device, and medium

Abstract

A voice generation method and apparatus, a product, a device, and a medium, which are applied to the technical field of voice generation. The method comprises: using a quantizer to discretize a voice feature vector of an original voice signal, obtaining a discrete symbol representation corresponding to the original voice signal (S11); extracting a text feature corresponding to a target text (S12); and inputting the text feature and the discrete symbol representation into a voice generation model, so that the voice generation model uses the text feature as a condition, and generates a target voice on the basis of the discrete symbol representation (S13). The quantizer is obtained by training a self-organizing map network by means of voice signal training samples. The method may restore an original voice feature more accurately, and improve the quality of the generated voice.

Description

A speech generation method, apparatus, product, device, and medium

[0001] This application claims priority to Chinese Patent Application No. 202411697778.2, filed on November 25, 2024, entitled "A Speech Generation Method, Apparatus, Product, Equipment and Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of speech generation technology, and in particular to a speech generation method, apparatus, product, device and medium. Background Technology

[0003] With the development of speech generation technology, more and more fields are applying it to complete specific tasks, such as customer service, education, and smart homes. Deep learning-based speech generation algorithms analyze the input text content through extensive data training and then convert it into audio signals using audio generation models. How to more accurately reproduce the original speech features and improve the quality of generated speech remains a challenge in the field of speech generation. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a speech generation method, apparatus, product, device, and medium that can more accurately restore the original speech features and improve the quality of generated speech. The specific solution is as follows:

[0005] In a first aspect, this application discloses a speech generation method, including:

[0006] The speech feature vector of the original speech signal is discretized using a quantizer to obtain the discrete symbol representation of the original speech signal. The quantizer is obtained by training a self-organizing map network with speech signal training samples.

[0007] Extract text features corresponding to the target text;

[0008] The text features and the discrete symbol representation are input into the speech generation model so that the speech generation model uses the text features as conditions and generates the target speech based on the discrete symbol representation.

[0009] Optionally, the training process of the quantizer includes:

[0010] Obtain speech signal training samples from the training sample set;

[0011] Extract the feature vectors from the speech signal training samples to obtain the input feature vector;

[0012] The input feature vector is input into a self-organizing map network. Based on the similarity between the weight vector corresponding to the neuron in the output layer of the self-organizing map network and the input feature vector, the winning neuron is determined from the neurons in the output layer.

[0013] The neighborhood neurons corresponding to the winning neuron are determined based on the neighborhood radius;

[0014] The weight vectors corresponding to the winning neuron and the neighboring neurons are adjusted based on the learning rate.

[0015] When the training termination condition is met, the current self-organizing map network is determined as the quantizer.

[0016] Optionally, the weight vectors corresponding to the neurons in the output layer are all represented based on the mean and variance, and all neurons in the output layer of the self-organizing map network share the same mean and variance.

[0017] Optionally, after adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate, the method further includes:

[0018] Reduce the learning rate and the neighborhood radius.

[0019] Optionally, adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate includes:

[0020] According to a preset principle, and based on the learning rate, the weight vectors corresponding to the winning neuron and the neighboring neurons are adjusted, wherein the preset principle is that the activation of the neighboring neurons decreases as the distance between them and the winning neuron increases.

[0021] Optionally, before discretizing the speech feature vector of the original speech signal using a quantizer, the method further includes:

[0022] Acquire the raw speech signal;

[0023] The original speech signal is segmented to obtain speech segments;

[0024] The speech segments are encoded separately to obtain the speech feature vectors of the speech segments.

[0025] Secondly, this application discloses a speech generation device, comprising:

[0026] The vector discretization module is used to discretize the speech feature vector of the original speech signal using a quantizer to obtain the discrete symbol representation of the original speech signal. The quantizer is obtained by training the self-organizing map network with speech signal training samples.

[0027] The text feature extraction module is used to extract the text features corresponding to the target text;

[0028] The target speech generation module is used to input the text features and the discrete symbol representation into the speech generation model, so that the speech generation model uses the text features as conditions and generates target speech based on the discrete symbol representation.

[0029] Thirdly, this application discloses a computer program product that, when executed, implements the aforementioned speech generation method.

[0030] Fourthly, this application discloses an electronic device, including a memory and a processor, wherein:

[0031] The memory is used to store computer programs;

[0032] The processor is used to execute the computer program to implement the aforementioned speech generation method.

[0033] Fifthly, this application discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned speech generation method.

[0034] As can be seen from the above scheme, this application provides a speech generation method, including: discretizing the speech feature vector of the original speech signal using a quantizer to obtain the discrete symbol representation corresponding to the original speech signal, wherein the quantizer is obtained by training a self-organizing map network using speech signal training samples; extracting text features corresponding to the target text; and inputting the text features and the discrete symbol representation into a speech generation model, so that the speech generation model uses the text features as conditions and generates the target speech based on the discrete symbol representation.

[0035] As can be seen, the beneficial effects of this application are as follows: by using a trained self-organizing map network as a quantizer to discretize the speech feature vector of the original speech signal, continuous features can be mapped onto code points in a discrete codebook, and the original speech signal can be represented as a series of discrete code points. These code points can more efficiently represent and store speech information. Furthermore, the utilization rate of the quantized codebook is improved through the self-organizing map network. Through unsupervised learning, similar features are clustered into close positions, thereby forming a topological structure that can reflect the relationship between speech features. The quantized code points will more effectively represent the original speech features, enabling the subsequent speech generation model to more accurately restore the original speech features and improve the quality of generated speech.

[0036] Correspondingly, the speech generation device, product, equipment, and readable storage medium provided in this application also have the above-mentioned technical effects. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 is a flowchart of a speech generation method provided in an embodiment of this application;

[0039] Figure 2 is a schematic diagram of the structure of a self-organizing mapping network provided in an embodiment of this application;

[0040] Figure 3 is a schematic diagram of a speech generation device provided in an embodiment of this application;

[0041] Figure 4 is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0043] Vector Quantization (VQ) is a process that maps points in a vector space to a finite number of representative points (codewords) for easier storage and computation. This method is widely used in data compression, pattern recognition, neural networks, and other fields. A crucial concept in vector quantization is the codebook, which consists of all codewords and can be viewed as a set of representative points in the feature space. Each codeword represents a class of features in the input data. Codebook collapse is a common problem in vector quantization, especially when training deep learning models or autoencoders. Codebook collapse refers to the situation where, during vector quantization, most or all data points are mapped to a very small number of codewords in the codebook, or even just one codeword. This results in most codewords in the codebook being underutilized, leading to a significant decrease in representational power.

[0044] Vector quantization (VQ) techniques are widely used in deep neural networks for learning discrete representations. The core idea is to quantize features into vectors called "codenumerical vectors" through a clustering-like operation, thereby achieving efficient data compression and meaningful feature extraction. This method has achieved significant results in tasks such as image generation, representation learning, and speech generation. However, the training process of VQ networks often faces several challenges, especially training instability and the "index collapse" phenomenon. The main reason for training instability lies in the significant difference between the model embedding and the codenumerical vector distribution. This difference leads to codenumerical gradient sparsity, causing some codenumerical vectors to fail to be effectively updated during training, thus affecting the overall performance of the model. Furthermore, the asymmetry of the commitment loss also leads to misalignment in codenumerical vector allocation, further exacerbating training instability.

[0045] In speech generation tasks, low codebook utilization during vector quantization affects the generation of speech commands. How to more accurately reconstruct the original speech features and improve the quality of generated speech is a challenge in the field. To address this, this application provides a speech generation scheme that can more accurately reconstruct the original speech features and improve the quality of generated speech.

[0046] Referring to Figure 1, this application discloses a speech generation method, including:

[0047] Step S11: Discretize the speech feature vector of the original speech signal using a quantizer to obtain the discrete symbol representation of the original speech signal. The quantizer is obtained by training the self-organizing map network with speech signal training samples.

[0048] In one optional implementation, the original speech signal can be acquired; the original speech signal can be segmented to obtain speech segments; each speech segment can be encoded to obtain a speech feature vector. A quantizer is used to discretize the speech feature vector of the speech segment; that is, the speech feature vector is input into the quantizer, and the output of the quantizer is the discrete symbol representation of the original speech signal.

[0049] The process involves using an encoder to encode speech segments, obtaining speech embeddings (i.e., embedding vectors), which serve as speech feature vectors. The encoder is the key component in converting speech segments into embeddings, and can include, but is not limited to, recurrent neural networks, long short-term memory networks, gated recurrent units, Transformers (i.e., transformer models), and convolutional neural networks.

[0050] In this embodiment, during speech quantization, the original speech signal is acquired and segmented to extract features more effectively. A quantizer is used to discretize the speech feature vector, mapping continuous features onto symbols in a discrete codebook. This process represents the original speech signal as a series of discrete symbols, which can more efficiently represent and store speech information. A self-organizing map (SOM) network is employed to improve the utilization of the quantized codebook. The SOM network can use unsupervised learning to cluster similar features into nearby positions, thus forming a topological structure that reflects the relationships between speech features.

[0051] In one optional implementation, the training process of the quantizer includes:

[0052] Step 000: Obtain speech signal training samples from the training sample set.

[0053] Step 001: Extract the feature vector of the speech signal training sample to obtain the input feature vector.

[0054] This embodiment utilizes an encoder to extract feature vectors from the speech signal training samples, thereby obtaining the input feature vector. Specifically, the speech signal training samples can be segmented first, and the encoder can then extract feature vectors from the speech segments.

[0055] Step 002: Input the input feature vector into the self-organizing map network, and determine the winning neuron from the neurons of the output layer based on the similarity between the weight vector corresponding to the neuron in the output layer of the self-organizing map network and the input feature vector.

[0056] In this embodiment, the cosine or Euclidean distance between the weight vectors corresponding to neurons in the output layer of the self-organizing map network and the input feature vectors can be calculated to obtain the similarity. The neuron corresponding to the weight vector with the highest similarity is selected as the winning neuron. The weight vectors can be represented using the original self-organizing map network representation.

[0057] Step 003: Determine the neighboring neurons corresponding to the winning neuron based on the neighborhood radius.

[0058] The neighborhood radius is used to determine the neighborhood range of the winning neuron, and thus the corresponding neighboring neurons.

[0059] Step 004: Adjust the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate.

[0060] This embodiment adjusts the weight vectors corresponding to the winning neuron and the neighboring neurons based on a preset principle and the learning rate. The preset principle is that the activation of the neighboring neurons decreases as the distance between them and the winning neuron increases. After adjusting the weight vectors of the winning neuron and the neighboring neurons based on the learning rate, the learning rate and the neighborhood radius are reduced.

[0061] For example, functions such as the Mexican hat function, top hat function, and chef's hat function can be used, and the weight vectors corresponding to the winning neuron and the neighboring neurons can be adjusted based on the learning rate. Nearest neighbors encourage each other, while distant neighbors inhibit each other.

[0062] Step 005: When the training termination condition is met, the current self-organizing map network is determined as the quantizer.

[0063] In this embodiment, during the quantization process, the codewords in the codebook are organized using a SOM network. Codewords with similar speech features are grouped together to form a semantically related topology. The SOM network adaptively adjusts the relationships between codewords, making fuller use of codeword characteristics during speech quantization, reducing codeword redundancy, and improving codebook utilization. Through this process, the quantized codewords more effectively represent the original speech features, enabling subsequent speech generation tasks to more accurately reconstruct the original speech content.

[0064] It's important to note that SOM transforms input data into discrete, low-dimensional data, which is then represented as local regions or active points within the network. After initialization, the next three crucial learning processes are competition, cooperation, and adaptation. The SOM network can map arbitrary-dimensional input patterns into one- or two-dimensional graphs at the output layer while preserving their topology. Through repeated learning of input patterns, the network can make the weight vector space consistent with the probability distribution of the input patterns, i.e., probability preservation. In the output layer of the neural network, neurons compete for responses to input patterns. The winning neuron adjusts its weights to better suit the competition. Specifically, the winning neuron applies excitatory feedback to neighboring neurons and inhibitory feedback to distant neurons. In a self-organizing map network, each neuron has a connection strength (weight w), with a value between [0,1] that is normalized during initialization. During the learning process, the input pattern (i.e., the input sample) also needs to be normalized to align with the weight w. Next, the distance between the input pattern and the weights w of all neurons is calculated; this distance can be represented by a dot product. The neuron with the smallest distance is called the winner, and this is the competitive process. This embodies the core concept of Kohonen networks—"winner-takes-all," meaning only the winner outputs and updates its weight w. However, if the brain only updates individual neurons, it may lead to a lack of diversity. In fact, activation areas can be divided into near and far neighbors. Near neighbors refer to neurons with a radius of approximately 50µm to 500µm centered on the neuron that emitted the signal; far neighbors are neurons with a radius of 200µm to 2mm. More distant neurons also produce weak stimuli, but the update magnitude decreases with increasing distance. The closer to the center, the more updates, and vice versa. This relationship between interaction strength and distance can be represented by a curve resembling a "Mexican hat." The size of the adaptation is controlled by the learning rate, which decays over time, slowing down the convergence speed of the SOM.

[0065] In the SOM training process, the set of all output neurons c can be denoted as C, and the connection weight vector between neuron c and the input layer neurons can be denoted as w. The training process of the SOM algorithm includes: (1) Initialization: In order to build the SOM network, it is necessary to initialize the weight vector of each neuron. For example, randomly set the value of each weight vector to ensure that they are distributed in the input data space. The number of parameters of each node is the same as the dimension of the input (input vector), otherwise the distance value cannot be obtained when calculating the dot product. Before multiplying the node weights with the input, it is generally necessary to normalize both; (2) Find the winner: For any input vector x provided to the network, find the neuron node that best matches it. Assume that the input is D-dimensional, that is, x = {x i The weight vector is w = {w_i, i = 1, ..., D}.i,j Let {j = 1, ..., N, i = 1, ..., D} be the total number of neurons, and let N be the total number of neurons. Then the discriminant function can be the Euclidean distance:

[0066] (3) After finding the winning neural node, the next step is to update the node's parameters, including updating the weights, learning rate, and neighborhood radius. This can be done using gradient descent: Δω ji =η(t)T j,I(x) (t)(x i -ω ji );

[0067] Where, Δω ji η(t) represents the degree of adjustment of the weight vector, and η(t) represents the learning rate, which decreases over time. j,I(x) (t) represents the update level of neighboring neurons determined by their distance from the winning neuron I(x); the greater the distance, the lower the update level. t represents time. Let σ(t) represent the distance between nodes i and j, thus causing the neuron's vector to align with the input vector. σ(t) represents the neighborhood radius, which gradually decreases over time. A smaller neighborhood radius means a smaller update scope. σ0 represents the preset initial neighborhood radius, and τ... σ This indicates the preset time adjustment parameters. (4) Iteration: As learning continues (samples are input repeatedly), after each iteration, the learning rate and neighborhood radius will be updated according to the preset decay function. This allows the model to be more focused in the early stages of training and then gradually expand, promoting a wider exploration of samples. The learning rate will continuously decrease, and the neighborhood will also continuously shrink. All ownership vectors will be separated from each other in the input vector space, each representing a type of pattern in the input space, ultimately forming a cluster.

[0068] Thus, Self-Organizing Maps (SOMs) map high-dimensional input data to a low-dimensional space (such as one- or two-dimensional) while preserving the data's topological structure. This allows similar input patterns to cluster near the output layer, avoiding information loss. During learning, neurons compete with each other, with only the neuron closest to the input pattern updating its weights. This "winner-takes-all" strategy ensures the network can effectively adjust for specific input patterns. The winning neuron not only updates its own weights but also applies excitatory feedback to its neighboring neurons, while distant neighbors apply inhibitory feedback. This mechanism enhances the connections between nearby neurons, promotes pattern diversity, and avoids over-updating of a single neuron. The learning rate and neighborhood radius gradually decrease during training, allowing the model to focus on details initially and then expand to a wider input space. This gradual convergence avoids overfitting and information loss. Through repeated iterations, the weight vectors of the SOM gradually separate in the input space, eventually forming a cluster structure where each neuron represents a pattern in the input space. In other words, SOM successfully avoids the codebook collapse problem by maintaining the topology, implementing competition and cooperation mechanisms, and dynamically adjusting learning parameters, enabling the network to effectively learn and represent the distribution of input data.

[0069] In another alternative implementation, the training process of the quantizer includes:

[0070] Step 000: Obtain speech signal training samples from the training sample set.

[0071] Step 001: Extract the feature vector of the speech signal training sample to obtain the input feature vector.

[0072] Step 002: Input the input feature vector into the self-organizing map network, and determine the winning neuron from the neurons of the output layer based on the similarity between the weight vector corresponding to the neuron in the output layer of the self-organizing map network and the input feature vector.

[0073] In this embodiment, the weight vectors corresponding to the neurons in the output layer are all represented based on the mean and variance, and all neurons in the output layer share the same mean and variance. This embodiment first determines the mean and variance of the weight vectors corresponding to all neurons in the output layer of the self-organizing map network, and then represents the weight vectors of all neurons based on these mean and variance. The formula is w = mean + std * w′, where w is the weight vector, mean is the mean, std is the variance, and w′ is the calculated value, ensuring that the weight vectors are represented based on the mean and variance, and that the values ​​of the weight vectors are consistent with their original values.

[0074] Step 003: Determine the neighboring neurons corresponding to the winning neuron based on the neighborhood radius.

[0075] Step 004: Adjust the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate.

[0076] In this embodiment, the learning rate can be used to adjust the weight vectors corresponding to the winning neuron and the neighboring neurons to obtain the target vector. Since the target vectors corresponding to multiple neurons are calculated, multiple adjustment values ​​of the mean and variance are obtained. In this embodiment, the multiple adjustment values ​​of the mean and variance can be weighted and updated based on the adjustment weight parameters corresponding to multiple neurons to obtain the weighted summed mean and the weighted summed variance. Then, the mean and variance in the weight vectors of all neurons in the output layer are updated to the weighted summed mean and the weighted summed variance.

[0077] Step 005: When the training termination condition is met, the current self-organizing map network is determined as the quantizer.

[0078] Thus, based on SOM optimization, a mechanism for sharing a global mean and variance is proposed to further improve codebook utilization. All codewords are normalized under the same global mean and variance, ensuring that each codeword shares a uniform distribution feature during quantization. Under this shared global mean and variance mechanism, all codewords are updated globally; if one codeword is updated, all codewords are updated accordingly. The uniform mean and variance reduce quantization unevenness caused by differences in codeword characteristics, while enhancing the generality of codewords, enabling the speech generation model to effectively acquire information in a broader feature space.

[0079] Step S12: Extract the text features corresponding to the target text.

[0080] In this embodiment, the target text, i.e. the text to be used to generate the target speech, is obtained, and the text features are extracted by an encoder.

[0081] Step S13: Input the text features and the discrete symbol representation into the speech generation model, so that the speech generation model uses the text features as conditions and generates target speech based on the discrete symbol representation.

[0082] The speech generation model is a deep learning model trained on training samples, capable of generating speech from text. In this embodiment, during the speech generation stage, the given text is used as a conditional input. An encoder extracts text features, which are then used as conditional inputs to the speech generation model to generate the corresponding speech output. The speech generation model converts the discrete symbol representation back into a continuous speech signal through a dequantization process. In this embodiment, the speech generation model can obtain the target speech through a decoder.

[0083] In this embodiment, because the code symbols are optimized by the SOM network and processed with shared global mean and variance, the generated speech has higher quality and naturalness. The final output speech will conform to the semantics of the input text and possess high-fidelity natural speech characteristics. It can more accurately restore the original speech features and improve the quality of the generated speech.

[0084] It should be noted that in biological nervous systems, there exists a phenomenon of lateral inhibition, where the excitation of one nerve cell inhibits the activity of surrounding nerve cells. This inhibition leads to competition among nerve cells, resulting in some winning and others losing. This manifests as the excitation of winning nerve cells and the inhibition of losing nerve cells. Self-organizing (competitive) neural networks are artificial neural networks that simulate the functions of these biological nervous systems. A self-organizing mapping neural network is essentially a two-layer neural network, containing an input layer and an output layer (or competition layer, computation layer), as shown in Figure 2, which is a schematic diagram of the structure of a self-organizing mapping network provided in an embodiment of this application. Its principle is as follows: the input layer simulates the retina that perceives external input information, and the output layer simulates the cerebral cortex that responds. The number of neurons in the output layer is typically the number of clusters, representing each cluster to be formed. During training, a "competitive learning" approach is used. For each input sample, a node that best matches it in the output layer is called the activation node, also known as the winning neuron. The parameters of the activation node are then updated using stochastic gradient descent. Simultaneously, nodes near the activation node also update their parameters appropriately based on their distance. This competition is achieved through lateral inhibitory connections (negative feedback paths) between neurons. Key components of the SOM algorithm include: Input layer: Contains feature vectors from the original data. Network structure: Typically a two-dimensional grid, where each neuron represents a weight vector or a class. Distance metric: Uses Euclidean distance to measure the similarity between the input vector and the weight vector; other distances can also be used. Learning rule: SOM uses competitive learning to update weights based on the distance between the input data and neurons. The main goal of SOM is to achieve topological mapping, such that similar input data is mapped to neighboring neurons. The output layer consists of a series of neurons and can be a one-dimensional or two-dimensional structure; that is, the output layer has a topological structure, and the network is fully connected, meaning each input node is connected to all output nodes. To put it another way, SOM can map high-dimensional input data to a one- or two-dimensional space, thus also achieving dimensionality reduction. In k-means, the value of k, i.e., the number of clusters, needs to be specified in advance. However, in SOM, a topology (the geometric relationship between clusters) needs to be specified in advance, and in the topology, each node represents a cluster.

[0085] Self-Organizing Maps (SOMs) are unsupervised machine learning algorithms designed to map high-dimensional data to a low-dimensional space (typically two-dimensional) while preserving the topological structure of the input data. SOMs ensure that sample points that are close to each other in the input space are also as close as possible in the mapped space. A key characteristic of SOMs is their ability to preserve the topological relationships of the input data. This means that when input vectors are close to each other, after training, their positions in the mapped space should also be relatively close. This property allows SOMs to effectively reflect the distribution and structure of the input data. SOM training is achieved by updating codebook vectors. Codebooks are vectors representing the feature space in SOMs. During training, SOMs maintain the topological relationships between input vectors, thus ensuring a uniform distribution of codebooks in the feature space. This uniformity helps reduce the risk of codebook collapse, preventing some codebooks from being overused while others remain idle. In SOM training, not only are the codebooks closest to the input vectors (called winner units, BMUs) updated, but their surrounding neighbor codebooks are also affected. This update method, using a neighborhood function, ensures a uniform distribution of codewords, preventing all data points from concentrating on a few codewords. This neighborhood update mechanism allows the entire codebook to better adapt to the distribution of the input data. SOM gradually reduces the learning rate during training, a strategy that helps stabilize the learning process. A higher learning rate allows the model to quickly adapt to the input data, but over-adjustment can make the model overly sensitive to noise or outliers. By gradually reducing the learning rate, SOM can make smaller adjustments in the later stages of training, thereby reducing the possibility of codebook collapse and ensuring greater model stability in later training phases. Thus, self-organizing maps, by maintaining the topological structure of the input data, employing a neighborhood update mechanism, and optimizing the training process through learning rate adjustment, form an effective low-dimensional feature map. This makes SOM widely applicable in fields such as data visualization, clustering, and pattern recognition. This application first combines the traditional machine learning SOM algorithm with the current neural network VQ technique, effectively solving the codebook collapse problem.

[0086] Referring to Figure 3, this application discloses a speech generation device, including:

[0087] Vector discretization module 11 is used to discretize the speech feature vector of the original speech signal using a quantizer to obtain the discrete symbol representation of the original speech signal, wherein the quantizer is obtained by training the self-organizing map network with speech signal training samples;

[0088] Text feature extraction module 12 is used to extract text features corresponding to the target text;

[0089] The target speech generation module 13 is used to input the text features and the discrete symbol representation into the speech generation model, so that the speech generation model uses the text features as conditions and generates target speech based on the discrete symbol representation.

[0090] In an optional implementation, the device further includes a quantizer training module, specifically used for:

[0091] Obtain speech signal training samples from the training sample set;

[0092] Extract the feature vectors from the speech signal training samples to obtain the input feature vector;

[0093] The input feature vector is input into a self-organizing map network. Based on the similarity between the weight vector corresponding to the neuron in the output layer of the self-organizing map network and the input feature vector, the winning neuron is determined from the neurons in the output layer.

[0094] The neighborhood neurons corresponding to the winning neuron are determined based on the neighborhood radius;

[0095] The weight vectors corresponding to the winning neuron and the neighboring neurons are adjusted based on the learning rate.

[0096] When the training termination condition is met, the current self-organizing map network is determined as the quantizer.

[0097] In one optional implementation, the weight vectors corresponding to the neurons in the output layer are all represented based on the mean and variance, and all neurons in the output layer share the same mean and variance.

[0098] The quantizer training module is further configured to: after adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate, reduce the learning rate and the neighborhood radius.

[0099] The quantizer training module is specifically used to adjust the weight vectors corresponding to the winning neuron and the neighboring neurons according to a preset principle and based on the learning rate. The preset principle is that the activation of the neighboring neurons decreases as the distance between them and the winning neuron increases.

[0100] Furthermore, the device also includes a speech encoding module, used for: acquiring the original speech signal; segmenting the original speech signal to obtain speech segments; and encoding the speech segments respectively to obtain the speech feature vector of the speech segment.

[0101] As can be seen, the embodiments of this application utilize a trained self-organizing map network as a quantizer to discretize the speech feature vector of the original speech signal. This enables the mapping of continuous features onto code elements in a discrete codebook, representing the original speech signal as a series of discrete code elements. These code elements can more efficiently represent and store speech information. Furthermore, the utilization rate of the quantized codebook is improved through the self-organizing map network. By using unsupervised learning, similar features are clustered into nearby positions, thereby forming a topological structure that reflects the relationship between speech features. The quantized code elements will more effectively represent the original speech features, enabling the subsequent speech generation model to more accurately restore the original speech features and improve the quality of the generated speech.

[0102] This application discloses a computer program product that, when executed, implements the speech generation method disclosed in the foregoing embodiments.

[0103] For details regarding the specific process of the above-mentioned speech generation method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0104] Referring to Figure 4, this application discloses an electronic device 20, including a processor 21 and a memory 22; wherein, the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, the speech generation method disclosed in the foregoing embodiments.

[0105] For details regarding the specific process of the above-mentioned speech generation method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0106] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0107] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26; wherein, the power supply 23 is used to provide operating voltage for the various hardware devices on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0108] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the speech generation method disclosed in the foregoing embodiments.

[0109] For details regarding the specific process of the above-mentioned speech generation method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0110] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0111] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0112] The above provides a detailed description of the speech generation method, apparatus, product, device, and medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A speech generation method, characterized in that, include: The speech feature vector of the original speech signal is discretized using a quantizer to obtain the discrete symbol representation of the original speech signal. The quantizer is obtained by training a self-organizing map network with speech signal training samples. Extract text features corresponding to the target text; The text features and the discrete symbol representation are input into the speech generation model, so that the speech generation model uses the text features as conditions and generates the target speech based on the discrete symbol representation.

2. The speech generation method according to claim 1, characterized in that, The training process of the quantizer includes: Obtain speech signal training samples from the training sample set; Extract the feature vectors from the speech signal training samples to obtain the input feature vector; The input feature vector is input into a self-organizing map network. Based on the similarity between the weight vector corresponding to the neuron in the output layer of the self-organizing map network and the input feature vector, the winning neuron is determined from the neurons in the output layer. The neighborhood neurons corresponding to the winning neuron are determined based on the neighborhood radius; The weight vectors corresponding to the winning neuron and the neighboring neurons are adjusted based on the learning rate. When the training termination condition is met, the current self-organizing map network is determined as the quantizer.

3. The speech generation method according to claim 2, characterized in that, The weight vectors corresponding to the neurons in the output layer are all represented based on the mean and variance, and all neurons in the output layer share the same mean and variance.

4. The speech generation method according to claim 2, characterized in that, After adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate, the process further includes: Reduce the learning rate and the neighborhood radius.

5. The speech generation method according to claim 2, characterized in that, The adjustment of the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate includes: According to a preset principle, and based on the learning rate, the weight vectors corresponding to the winning neuron and the neighboring neurons are adjusted, wherein the preset principle is that the activation of the neighboring neurons decreases as the distance between them and the winning neuron increases.

6. The speech generation method according to any one of claims 1 to 5, characterized in that, Before discretizing the speech feature vector of the original speech signal using a quantizer, the method further includes: Acquire the raw speech signal; The original speech signal is segmented to obtain speech segments; The speech segments are encoded separately to obtain the speech feature vectors of the speech segments.

7. A speech generation device, characterized in that, include: The vector discretization module is used to discretize the speech feature vector of the original speech signal using a quantizer to obtain the discrete symbol representation of the original speech signal. The quantizer is obtained by training the self-organizing map network with speech signal training samples. The text feature extraction module is used to extract the text features corresponding to the target text; The target speech generation module is used to input the text features and the discrete symbol representation into the speech generation model, so that the speech generation model uses the text features as conditions and generates target speech based on the discrete symbol representation.

8. A computer program product, characterized in that, When the computer program product is executed, it implements the speech generation method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, Includes memory and processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the speech generation method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the speech generation method as described in any one of claims 1 to 6.

Images