A method, device, product, equipment and medium for generating speech

Abstract

The present application discloses a speech generation method, device, product, equipment and medium, which are applied to the field of speech generation technology, including: discretizing the speech feature vector of the original speech signal by a quantizer to obtain a discrete code element representation corresponding to the original speech signal, wherein the quantizer is obtained by training a self-organizing map network with speech signal training samples; extracting text features corresponding to the target text; inputting the text features and the discrete code element representation into a speech generation model, so that the speech generation model uses the text features as conditions and generates the target speech based on the discrete code element representation. In this way, the original speech features can be restored more accurately and the quality of the generated speech can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech generation technology, and in particular to a speech generation method, device, product, equipment and medium. Background Art

[0002] With the development of speech generation technology, more and more fields are applying speech generation technology to complete specific tasks, such as customer service, education, smart home and other fields. The speech generation algorithm based on deep learning analyzes the input text content through a large amount of data training, and then converts it into an audio signal through an audio generation model. How to more accurately restore the original speech features and improve the quality of the generated speech is a problem faced by the speech generation field. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a speech generation method, device, product, equipment and medium, which can more accurately restore the original speech features and improve the quality of generated speech. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses a speech generation method, comprising:

[0005] Discretizing the speech feature vector of the original speech signal by using a quantizer to obtain a discrete code element representation corresponding to the original speech signal, wherein the quantizer is obtained by training a self-organizing map network using speech signal training samples;

[0006] Extract text features corresponding to the target text;

[0007] The text features and the discrete code element representation are input into a speech generation model so that the speech generation model generates target speech based on the discrete code element representation and taking the text features as conditions.

[0008] Optionally, the training process of the quantizer includes:

[0009] Acquire speech signal training samples from a training sample set;

[0010] Extracting a feature vector of the speech signal training sample to obtain an input feature vector;

[0011] Inputting the input feature vector into a self-organizing map network, and determining a winning neuron from neurons in the output layer based on similarity between a weight vector corresponding to neurons in an output layer of the self-organizing map network and the input feature vector;

[0012] Determine the neighboring neuron corresponding to the winning neuron based on the neighborhood radius;

[0013] Adjust the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate;

[0014] When the training end condition is met, the current self-organizing map network is determined as the quantizer.

[0015] Optionally, the weight vectors corresponding to the neurons in the output layer are all represented based on mean and variance, and all neurons in the output layer of the self-organizing map network share the same mean and variance.

[0016] Optionally, after adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate, the method further includes:

[0017] Reduce the learning rate and the neighborhood radius.

[0018] Optionally, adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate includes:

[0019] According to a preset principle and based on a learning rate, the weight vectors corresponding to the winning neuron and the neighboring neurons are adjusted, wherein the preset principle is that the excitation of the neighboring neurons decreases as the distance between the winning neuron and the neighboring neurons increases.

[0020] Optionally, before the speech feature vector of the original speech signal is discretized by the quantizer, the method further includes:

[0021] Obtaining the original speech signal;

[0022] Segmenting the original speech signal to obtain speech segments;

[0023] The speech segments are respectively encoded to obtain speech feature vectors of the speech segments.

[0024] In a second aspect, the present application discloses a speech generating device, comprising:

[0025] A vector discretization module, used for discretizing the speech feature vector of the original speech signal by using a quantizer to obtain a discrete code element representation corresponding to the original speech signal, wherein the quantizer is obtained by training a self-organizing map network using speech signal training samples;

[0026] A text feature extraction module is used to extract text features corresponding to the target text;

[0027] The target speech generation module is used to input the text features and the discrete code element representation into a speech generation model so that the speech generation model takes the text features as conditions and generates the target speech based on the discrete code element representation.

[0028] In a third aspect, the present application discloses a computer program product, which implements the aforementioned speech generation method when executed.

[0029] In a fourth aspect, the present application discloses an electronic device, including a memory and a processor, wherein:

[0030] The memory is used to store the computer program;

[0031] The processor is used to execute the computer program to implement the aforementioned speech generation method.

[0032] In a fifth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned speech generation method when executed by a processor.

[0033] It can be seen from the above scheme that the present application provides a speech generation method, including: using a quantizer to discretize the speech feature vector of the original speech signal to obtain a discrete code element representation corresponding to the original speech signal, wherein the quantizer is obtained by training a self-organizing map network using speech signal training samples; extracting text features corresponding to the target text; inputting the text features and the discrete code element representation into a speech generation model, so that the speech generation model uses the text features as conditions and generates the target speech based on the discrete code element representation.

[0034] It can be seen that the beneficial effects of the present application are: using the trained self-organizing map network as a quantizer to discretize the speech feature vector of the original speech signal, it is possible to map continuous features to code elements in a discrete code book, and represent the original speech signal as a series of discrete code elements. These code elements can more efficiently represent and store speech information, and the utilization rate of the quantization code book is improved through the self-organizing map network, and similar features are clustered to similar positions through unsupervised learning, thereby forming a topological structure that can reflect the relationship between speech features. The quantized code elements will more effectively represent the original speech features, so that the subsequent speech generation model can more accurately restore the original speech features and improve the quality of the generated speech.

[0035] Correspondingly, a speech generation device, product, equipment and readable storage medium provided by the present application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0037] Figure 1 A flow chart of a speech generation method provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of the structure of a self-organizing map network provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of a speech generating device provided in an embodiment of the present application;

[0040] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0042] VQ (Vector Quantization) is a process of mapping points in a vector space to a limited number of representative points (codewords) for easy storage and calculation. This method is widely used in fields such as data compression, pattern recognition, and neural networks. In the process of vector quantization, an important concept is the codebook, which consists of all codewords and can be regarded as a set of representative points in the feature space. Each codeword represents a class of features in the input data. Codebook collapse is a common problem in vector quantization, especially when training deep learning models or autoencoders. Codebook collapse refers to the process in which most or all data points are mapped to very few codewords in the codebook, or even to one codeword. This results in most codewords in the codebook not being fully utilized, resulting in a significant decrease in representation capability.

[0043] Vector quantization technology is widely used in deep neural networks to learn discrete representations. Its core idea is to quantize features into vectors called "codeword vectors" through clustering-like operations, thereby achieving efficient data compression and meaningful feature extraction. This method has achieved remarkable results in tasks such as image generation, representation learning, and speech generation. However, the training process of VQ networks often faces multiple challenges, especially training instability and "index collapse" phenomena. The main reason for training instability is that there is a significant difference between model embedding and codeword vector distribution. This difference leads to codebook gradient sparsity, which makes some codeword vectors fail to be effectively updated during training, thus affecting the overall performance of the model. In addition, the asymmetry of commitment loss also leads to misalignment of codeword vector distribution, further exacerbating training instability.

[0044] In the speech generation task, there is also the problem of low codebook utilization in the vector quantization process, which affects the generation of speech commands. How to more accurately restore the original speech features and improve the quality of the generated speech is a problem faced by the speech generation field. To this end, the present application provides a speech generation solution that can more accurately restore the original speech features and improve the quality of the generated speech.

[0045] See also Figure 1 As shown, the embodiment of the present application discloses a speech generation method, including:

[0046] Step S11: discretizing the speech feature vector of the original speech signal by using a quantizer to obtain a discrete code element representation corresponding to the original speech signal, wherein the quantizer is obtained by training a self-organizing map network using speech signal training samples.

[0047] In an optional implementation, an original speech signal may be obtained; the original speech signal may be segmented to obtain speech segments; the speech segments may be encoded respectively to obtain speech feature vectors of the speech segments. The speech feature vectors of the speech segments may be discretized using a quantizer, that is, the speech feature vectors are input into the quantizer, and the output of the quantizer is the discrete code element representation corresponding to the original speech signal.

[0048] The encoder can be used to encode the speech fragment to obtain a speech embedding (i.e., an embedding vector) as a speech feature vector. The encoder is a key component for converting speech fragments into embeddings, and may include but is not limited to recurrent neural networks, long short-term memory networks, gated recurrent units, transformers (i.e., converter models), and convolutional neural networks.

[0049] In this embodiment, when performing speech quantization, the original speech signal is obtained and segmented so as to more effectively extract features therein. The speech feature vector is discretized using a quantizer, and the continuous features are mapped to code elements in a discrete code book. Through this process, the original speech signal is represented as a series of discrete code elements, which can more efficiently represent and store speech information. A self-organizing map (SOM) network is used to improve the utilization rate of the quantization code book. The self-organizing map network can aggregate similar features to similar positions through unsupervised learning, thereby forming a topological structure that can reflect the relationship between speech features.

[0050] In an optional implementation, the training process of the quantizer includes:

[0051] Step 000: Acquire speech signal training samples from a training sample set.

[0052] Step 001: extract the feature vector of the speech signal training sample to obtain an input feature vector.

[0053] In this embodiment, the encoder may be used to extract the feature vector of the speech signal training sample to obtain the input feature vector. Specifically, the speech signal training sample may be segmented first, and the encoder may be used to extract the feature vector of the speech segment.

[0054] Step 002: Input the input feature vector into a self-organizing map network, and determine a winning neuron from the neurons in the output layer based on the similarity between the weight vector corresponding to the neurons in the output layer of the self-organizing map network and the input feature vector.

[0055] In this embodiment, the cosine distance or Euclidean distance between the weight vector corresponding to the neuron in the output layer of the self-organizing map network and the input feature vector can be calculated to obtain the similarity. The neuron corresponding to the weight vector with the highest similarity is taken as the winning neuron. Among them, the weight vector can be represented by the original self-organizing map network.

[0056] Step 003: Determine the neighboring neurons corresponding to the winning neuron based on the neighborhood radius.

[0057] Among them, the neighborhood radius is used to determine the neighborhood range of the winning neuron, and then determine the corresponding field neuron.

[0058] Step 004: Adjust the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate.

[0059] In this embodiment, the weight vectors corresponding to the winning neuron and the neighboring neurons can be adjusted according to a preset principle and based on a learning rate, wherein the preset principle is that the excitation of the neighboring neurons decreases as the distance between the winning neuron and the neighboring neurons increases. After adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate, the learning rate and the neighborhood radius are reduced.

[0060] For example, a Mexican hat function, a top hat function, a chef hat function, etc. can be used, and the weight vectors corresponding to the winning neuron and the neighboring neurons can be adjusted based on the learning rate. Near neighbors encourage each other, and distant neighbors inhibit each other.

[0061] Step 005: When the training end condition is met, the current self-organizing map network is determined as the quantizer.

[0062] In the quantization process of this embodiment, the code elements in the code book are organized through the SOM network, and code elements with similar speech features are grouped together to form a topological graph with semantic relevance. The SOM network adaptively adjusts the relationship between each code element, so that the characteristics of the code element are more fully utilized in the speech quantization process, code element redundancy is reduced, and the utilization rate of the code book is improved. Through this process, the quantized code element will more effectively represent the original speech features, so that the subsequent speech generation task can more accurately restore the original speech content.

[0063] It should be pointed out that SOM converts input data into discrete low-dimensional data, which is then represented as a local area or an active point in the network. After the initialization step is completed, the next three important learning processes are competition, collaboration, and adaptation. The SOM network can map any dimensional input pattern into a one-dimensional or two-dimensional graph in the output layer and keep its topological structure unchanged; the network can make the weight vector space consistent with the probability distribution of the input pattern through repeated learning of the input pattern, that is, probability preservation. In the output layer of the neural network, each neuron will compete for the response to the input pattern. The winning neuron will adjust its weight to make it more conducive to competition. Specifically, the winning neuron applies excitatory feedback to the neighboring neurons and inhibitory feedback to the distant neurons. In the self-organizing map network, each neuron has a connection strength (weight w), whose value is between [0,1] and is normalized at initialization. During the learning process, the input pattern (i.e., input sample) also needs to be normalized to keep it consistent with the weight w. Next, the distance between the input pattern and the weight w of all neurons is calculated, which can be represented by the dot product. The neuron with the smallest distance is called the winner, and this is the process of competition. This reflects the core concept of the Kohonen network - "winner takes all", that is, only the winner will output and update its weight w. However, if the brain only updates individual neurons, it may lead to a lack of diversity. In fact, the activation area can be divided into near neighbors and far neighbors. Near neighbors refer to neurons with a radius of about 50um to 500um centered on the neuron that sends the signal; far neighbors are neurons with a radius of 200um to 2mm. Neurons farther away will also produce weak excitations, but the update amplitude will decrease with increasing distance. The closer to the center, the more updates, and vice versa, the fewer updates. This relationship between interaction intensity and distance can be represented by a curve similar to a "Mexican hat". The size of the adaptation is controlled by the learning rate, which decays over time, which slows down the convergence rate of the SOM.

[0064] In the SOM training process, the set of all output neurons c can be recorded as C, and the connection weight vector between neuron c and the input layer neuron is w. The training process of the SOM algorithm includes: (1) Initialization: In order to build a S0M network, it is necessary to initialize the weight vector of each neuron. For example, randomly set the value of each weight vector to ensure that they are distributed in the input data space. The number of parameters of each node is the same as the dimension of the Input (input vector), otherwise the distance value cannot be obtained when calculating the dot product. Before multiplying the node weight and the input, both need to be normalized; (2) Find the winner: For any input vector x provided to the network, find the neuron node that best matches it. Assume that the input is D-dimensional, that is , the weight vector is , N is the total number of neurons, then the discriminant function can be the Euclidean distance:

[0065] ;

[0066] (3) After finding the winning neural node, the next step is to update the node parameters, including updating the weight, learning rate, and neighborhood radius. This can be done using the gradient descent method:

[0067] ;

[0068] ;

[0069] ;

[0070] in, Indicates the degree of adjustment of the weight vector, represents the learning rate, which decreases over time, It represents the updating degree of the neighboring neurons determined by the distance from the winning neuron I(x). The farther the distance, the lower the updating degree. t is the time, Represents the distance between nodes i and j, so that the neuron vector moves closer to the direction of the input vector. Represents the neighborhood radius. As time goes by, the neighborhood radius gradually decreases. The smaller the neighborhood radius, the smaller the scope of the update. represents the preset initial neighborhood radius, Represents the preset time adjustment parameter. (4) Iteration: As learning continues (cyclic input of samples), after each iteration, the learning rate and neighborhood radius will be updated according to the preset decay function. This allows the model to be more concentrated in the early stages of training and then gradually diffuse, promoting a wider exploration of samples. The learning rate will continue to decrease, the neighborhood will continue to shrink, and the ownership vectors will separate from each other in the input vector space, each representing a class of patterns in the input space, and eventually forming a cluster.

[0071] It can be seen that the self-organizing map maps high-dimensional input data to a low-dimensional space (such as one or two dimensions) and keeps the topological structure of the data unchanged. This allows similar input patterns to cluster near the output layer, avoiding information loss. During the learning process, there is competition between neurons, and only the neurons closest to the input pattern update their weights. This "winner takes all" strategy ensures that the network can effectively adjust to a specific input pattern. The winning neuron not only updates its own weight, but also applies excitatory feedback to its neighboring neurons, and inhibitory feedback to distant neighbors. This mechanism enhances the mutual connection between similar neurons, promotes the diversity of patterns, and avoids excessive updates of single neurons. The learning rate and neighborhood radius gradually decrease during the training process, allowing the model to focus on details in the early stage and then expand to a wider input space. This gradual convergence method avoids overfitting and information loss. Through repeated iterations, the weight vectors of the SOM gradually separate from each other in the input space, and finally form a clustering structure, with each neuron representing a pattern in the input space. That is, SOM successfully avoids the problem of codebook collapse by maintaining the topological structure, implementing competition and cooperation mechanisms, and dynamically adjusting learning parameters, enabling the network to effectively learn and represent the distribution of input data.

[0072] In another optional implementation, the training process of the quantizer includes:

[0073] Step 000: Acquire speech signal training samples from a training sample set.

[0074] Step 001: extract the feature vector of the speech signal training sample to obtain an input feature vector.

[0075] Step 002: Input the input feature vector into a self-organizing map network, and determine a winning neuron from the neurons in the output layer based on the similarity between the weight vector corresponding to the neurons in the output layer of the self-organizing map network and the input feature vector.

[0076] The weight vectors corresponding to the neurons in the output layer are all represented based on mean and variance, and all neurons in the output layer share the same mean and variance. In this embodiment, the mean and variance of the weight vectors corresponding to all neurons in the output layer of the self-organizing map network can be determined first, and the weight vectors corresponding to all neurons are represented based on the mean and variance. The formula used is , where w is the weight vector, mean is the mean, and std is the variance. The calculated value is the weight vector represented by the mean and variance, and the value of the weight vector is consistent with the original value.

[0077] Step 003: Determine the neighboring neurons corresponding to the winning neuron based on the neighborhood radius.

[0078] Step 004: Adjust the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate.

[0079] The embodiment of the present application can use the learning rate to adjust and calculate the weight vectors corresponding to the winning neuron and the neighboring neurons to obtain the target vector. Since the corresponding target vectors of multiple neurons are calculated, that is, multiple adjustment values ​​of the mean and the variance are obtained, the embodiment of the present application can perform weighted calculations and updates on the multiple adjustment values ​​of the mean and the variance based on the adjustment weight parameters corresponding to the multiple neurons, respectively, to obtain the weighted summed mean and the weighted summed variance, and then update the mean and variance in the weight vectors of all neurons in the output layer to the weighted summed mean and the weighted summed variance.

[0080] Step 005: When the training end condition is met, the current self-organizing map network is determined as the quantizer.

[0081] In this way, based on SOM optimization, a mechanism for sharing global mean and variance is proposed to further improve codebook utilization. All code elements are normalized under the same global mean and variance, so that each code element can share a unified distribution feature during quantization. Under the global mean and variance sharing mechanism, global management is implemented, and as long as one code element is updated, all code elements will be updated accordingly. The unified mean and variance reduce the quantization unevenness caused by differences in code element characteristics, and at the same time enhance the versatility of code elements, so that the speech generation model can effectively obtain information in a wider feature space.

[0082] Step S12: extracting text features corresponding to the target text.

[0083] This embodiment obtains the target text, that is, the text for which the target speech is to be generated, and extracts text features through an encoder.

[0084] Step S13: inputting the text features and the discrete code element representation into a speech generation model, so that the speech generation model takes the text features as conditions and generates target speech based on the discrete code element representation.

[0085] Among them, the speech generation model is a deep learning model that can generate speech from text after training with training samples. In the speech generation stage, the embodiment of the present application uses a given text as a conditional input, extracts text features through an encoder, and uses them as conditional inputs of the speech generation model to generate corresponding speech outputs. The speech generation model converts the discrete code element representation back to a continuous speech signal through a dequantization process. In this embodiment, the speech generation model can obtain the target speech through a decoder.

[0086] In this embodiment, since the code elements are optimized by the SOM network and share the global mean and variance processing, the generated speech has higher quality and naturalness. The final output speech will conform to the semantics of the input text and have high-fidelity natural speech characteristics. It can more accurately restore the original speech features and improve the quality of the generated speech.

[0087] It should be pointed out that in the biological nervous system, there is a lateral inhibition phenomenon, that is, after a nerve cell is excited, it will inhibit the rest of the surrounding nerve cells. This inhibition will cause competition between nerve cells, with the result that some win and others lose. The manifestation is that the winning nerve cells are excited and the losing nerve cells are inhibited. The self-organizing (competitive) neural network is an artificial neural network that simulates the functions of the above-mentioned biological nervous system. The self-organizing map neural network is essentially a two-layer neural network, including an input layer and an output layer (also called a competition layer, a calculation layer), see Figure 2 As shown, Figure 2 A schematic diagram of the structure of a self-organizing map network provided for an embodiment of the present application. The principle is that the input layer simulates the retina that perceives external input information, and the output layer simulates the cerebral cortex that responds. The number of neurons in the output layer is usually the number of clusters, representing each class that needs to be clustered. During training, the "competitive learning" method is adopted. Each input sample finds a node that best matches it in the output layer, which is called an activation node, also called a winning-neuron; then the parameters of the activation node are updated using the random gradient descent method; at the same time, the points adjacent to the activation node also update the parameters appropriately according to their distance from the activation node. This competition can be achieved through lateral inhibitory connections (negative feedback paths) between neurons. The key components of the SOM algorithm include: Input layer: Feature vector containing the original data. Network structure: Usually a two-dimensional grid, each neuron represents a weight vector, or a category. Distance metric: Use Euclidean distance to measure the similarity between the input vector and the weight vector, of course, other distances can also be used. Learning rule: SOM uses competitive learning to update weights based on the distance between the input data and the neuron. The main goal of SOM is to achieve topological mapping so that similar input data is mapped to adjacent neurons. The output layer consists of a series of neurons, which can be a one-dimensional structure or a two-dimensional structure, that is, the output layer is a topological structure, and the network is fully connected, that is, each input node is connected to all output nodes. Thinking in another way, SOM can map high-dimensional input data to one-dimensional or two-dimensional space, which means that it also plays a role in dimensionality reduction. In k-means (i.e., K-means algorithm), the value of k, that is, the number of clusters, needs to be specified in advance. However, in SOM, a topological structure (the geometric relationship between clusters) needs to be specified in advance, and in the topological structure, a node is a cluster.

[0088] Self-organizing map is an unsupervised learning machine learning algorithm that aims to map high-dimensional data to a low-dimensional space (usually two-dimensional) and maintain the topological structure of the input data. Through this mapping, SOM makes sample points that are close to each other in the input space as close as possible in the mapped space. The key feature of SOM is that it can maintain the topological relationship of the input data. This means that when the input vectors are close to each other, their positions in the mapped space should also be relatively close after training. This feature enables SOM to effectively reflect the distribution and structure of the input data. The training of SOM is achieved by updating codebook vectors. Codewords are vectors that represent the feature space in SOM. During the training process, SOM will maintain the topological relationship between input vectors so that the codewords are evenly distributed in the feature space. This uniformity helps to reduce the risk of codebook collapse, that is, to avoid some codewords being overused and causing other codewords to be idle. In the training of SOM, not only the codeword closest to the input vector (called the winner unit, BMU) will be updated, but also the surrounding neighbor codewords will be affected. This update method through the neighborhood function ensures the uniform distribution of codewords and avoids all data points being concentrated on a few codewords. This neighborhood update mechanism enables the entire codebook to better adapt to the distribution of input data. SOM gradually reduces the learning rate during training, and this strategy helps to stabilize the learning process. A higher learning rate can enable the model to quickly adapt to the input data, but over-adjustment may cause the model to be overly sensitive to noise or outliers. By gradually reducing the learning rate, SOM can make smaller adjustments in the later stages of training, thereby reducing the possibility of codebook collapse and ensuring that the model is more stable in the later stages of training. In this way, the self-organizing map maintains the topological structure of the input data, adopts a neighborhood update mechanism, and optimizes the training process through learning rate adjustment to form an effective low-dimensional feature map. This makes SOM have broad application potential in fields such as data visualization, clustering, and pattern recognition. This application first combines the SOM algorithm of traditional machine learning with the current neural network VQ technology to effectively solve the problem of codebook collapse.

[0089] See also Figure 3 As shown, the embodiment of the present application discloses a speech generating device, comprising:

[0090] A vector discretization module 11 is used to discretize the speech feature vector of the original speech signal by using a quantizer to obtain a discrete code element representation corresponding to the original speech signal, wherein the quantizer is obtained by training a self-organizing map network using speech signal training samples;

[0091] A text feature extraction module 12 is used to extract text features corresponding to the target text;

[0092] The target speech generation module 13 is used to input the text features and the discrete code element representation into a speech generation model, so that the speech generation model takes the text features as conditions and generates the target speech based on the discrete code element representation.

[0093] In an optional implementation, the device further includes a quantizer training module, specifically configured to:

[0094] Acquire speech signal training samples from a training sample set;

[0095] Extracting a feature vector of the speech signal training sample to obtain an input feature vector;

[0096] Inputting the input feature vector into a self-organizing map network, and determining a winning neuron from neurons in the output layer based on similarity between a weight vector corresponding to neurons in an output layer of the self-organizing map network and the input feature vector;

[0097] Determine the neighboring neuron corresponding to the winning neuron based on the neighborhood radius;

[0098] Adjust the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate;

[0099] When the training end condition is met, the current self-organizing map network is determined as the quantizer.

[0100] In an optional implementation, the weight vectors corresponding to the neurons in the output layer are all represented based on the mean and variance, and all neurons in the output layer share the same mean and variance.

[0101] The quantizer training module is further used to: after adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate, reduce the learning rate and the neighborhood radius.

[0102] The quantizer training module is specifically used to adjust the weight vectors corresponding to the winning neuron and the neighboring neurons according to a preset principle and based on a learning rate, wherein the preset principle is that the excitation of the neighboring neurons decreases as the distance between the winning neuron and the neighboring neurons increases.

[0103] Furthermore, the device also includes a speech encoding module, which is used to: obtain an original speech signal; segment the original speech signal to obtain speech segments; and encode the speech segments respectively to obtain speech feature vectors of the speech segments.

[0104] It can be seen that the embodiment of the present application uses the trained self-organizing map network as a quantizer to discretize the speech feature vector of the original speech signal, and can map continuous features to code elements in a discrete code book, and represent the original speech signal as a series of discrete code elements. These code elements can more efficiently represent and store speech information, and the utilization rate of the quantization code book is improved through the self-organizing map network, and similar features are clustered to similar positions through unsupervised learning, thereby forming a topological structure that can reflect the relationship between speech features. The quantized code elements will more effectively represent the original speech features, so that the subsequent speech generation model can more accurately restore the original speech features and improve the quality of generated speech.

[0105] The embodiment of the present application discloses a computer program product, which, when executed, implements the speech generation method disclosed in the above embodiment.

[0106] For the specific process of the above-mentioned speech generation method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be elaborated here.

[0107] See also Figure 4 As shown, an embodiment of the present application discloses an electronic device 20, including a processor 21 and a memory 22; wherein the memory 22 is used to store a computer program; and the processor 21 is used to execute the computer program, the speech generation method disclosed in the above embodiment.

[0108] For the specific process of the above-mentioned speech generation method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be elaborated here.

[0109] Furthermore, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0110] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26; wherein the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and an external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input / output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0111] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the speech generation method disclosed in the aforementioned embodiment.

[0112] For the specific process of the above-mentioned speech generation method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be elaborated here.

[0113] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0114] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0115] The above is a detailed introduction to a speech generation method, device, product, equipment and medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A speech generation method, characterized in that: include: Discretizing the speech feature vector of the original speech signal by using a quantizer to obtain a discrete code element representation corresponding to the original speech signal, wherein the quantizer is obtained by training a self-organizing map network using speech signal training samples; Extract text features corresponding to the target text; Inputting the text features and the discrete code element representation into a speech generation model, so that the speech generation model generates a target speech based on the discrete code element representation and taking the text features as conditions; The training process of the quantizer includes: Acquire speech signal training samples from a training sample set; Extracting a feature vector of the speech signal training sample to obtain an input feature vector; The input feature vector is input into the self-organizing map network, and based on the similarity between the weight vector corresponding to the neurons in the output layer of the self-organizing map network and the input feature vector, the winning neuron is determined from the neurons in the output layer; the weight vectors corresponding to the neurons in the output layer are all represented based on the mean and variance, and all neurons in the output layer share the same mean and variance; the formula is: , where w is the weight vector, mean is the mean, and std is the variance. is the calculated value, so that the weight vector is represented based on the mean and variance; Determine the neighboring neuron corresponding to the winning neuron based on the neighborhood radius; Adjust the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate; When the training end condition is met, the current self-organizing map network is determined as the quantizer.

2. The speech generation method according to claim 1, characterized in that: After adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate, the method further includes: Reduce the learning rate and the neighborhood radius.

3. The speech generation method according to claim 1, characterized in that: The adjusting the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate includes: According to a preset principle and based on a learning rate, the weight vectors corresponding to the winning neuron and the neighboring neurons are adjusted, wherein the preset principle is that the excitation of the neighboring neurons decreases as the distance between the winning neuron and the neighboring neurons increases.

4. The speech generation method according to any one of claims 1 to 3, characterized in that: Before the speech feature vector of the original speech signal is discretized by the quantizer, the method further comprises: Obtaining the original speech signal; Segmenting the original speech signal to obtain speech segments; The speech segments are respectively encoded to obtain speech feature vectors of the speech segments.

5. A speech generating device, characterized in that: include: A vector discretization module, used for discretizing the speech feature vector of the original speech signal by using a quantizer to obtain a discrete code element representation corresponding to the original speech signal, wherein the quantizer is obtained by training a self-organizing map network using speech signal training samples; A text feature extraction module is used to extract text features corresponding to the target text; A target speech generation module, used for inputting the text features and the discrete code element representation into a speech generation model, so that the speech generation model takes the text features as conditions and generates the target speech based on the discrete code element representation; The training process of the quantizer includes: Acquire speech signal training samples from a training sample set; Extracting a feature vector of the speech signal training sample to obtain an input feature vector; The input feature vector is input into the self-organizing map network, and based on the similarity between the weight vector corresponding to the neurons in the output layer of the self-organizing map network and the input feature vector, the winning neuron is determined from the neurons in the output layer; the weight vectors corresponding to the neurons in the output layer are all represented based on the mean and variance, and all neurons in the output layer share the same mean and variance; the formula is: , where w is the weight vector, mean is the mean, and std is the variance. is the calculated value, so that the weight vector is represented based on the mean and variance; Determine the neighboring neuron corresponding to the winning neuron based on the neighborhood radius; Adjust the weight vectors corresponding to the winning neuron and the neighboring neurons based on the learning rate; When the training end condition is met, the current self-organizing map network is determined as the quantizer.

6. A computer program product, characterized in that When the computer program product is executed, the speech generation method according to any one of claims 1 to 4 is implemented.

7. An electronic device, characterized in that: comprising a memory and a processor, wherein: The memory is used to store the computer program; The processor is used to execute the computer program to implement the speech generation method according to any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the speech generation method according to any one of claims 1 to 4 is implemented.

Images