Improved pathological voice production model and construction method thereof

Abstract

The application relates to an improved pathological voice generation model and a construction method thereof, which comprises the following steps: S1, using all available pathological / a / vowel speech records under normal pitch in a sound data set as a training data set; S2, a training process is divided into training of a generator and training of a discriminator, both of which are alternately performed, the generator comprises a latent mapping network module and a transposed convolution and multi-receptive field fusion module; the discriminator comprises four discriminators with different cycle settings, which are used for evaluating and training intermediate waveform outputs after each upsampling of the generator; and S3, inputting a random noise vector into the trained generator, converting the random noise vector into new audio data by the generator, performing standardization processing on the generated audio data, and repeating the above steps to generate multiple new data.

Description

Technical Field

[0001] This invention belongs to the field of audio generation, specifically relating to an improved pathological voice generation model and its construction method. Background Technology

[0002] Following the widespread success of generative adversarial networks (GANs) in image generation, audio generation has also attracted researchers' attention. In recent years, doctors and researchers have increasingly focused on the diagnosis of voice disorders and have adopted computer methods such as neural networks for analysis. More and more researchers are beginning to explore the use of GAN-based models to generate various audio formats, such as musical instrument sounds and natural environmental sounds.

[0003] Currently, the mainstream audio generation models fall into three main categories: neural networks, diffusion models, and generative adversarial networks (GANs). Most audio generation networks target topics such as speech and music. However, the problems with using neural networks (e.g., WaveNet) and diffusion models (e.g., DiffWave) in the generation process are: 1. They require significant computational resources and storage space, and necessitate careful tuning of many parameters. Existing GANs (e.g., WaveGAN) produce poor quality audio on pathological voice data, exhibiting an inability to control the characteristics or style of the generated audio and failing to generate audio with specific features tailored to specific conditions or needs. 2. They do not consider the periodic patterns in the audio data, resulting in a lack of strong correlation between adjacent periods, large frequency variations, and the generation of high-frequency hissing sounds, thus reducing the subjective quality of the audio. 3. When processing pathological voice data, they struggle to fully learn the multi-scale features and information at different frequencies, leading to the loss of key sound features and inconsistent changes in the audio's features and frequency distribution, resulting in audio that fails to accurately represent the pathological state.

[0004] Therefore, generating data for pathological voice datasets remains a challenging problem. Further research and exploration are needed to develop high-quality data generation methods suitable for pathological voice data, in order to improve the richness and diversity of the datasets and thus effectively improve the performance of classification networks. Summary of the Invention

[0005] The purpose of this invention is to provide an improved pathological voice generation model and its construction method to solve the problems of low quality of generated pathological voices in the existing technology when processing pathological voice data, such as the inability to control the generated audio features or styles, the lack of strong correlation between adjacent periods, the difficulty in fully learning multi-scale features and information at different frequencies.

[0006] To achieve the above objectives, the specific technical solution of the present invention is as follows:

[0007] A method for constructing an improved pathological voice generation model includes the following steps:

[0008] S1. Preprocessing: All available pathological / a / vowel speech records at normal pitch in the Saarbruecken sound dataset were used as the training dataset;

[0009] S2. Network Training: The training process is divided into generator training and discriminator training, which are carried out alternately. The generator includes: a latent mapping network module, which is used to generate voice data with specific features and attributes; a transposed convolution and multi-receptive field fusion module, which is used to capture sound features in different frequency ranges; the discriminator includes: four discriminators with different period settings, which are used to evaluate and train the intermediate waveform output after each upsampling of the generator.

[0010] S3. Data Generation: Input the random noise vector into the pre-trained generator. The generator converts the random noise vector into new audio data. Standardize the generated audio data. Repeat the above steps to generate multiple new data.

[0011] Furthermore, the latent network mapping module described in step S2 integrates conditional information and random vectors as input, using the conditional information to guide the generation process, enabling the generator to learn the generation rules of audio signals from random noise; the transposed convolution and multi-receptive field fusion module directly uses transposed convolution to upsample the original data, and inputs the sampled one-dimensional data into the multi-receptive field fusion module.

[0012] Furthermore, the latent mapping network module described in step S2 employs five fully connected layers and the LeakyReLU activation function. Its input is a 100-dimensional noise vector, and the conditional information is encoded through an embedding layer of dimension w, which is then concatenated with the latent variables.

[0013] Further, in step S2, the data after transposed convolution upsampling is input into three different residual blocks. Each residual block consists of a continuous one-dimensional convolution and a LeakyReLU activation function. The kernelsize and dilation parameters of the one-dimensional convolution are different to obtain different receptive fields and capture contextual information. Finally, the outputs of the three residual blocks are summed, and the resulting value is used as input to the next transposed convolution for upsampling.

[0014] Furthermore, the discriminators with different periods mentioned in step S2 take the intermediate waveform of the generator as input to generate corresponding discrimination results, and then add and fuse the outputs of different discriminators to obtain the final comprehensive discrimination result.

[0015] Further, in step S2: before the intermediate waveform data is input to the discriminator, the one-dimensional original audio data of length L is first reshaped into two-dimensional data of height L / p and width p. During the reshaping process, adjacent values ​​of samples are selected to retain the temporal information in the original audio data, maintain the continuity of the audio data, and reduce the dimensionality of the data. Subsequently, two-dimensional convolution is applied to the reshaped data, and the kernel size of the two-dimensional convolution is set to k*1 to process periodic samples independently.

[0016] Furthermore, weight normalization is applied after all convolutional layers to reparameterize the weights.

[0017] Furthermore, by utilizing the error signal ratio ε with pre-emphasis filtering... ESR and DC loss term ε DC Optimize loss function L G The total loss function of the generator (G,D) is given by the following equation:

[0018]

[0019] An improved pathological voice generation model constructed according to the above method includes a generator and a discriminator. The generator includes a latent mapping network module, transposed convolution, and a multi-receptive field fusion module; the discriminator consists of multiple discriminators with different period settings.

[0020] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0021] 1. This invention directly uses the original audio as network input, allowing the network to perceive and learn a wider frequency range, including details in the low, mid, and high frequency components. This enables the network to better capture subtle changes and dynamic range of the audio signal over time, thereby providing richer and more accurate resolution, preserving the fine details and dynamic range of the audio signal, and providing a more accurate and realistic audio representation.

[0022] 2. This invention incorporates a layered nested approach, decomposing the generator and discriminator into multiple smaller parts. In addition to the final upsampled output, the generator's output also feeds intermediate waveforms after each upsampling step into discriminators with different periods. Through the discrimination of multiple discriminators, the realism of the output waveform at the corresponding resolution is determined. This design helps the generator learn the mapping between low-frequency and high-frequency components of acoustic features, improving the quality and realism of the generated audio. It allows the model to simultaneously capture audio features at different time scales, using multiple convolutional and deconvolutional layers with different receptive fields to capture sound features within different frequency ranges, thereby enhancing its modeling capabilities.

[0023] 3. This invention draws on the idea of ​​image generation and designs a latent mapping network to help the generator learn the generation rules of audio signals from random noise. It uses a method that combines conditional information with random vectors as input. The conditional information can be used to guide the generation process, so that the generated voice data has specific features or attributes, thereby generating more natural and realistic audio signals and improving the quality and accuracy of the generated data.

[0024] 4. This invention applies weight normalization to the weight matrix, which improves convergence speed, training stability, reduces gradient vanishing and gradient exploding, and helps improve training effectiveness and generalization performance, provides stronger learning rate robustness, and is less sensitive to noise.

[0025] 5. This invention introduces the ESR method to optimize the loss function, which can guide the optimization process, improve the quality and naturalness of speech synthesis, and evaluate network performance and tuning strategies. Simultaneously, by introducing pre-emphasis filtering into the preprocessing of speech generation, the high-frequency signal components can be enhanced, thereby improving the high-frequency response of the signal and reducing high-frequency loss in synthesized speech. Attached Figure Description

[0026] Figure 1 This is a diagram of the overall network architecture of the present invention.

[0027] Figure 2 This is a diagram of the potential mapping network module architecture.

[0028] Figure 3 This is a diagram of the multi-sensory field fusion module architecture.

[0029] Figure 4 This is a schematic diagram of a multi-period discriminator.

[0030] Figure 5 Loss function calculation graph.

[0031] Figure 6 Comparison chart of generated results (real data on the left, generated data on the right). Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0033] The present invention provides a method for constructing an improved pathological voice generation model, comprising the following steps:

[0034] S1. Preprocessing: All available pathological / a / vowel speech records at normal pitch in the Saarbruecken sound dataset were used as the training dataset;

[0035] S2. Network Training: The training process is divided into generator training and discriminator training, which are performed alternately.

[0036] The generator comprises a latent mapping network module and a transposed convolution and multi-receptive field fusion module. The latent mapping network module helps the generator learn the generation patterns of audio signals from random noise, using a fusion of conditional information and random vectors as input. The conditional information guides the generation process, giving the generated voice data specific features or attributes. The transposed convolution and multi-receptive field fusion module does not use two-dimensional spectral features; instead, it directly upsamples the original data using transposed convolution, inputting the sampled one-dimensional data into the multi-receptive field fusion module to capture sound features within different frequency ranges.

[0037] The discriminator comprises four discriminators with different periods, used to evaluate and train the intermediate waveform output after each upsampling of the generator. These four discriminators each have different period settings. Each discriminator takes the generated intermediate waveform and conditional information as input and generates a corresponding discrimination result. Through the collaborative work of multiple periodic discriminators, the generator can progressively optimize the generated voice signal, making it closer to realistic pathological voice.

[0038] S3. Data Generation: Input the random noise vector into the pre-trained generator. The generator converts the random noise vector into new audio data. Standardize the generated audio data. Repeat the above steps to generate multiple new data.

[0039] In the generation of pathological voice data, conditional information includes pathological type, severity, and other relevant features. This conditional information is combined with a random noise vector, enabling the generator to learn generalized features of the audio signal, including its tone, rhythm, and frequency distribution, which are then integrated into the generation process.

[0040] like Figure 1 As shown, the pathological voice generation model includes a generator and a discriminator. The generator includes a latent mapping network module, a transposed convolution, and a multi-receptive field fusion module. The discriminator consists of multiple discriminators with different period settings.

[0041] refer to Figure 2As shown, firstly, the noise vector is input into the latent mapping network, which employs five fully connected layers and the LeakyReLU activation function. Its input is a 100-dimensional noise vector, and each fully connected layer contains a certain number of neurons, designed to learn the relationship between the noise vector and the latent space. Conditional information is encoded through an embedding layer of dimension w. The embedding layer's role is to convert the conditional information into a continuous vector form that can be processed by the generator and discriminator, while preserving the semantic features of the conditional information. Specifically, the embedding layer encodes the discrete features in the conditional information into continuous low-dimensional vector representations and concatenates them with the latent variables. This connection method allows the generator to simultaneously utilize information from both the conditional information and the latent variables to generate samples with specific conditions.

[0042] In audio generation, a latent mapping network, as part of the generator, helps the generator learn the generation patterns of audio signals from random noise, thereby generating more natural and realistic audio signals. Typically, a multi-layered neural network is used to encode the random noise, resulting in a vector representation of the latent space. This vector representation can be considered an abstract representation containing crucial information for generating the audio signal. The generator can decode this vector, mapping it back to the audio signal space to generate the audio signal.

[0043] refer to Figure 3 As shown, the vector output by the latent mapping network is passed through a deconvolution layer and then input into a multi-receptive-field fusion module. This module inputs the data after transposed convolution and upsampling into three different residual block structures. Each residual block consists of consecutive one-dimensional convolutions and LeakyReLU activation functions, but the kernel size and dilation parameters of the one-dimensional convolutions are different, thus obtaining different receptive fields and capturing a wider range of contextual information. Finally, the outputs of the three residual blocks are summed, and the resulting value is used as input to the next transposed convolution for upsampling.

[0044] refer to Figure 4 As shown, the intermediate outputs of the generator are input into different discriminators. Before the data is input into the discriminators, this invention first reshapes the one-dimensional original audio data of length L into two-dimensional data of height L / p and width p. During the reshaping process, selecting adjacent sample values ​​can preserve the temporal information in the original audio data, maintain the continuity of the audio data, and reduce the dimensionality of the data. Subsequently, a two-dimensional convolution is applied to the reshaped data, and the kernel size of the two-dimensional convolution is set to k*1 to independently process periodic samples. At the same time, the period is set to [2,3,5,7] to avoid overlap as much as possible. By using a multi-period discriminator, the difference between the synthesized audio and the real audio can be evaluated more accurately, and the generator can learn the characteristics of audio signals of different periods more accurately, thereby improving the quality of the generated audio.

[0045] Since audio consists of sinusoidal signals with different periods, each discriminator processes a portion of the periodic signal in the input audio. To train the generator, this invention outputs the intermediate waveform after each upsampling of the generator, and feeds it into multiple discriminators with different periods for training. The authenticity of the output waveform at the corresponding resolution is determined through the discrimination of multiple discriminators.

[0046] refer to Figure 5 As shown, this invention optimizes the loss function by using an error signal-to-signal ratio (ESR) with pre-emphasis filtering during training. A pre-emphasis filter is applied to both the output and target signals before calculating the ESR. For a training signal segment of length N, the pre-emphasis ESR is given by Equation 1:

[0047]

[0048] Where y p It is the target signal to be pre-emphasized, and This is the pre-emphasized output of the neural network. The pre-emphasis filter is chosen as a first-order high-pass filter with a transfer function.

[0049] H(z) = 1 - 0.85z -1 (2)

[0050] Furthermore, DC offset refers to the deviation of the DC component of a signal from its expected value. When DC offset exists in a speech signal, it can lead to problems such as signal distortion, bias, errors, and noise. Therefore, a DC term is added to the loss to reduce the DC offset in the model output. The DC loss term is given by Equation 3:

[0051]

[0052] The total loss function of the generator is given by Equation 4:

[0053]

Claims

1. A method for constructing an improved pathological voice generation model, characterized in that, Includes the following steps: S1. Preprocessing: All available pathological / a / vowel speech records at normal pitch in the Saarbruecken sound dataset were used as the training dataset; S2. Network Training: The training process is divided into generator training and discriminator training, which are carried out alternately. The generator includes: a latent mapping network module, which is used to generate voice data with specific features and attributes; a transposed convolution and multi-receptive field fusion module, which is used to capture sound features in different frequency ranges; the discriminator includes: four discriminators with different period settings, which are used to evaluate and train the intermediate waveform output after each upsampling of the generator. S3. Data Generation: Input the random noise vector into the pre-trained generator. The generator converts the random noise vector into new audio data. Standardize the generated audio data. Repeat the above steps to generate multiple new data. The latent mapping network module described in step S2 integrates conditional information and random vectors as input, using the conditional information to guide the generation process, enabling the generator to learn the generation rules of audio signals from random noise; the transposed convolution and multi-receptive field fusion module directly uses transposed convolution to upsample the original data, and inputs the sampled one-dimensional data into the multi-receptive field fusion module. The data after transposed convolution upsampling is input into three different residual blocks. Each residual block consists of a continuous one-dimensional convolution and a LeakyReLU activation function. The kernelsize and dilation parameters of the one-dimensional convolution are different to obtain different receptive fields and capture contextual information. Finally, the outputs of the three residual blocks are summed and the resulting value is used as input to the next transposed convolution for upsampling. Before the intermediate waveform data is input to the discriminator, the one-dimensional original audio data of length L is first reshaped into two-dimensional data with height L / p and width p. During the reshaping process, adjacent values ​​of samples are selected to retain the temporal information in the original audio data, maintain the continuity of the audio data, and reduce the dimensionality of the data. Subsequently, two-dimensional convolution is applied to the reshaped data, and the kernel size of the two-dimensional convolution is set to k*1 to process periodic samples independently.

2. The method for constructing the improved pathological voice generation model according to claim 1, characterized in that: The latent mapping network module described in step S2 employs five fully connected layers and the LeakyReLU activation function. Its input is a 100-dimensional noise vector, and the conditional information is encoded through an embedding layer of dimension w, which is then concatenated with the latent variables.

3. The method for constructing the improved pathological voice generation model according to claim 2, characterized in that: The four discriminators with different period settings mentioned in step S2 take the intermediate waveform of the generator as input to generate corresponding discrimination results, and then add and fuse the outputs of different discriminators to obtain the final comprehensive discrimination result.

4. The method for constructing the improved pathological voice generation model according to claim 3, characterized in that: After all convolutional layers, weight normalization is performed to reparameterize the weights.

5. The method for constructing the improved pathological voice generation model according to claim 4, characterized in that: Using the error signal ratio with pre-emphasis filtering and DC loss term Optimize the loss function The total loss function of the generator is given by the following equation: 。 6. An improved pathological voice generation model constructed according to the construction method of claim 1, characterized in that: It includes a generator and a discriminator. The generator includes a latent mapping network module, a transposed convolution, and a multi-receptive field fusion module; the discriminator consists of multiple discriminators with different period settings.

Images