A speaker voice information decoupling method based on unsupervised learning

Through the unsupervised learning encoder-decoder framework and gradient inversion layer, the speaker information in speech is extracted, which solves the problem of poor generalization performance in the existing technology, realizes effective processing and characterization of pathological speech, and improves the Voice intelligibility and naturalness.

CN118887958BActive Publication Date: 2024-12-03TIANJIN UNIVERSITY OF TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411390118.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-10-08
Publication Date
2024-12-03
Estimated Expiration
2044-10-08

AI Technical Summary

Technical Problem

The prior art has the problem of weak generalization performance when extracting speaker information in voice, especially when facing pathological speech, which is not effective and cannot effectively eliminate interference from speech information, resulting in low generalization and speech intelligibility. Low.

Method used

Adopting an encoder-decoder framework based on unsupervised learning, a multi-reference speech prompt encoder and gradient inversion layer were introduced. Through speech reconstruction and speaker classification joint training, the speaker irrelevant and related features were extracted to realize the speaker in the voice signal. Decoupling of information.

Benefits of technology

It improves the generalization performance of speaker representation, can effectively deal with complex speech such as pathological speech, improves speech intelligibility and naturalness, and is suitable for speaker recognition, voice conversion and other tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118887958B_ABST
    Figure CN118887958B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of speech analysis and processing, and proposes a method for decoupling speaker speech information based on unsupervised learning. The method does not require the use of text labels, and uses a multi-reference speech prompt encoder to perform fine-grained representation of speaker information. At the same time, combined with gradient-inverted speaker classification, joint training is performed under the encoder-decoder framework to further achieve the decoupling of speaker-related and speaker-independent information in speech information. The present invention can be used as a basic module to provide effective speaker information representation for related downstream applications such as speaker recognition, speech conversion, speech synthesis and other tasks.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of speech analysis and processing, and discloses a method for decoupling speaker information in speech based on unsupervised learning. Specifically, based on an encoder-decoder framework, a multi-reference acoustic prompts encoder (MRAPE) and a gradient reversal layer (GRL) speaker classification joint training are introduced to achieve decoupling of speaker information in speech signals and provide more effective speaker information representation for downstream tasks. Background Art

[0002] Effective extraction of speaker information from speech is the core technology in speaker recognition, speech synthesis, speech conversion and other fields. Existing methods for extracting speaker information usually directly extract low-dimensional vectors of speakers from speech to represent speakers through speech encoder pre-training models based on time-delay neural networks or residual networks, such as ECAPA-TDNN and Resnet. However, considering that speech contains other information such as speech, emotion, and rhythm in addition to speaker information, existing methods cannot effectively extract and represent speaker information in a targeted manner, and therefore have the defect of weak generalization performance.

[0003] In order to eliminate the interference of speech information features in speech, existing technologies usually use pre-trained automatic speech recognition (ASR) networks to model speech content information and use it as a reference for speaker speech information extraction. However, considering the low intelligibility of speech in people with articulation disorders, the pre-trained automatic speech recognition network cannot effectively obtain the speech content representation in their speech, and due to the lack of text labels corresponding to their speech, there are certain difficulties in training and fine-tuning the automatic speech recognition network. Therefore, the method of modeling speech information based on the automatic speech recognition pre-trained network has very limited effect in extracting speaker information for pathological speech. In addition, in addition to low intelligibility, some pathological speech also has characteristics such as discontinuous and unstable speech segments, which will cause the previous speaker feature extraction method based on the global target speech to be affected to a certain extent in the speaker information representation effect. Summary of the invention

[0004] Based on the above-mentioned situation, the present invention has developed a method for decoupling speaker information in speech based on unsupervised learning. Without the need for text labels, the decoupling of speaker speech information is guided by training two joint tasks of speech reconstruction and speaker classification, thereby obtaining more effective speaker representation information.

[0005] The technical solution of the present invention is a method for decoupling speaker information in speech based on unsupervised learning, which realizes speaker voice information decoupling based on an autoencoder framework consisting of an encoder and a decoder. The specific implementation steps are as follows:

[0006] S1 Data preprocessing and feature extraction: Before model training, the audio data is first batch-normalized to a unified format; on the one hand, it can effectively improve the model training speed and accelerate model convergence; on the other hand, it helps to improve the accuracy and generalization ability of the model; the processed audio data is converted into Mel-spectrogram filter bank features (FBank) through audio feature extraction As the main input data of the framework, for each audio data, the audio data of other different speech segments of the same speaker are also converted into Mel spectrum filter features as the reference features of the same speaker ;

[0007] S2 extracts speaker-independent information representation: First, the Mel spectrum filter bank features extracted by S1 Input into the basic speech encoder to extract basic speech information features including speaker, speech, prosody, and emotion ; Then, by introducing vocal tract length perturbation (VTLP) processing, speaker-independent speech representation with destroyed speaker information is obtained, and speaker classification training is performed in the speaker classifier after gradient inversion to further compress (minimize) the speaker information in this part of the representation information, and obtain the optimized speaker-independent speech representation , the loss of this part in training is .

[0008] The detailed steps of S2 are as follows:

[0009] S2-1 Mel-spectrum filter bank characteristics First, it is input into the basic speech encoder, which includes a convolution downsampling module and a self-attention encoding module; the convolution downsampling module is composed of two layers of convolutional neural network (CNN) modules; the self-attention encoding module (ConformerLayer) is a multi-layer structure, each layer includes a feedforward neural network, a multi-head self-attention module, and a convolution module. The basic speech features obtained after convolution downsampling s ;

[0010] S2-2 Basic speech features Frequency regularization is performed by perturbing the vocal tract length, thereby initially removing the coupling between speech content information and speaker information and obtaining speaker-independent speech representation. ;

[0011] S2-3 Using gradient reversal to train speaker classification to represent speaker-independent speech It is sent to the speaker classifier for speaker classification training. The loss of this part in training is ; Represent speaker-independent speech After the gradient flip layer, the loss is calculated by the speaker classifier ,when When it is the smallest, it means that the speaker information of the current speaker in the obtained optimal feature is the least, so the speaker-independent speech representation that further strips away the speaker information is obtained;

[0012] S3 extracts speaker-related features: While performing step S2, the Mel spectrum filter bank features obtained in step S1 are and the same speaker reference features The two are fed together into a multi-reference speech prompt encoder to extract fine-grained speaker-related features. , speaker classification training is performed through the speaker classifier so that The representation capability of speaker information is further enhanced. The detailed steps of S3 are as follows:

[0013] S3-1 first extracts speaker information reference features of the same speaker through a multi-reference speech prompt encoder; the multi-reference speech prompt encoder is composed of two convolution modules and a downsampling layer; the multi-reference speech prompt encoder extracts speaker information reference features of the same speaker The first convolution module then performs 16-fold downsampling to improve computational efficiency. The second convolution module further convolves the downsampled speaker reference features to aggregate the speaker information contained in different speech segments to form the final reference code of the same speaker. ;

[0014] S3-2 Same speaker reference code The Mel spectrum filter bank features of step S1 The timbre attention module is sent to the multi-reference speech prompt encoder to encode the speaker information of the current information; the timbre attention module is a multi-layer structured attention encoding module (Conformer Layer), each layer of which includes a feedforward neural network, a multi-head attention module, and a convolution module; the query information Q in the timbre attention module is different from the self-attention encoding module in the basic speech encoder described in S2-1, which is the Mel spectrum filter group feature currently being processed. , the index information K and the queried information V are both the same speaker reference code obtained by S3-1 ; The feature extraction of the timbre attention module will further extract the speaker information of the same speaker in the current pronunciation state to form speaker-related features .

[0015] S3-3 The speaker-related features obtained in step S3-2 are Sent to the speaker classifier, the training loss of this part during training is .when When it is the smallest, it indicates the obtained feature The speaker information contained in is optimized, thereby obtaining further optimized speaker-related features.

[0016] S4 represents the speaker-independent speech obtained in S2 Speaker-related features obtained from S3 After being spliced ​​together, they are sent to the speech decoder for decoding to obtain the reconstructed Mel spectrum filter bank feature information , and calculate the reconstructed Mel spectrum filter bank features Compared with the original Mel spectrum filter bank features The loss between .

[0017] The speech decoder of the present invention is a multi-layer structure of a self-attention encoding module (Transformer Layer), each layer of which includes a feedforward neural network and a multi-head attention module.

[0018] By jointly training the modules in S2, S3, and S4, and performing the joint loss of the three modules = + + After optimization, the speaker-independent speech representation after optimization decoupling can be obtained from the original speech signal. Speaker-dependent representation .

[0019] The speaker classifier in S3 of the present invention has the same structure as the speaker classifier in S2, and the parameters are independent of each other.

[0020] Speaker-related representation extracted by the present invention It can be used as speaker embedding features and applied to multiple speaker-related applications such as speaker recognition, speech conversion, and voice cloning.

[0021] Beneficial Effects

[0022] The present invention aims to address the defects of existing speaker representation technology in dealing with complex speech, such as poor speaker information extraction performance and low generalization in pathological speech of people with pronunciation disorders. The extracted speaker representation has better generalization performance and can cope with the characteristics of low intelligibility of pathological speech, discontinuous and unstable speech segments, etc.

[0023] The present invention can be used as a basic module to provide effective speaker information representation for related downstream applications such as speaker recognition, speech conversion, speech synthesis and other tasks.

[0024] Compared with zero-shot speech synthesis that integrates previous speaker representation extraction methods, it can significantly improve the target speaker similarity, speech intelligibility, naturalness, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 Training model structure for the present invention;

[0026] Figure 2 The internal structure of the multi-reference speech prompt encoder module. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with the accompanying drawings. Figure 1 As shown, the training model structure of the present invention is based on an unsupervised learning method for decoupling speaker information in speech, and realizes speaker speech information decoupling based on an autoencoder framework consisting of an encoder and a decoder.

[0028] S1 Data preprocessing and feature extraction: The audio data is resampled and amplitude normalized using the Python audio processing toolkit librosa, and the spectrum is pre-emphasized, windowed, and pre-processed. The processed audio data is converted into 80-dimensional Mel-spectrum filter bank features (FBank) through audio feature extraction. As the main input data of the framework. For each audio, 2000 frames of data are randomly extracted from other different speech clips of the same speaker and converted into Mel spectrum filter bank features and then spliced ​​together as the speaker reference features. .

[0029] S2 extracts speaker-independent information representation: First, the Mel spectrum filter bank features extracted by S1 First, it is input into the basic speech encoder to extract basic speech information features including speaker, speech, rhythm, and emotion. After the vocal tract length perturbation processing, the speaker-independent speech representation with the speaker information destroyed is obtained, and the speaker information in this part of the representation information is further compressed (minimized) through the speaker classification training after gradient inversion to obtain the optimized speaker-independent speech representation. .

[0030] The detailed steps of S2 are as follows:

[0031] S2-1 Mel spectrum filter bank features extracted by S1 First, it is input into the basic speech encoder to obtain the basic speech information features In the basic speech encoder, the Mel spectrum filter bank features First, the convolutional downsampling module is used to perform downsampling processing with twice the number of frames. The convolutional downsampling module is composed of two layers of two-dimensional convolutional neural network (CNN) modules, with a convolution kernel of 3 and a downsampling step of 2; the downsampled Mel spectrum features are then converted into basic speech information features through the self-attention encoding module 。 The self-attention encoding module consists of 6 self-attention encoding layers, each of which includes a feedforward neural network, a multi-head self-attention module, and a convolution module;

[0032] S2-2 Basic voice information features Frequency regularization is performed by perturbing the vocal tract length, and the frequency axis of the spectrum is distorted by randomly generating a distortion factor to remove the vocal tract difference and obtain speaker-independent speech representation. ;

[0033] S2-3 Using gradient reversal to train speaker classification to represent speaker-independent speech The speaker classifier is sent to the speaker classifier for speaker classification training. The speaker classifier is a linear layer that represents speaker-independent speech. Map to Speaker classification is performed in the feature space of is the total number of speakers used for training, is the embedding feature dimension. The embedding feature dimension used in the current training is 192. The loss of this part in training is , the additive angular margin loss (AAM-Softmax Loss) is used for calculation.

[0034] S3 Speaker-related feature extraction: While performing step S2, the Mel spectrum filter bank features obtained in step S1 are extracted. and the same speaker reference features The detailed structure of the multi-reference speech prompt encoder module is as follows: Figure 2 As shown;

[0035] The detailed steps of S3 are as follows:

[0036] S3-1 Reference features of the same speaker First, the speaker information reference code is extracted through a multi-reference speech prompt encoder; the multi-reference speech prompt encoder consists of two convolution modules and a downsampling layer, each convolution module contains 5 layers of basic convolution layers, each basic convolution layer is composed of 2 one-dimensional convolution layers connected; the basic convolution layers are connected in a residual connection manner. Speaker reference features After being sent to the first convolution module, it is downsampled by 16 times, and then passes through the second convolution module to form the final reference code of the same speaker. ;

[0037] S3-2 Reference features of the same speaker The Mel spectrum filter bank features obtained in step S1 The same speaker-related features are obtained by feeding them into the timbre attention module in the multi-reference speech prompt encoder module. The timbre attention module is a multi-layer structure, each layer includes a feedforward neural network, a multi-head attention module, and a convolution module; the query information Q in the timbre attention module is the feature of the currently processed Mel spectrum filter bank , the index information K and the queried information V are both the same speaker reference code obtained by S3-1 .

[0038] S3-3 The speaker-related features obtained in step S3-2 are The speaker classifier has the same structure as that in S2-3, and the parameters are independent of each other. The training loss of this part is , the additive angular margin loss (AAM-Softmax Loss) is used for calculation.

[0039] S4 multi-module joint training: speaker-independent speech representation obtained in S2 Speaker-related features obtained from S3 After being spliced ​​together, they are sent to the speech decoder for decoding to obtain the reconstructed speech Mel filter bank information , and calculate the reconstructed speech information With the original Mel filter bank The loss between .

[0040] The speech decoder is a multi-layer self-attention encoding module (Transformer Layer), each layer of which includes a feedforward neural network and a multi-head attention module.

[0041] By jointly training the modules in S2, S3, and S4, and performing the joint loss of the three modules = + + After optimization, the speaker-independent speech representation after optimization decoupling can be obtained from the original speech signal. Speaker-dependent representation .

[0042] Based on the speaker features extracted by the framework, the present invention combines the speech synthesis framework to construct a zero-sample speech cloning system for pathological speech. Due to the proposed speaker speech information decoupling method based on unsupervised learning, on the one hand, it can maximize the extraction of effective information characterizing the speaker, and on the other hand, exclude the interference of other irrelevant information, especially avoiding the leakage of discontinuous and incomplete speech information in pathological speech.

Claims

1. A speaker information decoupling method based on unsupervised learning, characterized in that: Based on the autoencoder framework consisting of an encoder and a decoder, the speaker's voice information is decoupled. The steps are as follows: S1: Data preprocessing and feature extraction: The audio data is batch-normalized to a unified format; the processed audio data is converted into Mel-spectrogram filter bank features through audio feature extraction. It is the main input data of the framework; At the same time, for each audio data, the audio data of other different speech segments of the same speaker are converted into Mel spectrum filter features, and the features are used as reference features of the same speaker. ; S2: Extract speaker-independent information representation: First, the Mel spectrum filter bank features extracted by S1 Input into the basic speech encoder to extract basic speech information features , and then introduce the vocal tract length perturbation processing to obtain the speaker-independent speech representation with the speaker information destroyed, and then perform speaker classification training through gradient inversion, further compress the speaker information in this part of the representation information, and obtain the optimized speaker-independent speech representation The loss of this part in training is ; S3: Extract speaker-related features: While performing step S2, the Mel spectrum filter bank features obtained in step S1 are and the same speaker reference features The two are fed into a multi-reference speech prompt encoder to extract fine-grained speaker-related features. , by performing speaker classification training on the speaker classifier, The representation ability of speaker information is further enhanced, and the training loss of this part in training is ; S4: Speaker-independent speech representation after optimizing S2 Speaker-related representation obtained with S3 After being spliced ​​together, they are sent to the speech decoder for decoding to obtain the reconstructed Mel spectrum filter bank feature information And calculate the reconstructed Mel spectrum filter bank features Compared with the original Mel spectrum filter bank features The loss between ; Joint training and joint loss of the three modules = + + Optimize and finally obtain the optimized decoupled speaker-independent speech representation from the original speech signal Speaker-dependent representation ; The S3 is as follows: S3-1 first extracts the speaker information reference features of the same speaker through the multi-reference speech prompt encoder to form the same speaker reference code ; S3-2 Same speaker reference code The Mel spectrum filter bank features of step S1 The timbre attention module in the multi-reference speech prompt encoder is sent together to encode the speaker information of the current information; the feature extraction of the timbre attention module will further extract the speaker information in the current pronunciation state of the same speaker to form speaker-related features ; S3-3 The speaker-related features obtained in step S3-2 are Sent to the speaker classifier, the training loss of this part during training is ,when When it is the smallest, it indicates the speaker-related features obtained The speaker information contained in is optimized, thereby obtaining further optimized speaker-related features; The timbre attention module is a multi-layer attention encoding module, each layer of which includes a feedforward neural network, a multi-head attention module, and a convolution module; The query information Q in the timbre attention module is the feature of the currently processed Mel spectrum filter bank. , the index information K and the queried information V are both the same speaker reference code obtained by S3-1 .

2. The method according to claim 1, characterized in that S2 calculates the loss of the speaker-independent speech representation whose speaker information is destroyed after gradient flipping by the speaker classifier. ,when When it is the smallest, it means that the speaker information of the current speaker in the optimized speaker-independent speech representation is minimized, thus obtaining a speaker-independent speech representation that further strips away the speaker information. .

3. The method according to claim 1, characterized in that The basic speech encoder includes a convolution downsampling module and a self-attention encoding module; the convolution downsampling module is composed of two layers of convolutional neural network modules; the self-attention encoding module is a multi-layer structure, each layer includes a feedforward neural network, a multi-head self-attention module, and a convolution module.

4. The method according to claim 1, characterized in that: The multi-reference speech prompt encoder is composed of two convolution modules and a downsampling layer; the multi-reference speech prompt encoder converts the speaker reference features The first convolution module then performs 16-fold downsampling to improve computational efficiency. The second convolution module further convolves the downsampled speaker reference features to aggregate the speaker information contained in different speech segments to form the final reference code of the same speaker. .