Speaker voice segmentation method based on non-local spatial U-Net and mixed features

Abstract

This invention belongs to the field of speech signal processing technology, specifically proposing a speaker speech segmentation method based on nonlocal spatial U-Net and hybrid features. This invention constructs a nonlocal spatial U-Net network, and by introducing a nonlocal spatial attention module, effectively captures long-range dependencies in the speech signal, enhancing the expressive power of spatial features. Simultaneously, a hybrid feature fusion mechanism is employed, combining time-frequency domain features and deep semantic features to enhance the discriminative power of speech features. Furthermore, a dynamic weight loss function is designed to optimize the fusion efficiency of features at different scales and balance the contribution of various speech segments. This invention can fully exploit the spatiotemporal correlations in speech signals, overcome the dependence of traditional methods on local and single features, and significantly improve the accuracy and robustness of speaker speech segmentation.

Description

Technical Field

[0001] This invention belongs to the field of speech signal processing technology, and specifically proposes a speaker speech segmentation method based on nonlocal spatial U-Net and hybrid features. Background Technology

[0002] To enhance the intelligence of speech signal processing and meet the high-precision segmentation requirements for multi-speaker speech in fields such as intelligent customer service, conference transcription, and forensic evidence collection, speaker speech segmentation technology is increasingly important in complex acoustic environments. Speaker speech segmentation aims to accurately divide the speech segments of different speakers from a continuous audio stream and label their identity information. Traditional methods based on handcrafted features (such as Mel-triangle filters, Pitch) and clustering algorithms (such as K-means, GMM) show significantly reduced segmentation performance in scenarios with low signal-to-noise ratios or overlapping multi-speaker conversations, and are difficult to adapt to the differences in acoustic characteristics among different speakers. Therefore, researching robust and adaptable speaker speech segmentation methods is of great significance for achieving accurate speech analysis and processing.

[0003] Existing speaker speech segmentation methods can be mainly divided into methods based on traditional signal processing and methods based on deep learning. Methods based on traditional signal processing typically rely on time-domain features such as short-time energy and zero-crossing rate, or frequency-domain features such as spectral centroid and harmonic structure, combined with dynamic time warping (DTW) or hidden Markov models (HMM) for segmentation. However, the performance of these methods deteriorates sharply under noise interference, speech overlap, or non-stationary background noise.

[0004] In recent years, deep learning-based speech segmentation methods have significantly improved segmentation accuracy through end-to-end training. For example, models based on recurrent neural networks (RNNs) or convolutional neural networks (CNNs) can automatically learn the temporal and spectral features of speech. However, current mainstream speaker speech segmentation methods still suffer from two key problems: firstly, algorithms based on fixed-scale analysis lack robustness in complex noise environments and are easily affected by background interference, leading to a decrease in segmentation accuracy; secondly, traditional feature extraction methods generate a large number of redundant segmentation points, which not only increases computational complexity but also affects the accuracy of segmentation boundaries. These problems severely restrict the application effectiveness of speech segmentation systems in real-world scenarios. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention proposes a speaker speech segmentation method based on nonlocal spatial U-Net and hybrid features.

[0006] The technical solution adopted in this invention is as follows:

[0007] A speaker speech segmentation method based on non-local spatial U-Net and hybrid features includes the following steps:

[0008] S1. Acquire the time-domain waveform based on the input speech signal, and preprocess the waveform to obtain the speech spectrum of the speech signal;

[0009] S2. Extract Mel-Cepstral Coefficients and Cepstral Coefficients, superimpose and normalize the two features in the frequency dimension, and combine them as a hybrid feature based on the high accuracy of Mel-Cepstral Coefficients and the strong robustness of Cepstral Coefficients for speaker speech segmentation.

[0010] S3. Construct an encoder model based on a nonlocal spatial U-Net network, design a nonlocal spatial module, and reduce the dimensionality of the spectral features of speech.

[0011] S4. Use KL divergence to measure the distance between two features;

[0012] S5. Repeat steps S3 and S4 until the distance exceeds the feature vector range; then, smooth the distance obtained in step S4 using a low-pass filter, and take a local extremum of the curve as the speech segmentation point.

[0013] This allows us to train a speaker speech segmentation model with high accuracy and stability based on nonlocal space U-Net and hybrid features.

[0014] Furthermore, the following steps are included:

[0015] The speech signal is preprocessed, specifically as follows:

[0016] The experimental speech signal samples were several minutes of multi-person dialogue audio recorded using a mobile phone; first, the audio files were converted into WAV files, and then noise was added to the noise library.

[0017] Based on the high accuracy and robustness of Mel-Cepstral Coefficient (MCC) features, the obtained WAV files are combined as hybrid features for speaker speech segmentation.

[0018] Obtaining Mel-frequency cepstral coefficients requires preprocessing the input speech signal. Sound waves travel through the vocal tract to the lips and then radiate into the air. During this propagation, higher-frequency components of the speech are more easily lost. To compensate for the suppression of speech by the vocal system and to facilitate the analysis of vocal tract parameters and the spectrum, a transfer function for boosting high frequencies is used. A first-order digital filter is used to pre-emphasize the original speech, as shown in the following formula.

[0019]

[0020] The value is between 0.9 and 1.0;

[0021] Let the original speech signal be The speech signal after passing through the pre-emphasis filter is :

[0022]

[0023] Speech framing uses a movable, finite-length window to weight the speech. Frames are not continuous but partially overlap. Assume there exists a frame of length... The audio signal is framed:

[0024]

[0025] in For frame shift, The frame length is such that the ratio of frame shift to frame length is between 0 and 0.5, ensuring a smooth transition between frames.

[0026] Speech signals can be abruptly truncated due to speech framing, resulting in discontinuous signals and distortion of the Fourier transform spectrum. This distortion is called spectral leakage. To prevent this, the speech signal after framing is windowed.

[0027] Window function Its function is to generate a function that takes all zero values ​​in a given interval. The Hanning window is one of the most commonly used window functions.

[0028]

[0029] After segmenting and windowing the original data, time-frequency analysis of the speech is performed using short-time Fourier transform:

[0030]

[0031] in, It refers to the window function; the short-time Fourier transform means using a window function. For speech signals The truncated signal is then subjected to a Fourier transform, i.e., the calculation is performed. t This moment is changing, and it is constantly evolving. t The values ​​are used to obtain the set of Fourier transforms at each time step. This function represents the speech signal in t Frequency components obtained near time The amplitude and phase; then each in the set t The energy spectrum is obtained by taking the absolute value or squaring the data at each moment.

[0032] To calculate the Mel-frequency cepstral coefficients, a range of frequencies was set for each frame of speech. M Filters ,in Representing the m The center frequency of the triangular filter;

[0033]

[0034] The logarithmic energy of the output of each triangular filter bank is:

[0035]

[0036] in, It is the first The first frame of the audio signal spectrum diagram The amplitude of the discrete Fourier transform coefficients of the term; when filtering the speech spectrum with a triangular filter bank, the product of the amplitude of each point within the bandwidth of each filter and the energy of the corresponding point in the spectrum is calculated and summed. The result is used as the output of this triangular filter, and the actual spectrum is mapped onto the Mel spectrum; finally, the output logarithmic energy is subjected to discrete cosine transform to obtain the Mel cepstral coefficients.

[0037]

[0038] Regarding the cepstral coefficient characteristics; the Gammatone filter, similar to the basilar membrane of the human cochlea, can effectively simulate similar frequency division physiological characteristics, establishing a cochlear-like auditory model, whose time-domain expression is:

[0039]

[0040] in, Represents phase, Represents the center frequency. Represents the order of the filter. For the first The bandwidth of each filter, For the center frequency, It is the filter gain;

[0041] The bandwidth of a fourth-order Gammatone filter can be expressed by the following formula:

[0042]

[0043] As the center frequency increases, the bandwidth of the filter also increases; among which ERB Representing the equivalent rectangular bandwidth, it is a psychoacoustic measure that determines the decay rate of each filter's impulse response. ERB With frequency fThe relationship is as follows:

[0044]

[0045] in ERB Represents the equivalent rectangular bandwidth. The center frequency.

[0046] Furthermore, in step S3, a nonlocal spatial module is designed to construct an encoder model based on a nonlocal spatial U-Net network to reduce the dimensionality of the speech spectral features. Specifically:

[0047] A nonlocal space-based U-Net network is constructed to reduce the dimensionality of speech spectral features, and the dimensionality-reduced features are used for subsequent speaker speech segmentation. Mel-frequency cepstral coefficients and cepstral coefficient features are used as speech features in the frequency domain. In order to be suitable for speech features, the number of channels of convolutional layers and transposed convolutions in the decoder and encoder of the original network are reduced respectively, and multiple linear layers are added between the decoder and encoder to obtain one-dimensional feature representation.

[0048] The nonlocal spatial module, designed before each upsampling block, can suppress feature responses from irrelevant background regions and improve performance. Within the nonlocal spatial module, convolution operations are performed on the feature maps generated by the upsampling and downsampling blocks at corresponding positions, and these features are then combined to obtain attention coefficients. In the nonlocal spatial module, these attention coefficients can increase the weights of speech feature regions, ensuring the extraction of effective feature regions. Therefore, adding the nonlocal spatial module to the skip connections of the encoder and decoder can improve the network model's recognition ability. The nonlocal spatial module, before each upsampling block, can suppress responses from irrelevant speech features and improve performance.

[0049] Specifically, decoder features It can be represented as:

[0050]

[0051] Encoder features It can be represented as:

[0052]

[0053] in, and They represent the first Feature maps of upsampled blocks and downsampled blocks in the layer. and They represent the first Layer-by-layer learning feature maps and The weight parameters, and Indicates the first The corresponding learning bias of the layer. H and W This represents the height and width of the feature map. Represents the ReLU activation function; after obtaining the features of the encoder and decoder, the attention coefficients are calculated using the following formula:

[0054]

[0055] in, and They represent the first Learnable weight parameters and biases are extracted from the feature map layer. This represents the Sigmoid activation function, which makes each output value range from 0 to 1. Furthermore, the learned weight parameters are all obtained using 1×1×1 convolutions. and The weight parameters of the feature map are learned and updated through backpropagation; the output features of the nonlocal spatial module can be represented as:

[0056]

[0057] To train the autoencoder, the TED-LIUM dataset was used. The corpus in the dataset was segmented into segments of fixed length, and Mel-Cepstral Coefficients and Cepstral Coefficients features were extracted and input into the encoder for training. During training, the loss of the model's output and input was calculated using mean squared error, and the model was optimized using Adam.

[0058] Furthermore, in step S4, KL divergence is used to measure the distance between two feature segments. .

[0059] Furthermore, in step S5, steps S3 and S4 are repeated until the range of the feature vector is exceeded; then, the distance obtained in step S4 is smoothed using a low-pass filter, and a local extremum of the curve is taken as the speech segmentation point.

[0060] The beneficial effects of this invention are as follows: This invention constructs a nonlocal spatial U-Net network, which effectively captures long-range dependencies in speech signals and enhances the expressive power of spatial features by introducing a nonlocal spatial attention module. Simultaneously, it employs a hybrid feature fusion mechanism, combining time-frequency domain features and deep semantic features to enhance the discriminative power of speech features. Furthermore, a dynamic weight loss function is designed to optimize the fusion efficiency of features at different scales and balance the contribution of various speech segments. This invention can fully exploit the spatiotemporal correlations in speech signals, overcome the dependence of traditional methods on local and single features, and significantly improve the accuracy and robustness of speaker speech segmentation. Attached Figure Description

[0061] Figure 1 This is a flowchart of the speaker speech segmentation algorithm based on nonlocal spatial U-Net and hybrid features of the present invention;

[0062] Figure 2 This is a noisy speech spectrum comparison diagram of the present invention;

[0063] Figure 3 This is a block diagram illustrating the principle of Mel-frequency cepstral coefficient (MFCC) feature extraction in this invention.

[0064] Figure 4 This is a block diagram illustrating the principle of cepstral coefficient (GFCC) feature extraction in this invention.

[0065] Figure 5 This is a schematic diagram of the autoencoder network structure based on U-Net of the present invention;

[0066] Figure 6 This is the non-local spatial module of the present invention;

[0067] Figure 7 This is a comparison chart of training losses for autoencoders with different structures according to the present invention;

[0068] Figure 8 The following are waveforms of speech segmentation using different algorithms in a noise-free environment according to this invention;

[0069] Figure 9 The waveforms of speech segmentation using different algorithms under babble (SNR=-5) noise are shown in this invention. Detailed Implementation

[0070] The following is in conjunction with the appendix Figure 1-9 The principles and features of the present invention are described, and the examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0071] refer to Figure 1 This invention provides a speaker speech segmentation method based on non-local spatial U-Net and hybrid features, the implementation steps of which are as follows:

[0072] S1: Acquire the time-domain waveform based on the input speech signal, and preprocess the waveform to obtain the speech spectrum of the speech signal.

[0073] S2. The extracted Mel-frequency cepstral coefficients (MFCC) and spectral cepstral coefficients (GFCC) are superimposed and normalized in the frequency dimension. Based on the high accuracy of the MFCC feature and the strong robustness of the GFCC feature, the two are combined as a hybrid feature for speaker speech segmentation.

[0074] S3. Construct an encoder model based on a nonlocal spatial U-Net network, design a nonlocal spatial module, and reduce the dimensionality of the speech spectral features.

[0075] S4 uses KL divergence to measure the distance between two features.

[0076] S5, repeat steps S3 and S4 until the range of the feature vector is exceeded. Then, the distance obtained in step S4 is smoothed using a low-pass filter, and a local extremum of the curve is taken as the speech segmentation point.

[0077] Using the above methods, a speaker speech segmentation model with high accuracy and stability based on non-local space U-Net and mixed features can be trained.

[0078] Furthermore, in step S1, the audio data is preprocessed, specifically as follows:

[0079] The experimental speech samples were several minutes of multi-person dialogue audio recorded using a mobile phone. First, the audio files were converted to WAV files, and then noise was added to a noise library.

[0080] Furthermore, in step S2, the WAV file obtained in step S1 is combined with Mel-frequency cepstral coefficients (MFCC) features based on their high accuracy and GFCC features on their robustness, to perform speaker speech segmentation. To illustrate the difference in speech expression between the two features in noisy environments, a noisy speech is generated by superimposing factory noise with a signal-to-noise ratio of 8 onto a clean speech segment. Its spectrogram and Gammatone spectrum are shown below. Figure 2 As shown in the figure, after the noise is superimposed, it can be clearly seen from the figure that the first and second formants of the low frequency in the spectrogram have been covered by noise, while the Gammatone spectrum still shows a clear formant trend. The Gammatone spectrum is more advantageous in noisy speech.

[0081] The feature extraction process for Mel-frequency cepstral coefficients (MFCC) is as follows: Figure 3 As shown, obtaining Mel-frequency cepstral coefficients (MFCCs) requires preprocessing the input speech signal. Sound waves radiate into the air through the vocal tract and lips; during this propagation, higher-frequency components of speech are more easily lost. To compensate for the suppression of speech by the vocal system and to flatten the frequency spectrum, facilitating the analysis of vocal tract parameters and the spectrum, a first-order digital filter that boosts high frequencies is typically used to pre-emphasize the original speech. The formula is as follows:

[0082]

[0083] The value is between 0.9 and 1.0.

[0084] Let the original speech signal be The speech signal after passing through the pre-emphasis filter is :

[0085]

[0086] A common method for speech framing is to weight the speech using a movable, finite-length window. Typically, frames are not continuous but partially overlap. For frame shift, The ratio of frame length and frame shift to frame length is between 0 and 0.5 to ensure smooth transitions between frames. Assume there exists a frame with a length of... The audio signal is framed:

[0087]

[0088] Speech signals can be abruptly truncated due to framing, resulting in discontinuous signals and distortion of the Fourier transform spectrum. This distortion is commonly referred to as spectral leakage. To prevent this, windowing is applied to the framing of the speech signal.

[0089] Window function Its function is to generate a function whose values ​​are all 0 in a given interval. The Hanning window is one of the most commonly used window functions.

[0090]

[0091] After segmenting and windowing the original data, time-frequency analysis of the speech is performed using short-time Fourier transform:

[0092]

[0093] in, It refers to a window function; the Short-Time Fourier Transform (STFT) means using a window function. For speech signals The signal is truncated, and then a Fourier transform is performed on the truncated local signal, i.e., the calculation is performed. t This moment is changing, and it is constantly evolving. t The values ​​are used to obtain the set of Fourier transforms at each time step. This function represents the speech signal in t Frequency components obtained near time The amplitude and phase. Then, each in the set... t The energy spectrum is obtained by taking the absolute value or squaring the data at each time point.

[0094] To calculate the Mel-frequency cepstral coefficients, a range of frequencies was set for each frame of speech. MFilters ,in Representing the m The center frequency of the triangular filter.

[0095]

[0096] The logarithmic energy of the output of each triangular filter bank is:

[0097]

[0098] in, It is the first The first frame of the audio signal spectrum diagram The magnitudes of the discrete Fourier transform (DFT) coefficients of the term are calculated. When filtering the speech spectrum using a Mel triangular filter bank, the product of the magnitude at each point within the bandwidth of each filter and the energy at the corresponding point in the spectrogram is calculated and summed. The result is used as the output of this triangular filter, and the actual spectrum is mapped onto the Mel spectrum. Finally, the output logarithmic energy is subjected to a discrete cosine transform (DCT) to obtain the Mel cepstral coefficients (MFCC).

[0099]

[0100] The block diagram of cepstral coefficient (GFCC) feature extraction principle is as follows: Figure 4 As shown. The Gammatone filter, similar to the basilar membrane of the human cochlea, can effectively simulate similar frequency division physiological characteristics, thereby establishing a cochlear-like auditory model. Its time-domain expression is:

[0101]

[0102] in, Represents phase, Represents the center frequency. Represents the order of the filter. For the first The bandwidth of each filter, For the center frequency, It is the filter gain.

[0103] The bandwidth of a fourth-order Gammatone filter can be expressed by the following formula:

[0104]

[0105] As can be seen from the above formula, the bandwidth of the filter increases with the increase of the center frequency. ERBEquivalent rectangular bandwidth is a psychoacoustic measure that determines the decay rate of each filter's impulse response. ERB With frequency f The relationship is as follows:

[0106]

[0107] Furthermore, in step S3, a nonlocal spatial module is designed to construct an encoder model based on a nonlocal spatial U-Net network, which reduces the dimensionality of the speech spectral features. Specifically,

[0108] A nonlocal spatial U-Net network is constructed to reduce the dimensionality of speech spectral features, and the dimensionality-reduced features are then used for subsequent speaker speech segmentation. Mel-triangle filter (MFCC) and cepstral coefficient (GFCC) features are used as speech features in the frequency domain. To be suitable for speech features, the number of channels in the convolutional layers and transposed convolutions in the decoder and encoder of the original network are reduced, respectively. Multiple linear layers are added between the decoder and encoder to obtain a one-dimensional feature representation. A schematic diagram of the U-Net-based autoencoder network structure is shown below. Figure 5 As shown.

[0109] The design of nonlocal spatial modules before each upsampling block can suppress the feature responses of irrelevant background regions and improve performance. Nonlocal spatial modules, such as... Figure 6 As shown, convolution operations are performed on the feature maps generated by the upsampling and downsampling blocks at corresponding positions, and these features are then combined to obtain attention coefficients. In the non-local spatial module, the attention coefficients can improve the weights of speech feature regions, ensuring the extraction of effective feature regions. Therefore, adding the non-local spatial module to the skip connections of the encoder and decoder can improve the recognition ability of the network model. The non-local spatial module, before each upsampling block, can suppress the response of irrelevant speech features and improve performance. In the non-local spatial module, convolution operations are performed on the feature maps generated by the upsampling and downsampling blocks at corresponding positions, and these features are then combined to obtain attention coefficients. Specifically, decoder features... They can be represented as:

[0110]

[0111] Encoder features They can be represented as:

[0112]

[0113] in, and They represent the first Feature maps of upsampled blocks and downsampled blocks in the layer. and They represent the first Layer-by-layer learning feature maps and The weight parameters, and Indicates the first The corresponding learning bias of the layer. H and W This represents the height and width of the feature map. This represents the ReLU activation function. After obtaining the features of the encoder and decoder, the attention coefficients can be calculated using the following formula:

[0114]

[0115] in, and They represent the first Learnable weight parameters and biases are used to extract feature maps from layers. This represents the Sigmoid activation function, which makes each output value range from 0 to 1. Furthermore, the learned weight parameters are all obtained using 1×1×1 convolutions. and The weight parameters of the feature map are learned and updated through backpropagation. The output features of the nonlocal spatial module can be represented as:

[0116]

[0117] To train the autoencoder, the TED-LIUM dataset was used. The corpus in the dataset was segmented to a fixed length, and Mel-triangular filter (MFCC) and cepstral coefficient (GFCC) features were extracted separately and input into the encoder for training. During training, the loss was calculated using mean squared error (MSE) on both the model's output and input, and the model was optimized using Adam.

[0118] In step S4, the distance between two feature segments is measured using KL divergence. .

[0119] In step S5, steps S3 and S4 are repeated until the range of the feature vector is exceeded. Then, the distance obtained in S4 is smoothed using a low-pass filter, and a local extremum of the curve is taken as the speech segmentation point.

[0120] The specific steps of the embodiments of the present invention are as follows: For S1: The experimental voice sample is a few minutes of multi-person dialogue audio recorded using a mobile phone. The audio file is converted into a mono WAV file with a sampling rate of 16KHz and a bit rate of 256kb / s using ffmpeg. The noise files selected are the three noise types: babble, volvo, and factory1 from the Noise-92 noise library.

[0121] For S2: To calculate the Mel frequency cepstral coefficients, 26 filters are set within the spectral range of each frame of speech in step S1. When using Mel-triangle filter (MFCC) features, due to the contribution factor, the first dimension is the energy parameter, and the computational load after the 14th dimension is too large. The changes between them are not significant, and their contributions can be basically ignored. The Mel-triangle filter (MFCC) feature extraction process is as follows: Figure 3 As shown. The Gammatone filter consists of 64 individual filters. Since the fourth-order Gammatone filter is closest to the human auditory response function, the value is set to 4 in this invention. The cepstral coefficient (GFCC) extraction process is similar to the Mel-triangle filter (MFCC) extraction process, except that the Mel-triangle filter is replaced by a Gammatone filter. The process is as follows: Figure 4 As shown. Finally, 20-dimensional Mel-triangle filter (MFCC) and cepstral coefficient (GFCC) features are extracted respectively, and the two features are superimposed and normalized in the frequency dimension.

[0122] For S3: To compare the training performance of different models, a linear encoder was selected (with the number of neurons being 4000, 2000, and 30 respectively). For the autoencoder designed in this invention, the maximum number of channels for the convolution and transposed convolution in the encoder and decoder of the model was set to 64 and 128 respectively for comparison. The above models were trained using a constant learning rate. lr =0.001 training, the loss of different models after 100 rounds is as follows Figure 7 As shown in the figure, the Unet encoder represents a maximum of 64 channels, and the Unet_big encoder represents 128 channels. Therefore, the autoencoder with a maximum of 64 channels is ultimately chosen as the feature dimensionality reduction model. Assuming we extract a vector from a frame, using a length of... d A window is used to extract a segment of features, and then the adjacent segment is extracted by shifting the frame by one frame. These segments are then input into the trained non-local space U-Net to obtain two one-dimensional feature vectors of length 30. In speech segmentation and clustering, 12-dimensional Mel-triangle filter (MFCC) features are used.

[0123] For S4, KL divergence is used to measure the distance between two features.

[0124] For S5, steps S3 and S4 are repeated until the range of the feature vector is exceeded. Then, the distance obtained in step S4 is smoothed using a low-pass filter, and a local extremum of the curve is taken as the speech segmentation point. To verify the effectiveness of the invention, the inventors also conducted the following simulation experiments.

[0125] refer to Figure 1 The flowchart of the entire network framework algorithm is shown below. The output is the violent event recognition result. The test results of the stabilized trained model are evaluated on the test set as shown below.

[0126] Table 1 shows that the purity of the binary classification-based speech segmentation method reached 95.36%, which is higher than the other three segmentation algorithms and 0.68% higher than the purity of the method of this invention. Figure 8 As can be seen, the method of this invention, compared to the binary classification segmentation algorithm, only segments some non-speech segments into speech segments, thus reducing the purity slightly. Therefore, the difference in purity between the two algorithms is negligible. In terms of algorithm efficiency, the segmentation algorithm based on the binary classification window takes 6.12 seconds, while the method of this invention takes 3.85 seconds. Under noise-free speech conditions, the method of this invention improves the running efficiency by approximately 37% without sacrificing segmentation accuracy.

[0127] Table 1. Segmentation results of different algorithms in a noise-free environment

[0128]

[0129] To compare the segmentation results of different types of noise with signal-to-noise ratios (SNRs) of -5, 0, and 5 added to different pairs of speech samples under different noise environments, Table 2 shows the segmentation results. Taking the babble noise at -5 dB as an example, the segmentation effect is as follows: Figure 9 As shown in Table 2, it can be seen that the signal-to-noise ratio (SNR) has a greater impact on different segmentation algorithms than the difference in noise type. Among the five segmentation algorithms, the SNR has the greatest impact on the BIC-based segmentation algorithm. As the SNR decreases, the purity of the segmentation continuously declines, while the segmentation results of the method of this invention remain almost unchanged under different speech environments. The BIC-based algorithm uses the original Mel-triangle filter (MFCC) as the input feature, which indicates that the hybrid feature proposed in this chapter, utilizing the dimensionality-reduced Mel-triangle filter (MFCC) and cepstral coefficients (GFCC) after autoencoder reduction, has good noise resistance.

[0130] Table 2. Segmentation performance of different algorithms under different noise levels.

[0131]

[0132] The specific embodiments of the present invention have been described in detail above. The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. The above embodiments are only used to illustrate the methods and are not intended to limit the technical solutions of the present invention. Rather, they are technical solutions consistent with the principles and novel features of the present invention and should all be within the scope of protection defined by the claims.

Claims

1. A speaker speech segmentation method based on non-local spatial U-Net and hybrid features, characterized in that, Includes the following steps: S1. Acquire the time-domain waveform based on the input speech signal, and preprocess the waveform to obtain the speech spectrum of the speech signal; S2. Extract Mel-Cepstral Coefficients and Cepstral Coefficients, superimpose and normalize the two features in the frequency dimension, and combine them as a hybrid feature based on the high accuracy of Mel-Cepstral Coefficients and the strong robustness of Cepstral Coefficients for speaker speech segmentation. S3. Construct an encoder model based on a nonlocal spatial U-Net network, design a nonlocal spatial module, and reduce the dimensionality of the mixed features of speech. Extract a vector from a frame, using a length of d The window takes one segment of features and then moves to the adjacent segment of features by 1 frame. The features are input into the trained non-local space U-Net respectively, resulting in two feature segments; S4. Use KL divergence to measure the distance between two features; S5. Repeat steps S3 and S4 until the range of the feature vector is exceeded; Then, the distance obtained in step S4 is smoothed using a low-pass filter, and a local extremum of the curve is taken as the speech segmentation point. This allows us to train a speaker speech segmentation model with high accuracy and stability based on nonlocal space U-Net and mixed features. The trained speech segmentation model is then tested on a test set.

2. The speaker speech segmentation method based on non-local spatial U-Net and hybrid features according to claim 1, characterized in that, Includes the following steps: Preprocessing of the speech signal, specifically: The experimental speech signal samples were several minutes of multi-person dialogue audio recorded using a mobile phone; first, the audio files were converted into WAV files, and then noise was added to the noise library. Based on the high accuracy and robustness of Mel-Cepstral Coefficient (MCC) features, the obtained WAV files are combined as hybrid features for speaker speech segmentation. Obtaining Mel-frequency cepstral coefficients requires preprocessing the input speech signal. Sound waves travel through the vocal tract to the lips and then radiate into the air. During this propagation, higher-frequency components of the speech are more easily lost. To compensate for the suppression of speech by the vocal system and to facilitate the analysis of vocal tract parameters and the spectrum, a high-frequency boosting transfer function is used. A first-order digital filter is used to pre-emphasize the original speech, as shown in the following formula. ； The value is between 0.9 and 1.0; Let the original speech signal be The speech signal after passing through the pre-emphasis filter is : ； Speech framing uses a movable, finite-length window to weight the speech. Typically, frames are not continuous but partially overlap. Assume there exists a frame of length... The audio signal is framed: ； in For frame shift, The frame length is such that the ratio of frame shift to frame length is between 0 and 0.5, ensuring a smooth transition between frames. Speech signals can be abruptly truncated due to speech framing, resulting in discontinuous signals and distortion of the Fourier transform spectrum. This distortion is commonly referred to as spectral leakage. To prevent this, the speech signal after framing is windowed. Window function Its function is to generate a function that takes all zero values ​​within a given interval. The Hanning window is one of the commonly used window functions. ； After segmenting and windowing the original data, time-frequency analysis of the speech is performed using short-time Fourier transform: ； in, It refers to the window function; the short-time Fourier transform means using a window function. For speech signals The truncated signal is then subjected to a Fourier transform, i.e., the calculation is performed. t This moment is changing, and it is constantly evolving. t The values ​​are used to obtain the set of Fourier transforms at each time step. This function represents the speech signal in t Frequency components obtained near time The amplitude and phase; then each in the set t The energy spectrum is obtained by taking the absolute value or squaring the data at each moment. To calculate the Mel-frequency cepstral coefficients, a range of frequencies was set for each frame of speech. M Filters ,in Representing the m The center frequency of the triangular filter; ； The logarithmic energy of the output of each triangular filter bank is: ； in, It is the first The first frame of the audio signal spectrum diagram The amplitude of the discrete Fourier transform coefficients of the term; when filtering the speech spectrum with a triangular filter bank, the product of the amplitude of each point within the bandwidth of each filter and the energy of the corresponding point in the spectrum is calculated and summed. The result is used as the output of this triangular filter, and the actual spectrum is mapped onto the Mel spectrum; finally, the output logarithmic energy is subjected to discrete cosine transform to obtain the Mel cepstral coefficients. ； Regarding the cepstral coefficient characteristics; the Gammatone filter, similar to the basilar membrane of the human cochlea, can effectively simulate similar frequency division physiological characteristics, establishing a cochlear-like auditory model, whose time-domain expression is: ； in, Represents phase, Represents the center frequency. Represents the order of the filter. For the first The bandwidth of each filter, For the center frequency, It is the filter gain; The bandwidth of a fourth-order Gammatone filter can be expressed by the following formula: ； As the center frequency increases, the bandwidth of the filter also increases; among which ERB Representing the equivalent rectangular bandwidth, it is a psychoacoustic measure that determines the decay rate of each filter's impulse response. ERB With frequency The relationship is as follows: ； in ERB Represents the equivalent rectangular bandwidth. The center frequency.

3. The speaker speech segmentation method based on non-local spatial U-Net and hybrid features according to claim 1, characterized in that, In step S3, a nonlocal spatial module is designed to construct an encoder model based on a nonlocal spatial U-Net network, which reduces the dimensionality of the speech spectral features. Specifically: A nonlocal space-based U-Net network is constructed to reduce the dimensionality of speech spectral features, and the dimensionality-reduced features are used for subsequent speaker speech segmentation. Mel-frequency cepstral coefficients and cepstral coefficient features are used as speech features in the frequency domain. In order to be suitable for speech features, the number of channels of convolutional layers and transposed convolutions in the decoder and encoder of the original network are reduced respectively, and multiple linear layers are added between the decoder and encoder to obtain one-dimensional feature representation. The nonlocal spatial module, designed before each upsampling block, can suppress feature responses from irrelevant background regions and improve performance. Within the nonlocal spatial module, convolution operations are performed on the feature maps generated by the upsampling and downsampling blocks at corresponding positions, and these features are then combined to obtain attention coefficients. In the nonlocal spatial module, these attention coefficients can increase the weights of speech feature regions, ensuring the extraction of effective feature regions. Therefore, adding the nonlocal spatial module to the skip connections of the encoder and decoder can improve the network model's recognition ability. The nonlocal spatial module, before each upsampling block, can suppress responses from irrelevant speech features and improve performance. Specifically, decoder features It can be represented as: ； Encoder features It can be represented as: ； in, and They represent the first Feature maps of upsampled blocks and downsampled blocks in the layer. and They represent the first Layer-by-layer learning feature maps and The weight parameters, and Indicates the first The corresponding learning bias of the layer, H and W This represents the height and width of the feature map. Represents the ReLU activation function; after obtaining the features of the encoder and decoder, the attention coefficients are calculated using the following formula: ； in, and They represent the first Learnable weight parameters and biases are extracted from the feature map layer. This represents the Sigmoid activation function, which makes each output value range from 0 to 1. Furthermore, the learned weight parameters are all obtained using 1×1×1 convolutions. and The weight parameters of the feature map are learned and updated through backpropagation; the output features of the nonlocal spatial module can be represented as: ； To train the encoder, the TED-LIUM dataset was used. The corpus in the dataset was segmented into segments of fixed length, and Mel-Cepstral Coefficients and Cepstral Coefficients features were extracted and input into the encoder for training. During training, the loss of the model's output and input was calculated using mean squared error, and the model was optimized using Adam.

4. The speaker speech segmentation method based on non-local spatial U-Net and hybrid features according to claim 1, characterized in that, In step S4, the distance between two feature segments is measured using KL divergence. .

5. The speaker speech segmentation method based on non-local spatial U-Net and hybrid features according to claim 1, characterized in that, In step S5, steps S3 and S4 are repeated until the range of the feature vector is exceeded; then, the distance obtained in step S4 is smoothed using a low-pass filter, and a local extremum of the curve is taken as the speech segmentation point.

Images