Single-channel voice separation method and device based on auditory modulation mechanism and comparative learning
Through auditory modulation mechanism and contrast learning method, the characteristics of speech segments are extracted and adaptively classified, which solves the problem of distinguishing between single-channel speech separation in complex environments, realizes accurate separation of mixed speech, and improves the accuracy of speech signal processing .
Patent Information
- Application Number
- CN202411225072.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-19
AI Technical Summary
In complex acoustic environments, single-channel speech separation technology is difficult to accurately distinguish the time and frequency points of the spectral map of different speakers, resulting in the impact of separation performance.
Using a method based on auditory modulation mechanism and contrast learning, voice fragments are generated through mutation point detection and matching methods, voice fragment features are extracted using triple attention comparison learning network, and unsupervised adaptive classification is performed through adaptive neural structure evolution network, combined with Binary masking performs speech reconstruction to achieve speech separation.
It effectively realizes the precise separation of mixed speech, improves the accuracy and reliability of speech signal processing, and expands the application range of deep learning beyond the traditional time-domain and time-frequency domain speech separation.
Smart Images

Figure CN118982989A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech signal processing, and in particular to a single-channel speech separation method and device based on an auditory modulation mechanism and contrastive learning. Background Art
[0002] In a complex acoustic environment, the target speaker's voice signal is often interfered by various noises, which seriously affects the recognition performance of the target voice. Speech separation technology can effectively remove noise interference in the actual environment and provide more accurate and reliable information for subsequent speech signal processing. The application scenarios of speech separation technology are very wide. For example, in the field of national defense and military, in the context of war environment and conference monitoring, the use of voiceprint recognition technology alone cannot accurately analyze whether there is a specific speaker in the conference recording intercepted from the enemy, and speech separation technology can improve the accuracy of voiceprint recognition. In the field of public safety, on the streets in a noisy multi-speaker scene, the use of speech separation technology can more accurately find specific words and lock pedestrians with dangerous intentions; in the field of smart homes, when controlling smart devices through voice commands, other family members are often communicating in language, so speech separation technology is needed to accurately obtain the target voice command, so as to correctly execute its intention.
[0003] Speech separation can be regarded as a basic preprocessing step in speaker identification, speech recognition, automatic speech translation and speech emotion recognition, and plays a fundamental role in the entire field of speech signal processing. From an application perspective, single-microphone devices have fewer deployment restrictions and do not have potential problems such as configuration stability of multi-microphone systems. They are easier to implement on devices, so single-channel speech separation is the most ideal research object. In complex acoustic environments, single-channel speech separation technology faces important challenges. One of the main disadvantages is that the time-frequency points of the spectrograms of different speakers have a high overlap rate, which makes it difficult for the separation algorithm to accurately distinguish them, thus affecting the separation performance. Summary of the invention
[0004] Purpose of the invention: The present invention discloses a single-channel speech separation method and device based on auditory modulation mechanism and contrastive learning. The method aims to solve the problem of speech separation of mixed speech of multiple speakers in a non-contact state. The original speech data of multiple speakers are collected, randomly mixed to construct a speech data set, and the mutation point detection and matching method are used to generate training set speech segments. Subsequently, the speech segment features are extracted through a triple attention contrastive learning network, and the similarity loss function is used for training to achieve the extraction of speaker discriminative features. Furthermore, an adaptive neural structure evolution network is used for unsupervised adaptive classification, and the speech segment is reconstructed in combination with binary masking to finally obtain the separated speech. This method comprehensively uses auditory modulation mechanism and contrastive learning to effectively achieve the accurate separation of mixed speech, providing an innovative solution for the field of speech signal processing.
[0005] Technical solution: The present invention is implemented by adopting the following technical solution: a single-channel speech separation method based on auditory modulation mechanism and contrastive learning, comprising the following steps:
[0006] Step A: Use a microphone to collect raw speech data of several speakers, randomly select speech signals of multiple different speakers to mix and construct a speech data set, and divide it into a training set, a validation set, and a test set;
[0007] Step B, using envelope detection to calculate the modulation amplitude spectrum and phase spectrum;
[0008] Step C, generating a speech segment on the modulation amplitude spectrum based on a mutation point detection and matching method;
[0009] Step C1, using a mutation point detection method to obtain a maximum point / minimum point connection line of the modulation amplitude spectrum;
[0010] Step C2, generating a speech segment using a mutation point matching method;
[0011] Step D, randomly selecting speech segments to construct triples;
[0012] Step E: online screening of hard / semi-hard triplets as training samples based on the Euclidean distance of positive and negative samples;
[0013] Step F: construct a triple attention contrastive learning network to extract speech segment features;
[0014] Step F1, design a triple attention mechanism, use channel attention to distinguish the importance of different channels, use spatial attention to aggregate spatial dependencies within channels, and capture long-distance dependencies of BLSTM layer outputs through multi-head self-attention;
[0015] Step F2: Use the spatial pyramid pooling method to unify the feature dimensions of speech segments of different sizes;
[0016] Step F3, designing a similarity loss function to reduce the distance between the features of the speech segments of the same speaker;
[0017] Step G, using the training samples described in step E to update the parameters of the triple attention contrastive learning network, and selecting the optimal hyperparameters through the validation set;
[0018] Step H, designing an adaptive neural structure evolution network to perform unsupervised adaptive classification of speech fragments;
[0019] Step I: Calculate binary masking based on the speech segment classification results, reconstruct the speech segments of different speakers, and obtain separated speech.
[0020] Step J: For the mixed speech to be separated in the test set, input the trained model to obtain the separated speech.
[0021] Furthermore, in step A, a microphone is used to collect raw speech data of several speakers, and speech signals of multiple different speakers are randomly selected to mix and construct a speech data set, which is divided into a training set, a validation set, and a test set. Specifically, the following method is adopted:
[0022] (1) Create two-speaker mixed and three-speaker mixed datasets, and the speech in the datasets is resampled to 8kHz.
[0023] (2) The training data and validation data are generated by randomly selecting sentences from different speakers and mixing them with a random signal-to-noise ratio ranging from 0 dB to 5 dB;
[0024] (3) The test data is generated by randomly selecting 16 sentences from different speakers in the training data and validation data and mixing them with a random signal-to-noise ratio ranging from 0dB to 5dB.
[0025] (4) The audio sampling frequency of the speech signals in the training set and the validation set is 8 kHz, and the sampling length is 4 s. If the speech length is less than 4 s, it is padded, and if the speech length is greater than 4 s, 4 s is cut from the beginning of the speech.
[0026] Furthermore, when the modulation amplitude spectrum is calculated by envelope detection in step B, the following method is specifically adopted:
[0027] (1) Use Gammatone filter to establish a 128-channel filter bank;
[0028] (2) Envelope detection using Hilbert transform based on incoherent demodulation;
[0029] (3) Obtain the modulation amplitude spectrum and phase spectrum through 1024-point Fourier transform;
[0030] (4) Smooth the modulation amplitude spectrum through a low-pass filter.
[0031] Furthermore, in step C1, the maximum / minimum point connection line of the modulation amplitude spectrum is obtained by using the mutation point detection method as follows:
[0032] (1) Calculate the first-order derivative of the smoothed modulation amplitude spectrum in each channel to obtain candidate maximum / minimum mutation points, where the maximum point corresponds to the maximum point and the minimum point corresponds to the minimum point; further filter the maximum points by setting a threshold; retain all the minimum points with the smallest value between adjacent maximum points, and delete the remaining minimum points.
[0033] (2) The average value of the distances between all adjacent maximum points in the current channel is used as the threshold; the set of maximum points of adjacent channels whose distances to the maximum point of the current channel are less than the threshold are screened, and the maximum point with the smallest distance in the set is selected for connection; the same screening and connection method is used for the minimum points; and the connection lines with a span of less than three adjacent channels are cancelled.
[0034] (3) Select the connecting line of the maximum points of each continuous channel, select the minimum points adjacent to the right side of the maximum points, and construct a minimum point set of size Z; select the minimum point connecting line that passes through the minimum point set with the largest number of points as the minimum point connecting line that best matches the maximum point connecting line; when all the maximum points of Z consecutive channels are successfully matched, the process ends; otherwise, repeat this process for the channels that fail to match; and use the area between the matching maximum points and the minimum point connecting line as the voice segment.
[0035] Furthermore, in step C2, the voice segment window is generated by using the mutation point matching method as follows:
[0036] (1) Obtaining coordinate data of irregular speech segments: First, the coordinate data of irregular graphics need to be collected. These coordinates can represent the boundary points or vertices of the irregular speech segments.
[0037] Calculate the minimum and maximum coordinate values: Traverse the coordinate data of the voice segment and find the minimum x-coordinate value and y-coordinate value, as well as the maximum x-coordinate value and y-coordinate value. This will become the upper left corner and lower right corner coordinates of the outer box.
[0038] (2) Calculate the width and height of the bounding box: The width and height of the bounding box can be calculated by the minimum and maximum coordinate values. The width is equal to the maximum x-coordinate value minus the minimum x-coordinate value, and the height is equal to the maximum y-coordinate value minus the minimum y-coordinate value.
[0039] (3) Create a rectangular outer frame: Use the outer frame width and height calculated in the previous step and the minimum coordinate value to create a rectangular outer frame as the voice segment window.
[0040] Furthermore, in the step D, speech segments are randomly selected to construct triplets, and triplets of training samples consisting of a benchmark sample, a positive sample and a negative sample are constructed, wherein the benchmark sample is a speech segment randomly selected from the training data set, the positive sample is a speech segment belonging to the same speaker as the benchmark sample, and the negative sample is a speech segment belonging to a different speaker than the benchmark sample.
[0041] Furthermore, in step E, the method for screening hard / semi-hard triplet training samples is: in the training process of each batch, the hard / semi-hard triplet training samples are screened online, wherein the semi-hard triplet satisfies:
[0042]
[0043] Where: and are the features extracted from the benchmark sample, positive sample and negative sample in the lth training sample respectively; γ is the threshold for controlling the distance difference between positive and negative samples.
[0044] A hard triple satisfies:
[0045]
[0046] Furthermore, the step F constructs a triple attention contrastive learning network to extract speech segment features; the step F1 establishes the discriminative features of speech segments extracted by the triple attention contrastive learning network, and the contrastive learning network model can still have good generalization performance when the amount of data is small. The steps are as follows:
[0047] (4) Spatial structural feature extraction design
[0048] Firstly, the convolutional layer of the contrastive learning network extracts the spatial structural features related to the frequency band of the speech segment by setting different convolution kernels; then, the batch normalization method is used to normalize the output of each convolutional layer to solve the problem of gradient disappearance during back propagation; finally, the LeakyReLU activation function is used to solve the problems of gradient disappearance and neuron "death".
[0049] (5) Triple Attention Design
[0050] ① Channel attention: First, the global spatial feature information in each channel is compressed through average pooling and maximum pooling to generate channel features; then, the feature weight of each channel domain is obtained through a shared network; then, the channel weight information is normalized through an activation function; finally, the channel weight information is multiplied by the input feature map to generate the feature map F′. The calculation method of channel attention is as follows:
[0051]
[0052] Where: σ is the sigmoid function; S is the shared network; W0 and W1 are the shared weight matrices; and Represent the maximum pooling feature and the average pooling feature respectively.
[0053] ② Spatial attention: First, the channel domain features of the input features are compressed into dual-channel features through global maximum pooling and global average pooling; then, the dual-channel features are compressed into single-channel features through convolution operations to eliminate the influence of inter-channel information distribution on spatial attention; then, the spatial weight information is normalized through the activation function; finally, the spatial weight information is multiplied with the corresponding elements of the input feature map to generate the feature map F″. The calculation method of spatial attention is as follows:
[0054]
[0055] Where: σ is the sigmoid function; c is the convolution operation; and Represent the maximum pooling feature and the average pooling feature respectively.
[0056] ③Multi-head self-attention: First, keep the width n of the input feature w and height n h The number of channels is transformed to n through 1×1 convolution. size ; Then, use the scaled dot product to calculate n head The attention weight matrix of each head; then, the output matrices of all heads are superimposed to obtain the multi-attention feature map X m , the calculation method is as follows:
[0057]
[0058] Where: n head is the number of self-attention heads; is the output matrix of the i-th head.
[0059] Finally, global average pooling (GAP) is used to calculate n size The eigenvector X of fusion , as the input of the fully connected layer in the contrastive learning network.
[0060] In step F2, in order to solve the problem that the dimensions of the speech segments input to the convolutional contrastive learning network are not uniform, while the number of neurons in the fully connected layer must be fixed, multi-scale spatial pyramid pooling is further used to unify the feature dimensions.
[0061] (1) First, the feature map v is divided into M windows of different scales. Each scale represents a layer of the pyramid. The window size of the pooling layer of the mth layer is:
[0062]
[0063] Where: win_w m and win_h m are the width and height of the pooling window of the mth layer, scale m is the scale of the mth layer, W fm and H fm are the width and height of the feature map v respectively;
[0064] (2) Then, for scale m The features in the pooling window of are subjected to the maximum pooling operation, expanded and concatenated to obtain R m dimensional feature vector:
[0065]
[0066] Where: D fm4 is the depth of the feature map v;
[0067] (3) Finally, the R-dimensional feature vector obtained by concatenating the feature vectors of all scales is used as the input of the fully connected layer:
[0068]
[0069] In step F3, the design goal of the similarity loss function is to map the speech segment to a new feature space, so that the reference samples belonging to the same speaker are closer to the positive samples in the feature space, and the reference samples belonging to different speakers are farther from the negative samples; at the same time, in order to avoid the positive and negative samples from gathering in a narrow feature space, the distance between the positive and negative samples is controlled by changing the threshold γ, that is, satisfying:
[0070] The designed similarity loss function is: By using the online screened hard / semi-hard triplet training samples, the triple attention contrastive learning network is trained through the similarity loss function to reduce the distance between the features of the speech segments of the same speaker and increase the distance between the features of the speech segments of different speakers, finally realizing the extraction of discriminative features of speech segments.
[0071] Furthermore, in step G, the training samples described in step E are used to update the parameters of the triple attention contrastive learning network, and the optimal hyperparameters are selected through the validation set;
[0072] (1) Randomly initialize parameters: Randomly initialize the parameters of the speech separation model to ensure that the model is in a baseline state when training begins.
[0073] (2) Predict the labels of training samples: Use the randomly initialized model to forward propagate the training samples to obtain the separated speech labels.
[0074] (3) Calculate the total loss: Calculate the loss between the model output and the true label, and use the similarity loss function to reduce the distance between the features of the same speaker's speech segments and expand the distance between the features of different speakers' speech segments.
[0075] (4) Update parameters: Use the stochastic gradient descent optimization algorithm to update the parameters of the triple attention contrastive learning network model according to the calculated total loss. This includes adjusting the parameters of the convolutional neural network, triple attention mechanism, spatial pyramid pooling, fully connected layer, etc.
[0076] (5) Determine whether it has converged: Check whether the total loss has converged. If the loss has not converged, that is, the training has not reached a stable state, perform the following steps.
[0077] (6) Update labels: Recompute the labels of training samples based on the current triple attention contrastive learning network model parameters. This may involve updating the choice of semi-hard / hard triplets to better reflect the performance of the current model.
[0078] (7) Repeat training: Return to step (3), recalculate the total loss, and update the model parameters again through the stochastic gradient descent optimization algorithm. This cycle will be repeated until the loss converges, indicating that the model has learned discriminative features.
[0079] Furthermore, in step H, the method for performing unsupervised adaptive classification of speech segments by the adaptive neural structure evolution network is:
[0080] (1) Weight vector W of the root node (0) Assign a random value in the interval [0,1];
[0081] (2) Set the initial learning rate η(0) and growth factor α to 0.5, and calculate the growth threshold G:
[0082] Where s is the total number of nodes in the growing self-organizing neural network;
[0083] (3) Select a sample vector v from the sample set (i) , where i is the sample number, and the weighted distance is used to find the sample with the same weight as f (n) The nearest competitive layer node j * ; n∈{1,2,...,N}
[0084] (4) Calculate v (i) With the competition layer * Error distance E between nodes:
[0085] Where R is v (i) If E>G, then go to step (5) to perform the growth operation, otherwise go to step (6) to perform the adjustment operation;
[0086] (5) Generate j * The child nodes of
[0087] (6) Adjust j * And its child node weights:
[0088]
[0089] in, For node j * The winning neighborhood.
[0090] (7) A large learning rate can speed up learning, but it is not easy to converge, so a dynamic learning rate from large to small can be used. The learning rate can be adjusted according to the following formula:
[0091] η(t+1)=λ.η(t)
[0092] Where λ is the adjustment factor, and its value range is (0,1).
[0093] (8) Repeat steps 3-7 until all samples are trained;
[0094] (9) Repeat step 8 to enter the next training cycle until no new nodes are generated in the network.
[0095] (10) Finally, reduce η(0) and fine-tune the weights of the nodes in the network. As in the training phase, find (i) The nearest node j * , and adjust the weights of its child nodes, but no new nodes are generated in this stage. The purpose is to adjust the learning rate and correct the error, and repeatedly train the nodes generated by steps (2) and (3). After the network training is completed, a two-dimensional tree structure is obtained. The growth threshold G determines whether new nodes are generated in the network, so the size of G will determine the growth of the network; and the size of G is determined by the growth factor α. When α is large, G is large, and low-level clustering is achieved; when α is small, G is small, and high-level clustering is achieved.
[0096] Furthermore, in the step I, speech reconstruction is performed on the speech segments in the same group based on the binary masking strategy to obtain the separated target speech.
[0097] (1) Based on the results of speech segment classification, the modulation amplitude spectrum of the nth speaker after separation is obtained using binary masking.
[0098] (2) Combined with the modulation phase spectrum of the mixed speech, the separated speech envelope is obtained using inverse Fourier transform:
[0099]
[0100] (3) Combine the carrier of the mixed voice to obtain the time domain signals of different channels:
[0101] y (n) (t,k)=e (n) (t,k).c(t,k)
[0102] (4) The separated speech is synthesized through an inverse gammatone filter.
[0103] In addition, step J provides a computer-readable storage medium, which stores a computer execution program of the speech separation model constructed by the construction method described in the first aspect. When the computer execution program is executed by the processor, an efficient speech separation method can be implemented.
[0104] Beneficial effects: This scheme introduces a triple attention mechanism, which is conducive to quickly screening out the key features of different speakers' speech from a large amount of feature information and removing interference information; it proposes a speech segment discriminative feature extraction algorithm to realize the speech feature extraction task of any speaker in a single network framework, improving the discriminativeness of speech segment features and the versatility of speech feature extraction algorithms. It proposes that the adaptive neural structure evolution network can simulate the load balancing process of neurons in the brain neural network, dynamically generate new sub-nodes of the output layer, and realize unsupervised adaptive classification of speech segments; it proposes a speech separation algorithm based on auditory modulation mechanism, which has certain advantages in solving speech separation problems in the modulation domain, and expands the application scope of deep learning beyond traditional time domain and time-frequency domain speech separation. BRIEF DESCRIPTION OF THE DRAWINGS
[0105] Figure 1 This is a block diagram of the speech separation principle described in an embodiment of the present invention;
[0106] Figure 2 It is a schematic diagram of calculating the modulation amplitude spectrum and phase spectrum according to an embodiment of the present invention.
[0107] Figure 3 This is a schematic diagram of speech segment classification according to an embodiment of the present invention.
[0108] Figure 4 This is a schematic diagram of speech reconstruction according to an embodiment of the present invention. DETAILED DESCRIPTION
[0109] This embodiment proposes a single-channel speech separation method based on auditory modulation mechanism and contrastive learning. Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, the following steps are included:
[0110] Step 1: Use a microphone to collect the original speech data of several speakers, randomly select the speech signals of multiple different speakers to mix and construct a speech data set, and divide it into a training set, a validation set, and a test set;
[0111] Step 2: Calculate the modulation amplitude spectrum and phase spectrum using envelope detection;
[0112] Step 3: Generate speech segments on the modulation amplitude spectrum based on mutation point detection and matching methods;
[0113] Step 4: Randomly select speech segments to construct triples;
[0114] Step 5: Online selection of hard / semi-hard triplets as training samples based on the Euclidean distance of positive and negative samples;
[0115] Step 6: Construct a triple attention contrastive learning network to extract speech segment features;
[0116] Step 7: Use the training samples described in step E to update the parameters of the triple attention contrastive learning network, and select the optimal hyperparameters through the validation set;
[0117] Step 8: Design an adaptive neural structure evolution network to perform unsupervised adaptive classification of speech fragments;
[0118] Step 9: Calculate binary masking based on the speech segment classification results, reconstruct the speech segments of different speakers, and obtain separated speech.
[0119] Step 10: For the mixed speech to be separated in the test set, input the trained model to obtain the separated speech.
[0120] Step 1: Use a microphone to collect the original speech data of several speakers, randomly select the speech signals of multiple different speakers to mix and construct a speech data set, and divide it into a training set, a validation set, and a test set;
[0121] The original speech data of several speakers are obtained, and randomly selected to construct a mixed speech data set, which is used as training data, test data and verification data respectively. The speech signals of different speakers are randomly selected from the original data, and they are randomly mixed in a mixed manner with different signal-to-noise ratios between 0dB and 5dB to generate the mixed speech of two speakers, construct a mixed speech data set, and divide it into training set, verification set and test set. The audio sampling frequency of the speech signal is 8kHz, and the sampling length is 4s; the speech length less than 4s is padded with zeros, and the speech length greater than 4s is cut from the beginning of the speech 4s. The training set, verification set and test set are divided into 8:1:1.
[0122] Step 2: Calculate the modulation amplitude spectrum and phase spectrum using envelope detection;
[0123] 1. Calculation method of modulation amplitude spectrum and phase spectrum
[0124] (1) A 128-channel filter bank is established using a Gammatone filter. Specifically:
[0125] ① Calculate the equivalent rectangular bandwidth (ERB) at a specific frequency according to the empirical formula proposed by Moore and Glasberg;
[0126] ERB(f)=24.7*(0.00437f+1)
[0127] ② Perform an integral operation on the inverse of the product of the equivalent rectangular bandwidth and the fixed frequency interval to obtain a function that maps the frequency to the channel number, and further solve the inverse function that maps the channel number to the frequency;
[0128] ③ Obtain the center frequency of all channels by solving the fixed frequency interval, and then obtain the impulse response gt(t,k) of the Gammatone filter corresponding to the kth channel;
[0129] ④The output of the kth channel is:
[0130] y(t,k)=y(t)*gt(t,k)
[0131] Where y(t) is the original input signal and * is the convolution operation.
[0132] (2) Envelope detection using Hilbert transform based on incoherent demodulation:
[0133]
[0134] Among them, e(t,k) and c k(t) are the envelope and carrier of the kth channel of the mixed speech signal; Hilbert[.] is the Hilbert transform; LPF[.] is the low-pass filter; the cut-off frequency is 20Hz.
[0135] The modulation amplitude spectrum Y is obtained by Fourier transform of 1024 points a (i,k):
[0136]
[0137] (3) In order to eliminate the interference of weak fluctuations in the modulation spectrum, the modulation amplitude spectrum is smoothed by a low-pass filter. The modulation amplitude spectrum of the kth channel is smoothed to obtain: Y s (i,k)=Y a (i,k)*g s (i)
[0138] where g s (i) is a low-pass FIR filter with a cutoff frequency of s Hz, and * is a convolution operation. The size of s determines the degree of smoothness after filtering.
[0139] Step 3: Generate speech segments on the modulation amplitude spectrum based on mutation point detection and matching method
[0140] Taking advantage of the small overlap of the modulation frequency ranges of different speakers, the clustering area of the frequency points of the same speaker is determined, effectively avoiding the confusion problem caused by the method based on a single frequency point. To solve this problem, a Gammatone filter bank is constructed to decompose the mixed speech signal into K channels; then, the envelope and carrier are extracted from the speech signal of each channel by envelope detection; then, the modulation amplitude spectrum and phase spectrum of each channel envelope are obtained by Fourier transform; then, the modulation amplitude spectrum is smoothed by a low-pass filter to obtain the smoothed modulation amplitude spectrum of the kth channel; finally, the speech segment window is obtained based on mutation point detection and matching to achieve effective extraction of speech segments.
[0141] like Figure 2 As shown, this embodiment uses the following method to generate voice segments from voice segments:
[0142] 1. Use the mutation point detection method to obtain the maximum / minimum point connection line of the modulation amplitude spectrum;
[0143] Use the mutation point detection method to obtain the maximum / minimum point connection of the modulation amplitude spectrum, specifically:
[0144] (1) Calculate the first-order derivative of the smoothed modulation amplitude spectrum in each channel to obtain candidate maximum / minimum mutation points, where the maximum point corresponds to the maximum point and the minimum point corresponds to the minimum point; further filter the maximum points by setting a threshold; retain all the minimum points with the smallest value between adjacent maximum points, and delete the remaining minimum points.
[0145] (2) The average value of the distances between all adjacent maximum points in the current channel is used as the threshold; the set of maximum points of adjacent channels whose distances to the maximum point of the current channel are less than the threshold are screened, and the maximum point with the smallest distance in the set is selected for connection; the same screening and connection method is used for the minimum points; and the connection lines with a span of less than three adjacent channels are cancelled.
[0146] 2. Generate speech segment window using mutation point matching method
[0147] For the line connecting the maximum points of each continuous channel, select the minimum points adjacent to the right side of the maximum point to construct a minimum point set of size Z; select the minimum point line that passes through the minimum point set with the most points as the minimum point line that best matches the maximum point line; when all the maximum points of Z consecutive channels are successfully matched, the process ends, otherwise repeat this process for the channels that failed to match; the area between the matched maximum point and the minimum point line is used as the speech segment. Select the largest rectangular area in the speech segment as the speech segment window, and use the modulation amplitude spectrum features in the speech segment window as the speech segment features, such as Figure 1 shown.
[0148] Step 4: Randomly select speech segments to construct triples
[0149] Construct triplet training samples: Randomly select benchmark samples in the training data set as reference points for speech segments. Positive samples are speech segments that belong to the same speaker as the benchmark samples. Negative samples are speech segments that belong to different speakers than the benchmark samples.
[0150] Step 5: Online screening of hard / semi-hard speech segment triplet training samples based on positive and negative sample Euclidean distance
[0151] This method of online selection of hard / semi-hard triplet training samples is designed to accelerate the convergence of the model and increase the focus on marginal samples. In this way, the model pays more attention to samples that are more sensitive to hard or semi-hard constraints during training, thereby accelerating the learning process of the model and improving the recognition and learning effect of key samples. This strategy helps to improve the robustness and performance of the model.
[0152] The specific steps are as follows:
[0153] (1) Online screening of hard / semi-hard triples:
[0154] In each training batch, the features of the benchmark samples, positive samples and negative samples are extracted through the feature extraction network.
[0155] For semi-hard triples, by controlling the threshold of the distance difference between positive and negative samples, the following conditions are met:
[0156]
[0157] Where: and are the features extracted from the benchmark sample, positive sample and negative sample in the lth training sample respectively; γ is the threshold for controlling the distance difference between positive and negative samples.
[0158] For hard triples, the following conditions are met:
[0159]
[0160] (2) Update network parameters:
[0161] Only the hard / semi-hard triplet training samples that pass the screening are used, instead of all the training samples, to update the parameters of the speech segment feature extraction network.
[0162] Step 6: Construct a triple attention contrastive learning network to extract speech fragment features
[0163] Audio is a time series data. Compared with images, it has temporal continuity and dynamics. In addition, the contrastive learning network model can still have good generalization performance when the amount of data is small. In this step, a convolutional neural network is introduced to highlight the spatial structural information of the speech segment. The triple attention mechanism is used to distinguish the importance of channels, aggregate spatial dependencies, and use multi-head self-attention to capture long-distance dependencies, thereby improving the expression ability of discriminative features. The spatial pyramid pooling method is introduced to unify the feature dimensions of speech segments of different sizes to solve the problem of fixed input dimensions of the fully connected layer. The similarity loss function is designed to reduce the distance between the features of speech segments of the same speaker, while expanding the distance between the features of speech segments of different speakers. The discriminative features extracted by the contrastive learning network are used as the input of the unsupervised adaptive classification method. The specific steps are as follows:
[0164] (1) Spatial structural feature extraction design
[0165] In the contrastive learning network, the convolution layer is designed to extract frequency-band-related spatial structural features from the speech segment by configuring different convolution kernels. The use of different convolution kernels enables the network to capture the frequency band information in the speech signal, thereby better understanding the spatial structure of the speech; the introduction of the batch normalization layer to normalize the output of each convolution layer helps to alleviate the gradient vanishing problem during back propagation. By keeping the mean and variance of each layer output stable, batch normalization helps to accelerate the network training process; the LeakyReLU function is used as the activation function to effectively solve the gradient vanishing and neuron "death" problems. LeakyReLU introduces a non-zero slope, so that the gradient can be passed even when the input is negative, which helps to improve the robustness of the network. Through the above design, the convolution layer can effectively extract the spatial structural features of the speech segment, and the use of batch normalization and LeakyReLU activation functions further enhances the training stability and learning ability of the network. This design enables the contrastive learning network to more effectively capture the relevant information of the speech signal and provide a more powerful feature representation for subsequent tasks.
[0166] (2) Triple Attention Design
[0167] ① Channel attention: First, the global spatial feature information in each channel is compressed through average pooling and maximum pooling to generate channel features; then, the feature weight of each channel domain is obtained through a shared network; then, the channel weight information is normalized through an activation function; finally, the channel weight information is multiplied by the input feature map to generate the feature map F′. The calculation method of channel attention is as follows:
[0168]
[0169] Where: σ is the sigmoid function; S is the shared network; W0 and W1 are the shared weight matrices; and Represent the maximum pooling feature and the average pooling feature respectively.
[0170] ② Spatial attention: First, the channel domain features of the input features are compressed into dual-channel features through global maximum pooling and global average pooling; then, the dual-channel features are compressed into single-channel features through convolution operations to eliminate the influence of inter-channel information distribution on spatial attention; then, the spatial weight information is normalized through the activation function; finally, the spatial weight information is multiplied with the corresponding elements of the input feature map to generate the feature map F″. The calculation method of spatial attention is as follows:
[0171]
[0172] Where: σ is the sigmoid function; c is the convolution operation; and Represent the maximum pooling feature and the average pooling feature respectively.
[0173] ③Multi-head self-attention: First, keep the width n of the input feature w and height n h The number of channels is transformed to n through 1×1 convolution. size ; Then, use the scaled dot product to calculate n head The attention weight matrix of each head; then, the output matrices of all heads are superimposed to obtain the multi-attention feature map X m , the calculation method is as follows:
[0174] Where: n head is the number of self-attention heads; is the output matrix of the i-th head.
[0175] Finally, global average pooling (GAP) is used to calculate n size The eigenvector X of fusion , as the input of the fully connected layer in the contrastive learning network.
[0176] (3) Unified design of feature dimensions
[0177] To address the problem that the dimensions of speech segments input to the convolutional contrastive learning network are not uniform, while the number of neurons in the fully connected layer must be fixed, multi-scale spatial pyramid pooling is further used to unify the feature dimensions.
[0178] First, the feature map v is divided using windows of M different scales. Each scale represents a layer of the pyramid. The window size of the pooling layer of the mth layer is:
[0179] Where: win_w m and win_h m are the width and height of the pooling window of the mth layer, scale m is the scale of the mth layer, W fm and H fm are the width and height of the feature map v respectively;
[0180] Then, for the scale m The features in the pooling window of are subjected to the maximum pooling operation, expanded and concatenated to obtain R m dimensional feature vector:
[0181] Where: D fm4 is the depth of the feature map v;
[0182] Finally, the R-dimensional feature vector obtained by concatenating the feature vectors of all scales is used as the input of the fully connected layer: (4) Similarity loss function design
[0183] The design goal of the similarity loss function is to map the speech segment to a new feature space, so that the benchmark samples belonging to the same speaker are closer to the positive samples in the feature space, and the benchmark samples belonging to different speakers are farther from the negative samples. At the same time, in order to avoid the aggregation of positive and negative samples in a narrow feature space, the distance between positive and negative samples is controlled by changing the threshold γ, that is, satisfying:
[0184] The designed similarity loss function is:
[0185] By using the online screened hard / semi-hard triplet training samples, the triple attention contrastive learning network is trained through the similarity loss function to reduce the distance between the features of the speech segments of the same speaker and increase the distance between the features of the speech segments of different speakers, finally realizing the extraction of discriminative features of speech segments.
[0186] Step 7: Update the triple attention contrast learning network parameters and select the optimal hyperparameters through the validation set
[0187] Use the training samples described in step 5 to update the parameters of the triple attention contrastive learning network, and select the optimal hyperparameters through the validation set;
[0188] (1) Randomly initialize parameters: Randomly initialize the parameters of the speech separation model to ensure that the model is in a baseline state when training begins.
[0189] (2) Predict the labels of training samples: Use the randomly initialized model to forward propagate the training samples to obtain the separated speech labels.
[0190] (3) Calculate the total loss: Calculate the loss between the model output and the true label, and use the similarity loss function to reduce the distance between the features of the same speaker's speech segments and expand the distance between the features of different speakers' speech segments.
[0191] (4) Update parameters: Back-propagate the error through the similarity loss function and update the parameters of the triple attention contrastive learning network. This includes adjusting the parameters of the convolutional neural network, triple attention mechanism, spatial pyramid pooling, fully connected layer, etc.
[0192] (5) Determine whether it has converged: Check whether the total loss has converged. If the loss has not converged, that is, the training has not reached a stable state, perform the following steps.
[0193] (6) Update labels: Recompute the labels of training samples based on the current triple attention contrastive learning network model parameters. This may involve updating the choice of semi-hard / hard triplets to better reflect the performance of the current model.
[0194] (7) Repeat training: Return to step (3), recalculate the total loss, and update the model parameters again. This cycle will be repeated until the loss converges, indicating that the model has learned discriminative features.
[0195] (8) Establishing a neural network and training the neural network. During training, constructing a triplet of training samples consisting of a reference sample, a positive sample, and a negative sample, and updating the network parameters through a back propagation algorithm to complete the network training.
[0196] Step 8: Unsupervised adaptive classification of speech fragments based on adaptive neural structure evolution network
[0197] Speech separation models are usually static network structures that are predefined before training. When the number of speakers in a mixed speech is unknown, a large number of attempts must be made to obtain a suitable network structure, which reduces the generalization of the speech separation model. To address this problem, the features of speech segments are used as input, a flexible tree structure is adopted, and the relationship between the growth threshold and the distance error is analyzed based on the root node of the initial state. The load balancing process of neurons in the brain neural network is simulated, and new child nodes of the output layer are dynamically generated to achieve adaptive classification of unsupervised speech segments. Figure 3 As shown, this embodiment uses the following method to classify the voice fragments:
[0198] (1) Initialize the root node, set the initial learning rate and growth factor to calculate the growth threshold: increase the weight vector W of the root node of the self-organizing neural network (0) Assign a random value in the interval [0,1]; set the initial learning rate η(0) and growth factor α to 0.5, and calculate the growth threshold G;
[0199] Where s is the total number of nodes in the growing self-organizing neural network;
[0200] (2) Sample selection and competition layer node search: Select a sample vector v from the sample set (i) , where i is the sample number, and the weighted distance is used to find the sample with the same weight as f (n) The nearest competitive layer node j * ; n∈{1,2,...,N}
[0201] (3) Calculate the error distance: Calculate v (i) With the competition layer * Error distance E between nodes:
[0202] Where R is v (i) If E>G, go to step (4) to perform the growth operation, otherwise go to step (4) to perform the adjustment operation;
[0203] (4) Generate child nodes and adjust weights: Generate j * The child nodes of Adjust j * And its child node weights:
[0204]
[0205] in, For node j * The winning neighborhood.
[0206] (5) Adjust the learning rate: A large learning rate can speed up the learning process, but it is not easy to converge. Therefore, a dynamic learning rate from large to small can be used:
[0207] η(t+1)=λ×η(t), where λ is an adjustment factor, ranging from (0,1), and is initialized to 0.5 in this embodiment;
[0208] (6) Repeat training: Repeat steps 1-5 until all samples are trained.
[0209] (7) Entering the next training cycle: Repeat step (2) to enter the next training cycle until no new nodes are generated in the network.
[0210] (8) Finally, reduce η(0) and fine-tune the weights of the nodes in the network. As in the training phase, find (i) The nearest node j * , and adjust the weights of its child nodes, but no new nodes are generated in this stage. The purpose is to adjust the learning rate and correct the error, and repeatedly train the nodes generated by steps (2) and (3). After the network training is completed, a two-dimensional tree structure is obtained. The growth threshold G determines whether new nodes are generated in the network, so the size of G will determine the growth of the network; and the size of G is determined by the growth factor α. When α is large, G is large, and low-level clustering is achieved; when α is small, G is small, and high-level clustering is achieved.
[0211] Step 9: Reconstruct the speech of the same type of speech fragments based on binary masking to obtain the separated speech
[0212] like Figure 4 As shown, this embodiment uses the following method to reconstruct the target speech:
[0213] (1) Based on the results of speech segment classification, the modulation amplitude spectrum of the nth speaker after separation is obtained using binary masking.
[0214] (2) Combined with the modulation phase spectrum of the mixed speech, the separated speech envelope is obtained using inverse Fourier transform:
[0215]
[0216] (3) Combine the carrier of the mixed voice to obtain the time domain signals of different channels:
[0217] y (n) (t,k)=e (n) (t,k).c(t,k)
[0218] (4) The separated speech is synthesized through an inverse gammatone filter.
[0219] Step 10: Generate a speech segment for the speech to be separated in the test set, input the trained model, and obtain the separated speech
[0220] The mixed speech generation segment to be separated in the test set is input into the speech separation model trained according to the single-channel speech separation method based on auditory modulation mechanism and contrastive learning; the separated speech is obtained. In different application scenarios, the speech separation model of the present invention can be used for speech separation. The specific process is as follows: Figure 1 shown.
[0221] The present invention also provides a device for a single-channel speech separation method based on an auditory modulation mechanism and contrastive learning, comprising a memory and a processor, wherein a computer program is stored in the memory, and the processor is used to implement the above method when executing the computer program.
[0222] Taking the WSJ0-2mix dataset as an example, PESQ and SDRi are selected as objective evaluation indicators, and the mean opinion score (MOS) is used as the subjective evaluation. The audio quality is evaluated by both objective and subjective evaluation methods.
[0223] Verify the correctness and effectiveness of the proposed speech separation method.
[0224] Since there is overlap between the speakers in the verification data and the training data, and there is no overlap between the speakers in the test data and the training data, objective and subjective comparative verification experiments of speaker-dependent and speaker-independent speech separation are carried out using the verification data and the test data respectively.
[0225] Objective comparative verification experiments are carried out on the proposed speech separation method and frequency domain speech separation methods, including NMF-LSTM, PIT, DAnet, Chimera, etc.
[0226] The results compared with the existing state-of-the-art models are shown in the following table:
[0227] Table 1 Comparative experimental results
[0228]
[0229]
[0230] From the results of PESQ and SDRi in the table, we can see that the speech separation model proposed in this paper outperforms the other six models on the WSJ0-2mix test set. The improvement is attributed to the generation of speech segments on the modulation amplitude spectrum based on mutation point detection and matching methods. In addition, this paper proposes a triple attention contrastive learning network model for feature extraction, which can effectively capture the interdependent features between points belonging to the same speaker in mixed speech, reduce the interference of external information, and provide more accurate input information for subsequent calculations, thereby effectively improving the separation effect.
[0231] The MOS method is used to divide the speech quality into 5 levels, and the speech quality is evaluated based on the actual auditory perception of the evaluator. Four separated speech sounds are randomly selected and recorded as S1 to S4. 40 participants are asked to score the separated speech sounds under different separation methods in the same evaluation environment, and the average score of all participants for each speech sound is taken as the final MOS score of the speech sound.
[0232] Table 2 MOS score table
[0233]
[0234] It can be observed from the MOS score table that the auditory experimental effect of this method is more ideal than that of the other six speech separation methods. The verification of the speech separation quality from the auditory perspective shows that the semantics and intelligibility of the separated audio are guaranteed, verifying the reliability of the speech separation effect of this method.
[0235] To sum up, the above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.
Claims
1. A single-channel speech separation method based on auditory modulation mechanism and contrastive learning, characterized in that: The following steps are involved: Step A: Use a microphone to collect raw speech data of several speakers, randomly select speech signals of multiple different speakers to mix and construct a speech data set, and divide it into a training set, a validation set, and a test set; Step B, using envelope detection to calculate the modulation amplitude spectrum and phase spectrum; Step C, generating a speech segment on the modulation amplitude spectrum based on a mutation point detection and matching method; Step C1, using a mutation point detection method to obtain a maximum point / minimum point connection line of the modulation amplitude spectrum; Step C2, generating a speech segment using a mutation point matching method; Step D, randomly selecting speech segments to construct triples; Step E: online screening of hard / semi-hard triplets as training samples based on the Euclidean distance of positive and negative samples; Step F: construct a triple attention contrastive learning network to extract speech segment features; Step F1, design a triple attention mechanism, use channel attention to distinguish the importance of different channels, use spatial attention to aggregate spatial dependencies within channels, and capture long-distance dependencies of BLSTM layer outputs through multi-head self-attention; Step F2: Use the spatial pyramid pooling method to unify the feature dimensions of speech segments of different sizes; Step F3, designing a similarity loss function to reduce the distance between the features of the speech segments of the same speaker; Step G, using the training samples described in step E to update the parameters of the triple attention contrastive learning network, and selecting the optimal hyperparameters through the validation set; Step H, designing an adaptive neural structure evolution network to perform unsupervised adaptive classification of speech fragments; Step I: Calculate binary masking based on the speech segment classification results, reconstruct the speech segments of different speakers, and obtain separated speech; Step J: For the mixed speech to be separated in the test set, input the trained model to obtain the separated speech.
2. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1 is characterized in that: In the step A, speech mixtures of different speakers are randomly selected from the data set, and the required training set, validation set and test set are generated by mixing them at various signal-to-noise ratios between 0 dB and 5 dB; the audio sampling frequency of the speech signals of the training set and the validation set is 8 kHz, and the sampling length is 4 s; the speech length less than 4 s is padded, and the speech length greater than 4 s is cut 4 s from the beginning of the speech.
3. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: When the modulation amplitude spectrum and phase spectrum are calculated by envelope detection in step B, the following method is specifically adopted: (1) Use Gammatone filter to establish a 128-channel filter bank; (2) Envelope detection using Hilbert transform based on incoherent demodulation; (3) Obtain the modulation amplitude spectrum and phase spectrum through 1024-point Fourier transform; (4) Smooth the modulation amplitude spectrum through a low-pass filter.
4. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In step C1, the mutation point detection method specifically includes: (1) Calculate the first-order derivative of the smoothed modulation amplitude spectrum in each channel to obtain candidate maximum and minimum mutation points, and further filter the maximum points by setting a threshold; retain all the minimum points with the smallest value between adjacent maximum points, and delete the remaining minimum points; (2) The average value of the distances between all adjacent maximum points in the current channel is used as the threshold; the set of maximum points of adjacent channels whose distances to the maximum point of the current channel are less than the threshold are screened, and the maximum point with the smallest distance in the set is selected for connection; the same screening and connection method is used for the minimum points; and the connection lines with a span of less than three adjacent channels are cancelled.
5. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In the step C2, a line connecting the maximum points of each continuous channel is selected, and a minimum point adjacent to the right side of the maximum point is selected to construct a minimum point set of size Z; a minimum point line passing through the minimum point set with the largest number of points is selected as the minimum point line that best matches the maximum point line; the process ends when all the maximum points of Z consecutive channels are successfully matched, otherwise this process is repeated for the channels that failed to match; and the area between the matched maximum point and the minimum point line is used as a voice segment.
6. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In step C2, the largest rectangular area in the speech segment is selected as the speech segment window; the speech segment window is calculated as follows: (1) Obtaining coordinate data of irregular speech segments: First, the coordinate data of irregular graphics need to be collected; these coordinates can represent the boundary points or vertices of the irregular speech segments; (2) Calculate the minimum and maximum coordinate values: traverse the coordinate data of the speech segment and find the minimum x-coordinate value and y-coordinate value, as well as the maximum x-coordinate value and y-coordinate value; this will become the coordinates of the upper left corner and lower right corner of the outer box; Calculate the width and height of the bounding box: The width and height of the bounding box can be calculated through the minimum and maximum coordinate values; the width is equal to the maximum x coordinate value minus the minimum x coordinate value, and the height is equal to the maximum y coordinate value minus the minimum y coordinate value; (3) Create a rectangular outer frame: Use the outer frame width and height calculated in the previous step and the minimum coordinate value to create a rectangular outer frame as the voice segment window.
7. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In the step D, speech segments are randomly selected to construct triplets, and a triplet training sample consisting of a benchmark sample, a positive sample and a negative sample is constructed, wherein the benchmark sample is a speech segment randomly selected from the training data set, the positive sample is a speech segment belonging to the same speaker as the benchmark sample, and the negative sample is a speech segment belonging to a different speaker than the benchmark sample.
8. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In step E, hard / semi-hard triples are selected as training samples: In the training process of each batch, hard / semi-hard triples are selected online as training samples, wherein the semi-hard triples satisfy: Where: and are the characteristics of the benchmark sample, positive sample and negative sample in the lth training sample respectively; γ is the threshold for controlling the distance between positive and negative samples; A hard triple satisfies: The screened hard / semi-hard triplet training samples are used as input to the feature extraction network.
9. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In step F1, the discriminative features of speech segments extracted by the triple attention contrastive learning network are established; the contrastive learning network model can still have good generalization performance when the amount of data is small; the specific steps are as follows: (1) Spatial structural feature extraction design First, the convolutional layer of the contrastive learning network extracts the spatial structural features related to the frequency band of the speech segment by setting different convolution kernels; Then, the batch normalization method is used to normalize the output of each convolutional layer to solve the problem of gradient disappearance during back propagation; finally, the LeakyReLU activation function is used to solve the problem of gradient disappearance and neuron "death"; (2) Triple Attention Design The triple attention design includes: channel attention, spatial attention and multi-head self-attention; the details are as follows: ① First, channel attention compresses the global spatial feature information in each channel through average pooling and maximum pooling to generate channel features; then, the feature weight of each channel domain is obtained through a shared network; after that, the channel weight information is normalized through an activation function; finally, the channel weight information is multiplied with the input feature map to generate a feature map F′; ② Secondly, spatial attention compresses the channel domain features of the input features into dual-channel features through global maximum pooling and global average pooling; then, the dual-channel features are compressed into single-channel features through convolution operations to eliminate the influence of inter-channel information distribution on spatial attention; then, the spatial weight information is normalized through the activation function; finally, the spatial weight information is multiplied with the corresponding elements of the input feature map to generate the feature map F″; ③ Then, multi-head self-attention maintains the width n of the input feature w and height n h The number of channels is transformed to n through 1×1 convolution. size ; Then, use the scaled dot product to calculate n head The attention weight matrix of each head; then, the output matrices of all heads are superimposed to obtain the multi-attention feature map X m , ④Finally, use global average pooling (GAP) to calculate n size The eigenvector X of fusion , as the input of the fully connected layer in the contrastive learning network.
10. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In step F2, in order to solve the problem that the dimensions of the speech segments input to the convolutional contrastive learning network are not uniform, and the number of neurons in the fully connected layer must be fixed, multi-scale spatial pyramid pooling is further used to unify the feature dimensions; specifically, as follows: (1) First, the feature map v is divided into M windows of different scales, each scale represents a layer of the pyramid; (2) Then, the maximum pooling operation is performed on the features in the pooling window, and they are expanded and concatenated to obtain R m dimensional feature vector: (3) Finally, the R-dimensional feature vector obtained by concatenating the feature vectors of all scales is used as the input of the fully connected layer.
11. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: The similarity loss function used in step F3 is: In the feature space, the distance between the benchmark samples and positive samples belonging to the same speaker is shortened, and the distance between the benchmark samples and negative samples belonging to different speakers is pushed away. At the same time, in order to avoid the positive and negative samples from gathering in a narrow feature space, the threshold γ is used to control the distance between the positive and negative samples, so as to achieve the extraction of high-quality discriminative features of speech segments. The goal of network training is to minimize the network loss function.
12. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In step G, the method for updating the parameters of the triple attention contrastive learning network is: through the back propagation algorithm, the back conduction of the gradient is used to update the parameters of the triple attention contrastive learning network, and the optimal hyperparameters are selected through the verification set.
13. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In the step H, the adaptive neural structure evolution network includes the following process: (1) Weight vector W of the root node (0) Assign a random value in the interval [0,1]; (2) Set the initial learning rate η(0) and growth factor α to 0.5, and calculate the growth threshold G: (3) Select a sample vector v from the sample set (i) , where i is the sample number, and the weighted distance is used to find the sample with the same weight as f (n) The nearest competitive layer node j * ; n∈{1,2,...,N} (4) Calculate v (i) With the competition layer * The error distance E between nodes. If E>G at this time, go to step (4) to perform the growth operation, otherwise go to step (6) to perform the adjustment operation; (5) Generate j * The child nodes of (6) Adjust j * And its child node weights: (7) A large learning rate can speed up learning, but it is not easy to converge, so a dynamic learning rate from large to small can be used; (8) Repeat steps 3-7 until all samples are trained; (9) Repeat step 8 to enter the next training cycle until no new nodes are generated in the network; (10) Finally, reduce η (0) And fine-tune the weights of the nodes in the network; as in the training stage, find (i) The nearest node j * , and adjust the weights of its child nodes, but no new nodes are generated at this stage; The purpose is to adjust the learning rate and correct the error, and repeatedly train the nodes generated by steps (2) and (3); after the network training is completed, a two-dimensional tree structure is obtained; the growth threshold G determines whether new nodes are generated in the network, so the size of G will determine the growth of the network; and the size of G is determined by the growth factor α. When α is large, G is large, and low-level clustering is achieved; when α is small, G is small, and high-level clustering is achieved.
14. The single-channel speech separation method based on auditory modulation mechanism and contrastive learning according to claim 1, characterized in that: In the step I, speech reconstruction is performed on the speech segments in the same group based on binary masking to obtain separated target speech; (1) Based on the results of speech segment classification, the modulation amplitude spectrum of the separated nth speaker is obtained by using binary masking, and then combined with the modulation phase spectrum of the mixed speech, the separated speech envelope is obtained by inverse Fourier transform; (2) Combine the carrier of the mixed speech to obtain the time domain signals of different channels, and then synthesize them through the inverse gammatone filter to obtain the separated speech.
15. A single-channel speech separation device based on auditory modulation mechanism and contrastive learning, characterized in that: The device comprises at least one processor and at least one memory, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 13 is implemented.
Citation Information
Cited By
Streaming speech recognition method, device and equipment based on adaptive AI large model
CN119252234A
Unsupervised video abstract generation method based on comparative learning and attention-driven adversarial network
CN119693862A
Audio watermark generation method, device and equipment and computer storage medium
CN120236593A
