Voice separation method, device and electronic device

Abstract

The present disclosure provides a method, apparatus, and electronic device for voice separation, which relate to the fields of artificial intelligence technologies such as speech technology and voiceprint recognition. The specific implementation is as follows: First, the audio to be processed is segmented to obtain multiple audio segments, and the target features corresponding to each of the multiple audio segments are respectively determined; the multiple target features are clustered to obtain multiple initial clustering results; then, according to the directed graphs corresponding to the respective initial clustering results, the multiple initial clustering results are fused to obtain multiple target clustering results. The present disclosure can improve the accuracy of the voice separation result.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technologies, and in particular, to artificial intelligence technology fields such as voice technology, voiceprint recognition, intelligent customer service, etc. Specifically, it relates to a method, apparatus, and electronic device for separating human voices from audio. Background Art

[0002] In many scenarios, such as intelligent customer service, conference discussions, interview conversations, public security interrogations, and variety shows, etc., during the audio recording process, there are often multiple speakers involved. Therefore, the recorded audio includes the audio of multiple speakers.

[0003] When it is determined according to business requirements that it is necessary to separate the audio segments corresponding to different speakers from the recorded audio for separating human voices from the audio, how to achieve the separation of human voices from the audio is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, and electronic device for separating human voices from audio, which can achieve the separation of human voices from the audio and improve the accuracy of the human voice separation result.

[0005] According to a first aspect of the present disclosure, there is provided a method for separating human voices from audio, which may include:

[0006] Performing segmentation processing on the audio to be processed to obtain a plurality of audio segments, and respectively determining the target features corresponding to each of the plurality of audio segments.

[0007] Clustering the plurality of target features to obtain a plurality of initial clustering results.

[0008] Fusing the plurality of initial clustering results according to the directed graph corresponding to each initial clustering result to obtain a plurality of target clustering results, so as to separate human voices from the audio to be processed.

[0009] According to a second aspect of the present disclosure, there is provided a device for separating human voices from audio, which may include:

[0010] A processing unit, configured to perform segmentation processing on the audio to be processed to obtain a plurality of audio segments, and respectively determine the target features corresponding to each of the plurality of audio segments.

[0011] A clustering unit, configured to cluster the plurality of target features to obtain a plurality of initial clustering results.

[0012] A fusion unit, configured to fuse the plurality of initial clustering results according to the directed graph corresponding to each initial clustering result to obtain a plurality of target clustering results, so as to separate human voices from the audio to be processed.[[ID=4I]]

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, which may include:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the voice separation method described in the first aspect above.

[0017] According to a fourth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute the voice separation method described in the first aspect above.

[0018] According to a fifth aspect of the present disclosure, there is provided a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the electronic device to execute the voice separation method described in the first aspect above.

[0019] According to the technical solution of the present disclosure, voice separation of audio can be achieved, and the accuracy of the voice separation result is improved.

[0020] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them:

[0022] Figure 1 is a schematic flowchart of a voice separation method provided according to a first embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a directed graph corresponding to an initial clustering result provided by an embodiment of the present disclosure;

[0024] Figure 3 is a schematic flowchart of a method for fusing a plurality of initial clustering results to obtain a plurality of target clustering results provided according to a second embodiment of the present disclosure;

[0025] Figure 4It is a schematic diagram provided by an embodiment of the present disclosure for fusing the initial clustering result 1 and the initial clustering result 2;

[0026] Figure 5 It is a schematic diagram provided by an embodiment of the present disclosure for four new paths existing relative to the initial clustering result 1;

[0027] Figure 6 It is a schematic diagram provided by an embodiment of the present disclosure for four new paths existing relative to the initial clustering result 2;

[0028] Figure 7 It is a schematic flowchart of a method for determining target features corresponding to multiple audio segments according to the third embodiment of the present disclosure;

[0029] Figure 8 It is a schematic flowchart of a method for optimizing network parameters of a feature enhancement model according to the fourth embodiment of the present disclosure;

[0030] Figure 9 It is a schematic structural diagram of a voice separation device according to the fifth embodiment of the present disclosure;

[0031] Figure 10 It is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Detailed implementation manners

[0032] The following makes an explanation of exemplary embodiments of the present disclosure with reference to the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0033] In the embodiments of the present disclosure, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the access relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. In the written description of the present disclosure, the character " / " generally represents an "or" relationship between the associated objects before and after. In addition, in the embodiments of the present disclosure, "first", "second", "third", "fourth", "fifth", and "sixth" are only used to distinguish the content of different objects and have no other special meanings.

[0034] The technical solutions provided by the embodiments of the present disclosure can be applied to scenarios such as speech processing. For example, in many scenarios, such as intelligent customer service, conference discussions, interviews, police interrogations, and variety shows, multiple speakers often participate in the audio recording process. Therefore, the recorded audio includes audio from multiple speakers.

[0035] When business needs dictate the need to separate audio clips corresponding to different speakers from recorded audio to isolate the human voice, the existing technology typically uses the Bayesian Information Criterion as a similarity metric for voice separation. However, this method generally achieves good separation results for audio of short conversations. However, for audio of long conversations, it cannot describe the distribution of data from different speakers, resulting in poor voice separation results.

[0036] In order to improve the accuracy of the voice separation results, when performing voice separation on audio, it is possible to consider introducing a directed graph-based clustering method into the voice separation scenario. In this way, a progressive clustering method can be adopted, from coarse to fine, to cluster multiple audio clips in the audio until the final clustering result is obtained, which can improve the accuracy of the voice separation results.

[0037] Based on the above technical concepts, the embodiments of the present disclosure provide a method for voice separation. Below, the method provided by the present disclosure will be described in detail through specific embodiments. It is understood that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0038] Example 1

[0039] Figure 1 This is a flow chart of a method for separating human voices according to the first embodiment of the present disclosure. The method for separating human voices can be executed by software and / or hardware devices. For example, the hardware device can be a terminal or a server. Figure 1 As shown, the human voice separation method may include:

[0040] S101 : Segment the audio to be processed to obtain multiple audio segments, and determine target features corresponding to each of the multiple audio segments.

[0041] For example, when segmenting the audio to be processed, an existing audio segmentation algorithm can be used to segment the audio to be processed, or an audio segmentation model based on deep learning can be used to segment the audio to be processed. The specific settings can be made according to actual needs. Here, the embodiments of the present disclosure do not make specific restrictions on the segmentation processing method of the audio to be processed.

[0042] For example, when segmenting the audio to be processed, the audio data within a preset duration can be used as an audio segment and segmented out every preset duration starting from the starting point of the audio to be processed. In this way, multiple audio segments can be obtained. It is also possible to use a non-fixed preset duration as the basis, starting from the starting point of the audio to be processed, and segment out the audio data within each preset duration as an audio segment every preset duration. In this way, multiple audio segments can be obtained, which can be specifically set according to actual needs.

[0043] After the segmentation process and obtaining multiple audio segments corresponding to the audio to be processed, the target features corresponding to each of the multiple audio segments can be determined respectively. Among them, the target features can represent the corresponding audio segments. After obtaining multiple target features, since each target feature among the multiple target features can uniquely refer to its corresponding audio segment, that is, the target feature and its corresponding audio segment are in one-to-one correspondence. Therefore, when performing the clustering operation, these multiple target features can be used as the basis for clustering, and the multiple target features are clustered, that is, the following S102 - S103 are executed; to achieve the clustering of multiple audio segments, thereby achieving the separation of the human voice in the entire audio.

[0044] S102. Cluster the multiple target features to obtain multiple initial clustering results.

[0045] Among them, the multiple initial clustering results can be understood as the clustering results obtained by roughly clustering the multiple target features. One initial clustering result is a cluster. These multiple initial clustering results are the basis for subsequent fine clustering and perform clustering processing on the multiple target features.

[0046] For example, when clustering the multiple target features to obtain multiple initial clustering results, the similarity scores between every two target features can be determined first, and based on the similarity scores between every two target features, the multiple target features are clustered to obtain multiple initial clustering results.

[0047] For example, when determining the similarity score between two target features, the similarity score between the two target features can be determined based on the cosine distance between the two target features; the probability linear discriminant analysis (PLDA) model can also be used to determine the similarity score between the two target features, and other methods can also be used to determine the similarity score between the two target features, which can be specifically set according to actual needs.

[0048] To facilitate understanding of how to perform an initial clustering on the multiple target features based on the similarity scores between pairwise target features to obtain multiple initial clustering results, hereinafter, an example will be given with the number of multiple target features being M.

[0049] When clustering the M initial clustering results, for any two of the M target features, determine the similarity score between the two target features to obtain an M×M similarity matrix. It can be understood that the score value of the similarity score between target feature a and target feature b among the M target features is the same as the score value of the similarity score between target feature b and target feature a.

[0050] When roughly clustering the M target features according to the similarity scores between any two target features, N target features can be randomly selected from the M target features as preset features, where N is less than M. From the N preset features, according to the random selection principle, select one preset feature, which can be denoted as preset feature 1, and from the remaining M−N target features, determine the K target features closest to preset feature 1, and divide preset feature 1 and the K target features closest to preset feature 1 into one cluster to obtain an initial clustering result corresponding to preset feature 1; then from the N−1 preset features, where the N−1 preset features refer to the preset features after removing preset feature 1 from the above N preset features, according to the random selection principle, select one preset feature, which can be denoted as preset feature 2, and from the remaining M−N−K target features, where the K target features refer to the K target features closest to preset feature 1, determine the K target features closest to preset feature 2, and divide preset feature 2 and the K target features closest to preset feature 2 into one cluster to obtain an initial clustering result corresponding to preset feature 2, and so on, until the last target feature among the N target features, that is, an initial clustering result corresponding to preset feature N is obtained, so as to roughly cluster the above M target clustering results into N clusters.

[0051] Among them, the value of K can be 1, or 2, or 3, etc., and can be specifically set according to actual needs. Here, for the value of K, the embodiments of the present disclosure do not make specific limitations. It should be noted that when the number of M−N target features is small, in view of the fact that target features cannot be reused, that is, they cannot belong to different initial clustering results at the same time, therefore, among the N target features, for the preset feature that is finally clustered, the corresponding initial clustering result may only include this preset feature.

[0052] After roughly clustering multiple target features to obtain multiple initial clustering results, in the embodiments of the present disclosure, in order to improve the accuracy of the vocal separation result, a clustering method based on a directed graph can be introduced into the vocal separation scenario, so that a progressive clustering method can be adopted, and then fine clustering is performed on the multiple initial clustering results obtained by rough clustering, that is, the following S103 is executed to perform vocal separation on the audio to be processed.

[0053] S103. According to the directed graphs corresponding to the respective initial clustering results, fuse the multiple initial clustering results to obtain multiple target clustering results, so as to perform vocal separation on the audio to be processed.

[0054] Among them, the similarity between the target features included in the target clustering result is relatively high.

[0055] Exemplarily, when constructing the directed graph corresponding to the initial clustering result, for this initial clustering result, the similarity scores between any two features in the initial clustering result can be determined respectively, and the edge weights between the two features can be determined based on the similarity scores between the two features, so as to construct the directed graph corresponding to the initial clustering result.

[0056] Combined with the description in S102 above, taking the initial clustering result corresponding to the preset feature 1 as an example, the initial clustering result corresponding to the preset feature 1 includes the preset feature 1, which can be denoted as feature 1, and K target features corresponding to the preset feature 1. Assuming that the value of K is 2, and the two target features can be denoted as feature 2 and feature 3, then when constructing the directed graph corresponding to the initial clustering result corresponding to the preset feature 1, the similarity scores between any two of the three features can be determined respectively, and the edge weights between the two features can be determined based on the similarity scores between the two features, so as to construct the directed graph corresponding to the initial clustering result. Exemplarily, reference can be made to Figure 2 shown in Figure 2 FIG. 16 is a schematic diagram of a directed graph corresponding to an initial clustering result provided by an embodiment of the present disclosure. It can be seen that the constructed directed graph includes three feature nodes. Among them, the edge weight P12 between feature 1 and feature 2 is determined based on the similarity between feature 1 and feature 2, the edge weight P13 between feature 1 and feature 3 is determined based on the similarity between feature 1 and feature 3, and the edge weight P23 between feature 2 and feature 3 is determined based on the similarity between feature 2 and feature 3, so as to construct the directed graph corresponding to the initial clustering result.

[0057] After constructing the directed graph corresponding to each initial clustering result, multiple initial clustering results can be fused according to the directed graph corresponding to each initial clustering result to cluster multiple target features and obtain multiple target clustering results. This clustering process can be understood as fine clustering to perform vocal separation on the audio to be processed. It can be understood that, given that each target feature can uniquely refer to its corresponding audio segment, that is, the target feature has a one-to-one correspondence with its corresponding audio segment, therefore, after clustering multiple target features and obtaining multiple target clustering results corresponding to the multiple target features, the clustering results of the corresponding multiple audio segments can be obtained, thereby achieving vocal separation on the audio to be processed.

[0058] It can be seen that in the disclosed embodiment, when performing human voice separation on audio, the audio to be processed can be first segmented to obtain multiple audio segments, and the target features corresponding to each of the multiple audio segments can be determined respectively; the multiple target features can be clustered to obtain multiple initial clustering results; and then, based on the directed graph corresponding to each initial clustering result, the multiple initial clustering results can be fused to obtain multiple target clustering results. By introducing the directed graph-based clustering method into the human voice separation scenario, a progressive clustering method can be adopted, first roughly clustering the multiple target features, and then finely clustering the multiple initial clustering results obtained by the rough clustering, so as to perform human voice separation on the audio to be processed, thereby improving the accuracy of the human voice separation result.

[0059] Based on the above Figure 1 In order to facilitate understanding of how to fuse multiple initial clustering results to obtain multiple target clustering results according to the directed graph corresponding to each initial clustering result in the embodiment shown in the above S103, the following will be described. Figure 3 The second embodiment shown is described in detail.

[0060] Example 2

[0061] Figure 3 This is a flow chart of a method for fusing multiple initial clustering results to obtain multiple target clustering results according to the second embodiment of the present disclosure. This method can also be executed by software and / or hardware devices. For example, the hardware device can be a terminal or a server. Figure 3 As shown, the method may include:

[0062] S301: Select a first initial clustering result from a plurality of initial clustering results, and determine a target initial clustering result corresponding to the first initial clustering result according to directed graphs corresponding to the initial clustering results.

[0063] Among them, the target initial clustering result can be understood as an initial clustering result that is fused with the first initial clustering result among other initial clustering results except the first initial clustering result in multiple initial clustering results.

[0064] Exemplarily, at least two target features are included in the initial clustering results. When determining the target initial clustering result corresponding to the first initial clustering result according to the directed graphs corresponding to the respective initial clustering results, the incremental similarity score after the fusion of the first initial clustering result and other initial clustering results except the first initial clustering result in multiple initial clustering results can be determined respectively according to the directed graphs corresponding to the respective initial clustering results. The edge weight of the directed graph is based on the similarity score between the two target features corresponding to the edge; and the other initial clustering result corresponding to the maximum incremental similarity score is determined as the target initial clustering result. Exemplarily, the calculation of the edge weight of the directed graph can refer to the following formula 1:

[0065]

[0066] Where, [W] ij represents the edge weight between target feature i and target feature j, s(i, j) represents the similarity score between target feature i and target feature j, y j represents target feature j, represents the initial clustering result to which target feature i belongs, and m represents the number of target features included in this initial clustering result.

[0067] Exemplarily, when determining the incremental similarity score after the fusion of the first initial clustering result and other initial clustering results respectively according to the directed graphs corresponding to the respective initial clustering results, the directed graph corresponding to the first initial clustering result and the directed graphs corresponding to other initial clustering results can be aggregated first to obtain an aggregated directed graph; and the new paths of the aggregated directed graph relative to the directed graph corresponding to the first initial clustering result and the directed graphs corresponding to other initial clustering results are determined respectively; then the incremental similarity score is determined according to the weights of the new paths.

[0068] When determining the incremental similarity score after the fusion of the first initial clustering result and other initial clustering results, exemplarily, it can be referred to Figure 4 as shown Figure 4It is a schematic diagram for fusing the initial clustering result 1 and the initial clustering result 2 provided by an embodiment of the present disclosure. Assume that the initial clustering result 1 is the first initial clustering result, and the first initial clustering result includes target feature 1, target feature 2, and target feature 3. The initial clustering result 2 is other initial clustering results, and the other initial clustering results include target feature 4, target feature 5, and target feature 6. After fusing the initial clustering result 1 and the initial clustering result 2, a corresponding aggregated directed graph will be obtained. Then, it is determined that there are multiple new paths in the aggregated directed graph relative to the initial clustering result 1, and there are multiple new paths relative to the initial clustering result 2. Among them, there are 4 new paths in the aggregated directed graph relative to the initial clustering result 1. For example, refer to Figure 5 as shown Figure 5 It is a schematic diagram of 4 new paths existing relative to the initial clustering result 1 provided by an embodiment of the present disclosure. Refer to Figure 5 the dashed line shown in Figure 6 as shown Figure 6 It is a schematic diagram of 4 new paths existing relative to the initial clustering result 2 provided by an embodiment of the present disclosure. Refer to Figure 6 the dashed line shown in

[0069] After determining the similarity score increment after fusing the first initial clustering result and the other initial clustering results, the initial clustering result with the largest similarity score increment can be determined from multiple similarity score increments, and the initial clustering result with the largest similarity score increment is determined as the target initial clustering result.

[0070] After determining the target initial clustering result corresponding to the first initial clustering result according to the directed graph corresponding to each initial clustering result in this way, the following S302 can be executed:

[0071] S302. Combine the first initial clustering result and the target initial clustering result to obtain a combined clustering result, and use the combined clustering result as the new initial clustering result.

[0072] When combining the first initial clustering result and the target initial clustering result to obtain a new combined clustering result and using this new combined clustering result as a new initial clustering result, the number of the new initial clustering results is one less than that before this combination operation.

[0073] S303. Repeat the above steps until the number of clusters after combination is the target number, and determine the clusters after combination as the target clustering result.

[0074] Among them, the target number can be preset. For example, when setting the target number, the value of the target number can be calculated according to the number of clusters obtained in the previous round of clustering, calculate the similarity matrix A, further calculate the eigenvalues of the similarity matrix A, then sort the eigenvalues from large to small, and accumulate them to a preset threshold to obtain the corresponding number of clusters, and this number of clusters is the target number.

[0075] It can be seen that in the embodiments of the present disclosure, when combining multiple initial clustering results, the first initial clustering result can be first selected from the multiple initial clustering results, and according to the directed graphs corresponding to the respective initial clustering results, the target initial clustering result corresponding to the first initial clustering result is determined; then the first initial clustering result and the target initial clustering result are combined to obtain a combined clustering result, and the combined clustering result is used as the new initial clustering result; repeat the above steps until the number of clusters after combination is the target number, and determine the clusters after combination as the target clustering result. In this way, the path integration clustering method based on the directed graph is used to perform fine clustering on multiple initial clustering results, which can improve the accuracy of the clustering results.

[0076] Based on any of the above embodiments, in order to facilitate understanding of how to determine the target features corresponding to multiple audio segments in S101 above, hereinafter, the following Figure 7 Example three shown will be described in detail.

[0077] Example three

[0078] Figure 7 is a flowchart of a method for determining the target features corresponding to multiple audio segments according to the third embodiment of the present disclosure. This method can also be executed by software and / or hardware devices. For example, the hardware device can be a terminal or a server. For example, as shown in Figure 7 shown, this method may include:

[0079] S701. Respectively determine the initial features corresponding to multiple audio segments.

[0080] Exemplarily, when respectively determining the initial features corresponding to multiple audio segments, existing feature extraction algorithms can be used to respectively determine the initial features corresponding to multiple audio segments; or a feature extraction model based on deep learning can be used to respectively extract the initial features corresponding to multiple audio segments, which can be specifically set according to actual needs.

[0081] Taking the example of using a feature extraction model based on deep learning to respectively extract the initial features corresponding to multiple audio segments, exemplarily, multiple audio segments are input into the feature extraction model to obtain the initial features corresponding to multiple audio segments respectively. The feature extraction model is obtained by removing the fully connected layer of the prediction model, and the prediction model is used to predict the probability that the audio belongs to each target user.

[0082] Exemplarily, the prediction model can be composed of a multi-layer time-delay neural network (TDNN), a full connection (FC) layer, and an activation layer. The output of the prediction model is the probability that the audio is each speaker, and the input is the feature of the audio, such as Mel-Frequency Cepstral Coefficients (MFCC) or others such as perceptual linear predictive (PLP), Fbank, etc. Generally, the dimension can be taken as 20. Exemplarily, the duration of the audio can be 25 ms, the frame shift is 10 ms, and a certain number of contexts are generally involved in the calculation, such as 2 frames each. The loss function for training can be the cross entropy (CE) loss function, and the prediction model is trained according to the cross entropy loss.

[0083] For example, when training a prediction model, multiple audio samples can be obtained first. The audio sample is only the audio of one speaker and can be open-source data, such as Aishell, Librispeech, SRE04 - 08, etc. Each audio sample is preprocessed, including noise removal. The noise can include environmental noise, busy tone, ringtone, silence, etc., to obtain a denoised audio sample. Then, the denoised audio sample is framed, and low-level features are extracted from each frame, such as 20-dimensional MFCC, PLP, or Fbank features. After first-order and second-order differences, and then regularization, that is, normal distribution regularization, so that each frame of audio sample extracts features of 20 * 3 dimensions. Each frame needs to combine a certain number of contexts and then be input into the initial prediction model for forward calculation. The output is the probability of the audio sample being each speaker. Then, the cross-entropy loss function is calculated, and the network parameters of the initial prediction model are updated backward according to the stochastic gradient descent criterion until the updated prediction model converges. The prediction model at the time of convergence is determined as the finally trained prediction model.

[0084] After training to obtain the prediction model, the fully connected layer of the prediction model can be removed, and the speaker voice spectrum features are extracted from the last TDNN layer as the initial features of the audio sample, and the dimension can be taken as 128. Usually, the initial features are obtained by accumulating and averaging multiple frames. For example, it can be calculated by accumulating and averaging every 50 frames, so that the initial features corresponding to each of the multiple audio segments can be obtained.

[0085] After respectively determining the initial features corresponding to each of the multiple audio segments, the initial features corresponding to each of the multiple audio segments can be subjected to feature enhancement processing to obtain the target features corresponding to each of the multiple audio segments, that is, execute the following S702:

[0086] S702. Perform feature enhancement processing on the initial features corresponding to each of the multiple audio segments to obtain the target features corresponding to each of the multiple audio segments.

[0087] For example, when performing feature enhancement processing on the initial features corresponding to each of the multiple audio segments, existing feature enhancement algorithms can be used to perform feature enhancement processing on the initial features corresponding to each of the multiple audio segments; or a feature enhancement model based on deep learning can be used to perform feature enhancement processing on the initial features corresponding to each of the multiple audio segments, which can be specifically set according to actual needs.

[0088] Exemplarily, in the embodiments of the present disclosure, the feature enhancement model may be composed of a 1-layer deep neural network (DNN), that is, including an input layer, a hidden layer, and an output layer. Among them, there is a transformation matrix M1 between the input layer and the hidden layer, and a transformation matrix M2 between the hidden layer and the output layer. The input of the transformation matrix M1 is the initial features corresponding to each of the multiple audio segments, and the output is the enhanced target features. The whole process is equivalent to performing two transformations on the initial features to obtain the enhanced target features.

[0089] It should be noted that the transformation matrix M1 can be initialized through a whitening (whitening transformation, WT) model, and the transformation matrix M2 can be initialized through a principal component analysis (PCA) model. Among them, the WT model and the PCA model can be trained based on the initial features obtained in the above S701.

[0090] Taking the example of using a feature enhancement model based on deep learning to perform feature enhancement processing on the initial features corresponding to each of the multiple audio segments, the multiple initial features can be input into the feature enhancement model, the multiple initial features are whitened through the first transformation matrix in the feature enhancement model, and the dimensionality reduction processing is performed on the whitening processing result through the second transformation matrix in the feature enhancement model, so as to obtain the target features corresponding to each of the multiple audio segments.

[0091] It can be seen that in the embodiments of the present disclosure, when obtaining the target features corresponding to each of the multiple audio segments, the initial features corresponding to each of the multiple audio segments can be determined first, and then the feature enhancement processing is performed on the initial features corresponding to each of the multiple audio segments to obtain the target features corresponding to each of the multiple audio segments. In this way, through the feature enhancement processing, the obtained target features can include more audio segment information, improving the accuracy of the target features.

[0092] In addition, in the embodiments of the present disclosure, if the target features are obtained based on the feature enhancement model, the network parameters of the feature enhancement model can be further optimized based on the target clustering results obtained in the above embodiments. This can not only improve the accuracy of the feature enhancement model, but also give full play to the role of each clustering result, assisting in the learning and updating of the network parameters in the feature enhancement model. In this way, the feature extraction and clustering are connected in series to form an end-to-end framework, avoiding the situation of mismatching between training and testing. Please refer to the following Embodiment 4.

[0093] Embodiment 4

[0094] Figure 8It is a schematic flowchart of a method for optimizing the network parameters of a feature enhancement model according to the fourth embodiment of the present disclosure. This method can also be executed by software and / or hardware devices. For example, the hardware device can be a terminal or a server. Exemplarily, please refer to Figure 7 As shown, the method may include:

[0095] S801. Respectively determine the self-supervised loss functions corresponding to each target feature in the target clustering result, and determine the self-supervised loss function corresponding to the target clustering result according to the self-supervised loss functions corresponding to each target feature.

[0096] Exemplarily, when determining the self-supervised loss functions corresponding to each target feature, for any target feature, a self-supervised loss function between intra-class samples and inter-class samples can be adopted. That is, respectively determine the first Euclidean distance between the target feature and other target features in the target clustering result except the target feature; and determine the second Euclidean distance between the target feature and each target feature in other target clustering results except the target clustering result. Among them, the first Euclidean distance is used to represent the self-supervised loss function between intra-class samples, and the second Euclidean distance is used to represent the self-supervised loss function between inter-class samples, so as to narrow the distance between samples of the same category and expand the distance between samples of different categories as a supervision signal. Then, according to the first Euclidean distance and the second Euclidean distance, determine the self-supervised loss function corresponding to the target feature, thereby obtaining the self-supervised loss function corresponding to the target feature.

[0097] Exemplarily, when determining the self-supervised loss function corresponding to the target clustering result according to the self-supervised loss functions corresponding to each target feature, it is possible to determine the sum of the losses of the self-supervised loss functions corresponding to each target feature included in the target clustering result, and determine the sum of the losses as the self-supervised loss function corresponding to the target clustering result, thereby determining the self-supervised loss function corresponding to the target clustering result.

[0098] S802. Update the network parameters of the feature enhancement model according to the self-supervised loss functions corresponding to each target clustering result.

[0099] Exemplarily, when updating the network parameters of the feature enhancement model according to the self-supervised loss functions corresponding to each target clustering result, it is possible to first determine the average loss function corresponding to multiple target clustering results according to the self-supervised loss functions corresponding to each target clustering result. Exemplarily, please refer to the following formula 2, and update the network parameters of the feature enhancement model according to the average loss function.

[0100]

[0101] Among them, L TThe average loss function corresponding to multiple target clustering results, N represents the number of target clustering results, j represents the j-th target clustering result, and y j，1 represents a target feature in the j-th target clustering result, and y j，2 represents another target feature in the j-th target clustering result, and y k≠j，2 represents a target feature in the k-th target clustering result.

[0102] It can be understood that after updating the network parameters of the feature enhancement model according to the average loss function, if the updated feature enhancement model converges, the updated feature enhancement model is directly determined as the finally trained feature enhancement model; if the updated feature enhancement model does not converge, the above steps are executed again until the updated feature enhancement model converges, and the feature enhancement model at the time of convergence is determined as the finally trained feature enhancement model, so as to obtain the final feature enhancement model. The feature enhancement model obtained in this way has good feature enhancement ability, thus effectively improving the robustness of the feature enhancement model.

[0103] Embodiment Five

[0104] Figure 9 is a schematic structural diagram of a voice separation device 90 according to the fifth embodiment of the present disclosure. For example, please refer to Figure 9 as shown. The voice separation device 90 may include:

[0105] A processing unit 901, configured to perform segmentation processing on the audio to be processed to obtain a plurality of audio segments, and respectively determine target features corresponding to the plurality of audio segments.

[0106] A clustering unit 902, configured to cluster the plurality of target features to obtain a plurality of initial clustering results.

[0107] A fusion unit 903, configured to fuse the plurality of initial clustering results according to the directed graphs corresponding to the respective initial clustering results to obtain a plurality of target clustering results for voice separation of the audio to be processed.

[0108] Optionally, the fusion unit 903 includes a first fusion module, a second fusion module, and a third fusion module.

[0109] The first fusion module is configured to select a first initial clustering result from the plurality of initial clustering results, and determine a target initial clustering result corresponding to the first initial clustering result according to the directed graphs corresponding to the respective initial clustering results.

[0110] The second fusion module is configured to fuse the first initial clustering result and the target initial clustering result to obtain a fused clustering result, and use the fused clustering result as a new initial clustering result.

[0111] The third fusion module is used to repeatedly execute the above steps until the number of the fused clusters reaches the target number, and determine the fused clusters as the target clustering result.

[0112] Optionally, the initial clustering result includes at least two target features; the first fusion module includes a first fusion sub-module and a second fusion sub-module.

[0113] The first fusion sub-module is used to respectively determine the similarity score increment after the first initial clustering result is fused with other initial clustering results except the first initial clustering result among the multiple initial clustering results according to the directed graphs corresponding to the respective initial clustering results; wherein, the edge weights of the directed graphs are based on the similarity scores between the two target features corresponding to the edges.

[0114] The second fusion sub-module is used to determine the other initial clustering result corresponding to the maximum similarity score increment as the target initial clustering result.

[0115] Optionally, the first fusion sub-module is further used to respectively aggregate the directed graph corresponding to the first initial clustering result and the directed graphs corresponding to other initial clustering results to obtain an aggregated directed graph; determine the new paths of the aggregated directed graph relative to the directed graph corresponding to the first initial clustering result and the directed graphs corresponding to other initial clustering results respectively; and determine the similarity score increment according to the weights of the new paths.

[0116] Optionally, the processing unit 901 includes a first processing module and a second processing module.

[0117] The first processing module is used to respectively determine the initial features corresponding to the multiple audio segments.

[0118] The second processing module is used to input the multiple initial features into a feature enhancement model, perform whitening processing on the multiple initial features through a first transformation matrix in the feature enhancement model, and perform dimensionality reduction processing on the whitening processing result through a second transformation matrix in the feature enhancement model to obtain the target features corresponding to the multiple audio segments respectively.

[0119] Optionally, the voice separation device 90 further includes a determination unit and an update unit.

[0120] The determination unit is used to respectively determine the self-supervised loss functions corresponding to the respective target features in the target clustering result, and determine the self-supervised loss function corresponding to the target clustering result according to the self-supervised loss functions corresponding to the respective target features.

[0121] The update unit is used to update the network parameters of the feature enhancement model according to the self-supervised loss functions corresponding to the respective target clustering results.

[0122] Optionally, the determination unit includes a first determination module, a second determination module, and a third determination module.

[0123] The first determination module is configured to determine, for any target feature, a first Euclidean distance between the target feature and other target features except the target feature in the target clustering result.

[0124] The second determination module is configured to determine a second Euclidean distance between the target feature and each target feature in other target clustering results except the target clustering result.

[0125] The third determination module is configured to determine a self-supervised loss function corresponding to the target feature according to the first Euclidean distance and the second Euclidean distance.

[0126] Optionally, the first processing module includes a first processing sub-module.

[0127] The first processing sub-module is configured to input a plurality of audio segments into a feature extraction model to obtain initial features corresponding to the plurality of audio segments respectively; wherein, the feature extraction model is obtained by removing the fully connected layer of a prediction model, and the prediction model is used to predict the probability that an audio belongs to each target user.

[0128] The voice separation device 90 provided by the embodiments of the present disclosure can execute the technical solutions of the voice separation method shown in any of the above embodiments. Its implementation principle and beneficial effects are similar to those of the voice separation method. For details, refer to the implementation principle and beneficial effects of the voice separation method, which will not be elaborated here.

[0129] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0130] According to an embodiment of the present disclosure, the present disclosure also provides a computer program product, which includes: a computer program. The computer program is stored in a readable storage medium. At least one processor of the electronic device can read the computer program from the readable storage medium, and the at least one processor executes the computer program to enable the electronic device to execute the solution provided in any of the above embodiments.

[0131] Figure 10FIG. 0 is a schematic block diagram of an electronic device 100 provided by an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0132] As Figure 10 shown, the device 100 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 100 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0133] A plurality of components in the device 100 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0134] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 executes the various methods and processes described above, such as the voice separation method. For example, in some embodiments, the voice separation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 100 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the voice separation method described above can be executed. Alternatively, in other embodiments, the computing unit 1001 can be configured to execute the voice separation method in any other suitable manner (e.g., by means of firmware).

[0135] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to the processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the program code is executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as an independent software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of this disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0139] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0140] A computer system may include a client and a server. The client and the server are generally far away from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system, and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS"). The server may also be a server of a distributed system or a server combined with a blockchain.

[0141] It should be understood that various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and no limitations are imposed herein.

[0142] The above specific embodiments do not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for voice separation, comprising: Performing segmentation processing on the audio to be processed to obtain a plurality of audio segments, and respectively determining target features corresponding to each of the plurality of audio segments; Clustering the plurality of target features to obtain a plurality of initial clustering results; Selecting a first initial clustering result from the plurality of initial clustering results, and determining a target initial clustering result corresponding to the first initial clustering result according to the directed graphs corresponding to the initial clustering results; Fusing the first initial clustering result and the target initial clustering result to obtain a fused clustering result, and using the fused clustering result as a new initial clustering result; Repeating the above steps until the number of the fused clustering results is the target number, and determining the fused clustering result as the target clustering result to perform voice separation on the audio to be processed.

2. The method according to claim 1, wherein Each of the initial clustering results includes at least two target features; The determining the target initial clustering result corresponding to the first initial clustering result according to the directed graphs corresponding to the initial clustering results includes: Respectively determining the similarity score increments after fusing the first initial clustering result with other initial clustering results except the first initial clustering result among the plurality of initial clustering results according to the directed graphs corresponding to the initial clustering results; wherein, the edge weights of the directed graphs are determined based on the similarity between two target features corresponding to the edge; Determining the other initial clustering result corresponding to the maximum similarity score increment as the target initial clustering result.

3. The method according to claim 2, wherein The respectively determining the similarity score increments after fusing the first initial clustering result with other initial clustering results except the first initial clustering result among the plurality of initial clustering results according to the directed graphs corresponding to the initial clustering results includes: Aggregating the directed graph corresponding to the first initial clustering result and the directed graphs corresponding to the other initial clustering results respectively to obtain an aggregated directed graph; Determining the new paths of the aggregated directed graph relative to the directed graph corresponding to the first initial clustering result and the directed graphs corresponding to the other initial clustering results respectively; Determining the similarity score increments according to the weights of the new paths.

4. The method according to claim 1, wherein, The respectively determining the target features corresponding to each of the plurality of audio segments includes: Respectively determining the initial features corresponding to each of the plurality of audio segments; Inputting the plurality of initial features into a feature enhancement model, performing whitening processing on the plurality of initial features through a first transformation matrix in the feature enhancement model, and performing dimensionality reduction processing on the whitening processing result through a second transformation matrix in the feature enhancement model to obtain the target features corresponding to each of the plurality of audio segments.

5. The method according to claim 4, wherein The method further includes: Respectively determining self-supervised loss functions corresponding to the target features in the target clustering result, and determining a self-supervised loss function corresponding to the target clustering result according to the self-supervised loss functions corresponding to the target features; Updating the network parameters of the feature enhancement model according to the self-supervised loss function corresponding to the target clustering result.

6. The method according to claim 5, wherein, The respectively determining the self-supervised loss functions corresponding to the target features in the target clustering result includes: For any target feature, determine the first Euclidean distance between the target feature and other target features in the target clustering result except the target feature; Determine the second Euclidean distance between the target feature and each target feature in other target clustering results except the target clustering result; Determine the self-supervised loss function corresponding to the target feature according to the first Euclidean distance and the second Euclidean distance.

7. The method according to any one of claims 4-6, wherein, The separately determining the initial features corresponding to the multiple audio segments includes: Input the multiple audio segments into a feature extraction model to obtain the initial features corresponding to the multiple audio segments respectively; wherein, the feature extraction model is obtained by removing the fully connected layer of a prediction model, and the prediction model is used to predict the probability that an audio belongs to each target user.

8. A voice separation device, comprising: A processing unit, configured to perform segmentation processing on the audio to be processed to obtain multiple audio segments, and separately determine the target features corresponding to the multiple audio segments; A clustering unit, configured to cluster the multiple target features to obtain multiple initial clustering results; A fusion unit, configured to fuse the multiple initial clustering results according to the directed graphs corresponding to the initial clustering results to obtain multiple target clustering results for voice separation of the audio to be processed; The fusion unit includes a first fusion module, a second fusion module, and a third fusion module; The first fusion module is configured to select a first initial clustering result from the multiple initial clustering results, and determine the target initial clustering result corresponding to the first initial clustering result according to the directed graphs corresponding to the initial clustering results; The second fusion module is configured to fuse the first initial clustering result and the target initial clustering result to obtain a fused clustering result, and use the fused clustering result as a new initial clustering result; The third fusion module is configured to repeat the above steps until the number of clusters after fusion is the target number, and determine the fused clusters as the target clustering results.

9. The device according to claim 8, wherein, At least two target features are included in the initial clustering result; the first fusion module includes a first fusion sub-module and a second fusion sub-module; The first fusion sub-module is configured to respectively determine the similarity score increments after fusing the first initial clustering result with other initial clustering results except the first initial clustering result in the multiple initial clustering results according to the directed graphs corresponding to the initial clustering results; wherein, the edge weights of the directed graphs are based on the similarity scores between the two target features corresponding to the edges; The second fusion sub-module is configured to determine the other initial clustering result corresponding to the maximum similarity score increment as the target initial clustering result.

10. The device according to claim 9, wherein, The first fusion sub-module is further configured to aggregate the directed graph corresponding to the first initial clustering result and the directed graphs corresponding to the other initial clustering results respectively to obtain an aggregated directed graph; determine the new paths of the aggregated directed graph with respect to the directed graph corresponding to the first initial clustering result and the directed graphs corresponding to the other initial clustering results respectively; and determine the similarity score increment according to the weights of the new paths.

11. The device according to claim 8, wherein, The processing unit includes a first processing module and a second processing module; The first processing module is configured to respectively determine the initial features corresponding to the multiple audio segments; The second processing module is configured to input the multiple initial features into a feature enhancement model, perform whitening processing on the multiple initial features through a first transformation matrix in the feature enhancement model, and perform dimensionality reduction processing on the whitening processing result through a second transformation matrix in the feature enhancement model to obtain the target features corresponding to the multiple audio segments respectively.

12. The apparatus according to claim 11, wherein, The apparatus further includes a determination unit and an update unit; The determination unit is configured to respectively determine the self-supervised loss functions corresponding to the target features in the target clustering result, and determine the self-supervised loss function corresponding to the target clustering result according to the self-supervised loss functions corresponding to the target features; The update unit is configured to update the network parameters of the feature enhancement model according to the self-supervised loss functions corresponding to the target clustering results.

13. The apparatus according to claim 12, wherein, The determination unit includes a first determination module, a second determination module, and a third determination module; The first determination module is configured to, for any target feature, determine the first Euclidean distance between the target feature and the other target features in the target clustering result except the target feature; The second determination module is configured to determine the second Euclidean distance between the target feature and the target features in other target clustering results except the target clustering result; The third determination module is configured to determine the self-supervised loss function corresponding to the target feature according to the first Euclidean distance and the second Euclidean distance.

14. The apparatus according to any one of claims 11 - 13, wherein, The first processing module includes a first processing sub-module; The first processing sub-module is configured to input the multiple audio segments into a feature extraction model to obtain the initial features corresponding to the multiple audio segments respectively; wherein, the feature extraction model is obtained by removing the fully connected layer of a prediction model, and the prediction model is used to predict the probability that an audio belongs to each target user.

15. An electronic device, comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the voice separation method according to any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to execute the voice separation method according to any one of claims 1-7.

17. A computer program product, comprising a computer program which, when executed by a processor, implements the steps of the voice separation method according to any one of claims 1-7.

Images