Speaker separation method, device, electronic device and storage medium

Abstract

The present disclosure provides a speaker separation method, device, electronic device and storage medium, which relate to the field of artificial intelligence, especially to the field of voice technology and intelligent customer service. The specific implementation scheme is: frame the target audio to obtain multiple continuous audio frames; predict the semantic information and speaker change point information of the current audio frame through a change point prediction model based on the audio features of the current audio frame and the semantic information of the previous audio frame; segment the target audio according to the semantic information and the speaker change point information to obtain multiple audio segments; determine the first similarity matrix corresponding to the audio segment; construct the second similarity matrix corresponding to the audio segment according to the speaker change point information and the first similarity matrix; cluster the second similarity matrix to obtain the result of classifying the audio segment according to the speaker. The present disclosure can improve the accuracy of speaker separation of audio.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of voice technology and intelligent customer service, and specifically to a speaker separation method, device, electronic device and storage medium. Background Art

[0002] In scenarios such as intelligent customer service, conference discussions, interviews, police interrogations, and variety shows, there are usually multiple speakers on a single channel. It is usually necessary to separate the speakers of the recorded audio to obtain the audio data of each person speaking, and then conduct targeted analysis. Summary of the Invention

[0003] The present disclosure provides a method, device, electronic device, and storage medium for speaker separation.

[0004] According to one aspect of the present disclosure, a method for separating speakers is provided, comprising:

[0005] Frame the target audio to obtain multiple continuous audio frames;

[0006] Based on the audio features of the current audio frame and the semantic information of the previous audio frame, the semantic information of the current audio frame and the speaker change point information are predicted by the change point prediction model;

[0007] Segmenting the target audio according to the semantic information and the speaker change point information to obtain multiple audio segments;

[0008] Determining a first similarity matrix corresponding to the audio segment;

[0009] constructing a second similarity matrix corresponding to the audio segment according to the speaker change point information and the first similarity matrix;

[0010] Clustering is performed on the second similarity matrix to obtain a result of classifying the audio segments according to the speakers.

[0011] If the current frame is the first audio frame among the multiple consecutive audio frames, predicting the semantic information and speaker change point information of the current audio frame using the change point prediction model includes:

[0012] According to the audio features and the start symbol of the current frame, semantic information and speaker change point information of the current audio frame are predicted by a change point prediction model.

[0013] The step of segmenting the target audio according to the semantic information and the speaker change point information includes:

[0014] If the semantic information of one of two consecutive audio frames is empty, the space between the two audio frames is the split point;

[0015] If the speaker change point information corresponding to an audio frame meets the set conditions, the audio frame and the previous audio frame are separated by a split point;

[0016] The target audio is segmented according to the segmentation points.

[0017] After the target audio is segmented according to the segmentation points to obtain a plurality of audio segments, the method further comprises: selecting all audio segments whose semantic information is not empty from the plurality of audio segments;

[0018] Correspondingly, determining the first similarity matrix corresponding to the audio segment includes: determining the first similarity matrix of all audio segments whose semantic information is not empty.

[0019] The determining of the first similarity matrix corresponding to all audio segments includes:

[0020] Predicting speaker features of each of the audio segments using a feature encoding model;

[0021] Based on the speaker features, similarities between two audio segments are calculated to obtain a first similarity matrix.

[0022] The step of constructing a second similarity matrix corresponding to the audio segment based on the speaker change point information and the first similarity matrix includes:

[0023] Constructing a first constraint matrix according to the speaker change point information, and normalizing the first similarity matrix to obtain a normalized similarity matrix;

[0024] constructing a second constraint matrix based on the normalized similarity matrix and the first constraint matrix;

[0025] A second similarity matrix is ​​constructed according to the second constraint matrix and the first similarity matrix.

[0026] The number of rows and columns of the first constraint matrix are both the number of audio segments whose semantic information is not empty, and the elements in the first constraint matrix are used to indicate whether two audio segments are connected and / or adjacent.

[0027] The step of constructing a first constraint matrix according to the speaker change point information includes:

[0028] If there is a semantically empty audio segment between the current audio segment and the next adjacent audio segment, determining whether the speaker change point information corresponding to the semantically empty audio segment meets the set condition; if the set condition is not met, determining that the current audio segment and the next audio segment are connected; if the set condition is met, determining that the current audio segment and the next audio segment are not connected;

[0029] If there is no semantically empty audio segment between the current audio segment and the next adjacent audio segment, it is determined that the current audio segment and the next audio segment are not connected;

[0030] The first constraint matrix is ​​constructed according to whether two adjacent audio segments are connected.

[0031] According to another aspect of the present disclosure, a model training method is provided, comprising:

[0032] Collect multiple sample audio data;

[0033] Acquire audio features of each piece of sample audio data, and mark speaker change points for each piece of sample audio data;

[0034] Convert each piece of the sample audio data to obtain corresponding sample text data, and perform semantic annotation on the sample text data;

[0035] A change point prediction model is trained according to the semantically annotated sample text data, the audio features, and the annotated speaker change points until convergence.

[0036] The step of training the change point prediction model based on the semantically annotated sample text data, the audio features, and the annotated speaker change points includes:

[0037] Predicting speaker change points on the sample audio data using a change point prediction model based on the semantically annotated sample text data and the audio features;

[0038] Calculate the loss value based on the marked speaker change points and the predicted speaker change points;

[0039] The parameters of the change point prediction model are adjusted according to the loss value.

[0040] According to another aspect of the present disclosure, there is provided a speaker separation device, comprising:

[0041] A framing module is used to frame the target audio to obtain multiple continuous audio frames;

[0042] A change point prediction module is used to predict the semantic information of the current audio frame and the speaker change point information through a change point prediction model based on the audio features of the current audio frame and the semantic information of the previous audio frame;

[0043] a segmentation module, configured to segment the target audio according to the semantic information and the speaker change point information to obtain a plurality of audio segments;

[0044] A similarity calculation module, configured to determine a first similarity matrix corresponding to the audio segment;

[0045] a constraint module, configured to construct a second similarity matrix corresponding to the audio segment based on the speaker change point information and the first similarity matrix;

[0046] A clustering module clusters the second similarity matrix to obtain a result of classifying the audio segments according to the speakers.

[0047] According to another aspect of the present disclosure, there is provided a model training device, comprising:

[0048] A data acquisition module, used to collect multiple sample audio data;

[0049] A feature extraction module, configured to obtain audio features of each piece of sample audio data;

[0050] A first annotation module is used to annotate speaker change points for each piece of sample audio data;

[0051] A second annotation module is used to convert each of the sample audio data to obtain corresponding sample text data, and to perform semantic annotation on the sample text data;

[0052] A training module is used to train a change point prediction model according to the semantically annotated sample text data, the audio features and the annotated speaker change points until convergence.

[0053] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0054] at least one processor; and

[0055] a memory communicatively connected to the at least one processor; wherein,

[0056] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned speaker separation method and / or model training method.

[0057] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above-mentioned speaker separation method and / or model training method.

[0058] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned speaker separation method and / or model training method when executed by a processor.

[0059] The above-mentioned speaker separation scheme combines voice information to predict speaker change points, and segments the audio according to the voice information and speaker change points to obtain audio segments; by constraining the similarity matrix of the audio segments through the speaker change points, the accuracy of classifying audio segments according to the speaker can be improved.

[0060] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0062] Figure 1 is a schematic diagram of a method for separating speakers according to an embodiment of the present disclosure;

[0063] Figure 2 is a schematic diagram of a method for determining a first similarity matrix according to an embodiment of the present disclosure;

[0064] Figure 3 is a schematic diagram of a second similarity matrix determination method according to an embodiment of the present disclosure;

[0065] Figure 4 is a schematic diagram of a model training method according to an embodiment of the present disclosure;

[0066] Figure 5 is a structural diagram of a speaker separation device according to an embodiment of the present disclosure;

[0067] Figure 6 is a structural diagram of a speaker separation device according to another embodiment of the present disclosure;

[0068] Figure 7 is a structural diagram of a training device for a model according to another embodiment of the present disclosure;

[0069] Figure 8 It is a block diagram of an electronic device used to implement the speaker separation method and / or model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0070] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0071] In scenarios such as intelligent customer service, conference discussions, interviews, police interrogations, and variety shows, the process of speaker separation in recorded audio generally includes: audio preprocessing to remove non-human sounds such as silence and noise, then cutting the audio into segments of equal length and extracting audio features such as MFCC (Mel-scale Frequency Cepstral Coefficients), PLP (Perceptual Linear Predictive) features, or Fbank (FilterBank) features. Speaker features are then extracted, such as x-vector features or d-vector features. Speaker features are then clustered, such as K-means, spectral clustering, or hierarchical clustering, to obtain the audio of each speaker.

[0072] In the above process, the equal-length segments obtained by uniform cutting are generally short and numerous, resulting in a large amount of calculation. In addition, each segment contains relatively little effective information, which is fragmented and incomplete. In addition, since the existing speaker separation scheme does not have a good speaker change point for detection, it is assumed that each segment obtained by uniform cutting has only one speaker, which is inconsistent with the actual situation. Furthermore, when clustering speaker features using the above-mentioned traditional clustering method, no additional constraints are added, and the accuracy is limited.

[0073] To this end, an example of the present disclosure provides a method for separating speakers, such as Figure 1 As shown, the method includes:

[0074] Step 101: Frame the target audio to obtain multiple continuous audio frames.

[0075] In this example, the target audio contains the speech of multiple speakers, and the identities of the speakers are not known in advance when performing speaker separation.

[0076] Unlike video data, audio data lacks the concept of frames. However, for transmission and storage purposes, collected audio data is segmented. To enable batch processing in computer programs, audio data can be segmented according to a specified length (time period or number of samples) and structured into a programmatic data structure, known as framing. In this disclosure, there are no restrictions on the duration or frame shift of each frame used to segment the target audio.

[0077] After framing, multiple continuous audio frames are obtained.

[0078] Step 102 : Predicting semantic information and speaker change point information of the current audio frame using a change point prediction model based on the audio features of the current audio frame and the semantic information of the previous audio frame.

[0079] In this example, a change point prediction model is provided that combines the semantic information of audio data to predict speaker change points. This model can utilize models such as Transformer Transducer, GMM-HMM, LSTM-CTC, or Chain to achieve speaker change point prediction. The input of the change point prediction model is the semantic information of the previous frame and the audio features of the current frame. The output is the semantic information of the current frame and the speaker change point information.

[0080] As mentioned above, the audio features may be MFCC features, PLP features, or Fbank features, etc.

[0081] Speaker change point information can include the probability of the speaker change point and the time information corresponding to the speaker change point. Since the duration of the target audio, the duration of each frame, and the frame shift are known, and the speaker change point corresponds to an audio frame, the time information corresponding to the speaker change point is the time information of the audio frame, including the start time, duration, and / or end time.

[0082] Semantic information can be information such as phonemes, words, phrases or sentences.

[0083] In the example, since the speaker change point is predicted based on semantic information, the accuracy of the prediction result of the speaker change point is improved.

[0084] Step 103 : segment the target audio according to the semantic information and the speaker change point information to obtain multiple audio segments.

[0085] The change point prediction model predicts the semantic information and speaker change point information of each audio frame, based on which the target audio can be segmented. The segmented audio segments vary in length, and each audio segment contains the speech of only one speaker.

[0086] Since speaker change points are predicted based on semantic information and audio segmentation is also based on semantic information, the audio frequency band segmentation results are more reasonable and more in line with actual conditions, which helps to improve the accuracy of audio segmentation in the future.

[0087] Step 104: Determine a first similarity matrix corresponding to the audio segment.

[0088] Step 105 : constructing a second similarity matrix corresponding to the audio segment according to the speaker change point information and the first similarity matrix.

[0089] Step 106: cluster the second similarity matrix to obtain a result of classifying the audio segments according to the speakers.

[0090] From steps 104 to 106 , it can be seen that the present disclosure again uses the speaker change point information to constrain the similarity matrix of the audio clips, optimizes the first similarity matrix into a second similarity matrix, and performs clustering based on the second similarity matrix, thereby obtaining a more accurate classification result.

[0091] In the above example of the present disclosure, the speaker change points are predicted based on semantic information, and the audio is segmented based on the predicted speaker change points in combination with the semantic information. As a result, the audio segments are reasonably segmented, and each audio segment only contains the voice of one speaker, which is more in line with the actual situation and is conducive to improving the accuracy of classifying audio segments according to the speaker, rather than the uniform segmentation method of the prior art that assumes that each audio segment only contains the voice of one speaker; and, compared with the uniform segmentation in the prior art that splits the voice of one speaker into different audio segments, each audio segment of the present disclosure is reasonably segmented in a supervised manner, and the speaker voice data it contains is more complete, which helps to extract more accurate speaker features, and reduces the number of audio segments, saving computational complexity. In addition, when clustering audio segments, the speaker change point information is used as a constraint to further improve the accuracy of classification.

[0092] In one example, when the above-mentioned step 102 predicts the semantic information and speaker change point information of the current audio frame, if the current frame is the first audio frame among multiple consecutive audio frames, the semantic information and speaker change point information of the current audio frame are predicted through the change point prediction model based on the audio features and start symbol of the current frame.

[0093] Because the current frame is the first audio frame, there is no historical semantic information. Therefore, when inputting the change point prediction model, the start symbol and the audio features of the current frame can be used. The start symbol here is the identifier of the beginning of the target audio.

[0094] In one example, the above step 103 segments the target audio according to the semantic information and the speaker change point information in the following manner:

[0095] If the semantic information of one of two consecutive audio frames is empty, the space between the two audio frames is the split point;

[0096] If the speaker change point information corresponding to an audio frame meets the set conditions, then the audio frame and the previous audio frame are separated by a split point. Here, the change point information meeting the set conditions can be the probability of the speaker change point meeting the threshold. Generally, the higher the probability, the higher the probability that the two speakers' voices are switched.

[0097] Split the target audio according to the split point.

[0098] For example, a target audio segment consists of 10 audio frames, numbered 1-10. A frame traversal reveals that frames 1-3 have predicted semantic information, while frame 4 has no predicted semantic information. Therefore, the first segmentation point is between frames 3 and 4. Frame 5 has no predicted semantic information, while frame 6 has predicted semantic information. Therefore, the second segmentation point is between frames 5 and 6. Frames 7-10 all have predicted semantic information, but the probability of the speaker change point corresponding to frame 9 meets the threshold. Therefore, the third segmentation point is between frames 8 and 9. This yields four audio segments: Segment 1 (frames 1-3), Segment 2 (frames 4-5), Segment 3 (frames 6-8), and Segment 4 (frames 9-10). Only Segment 2 lacks semantic information.

[0099] In summary, when an audio frame containing no semantic information appears, segmentation can be performed without considering the speaker change point information; when the speaker change point of an audio frame containing semantic information meets the set conditions, segmentation is performed.

[0100] Since speaker change points are predicted based on semantic information and audio segmentation is also based on semantic information, the audio frequency band segmentation results are more reasonable and more in line with actual conditions, which helps to improve the accuracy of audio segmentation in the future.

[0101] In one example, after segmenting the target audio according to the segmentation points to obtain multiple audio segments, all audio segments with non-null semantic information can be selected from the multiple segmented audio segments to determine a first similarity matrix for all audio segments with non-null semantic information. For example, segments 1, 3, and 4 are selected to calculate the first similarity matrix. This is because the first similarity matrix is ​​used to classify audio segments to identify all audio segments from the same speaker. Segments with null semantic information, i.e., segments that do not contain any speech, do not require cluster analysis. Therefore, when calculating the similarity matrix, these segments that do not contain speech need to be filtered out.

[0102] In one example, the above step 104 determines the first similarity matrix corresponding to all audio clips, such as Figure 2 Shown, including:

[0103] Step 201: predict speaker features of each audio segment using a feature encoding model;

[0104] Step 202: Calculate the similarity between each pair of audio segments based on the speaker features to obtain a first similarity matrix.

[0105] This disclosure provides another feature encoding model for predicting speaker features for audio clips. First, for each audio clip, the audio features are divided into frames and extracted. The audio features of all audio frames in the audio clip are then fed into the feature encoding model to predict the speaker features corresponding to the audio clip.

[0106] Here, the speaker features can be x-vector features or d-vector features, and each audio clip obtains a speaker feature (vector). For n audio clips, we can obtain n speaker features. By calculating the similarity between each pair, we can construct an nxn similarity matrix. Taking clips 1, 3, and 4 above as an example, the constructed 3x3 first similarity matrix is: Among them, A 11 Indicates the similarity between fragment 1 and fragment 1, A 13 Indicates the similarity between fragment 1 and fragment 3, A 14 Indicates the similarity between segment 1 and segment 4, and so on.

[0107] In the present invention, each audio clip is reasonably segmented in a supervised manner, and the speaker voice data contained therein is more complete, and the speaker features extracted based on this are more accurate.

[0108] The above-mentioned method of framing each audio segment and extracting audio features can be the same as the method of framing the target audio and extracting audio features in step 101. The framing results and audio feature extraction results of the target audio can also be directly reused. This depends on whether the speaker change point prediction model and feature encoding model use the same framing method and audio feature extraction method during training. Otherwise, the respective applicable framing methods and audio feature extraction methods are used.

[0109] In one example, the second similarity matrix corresponding to the audio segment is constructed based on the speaker change point information and the first similarity matrix, such as Figure 3 Shown, including:

[0110] Step 301: construct a first constraint matrix based on speaker change point information, and perform normalization processing on the first similarity matrix to obtain a normalized similarity matrix.

[0111] In one example, the first constraint matrix is ​​denoted as Q. The number of rows and columns in the first constraint matrix is ​​the number of audio segments whose semantic information is not empty. The elements in the first constraint matrix are used to indicate whether two audio segments are connected and / or adjacent. As described above, the semantic information of segments 1, 3, and 4 is not empty. Therefore, a 3x3 first constraint matrix is ​​constructed for segments 1, 3, and 4 as follows: Q 11 Indicates whether fragment 1 and fragment 1 are adjacent and / or connected, Q 13 Indicates whether fragment 1 and fragment 3 are adjacent and / or connected, Q 14 Indicates whether fragment 1 and fragment 4 are adjacent and / or connected, and so on.

[0112] In this example, the first similarity matrix is ​​recorded as A, and A is normalized to obtain a normalized similarity matrix, which is recorded as Matrix normalization can be done using the following formula: Where D is the degree matrix of A.

[0113] Step 302: construct a second constraint matrix based on the normalized similarity matrix and the first constraint matrix.

[0114] The second constraint matrix can be calculated using the following formula, denoted as Q * :

[0115] Wherein, I is the unit matrix, α is the coefficient factor, which can generally be 0.4. The present disclosure does not impose any restriction on the value of α.

[0116] Step 303: construct a second similarity matrix according to the second constraint matrix and the first similarity matrix.

[0117] The second similarity matrix can be calculated using the following formula:

[0118]

[0119] Where i is the i-th row in the matrix and j is the j-th column in the matrix.

[0120] In one example, constructing a first constraint matrix Q according to speaker change point information includes:

[0121] If there is a semantically empty audio segment between the current audio segment and the next adjacent audio segment, determining whether the speaker change point information corresponding to the semantically empty audio segment meets a set condition; if the set condition is not met, determining that the current audio segment and the next audio segment are connected; if the set condition is met, determining that the current audio segment and the next audio segment are not connected;

[0122] If there is no semantically empty audio segment between the current audio segment and the next adjacent audio segment, it is determined that the current audio segment and the next audio segment are not connected;

[0123] A first constraint matrix is ​​constructed according to whether two adjacent audio segments are connected.

[0124] It should be noted that the determination of whether the speaker change point information corresponding to a semantically empty audio segment meets the set conditions is the same as the set conditions used to determine the segmentation points described above. In one example, if the speaker change point probability meets a threshold, the set conditions can be determined to be met; if the speaker change point probability does not meet the threshold, the set conditions can be determined to be not met.

[0125] The first constraint matrix is For example, after excluding segment 2 from the four segments, segment 1 and segment 3 are adjacent, segment 3 and segment 4 are adjacent, and there is segment 2 with no semantic information between segment 1 and segment 3. Then the speaker change point probability of segment 2 is calculated. Segment 2 includes two audio frames (frame 4 and frame 5). In one example, the average of the speaker change point probabilities of frames 4 and 5 is taken as the speaker change point probability of segment 2. Assuming that the threshold is not met, it is considered that segment 1 and segment 3 need to be connected. Accordingly, Q 13 and Q 31 The value of is set to 1; Segments 3 and 4 are continuous, and there is no semantically empty audio segment. It is considered that Segments 3 and 4 do not need to be connected. This is because the segmentation of Segments 3 and 4 is based on the speaker change point probability of frame 9. Since the speaker change point probability of frame 9 meets the threshold, it is equivalent to that Segments 3 and 4 do not belong to the same speaker. The corresponding Q 34 and Q 43 The value of can be set to -1. If fragments 1 and 4 are not adjacent, then Q 14 and Q 41 The value of can be set to 0. In addition, Q 11 , Q 33 and Q 44 Representing fragment 1 itself, fragment 3 itself, and fragment 4 itself, they can be set to 1 respectively, so the first constraint matrix is:

[0126] It should be noted that the values ​​of the elements in the first constraint matrix Q have three cases: connected (1), not connected (-1) and not adjacent (0). The second constraint matrix Q obtained by optimizing the first constraint matrix Q is * In the example, the value of each element satisfies Q * ij ≥0 or Q * ij<0, where the case where it is greater than or equal to 0 is not limited to 1, and the case where it is less than 0 is not limited to -1.

[0127] In the above example, a constraint matrix is ​​constructed based on speaker change points, and the similarity matrix of the audio segments is optimized. This effectively increases the similarity between audio segments with high similarity and reduces the similarity between audio segments with low similarity, thereby obtaining a second similarity matrix. The classification results obtained based on this second similarity matrix are more accurate.

[0128] In one example, the process of clustering the second similarity matrix in step 106 may adopt spectral clustering, K-means, hierarchical clustering, etc., which will not be described in detail in this disclosure.

[0129] In order to achieve the above speaker change point prediction, Figure 4 As shown, the present disclosure provides a model training method for obtaining a change point prediction model, the method comprising:

[0130] Step 401: Collect multiple audio sample data.

[0131] These sample audio data may be the voices of different speakers, and each piece of sample audio data contains the voices of at least two speakers.

[0132] Step 402: Acquire audio features of each audio sample data and annotate speaker change points for each audio sample data.

[0133] Each sample audio data is divided into frames and audio features are extracted. Specifically, the above-mentioned framing method and audio feature extraction method can be adopted, which will not be repeated here.

[0134] Each audio sample is marked with a speaker change point, where a speaker change point indicates that the speaker has switched.

[0135] Step 403: convert each piece of sample audio data to obtain corresponding sample text data, and perform semantic annotation on the sample text data.

[0136] The present disclosure combines the semantic information of audio data to train the speaker change point model, which requires converting the sample audio data to obtain the corresponding sample text data, and semantically annotating the sample text data as the input of the speaker change point model.

[0137] The present disclosure does not limit the methods of semantic annotation and speaker change point annotation.

[0138] Step 404 : Training the change point prediction model based on the semantically annotated sample text data, audio features, and annotated speaker change points until convergence.

[0139] The training process for a sample audio data includes:

[0140] 1. Based on the semantically annotated sample text data and audio features, the speaker change point prediction model is used to predict the speaker change point of the sample audio data;

[0141] 2. Calculate the loss value based on the marked speaker change points and the predicted speaker change points. The loss value can be calculated using the loss function RNN-T (Recurrent Neural Network Transducer);

[0142] 3. Adjust the parameters of the change point prediction model based on the loss value.

[0143] The training process is repeated until the model converges.

[0144] During the model training process, the data annotation cost is reduced. The traditional end-to-end speaker change point detection model needs to annotate the start and end time of each segment of each person. However, in the present disclosure, semantic information is combined to identify change points, and only the semantic information of the text and the speaker change points need to be annotated.

[0145] In order to achieve the above-mentioned speaker feature extraction, the present disclosure also provides a training method for a feature coding model, including collecting the voices of different speakers as sample audio data, and each sample audio data contains only the voice of one speaker. The sample audio data is preprocessed, including removing noise (including environmental noise, busy tone, ringtone, silence, etc.) to obtain clean audio, and then the audio is framed, and low-level features (i.e., the above-mentioned MFCC and other audio features) are extracted from each frame and input into the feature coding model. The feature coding model includes a multi-layer LSTM (long-short term memory, long short-term memory model), a Linear mapping layer and an FC (full connection, full connection layer). The speaker features (x-vector or d-vector) can be extracted through the LSTM and Linear mapping layers, and then forward calculation is performed through the FC layer to predict the probability of the speaker; then, the loss value of the speaker prediction is calculated based on the cross entropy, and the parameters of the feature coding model are updated in reverse according to the SGD (stochastic gradient descent) criterion. This process is repeated for each sample audio data, so that the model is iterated until convergence.

[0146] In the above-mentioned speaker separation method, speaker features are extracted from audio clips through a converged feature encoding model, using only LSTM and Linear mapping layers.

[0147] In order to implement the above-mentioned speaker separation method, Figure 5 As shown, the present disclosure also provides a speaker separation device, comprising:

[0148] A framing module 10 is used to frame the target audio to obtain multiple continuous audio frames;

[0149] a change point prediction module 20 for predicting the semantic information of the current audio frame and the speaker change point information using a change point prediction model based on the audio features of the current audio frame and the semantic information of the previous audio frame;

[0150] A segmentation module 30 is used to segment the target audio according to semantic information and speaker change point information to obtain multiple audio segments;

[0151] A similarity calculation module 40 is configured to determine a first similarity matrix corresponding to the audio segment;

[0152] A constraint module 50, configured to construct a second similarity matrix corresponding to the audio segment based on the speaker change point information and the first similarity matrix;

[0153] The clustering module 60 clusters the second similarity matrix to obtain a result of classifying the audio segments according to the speakers.

[0154] According to the method of claim 1, if the current frame is the first audio frame among the multiple consecutive audio frames, the semantic information and speaker change point information of the current audio frame are predicted by a change point prediction model based on the audio features and start symbol of the current frame.

[0155] In one example, the change point prediction module 20 is further configured to, if the current frame is the first audio frame among the multiple consecutive audio frames, predict the semantic information and speaker change point information of the current audio frame based on the audio features and start symbol of the current frame.

[0156] In one example, the segmentation module 30 is also used to: if the semantic information of one audio frame in two consecutive audio frames is empty, then the segmentation point is between the two audio frames; if the speaker change point information corresponding to an audio frame meets the set conditions, then the segmentation point is between the audio frame and the previous audio frame; and the target audio is segmented according to the segmentation point.

[0157] In one example, the segmentation module 30 is further configured to segment the target audio according to the segmentation points to obtain multiple audio segments, and then select all audio segments whose semantic information is not empty;

[0158] Correspondingly, the similarity calculation module 40 is further configured to determine a first similarity matrix of all audio segments whose semantic information is not empty.

[0159] In one example, if Figure 6As shown, the apparatus further includes: a feature extraction module 70, configured to extract speaker features of each of the audio segments through a feature coding model;

[0160] The similarity calculation module 40 is further configured to calculate the similarity between any two audio segments based on the speaker features to obtain a first similarity matrix.

[0161] In one example, the constraint module 50 is also used to construct a first constraint matrix based on the speaker change point information, and standardize the first similarity matrix to obtain a standardized similarity matrix; construct a second constraint matrix based on the standardized similarity matrix and the first constraint matrix; and construct a second similarity matrix based on the second constraint matrix and the first similarity matrix.

[0162] The number of rows and columns of the first constraint matrix are both the number of audio segments whose semantic information is not empty, and the elements in the first constraint matrix are used to indicate whether two audio segments can be connected and whether they are adjacent.

[0163] In one example, when constructing the first constraint matrix, the constraint module 50 is further configured to determine whether the speaker change point information corresponding to the semantically empty audio segment satisfies a set condition if a semantically empty audio segment exists between the current audio segment and the next adjacent audio segment; if the set condition is not satisfied, determine that the current audio segment and the next audio segment are connected; if the set condition is satisfied, determine that the current audio segment and the next audio segment are not connected;

[0164] If there is no semantically empty audio segment between the current audio segment and the next adjacent audio segment, it is determined that the current audio segment and the next audio segment are not connected;

[0165] The first constraint matrix is ​​constructed according to whether two adjacent audio segments are connected.

[0166] In order to implement the above-mentioned model training method, an example of the present disclosure provides a model training device, such as Figure 7 Shown include:

[0167] The data collection module 701 is used to collect multiple audio sample data;

[0168] A feature extraction module 702 is configured to obtain audio features of each piece of sample audio data;

[0169] A first marking module 703 is configured to mark speaker change points for each piece of sample audio data;

[0170] The second annotation module 704 is used to convert each of the sample audio data to obtain corresponding sample text data, and perform semantic annotation on the sample text data;

[0171] The training module 705 is configured to train the change point prediction model according to the semantically annotated sample text data, the audio features, and the annotated speaker change points until convergence.

[0172] Among them, when the change point prediction model is trained based on the semantically annotated sample text data, the audio features and the annotated speaker change points, the training module 705 is also used to predict the speaker change points of the sample audio data through the change point prediction model based on the semantically annotated sample text data and the audio features; calculate the loss value based on the annotated speaker change points and the predicted speaker change points; and adjust the parameters of the change point prediction model according to the loss value.

[0173] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0174] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, comprising

[0175] at least one processor; and

[0176] a memory communicatively connected to the at least one processor; wherein,

[0177] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned speaker separation method and / or model training method.

[0178] According to an embodiment of the present disclosure, the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the above-mentioned speaker separation method and / or model training method.

[0179] According to an embodiment of the present disclosure, the present disclosure further provides a computer program product, including a computer program, wherein the computer program is executed by a processor to perform the above-mentioned speaker separation method and / or model training method.

[0180] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0181] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the device 800 can also be stored in the RAM 803. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0182] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0183] The computing unit 801 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the speaker separation method and / or the model training method. For example, in some embodiments, the speaker separation method and / or the model training method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the speaker separation method and / or the model training method described above can be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute the above-mentioned speaker separation method and / or model training method in any other appropriate manner (for example, by means of firmware).

[0184] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0185] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0186] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0187] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0188] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0189] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0190] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0191] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for separating speakers, comprising: Frame the target audio to obtain multiple continuous audio frames; Based on the audio features of the current audio frame and the semantic information of the previous audio frame, the semantic information of the current audio frame and the speaker change point information are predicted by the change point prediction model; Segmenting the target audio according to the semantic information and the speaker change point information to obtain multiple audio segments; Determining a first similarity matrix corresponding to the audio segment; constructing a second similarity matrix corresponding to the audio segment according to the speaker change point information and the first similarity matrix; Clustering is performed on the second similarity matrix to obtain a result of classifying the audio segments according to the speakers.

2. The method according to claim 1, wherein If the current frame is the first audio frame among the multiple consecutive audio frames, predicting the semantic information and speaker change point information of the current audio frame using the change point prediction model includes: According to the audio features and the start symbol of the current frame, semantic information and speaker change point information of the current audio frame are predicted by a change point prediction model.

3. The method according to claim 1, wherein Segmenting the target audio according to the semantic information and the speaker change point information includes: If the semantic information of one of two consecutive audio frames is empty, the space between the two audio frames is the split point; If the speaker change point information corresponding to an audio frame meets the set conditions, the audio frame and the previous audio frame are separated by a split point; The target audio is segmented according to the segmentation points.

4. The method according to claim 3, wherein: After segmenting the target audio according to the segmentation points to obtain multiple audio segments, the method further includes: Selecting all audio segments whose semantic information is not empty from the multiple audio segments; Accordingly, determining the first similarity matrix corresponding to the audio segment includes: A first similarity matrix of all audio segments whose semantic information is not empty is determined.

5. The method according to claim 1 or 4, wherein: The determining of the first similarity matrix corresponding to all audio segments includes: Predicting speaker features of each of the audio segments using a feature encoding model; Based on the speaker features, similarities between two audio segments are calculated to obtain a first similarity matrix.

6. The method according to claim 4, wherein: The step of constructing a second similarity matrix corresponding to the audio segment according to the speaker change point information and the first similarity matrix includes: Constructing a first constraint matrix according to the speaker change point information, and normalizing the first similarity matrix to obtain a normalized similarity matrix; constructing a second constraint matrix based on the normalized similarity matrix and the first constraint matrix; A second similarity matrix is ​​constructed according to the second constraint matrix and the first similarity matrix.

7. The method according to claim 6, wherein: The number of rows and columns of the first constraint matrix are both the number of audio segments whose semantic information is not empty. The elements in the first constraint matrix are used to indicate whether two audio segments are connected and / or adjacent.

8. The method according to claim 7, wherein: The step of constructing a first constraint matrix according to the speaker change point information includes: If there is a semantically empty audio segment between the current audio segment and the next adjacent audio segment, determining whether the speaker change point information corresponding to the semantically empty audio segment meets the set condition; if the set condition is not met, determining that the current audio segment and the next audio segment are connected; if the set condition is met, determining that the current audio segment and the next audio segment are not connected; If there is no semantically empty audio segment between the current audio segment and the next adjacent audio segment, it is determined that the current audio segment and the next audio segment are not connected; The first constraint matrix is ​​constructed according to whether two adjacent audio segments are connected.

9. The speaker separation method according to claim 1, wherein the training method of the change point prediction model comprises: Collect multiple sample audio data; Acquire audio features of each piece of sample audio data, and mark speaker change points for each piece of sample audio data; Convert each piece of the sample audio data to obtain corresponding sample text data, and perform semantic annotation on the sample text data; A change point prediction model is trained according to the semantically annotated sample text data, the audio features, and the annotated speaker change points until convergence.

10. The method according to claim 9, wherein: The training of the change point prediction model based on the semantically annotated sample text data, the audio features, and the annotated speaker change points includes: Predicting speaker change points on the sample audio data using a change point prediction model based on the semantically annotated sample text data and the audio features; Calculate the loss value based on the marked speaker change points and the predicted speaker change points; The parameters of the change point prediction model are adjusted according to the loss value.

11. A speaker separation device, comprising: A framing module is used to frame the target audio to obtain multiple continuous audio frames; A change point prediction module is used to predict the semantic information of the current audio frame and the speaker change point information through a change point prediction model based on the audio features of the current audio frame and the semantic information of the previous audio frame; a segmentation module, configured to segment the target audio according to the semantic information and the speaker change point information to obtain a plurality of audio segments; A similarity calculation module, configured to determine a first similarity matrix corresponding to the audio segment; a constraint module, configured to construct a second similarity matrix corresponding to the audio segment based on the speaker change point information and the first similarity matrix; A clustering module clusters the second similarity matrix to obtain a result of classifying the audio segments according to the speakers.

12. The apparatus according to claim 11, further comprising: A data acquisition module, used to collect multiple sample audio data; A feature extraction module, configured to obtain audio features of each piece of sample audio data; A first annotation module is used to annotate speaker change points for each piece of sample audio data; A second annotation module is used to convert each of the sample audio data to obtain corresponding sample text data, and to perform semantic annotation on the sample text data; A training module is used to train a change point prediction model according to the semantically annotated sample text data, the audio features and the annotated speaker change points until convergence.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 10.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-10.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images