Speaker segmentation and clustering method, device, equipment and storage medium

Abstract

The present invention relates to artificial intelligence technology, and provides a speaker segmentation clustering method, device, equipment and storage medium, the method comprising: obtaining an audio stream; inputting the audio stream into at least two speech segmentation models to obtain speech sets respectively; classifying the speech sets to obtain multiple groups of similar speech sets; performing similarity score processing on the similar speech sets to obtain a similarity score matrix; clustering the speech sets according to the similarity score matrix to obtain speech clusters; establishing a speaker classification model according to the speech clusters; aligning the speech frames in the audio stream with the speakers according to the speaker classification model. According to the scheme provided by the embodiment of the present invention, segmentation is performed using different fixed segmentation durations to obtain speech sets respectively, and then through classification processing and similarity score processing, it is achieved that both the credibility of speaker features and the accuracy of speaker label prediction are guaranteed, so that the clustering result is better.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, but is not limited to it, and in particular to a speaker segmentation and clustering method, device, equipment and storage medium. Background Art

[0002] Speaker Diarization is a technology that distinguishes the voices of different speakers according to their identities, solving the problem of "who spoke and when".

[0003] At present, in the segmentation process, the audio stream is segmented according to a fixed single segmentation time, or the audio stream is segmented according to a variable segmentation time to obtain multiple speech segments. In the clustering process, the speaker label is predicted based on the similarity results. The length of the speech segment largely determines the quality of the clustering result. If the length of the speech segment is too short, the speech segment contains less speaker feature information, and the credibility of the extracted speaker feature is low. If the length of the speech segment is too long, a speech segment is likely to contain the voices of multiple speakers, reducing the accuracy of speaker label prediction. Therefore, it is impossible to ensure both the credibility of speaker features and the accuracy of speaker label prediction, resulting in poor clustering results. Summary of the invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The embodiments of the present invention provide a speaker segmentation and clustering method, device, and storage medium, which not only ensure the credibility of speaker features, but also ensure the accuracy of speaker label prediction, and can achieve better clustering results.

[0006] In a first aspect, an embodiment of the present invention provides a speaker segmentation and clustering method, comprising: obtaining an audio stream, wherein speech frames in the audio stream contain speech from at least one speaker; inputting the audio stream into at least two speech segmentation models to obtain speech sets respectively, wherein the speech segmentation models are configured to perform speech segmentation according to a fixed segmentation duration, and the segmentation durations of any two speech segmentation models are different; classifying the speech sets to obtain multiple groups of similar speech sets; performing similarity score processing on the similar speech sets to obtain a similarity score matrix; clustering the speech sets according to the similarity score matrix to obtain speech clusters; establishing a speaker classification model according to the speech clusters; and aligning the speech frames in the audio stream with the speakers according to the speaker classification model.

[0007] In some embodiments, after the step of inputting the audio stream into at least two speech segmentation models to obtain speech sets respectively, it also includes: according to the segmentation time length of each speech segmentation model, respectively time-shifting the segmentation starting point of each speech segmentation model; inputting the audio stream into each of the time-shifted speech segmentation models to respectively obtain auxiliary speech sets, wherein the auxiliary speech sets correspond one-to-one to the speech sets; merging the auxiliary speech sets with the corresponding speech sets to update the speech sets.

[0008] In some embodiments, the time shift length of the segmentation starting point of the speech segmentation model is half of the corresponding segmentation time length of the speech segmentation model.

[0009] In some embodiments, each of the speech sets contains at least two speech segments; the speech sets are classified to obtain multiple groups of similar speech sets, including: taking the speech set obtained by the speech segmentation model with the shortest segmentation time in the speech segmentation model as the benchmark speech set; calculating the benchmark length center distance between each speech segment of the benchmark speech set and each speech segment of all the speech sets in turn; for any speech segment of the benchmark speech set, taking the speech segment with the smallest benchmark length center distance in each speech set to form a similar speech set.

[0010] In some embodiments, the similar speech sets are processed for similarity scores to obtain a similarity score matrix, including: performing feature extraction on any two of the similar speech sets to obtain a first feature vector set and a second feature vector set, respectively; inputting the first feature vector set and the second feature vector set into a score calculation model to obtain a similarity score; performing similarity calculation on the first feature vector set and the second feature vector set to obtain a similarity matrix; and obtaining a similarity score matrix based on the similarity scores and the similarity matrix.

[0011] In some embodiments, the training process of the score calculation model is as follows: obtaining a training audio stream and a speaker label, wherein the speech frames in the training audio stream contain speech from at least one speaker, and the speaker label is used to characterize the speaker to which the speech frames in the training audio stream are aligned; inputting the training audio stream into at least two of the speech segmentation models to obtain speech training sets respectively; classifying the speech training sets to obtain multiple groups of similar speech training sets; performing feature extraction on any two of the similar speech training sets to obtain a first feature vector training set and a second feature vector training set respectively; inputting the first feature vector training set and the second feature vector training set into the score calculation model to obtain a similarity training score; obtaining a predicted similarity based on the similarity training score, the first feature vector training set and the second feature vector training set; obtaining a true similarity through similarity calculation based on the speaker label, the speaker label, the training audio stream, the first feature vector training set and the second feature vector training set; determining a loss function based on the true similarity and the predicted similarity; and updating the score calculation model based on the loss function.

[0012] In some embodiments, the first input of the score calculation model is the first feature vector set, the second input of the score calculation model is the second feature vector set, the output of the score calculation model is the similarity score, the score calculation model includes a difference calculation module and a score calculation module, the difference calculation module includes a first fusion layer, a second fusion layer, at least two parallel first feedforward neural networks and at least two parallel second feedforward neural networks, the first feature vector set is used as the input of the first feedforward neural network, the second input of the score calculation model is used as the input of the second feedforward neural network, the output of the first feedforward neural network is used as the input of the first fusion layer, the output of the second feedforward neural network is used as the input of the second fusion layer, the output of the first fusion layer and the output of the second fusion layer are subtracted as the output of the difference calculation module, the output of the difference calculation module is used as the input of the score calculation module, the output of the score calculation module is used as the output of the score calculation model, the first fusion layer is used to splice the output of the first feedforward neural network, and the second fusion layer is used to splice the output of the second feedforward neural network.

[0013] In a second aspect, the present invention also provides a speaker segmentation and clustering device, comprising: an acquisition unit, used to acquire an audio stream, wherein the speech frames in the audio stream contain speech from at least one speaker; a segmentation unit, used to input the audio stream into at least two speech segmentation models to obtain speech sets respectively, wherein the speech segmentation models are configured to perform speech segmentation according to a fixed segmentation time, and the segmentation time lengths of any two speech segmentation models are different; a classification unit, used to classify the speech sets to obtain multiple groups of similar speech sets; a score calculation unit, used to perform similarity score processing on the similar speech sets to obtain a similarity score matrix; a clustering unit, used to cluster the speech sets according to the similarity score matrix to obtain speech clusters; a modeling unit, used to establish a speaker classification model according to the speech clusters; an alignment unit, used to align the speech frames in the audio stream with the speakers according to the speaker classification model.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the speaker segmentation and clustering method as described in the first aspect is implemented.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the speaker segmentation and clustering method as described in the first aspect.

[0016] The embodiment of the present invention includes: obtaining an audio stream, wherein the speech frames in the audio stream contain speech from at least one speaker; inputting the audio stream into at least two speech segmentation models to obtain speech sets respectively, wherein the speech segmentation models are configured to perform speech segmentation according to a fixed segmentation time, and the segmentation time of any two speech segmentation models is different; classifying the speech sets to obtain multiple groups of similar speech sets; performing similarity score processing on the similar speech sets to obtain a similarity score matrix; clustering the speech sets according to the similarity score matrix to obtain speech clusters; establishing a speaker classification model according to the speech clusters; aligning the speech frames in the audio stream with the speaker according to the speaker classification model. According to the scheme provided by the embodiment of the present invention, different fixed segmentation time lengths are used for segmentation to obtain speech sets respectively, and then similar speech sets are obtained by classification processing, and then similar speech sets are processed by similarity score processing to obtain a similarity score matrix, which ensures both the credibility of speaker features and the accuracy of speaker label prediction, so that the clustering result is better.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0019] Figure 1 is a flow chart of a speaker segmentation and clustering method provided by an embodiment of the present invention;

[0020] Figure 2 yes Figure 1 Specific method flow chart after step S120;

[0021] Figure 3 yes Figure 2 The time distribution diagram of the updated speech set in step S230;

[0022] Figure 4 yes Figure 1 Specific method flow chart of step S130;

[0023] Figure 5 yes Figure 1 Specific method flow chart of step S140;

[0024] Figure 6 yes Figure 5 A structural diagram of the score calculation model in step S520;

[0025] Figure 7 yes Figure 5 Flow chart of the training process of the score calculation model in step S520;

[0026] Figure 8 yes Figure 7 The time distribution diagram of the speech of all speakers in the training audio stream of step S710;

[0027] Fig. 9 is a structural diagram of a speaker segmentation and clustering device provided by another embodiment of the present invention;

[0028] Fig.10 It is a structural diagram of an electronic device provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0030] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0031] The present invention provides a speaker segmentation and clustering method, device, equipment and storage medium, the method comprising: obtaining an audio stream, wherein the speech frames in the audio stream contain speech from at least one speaker; inputting the audio stream into at least two speech segmentation models to obtain speech sets respectively, wherein the speech segmentation models are configured to perform speech segmentation according to a fixed segmentation time, and the segmentation time of any two speech segmentation models is different; classifying the speech sets to obtain multiple groups of similar speech sets; performing similarity score processing on the similar speech sets to obtain a similarity score matrix; clustering the speech sets according to the similarity score matrix to obtain speech clusters; establishing a speaker classification model according to the speech clusters; aligning the speech frames in the audio stream with the speakers according to the speaker classification model. According to the scheme provided by the embodiment of the present invention, segmentation is performed using different fixed segmentation time lengths to obtain speech sets respectively, and then similar speech sets are obtained by classification processing, and then similar speech sets are processed by similarity score processing to obtain a similarity score matrix, which realizes both the credibility of speaker features and the accuracy of speaker label prediction, so that the clustering result is better.

[0032] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.

[0033] Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. With the research and progress of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, robots, smart medical care, etc. It is believed that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0034] Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0035] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.

[0036] It should be noted that the speaker segmentation and clustering method in the embodiment of the present invention is suitable for voice scenarios of multi-person conversations, and can be applied to many voice-related product applications, such as media interview systems, intelligent customer service systems, multi-person conference systems, smart court systems, intelligent education systems, etc. The speaker segmentation and clustering method can distinguish the voices of different speakers according to the identity of the speaker, solving the problem of "who spoke and when".

[0037] First, several nouns involved in the present invention are analyzed:

[0038] Clustering is to divide a data set into different classes or clusters according to a specific standard, so that the similarity of data objects in the same cluster is as large as possible, and the difference of data objects in different clusters is as large as possible. That is, after clustering, data of the same class is gathered together as much as possible, and data of different classes are separated as much as possible.

[0039] Latent Dirichlet allocation (LDA) is a topic model that can give the topic of each document in a document set in the form of probability distribution. At the same time, it is an unsupervised learning algorithm that does not require manually annotated training sets during training. All it needs is a document set and the number of specified topics k.

[0040] The probabilistic LDA algorithm (Probabilistic Linear Discriminant Analysis, PLDA) is a channel compensation algorithm used to score speaker features pairwise. The higher the score, the higher the similarity, and the more likely it is that the voice is from the same person.

[0041] Agglomerative Hierarchical Clustering (AHC) calculates the similarity between two types of data points, combines the two most similar data points among all data points, and iterates this process repeatedly.

[0042] Mel-Frequency Cepstral Coefficients (MFCC) are a set of key coefficients used to create Mel-Frequency Cepstral, and are one of the methods required to extract speech feature parameters.

[0043] Multilayer Perceptron (MLP) is a feedforward artificial neural network model that maps multiple input data sets to a single output data set.

[0044] The mean-square error (MSE) is a measure of the difference between the estimator and the estimated quantity. Let t be an estimator of the population parameter θ determined based on the sample, (θ-t) 2 The mathematical expectation of is called the mean square error of the estimator t, which is equal to σ 2 +b 2 , where σ 2 and b are the variance and bias of the estimator t, respectively.

[0045] The normalized exponential function, or Softmax function, is a generalization of the logic function. It can "compress" a K-dimensional vector z containing any real number into another K-dimensional real vector σ(z), so that each element ranges between (0,1) and the sum of all elements is 1. This function is more common in multi-classification problems.

[0046] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0047] like Figure 1 As shown, Figure 1 1 is a flow chart of a speaker segmentation and clustering method provided by an embodiment of the present invention. The method includes but is not limited to the following steps:

[0048] Step S110, obtaining an audio stream, wherein the speech frames in the audio stream contain speech from at least one speaker;

[0049] It should be noted that the microphone can convert sound signals into electrical signals. After obtaining the speech to be processed through the microphone, the speech to be processed is obtained, and then the speech to be processed is input into the speech detection model, and all outputs of the speech detection model are spliced ​​in sequence to obtain an audio stream; wherein the speech detection model can perform speech detection on the speech to be processed, and divide the speech to be processed frame by frame into two categories: speech with someone speaking and speech without someone speaking. The speech without someone speaking includes but is not limited to pure silence, ambient noise, music and sound effects; the audio stream belongs to speech with someone speaking, and therefore, the speech frames in the audio stream contain speech from at least one speaker.

[0050] Step S120, inputting the audio stream into at least two speech segmentation models to obtain speech sets respectively, wherein the speech segmentation models are configured to perform speech segmentation according to a fixed segmentation time, and the segmentation time lengths of any two speech segmentation models are different;

[0051] It is understandable that all speech segmentation models perform speech segmentation according to a fixed segmentation time, and different speech segmentation models have different segmentation times. The speech set is sorted according to the segmentation time of the corresponding speech segmentation model to prepare for the subsequent calculation of the score of each element of the speech set.

[0052] It should be noted that the segmentation time length of the speech segmentation model can be selected in the range of 0.5s to 2s.

[0053] Step S130, classifying the speech set to obtain multiple groups of similar speech sets;

[0054] It can be understood that classification processing refers to classifying the elements of a speech set, first taking one of the speech sets, and then taking the elements in the speech sets as benchmark elements in turn, judging the degree of closeness between the elements of the other speech set and the benchmark elements, and classifying the elements in the other speech set with the highest degree of closeness to the benchmark elements into the same category, thereby obtaining multiple groups of similar speech sets. Therefore, the number of elements of a similar speech set is the number of speech sets, and the elements of a similar speech set are taken from different speech sets.

[0055] It is worth noting that in the classification process, one of the speech sets is first taken, and then the elements in the speech set are taken as benchmark elements in turn to judge the degree of proximity between the elements of the other speech set and the benchmark elements. When there are two elements with the highest degree of proximity, since the two elements appear at different times, the element with the earlier time is taken as the closest element, which can ensure the uniqueness of the similar speech sets.

[0056] Step S140, performing similarity score processing on the similar speech set to obtain a similarity score matrix;

[0057] It should be noted that similarity score processing refers to score calculation based on speech segmentation effect, which can improve the effect of speech segmentation, ensure the credibility of speaker characteristics, and ensure the accuracy of speaker label prediction; different speech sets are output by speech segmentation models with different segmentation times, and the speech segmentation effects of different speech sets will be different, and the speech segmentation effects of different elements in the same speech set will also be different; each similar speech set contains an element of each speech set, and similarity score processing for similar speech sets is equivalent to calculating the scores of corresponding elements in the speech set; in the similarity score processing, the better the speech segmentation effect of the similar speech set, the higher the score; the worse the speech segmentation effect of the similar speech set, the lower the score.

[0058] It can be understood that the similarity score matrix is ​​obtained by the scores of corresponding elements in the speech set, and can be used in the subsequent clustering process. Compared with directly clustering without calculating the scores of corresponding elements in the speech set, clustering using the similarity score matrix has better clustering results.

[0059] Step S150, clustering the speech set according to the similarity score matrix to obtain speech clusters;

[0060] It can be understood that the similarity score matrix is ​​derived from the speaker features corresponding to each speech segment in the audio stream, and clustering is performed using the similarity score matrix, and the clustering result is good.

[0061] In specific practice, in clustering processing, the similarity score matrix and PLDA are combined to score the speech segments segmented by the audio stream in pairs, and then AHC is used to merge the categories with high similarity to obtain speech clusters. The speech of the same speaker is in one speech cluster, and the speech of different speakers is in different speech clusters.

[0062] Step S160, establishing a speaker classification model based on the speech clusters;

[0063] It is understandable that different speech clusters can represent different speakers. A speaker classification model is established using speech clusters, and the speaker classification model can distinguish speech segments of different speakers.

[0064] Step S170: aligning speech frames in the audio stream with the speaker according to the speaker classification model.

[0065] It can be understood that by processing the audio stream using the speaker classification model, the speech frames in the audio stream are aligned with the speakers, thereby distinguishing the speech segments of different speakers in the audio stream.

[0066] In addition, refer to Figure 2 , in one embodiment, Figure 1 After step S120 in the illustrated embodiment, the following steps are also included but not limited to:

[0067] Step S210, according to the segmentation duration of each speech segmentation model, respectively time-shift the segmentation starting point of each speech segmentation model accordingly;

[0068] It is understandable that after the audio stream is segmented using a speech segmentation model, there may be a connection between two adjacent speech segments, and the speech near the segmentation point of the audio stream may have better speaker feature information. If it is directly segmented into two speech segments, the speaker feature information will be destroyed, affecting the subsequent clustering results; therefore, according to the segmentation time of each speech segmentation model, the segmentation starting point of each speech segmentation model is time-shifted respectively, which can retain the speech near the segmentation point and achieve better clustering results.

[0069] Step S220, inputting the audio stream into each of the time-shifted speech segmentation models to obtain auxiliary speech sets respectively, wherein the auxiliary speech sets correspond one to one to the speech sets;

[0070] It can be understood that the segmentation starting point of each speech segmentation model is time-shifted, and then each time-shifted speech segmentation model is used to perform a new round of speech segmentation to obtain auxiliary speech sets, respectively, so that the elements of the auxiliary speech set and the corresponding speech set have overlapping parts, and the speech near the segmentation point can be retained, and the clustering result is better.

[0071] Step S230: merge the auxiliary voice set with the corresponding voice set to update the voice set.

[0072] It is understandable that both the auxiliary speech set and the speech set are used for subsequent classification processing, and after the combined processing, the operation efficiency can be improved.

[0073] In addition, refer to Figure 3 , in one embodiment, Figure 3 yes Figure 2 In the time distribution diagram of the updated speech set in step S230, the time shift length of the segmentation starting point of the speech segmentation model is half of the segmentation time length of the corresponding speech segmentation model.

[0074] It can be understood that the first round of speech segmentation is performed on the audio stream using three speech segmentation models with fixed segmentation time, and the first old speech set OC1, the second old speech set OC2 and the third old speech set OC3 are obtained respectively, wherein the segmentation time of the first speech segmentation model is 0.5s, the segmentation time of the second speech segmentation model is 1s, and the segmentation time of the third speech segmentation model is 1.5s; then, according to the segmentation time, the segmentation starting point of the corresponding speech segmentation model is time-shifted, the segmentation starting point of the first speech segmentation model is time-shifted by 0.25s, the segmentation starting point of the second speech segmentation model is time-shifted by 0.5s, and the segmentation starting point of the third speech segmentation model is time-shifted by 1s. The segmentation starting point is time-shifted by 1s; then, the time-shifted speech segmentation model is used to perform a second round of speech segmentation on the audio stream, and the first auxiliary speech set SC1, the second auxiliary speech set SC2 and the third auxiliary speech set SC3 are obtained respectively; finally, the auxiliary speech set is merged with the corresponding speech set to update the speech set. The first speech set C1 is obtained by merging the first old voice set OC1 and the first auxiliary voice set SC1, the second voice set C2 is obtained by merging the second old voice set OC2 and the second auxiliary voice set SC2, and the third voice set C3 is obtained by merging the third old voice set OC3 and the third auxiliary voice set SC3.

[0075] In addition, in one embodiment, each speech set includes at least two speech segments;

[0076] It can be understood that when the audio stream is segmented using the speech segmentation model, the audio stream has at least one segmentation point, and therefore the obtained speech set contains at least two speech segments.

[0077] In addition, refer to Figure 4 , in one embodiment, Figure 1 Step S130 in the illustrated embodiment further includes but is not limited to the following steps:

[0078] Step S410, taking a speech set obtained by a speech segmentation model with the shortest segmentation time among the speech segmentation models as a reference speech set;

[0079] Step S420, sequentially calculating the reference length center distance between each speech segment of the reference speech set and each speech segment of all speech sets;

[0080] It can be understood that the length center distance is the distance between the times at which the centers of two speech segments are located, and the reference length center distance refers to the distance between the times at which the centers of the speech segments of the reference speech set and the speech segments of the speech set are located.

[0081] Step S430: for any speech segment of the reference speech set, take the speech segment with the smallest reference length center distance in each speech set to form a similar speech set.

[0082] It can be understood that a similar speech set is composed of k speech segments, k is the number of speech sets, and the speech segments and speech sets correspond one to one.

[0083] In addition, refer to Figure 5 , in one embodiment, Figure 1 Step S140 in the illustrated embodiment further includes but is not limited to the following steps:

[0084] Step S510, performing feature extraction on any two similar speech sets to obtain a first feature vector set and a second feature vector set respectively;

[0085] It should be noted that the steps of feature extraction are as follows: acoustic feature extraction is performed on a speech segment of a similar speech set, and then feature vector extraction is performed to obtain a first feature vector set; then acoustic feature extraction is performed on a speech segment of another similar speech set, and then feature vector extraction is performed to obtain a second feature vector set.

[0086] It can be understood that the method of acoustic feature extraction includes but is not limited to using a method based on MFCC feature extraction to extract the features of a speech segment. Since MFCC is based on the extraction method of the cepstrum, the MFCC features are more in line with the principles of human hearing and the effect of speech feature extraction is better. The MFCC feature extraction technology is well known to those skilled in the art and will not be elaborated here.

[0087] It is worth noting that the process of feature vector extraction is dimensionality reduction processing. The vectors in the first feature vector set and the second feature vector set are all fixed-length embedding vectors, which can effectively represent the individual characteristics of speech. The vectors in the first feature vector set and the second feature vector set are all feature vector x-vectors.

[0088] Step S520, inputting the first feature vector set and the second feature vector set into a score calculation model to obtain a similarity score;

[0089] It can be understood that the score calculation model is a trained machine learning model that can effectively calculate the similarity score between the first feature vector set and the second feature vector set.

[0090] Step S530, performing similarity calculation on the first eigenvector set and the second eigenvector set to obtain a similarity matrix;

[0091] It is understandable that the similarity matrix can be obtained by calculating the cosine distance between the first eigenvector set and the second eigenvector set. Similarity calculation is a technique well known to those skilled in the art and will not be elaborated herein.

[0092] Step S540: obtaining a similarity score matrix according to the similarity score and the similarity matrix.

[0093] It can be understood that similarity score processing refers to score calculation based on the speech segmentation effect. The similarity score is used to score the similarity matrix to obtain the similarity score matrix, which can improve the effect of speech segmentation, ensure the credibility of the speaker's characteristics, and ensure the accuracy of the speaker label prediction. The similarity score matrix is ​​subsequently used for clustering, and the clustering result is better.

[0094] In addition, refer to Figure 6 , in one embodiment, Figure 6 yes Figure 5 The structural diagram of the score calculation model of step S520 in the illustrated embodiment, the first input of the score calculation model is the first feature vector set, the second input of the score calculation model is the second feature vector set, the output of the score calculation model is the similarity score, the score calculation model includes a difference calculation module 610 and a score calculation module 620, the difference calculation module 610 includes a first fusion layer 612, a second fusion layer 614, at least two parallel first feedforward neural networks 611 and at least two parallel second feedforward neural networks 613, the first feature vector set is used as the input of the first feedforward neural network 611, and the second input of the score calculation model is used as The input of the second feedforward neural network 613 and the output of the first feedforward neural network 611 are used as the input of the first fusion layer 612, the output of the second feedforward neural network 613 is used as the input of the second fusion layer 614, the output of the first fusion layer 612 and the output of the second fusion layer 614 are subtracted and used as the output of the difference calculation module 610, the output of the difference calculation module 610 is used as the input of the score calculation module 620, and the output of the score calculation module 620 is used as the output of the score calculation model, the first fusion layer 612 is used to splice the output of the first feedforward neural network 611, and the second fusion layer 614 is used to splice the output of the second feedforward neural network 613.

[0095] It can be understood that the number of first feedforward neural networks 611 and second feedforward neural networks 613 is equal to the number of speech sets, each first feedforward neural network 611 processes an element of the first feature vector set, and each second feedforward neural network 613 processes an element of the second feature vector set, which can be performed synchronously with high processing efficiency; the first fusion layer 612 can splice the outputs of all first feedforward neural networks 611 in sequence to obtain an intermediate vector; the second fusion layer 614 can splice the outputs of all second feedforward neural networks 613 in sequence to obtain another intermediate vector; the difference calculation module 610 is used to calculate the difference between two intermediate vectors; the score calculation module 620 can calculate the similarity score between the first feature vector set and the second feature vector set based on the difference between the two intermediate vectors.

[0096] In specific practice, the first feedforward neural network 611 and the second feedforward neural network 613 adopt an MLP neural network model; the score calculation module 620 includes a fully connected layer and a Softmax function connected in sequence.

[0097] In addition, refer to Figure 7 , in one embodiment, Figure 7 yes Figure 5 The training process of the score calculation model in step S520 in the illustrated embodiment is as follows:

[0098] Step S710, obtaining a training audio stream and a speaker label, wherein the speech frames in the training audio stream contain speech from at least one speaker, and the speaker label is used to characterize the speaker to which the speech frames in the training audio stream are aligned;

[0099] It is understandable that clustering belongs to unsupervised learning. When obtaining the training audio stream, it is also necessary to obtain the speaker label corresponding to the training audio stream.

[0100] It should be noted that the number of speakers in the training audio stream needs to be the same as the number of speakers in the speech to be processed to ensure the accuracy of the score calculation model.

[0101] Step S720, inputting the training audio stream into at least two speech segmentation models to obtain speech training sets respectively;

[0102] It can be understood that the speech segmentation model mentioned in this step is the same as the speech segmentation model mentioned in the above steps, and no further details will be given here.

[0103] Step S730, classifying the speech training set to obtain multiple groups of similar speech training sets;

[0104] It can be understood that the classification process mentioned in this step is the same as the classification process mentioned in the above steps, and will not be described in detail here.

[0105] Step S740, performing feature extraction on any two similar speech training sets to obtain a first feature vector training set and a second feature vector training set respectively;

[0106] It can be understood that the feature extraction mentioned in this step is the same as the feature extraction mentioned in the above steps, and will not be elaborated here.

[0107] Step S750, inputting the first feature vector training set and the second feature vector training set into a score calculation model to obtain a similarity training score;

[0108] It can be understood that the score calculation model mentioned in this step is the same as the score calculation model mentioned in the above steps, and will not be described in detail here.

[0109] Step S760, obtaining a predicted similarity according to the similarity training score, the first feature vector training set and the second feature vector training set;

[0110] In specific practice, the calculation formula for predicted similarity is as follows:

[0111]

[0112]

[0113] Where N is the total number of pairs of the first feature vector training set and the second feature vector training set, is the average output of the Softmax function for each pair of the first feature vector training set and the second feature vector training set, c = (c 1 ,c 2 ,c 3 ) is the cosine distance between each pair of the first eigenvector training set and the second eigenvector training set, y n is the predicted similarity between the nth pair of the first feature vector training set and the second feature vector training set.

[0114] Step S770, obtaining true similarity through similarity calculation according to the speaker label, the training audio stream, the first feature vector training set and the second feature vector training set;

[0115] It should be noted that the training audio stream includes speech frames corresponding to the first feature vector training set and speech frames corresponding to the second feature vector training set. According to the speaker label, the proportion of speech of different speakers in the above two speech frames is obtained, thereby calculating the similarity of the two speech frames.

[0116] It is understandable that the true similarity can be obtained by calculating the cosine distance. Similarity calculation is a technique well known to those skilled in the art and will not be described in detail herein.

[0117] Step S780, determining a loss function according to the real similarity and the predicted similarity;

[0118] In specific practice, the loss function adopts the MSE loss function, as follows:

[0119]

[0120] Where N is the total number of pairs of the first feature vector training set and the second feature vector training set, y n is the predicted similarity of the nth pair of the first feature vector training set and the second feature vector training set, d n is the true similarity between the nth pair of the first feature vector training set and the second feature vector training set.

[0121] Step S790, updating the score calculation model according to the loss function.

[0122] It should be noted that updating the score calculation model through the loss function can effectively reduce the gap between the predicted similarity and the actual similarity, making the similarity score calculated by the score calculation model more accurate.

[0123] It can be understood that after being trained, the score calculation model can effectively calculate the similarity score between the first feature vector set and the second feature vector set.

[0124] In addition, refer to Figure 8 , in one embodiment, Figure 8 yes Figure 7 The time distribution diagram of the speech of all speakers in the training audio stream in step S710.

[0125] It can be understood that, taking two speakers as an example, the training audio stream contains the speech of the first speaker spk1 and the second speaker spk2. Assuming that the speech segment A is the speech frame corresponding to the first feature vector training set, and the speech segment B is the speech frame corresponding to the second feature vector training set, according to the speaker label, the proportion of the speech of different speakers in the above two speech frames is obtained, as follows:

[0126] v A =(spk1,spk2)=(0,0.5),

[0127] v B =(spk1,spk2)=(0.5,0.25),

[0128] Then, the similarity between the two speech frames is calculated by calculating the cosine distance, as follows:

[0129] d = cos(v A ,v B ),

[0130] Since the proportion of the speaker's voice is a positive value, the cosine distance between the two ranges from 0 to 1. Therefore, the cosine distance d needs to be normalized to be within the range of 0 to 1.

[0131] In addition, refer to Fig. 9 The present invention also provides a speaker segmentation and clustering device 900, comprising:

[0132] An acquisition unit 910 is configured to acquire an audio stream, wherein a speech frame in the audio stream includes speech from at least one speaker;

[0133] A segmentation unit 920, configured to input the audio stream into at least two speech segmentation models to obtain speech sets respectively, wherein the speech segmentation models are configured to perform speech segmentation according to a fixed segmentation time, and the segmentation time lengths of any two speech segmentation models are different;

[0134] The classification unit 930 is used to classify the speech set to obtain multiple groups of similar speech sets;

[0135] A score calculation unit 940 is used to perform similarity score processing on the similar speech set to obtain a similarity score matrix;

[0136] A clustering unit 950 is used to perform clustering processing on the speech set according to the similarity score matrix to obtain speech clusters;

[0137] A modeling unit 960, for establishing a speaker classification model according to speech clusters;

[0138] The alignment unit 970 is used to align the speech frames in the audio stream with the speaker according to the speaker classification model.

[0139] In addition, refer to Fig.10 An embodiment of the present invention further provides an electronic device, the electronic device 1000 comprising: a memory 1010, a processor 1020, and a computer program stored in the memory 1010 and executable on the processor 1020.

[0140] The processor 1020 and the memory 1010 may be connected via a bus or other means.

[0141] The non-transient software program and instructions required to implement the speaker segmentation and clustering method of the above embodiment are stored in the memory 1010. When executed by the processor 1020, the speaker segmentation and clustering method of the above embodiment is executed, for example, the above described Figure 1 Steps S110 to S170 of the method, Figure 2 Steps S210 to S230 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S510 to S540 of the method, Figure 7 The method steps S710 to S790 in the above-described device embodiment are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0142] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. The computer-executable instructions are executed by a processor or a controller, for example, by a processor in the above electronic device embodiment, so that the processor can execute the speaker segmentation and clustering method in the above embodiment, for example, execute the above described Figure 1 Steps S110 to S170 of the method, Figure 2 Steps S210 to S230 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S510 to S540 of the method, Figure 7Method steps S710 to step S790 in the method. It will be appreciated by those skilled in the art that all or some of the steps and systems in the method disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0143] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A speaker segmentation and clustering method, characterized in that: include: Acquire an audio stream, wherein the speech frames in the audio stream contain speech from at least one speaker; Inputting the audio stream into at least two speech segmentation models to obtain speech sets respectively, wherein the speech segmentation models are configured to perform speech segmentation according to a fixed segmentation duration, and the segmentation durations of any two speech segmentation models are different; Classifying the speech set to obtain multiple groups of similar speech sets, wherein the classification refers to classifying the elements of the speech set, taking the elements in one of the speech sets as reference elements in turn, judging the degree of proximity between the elements of another speech set and the reference elements, classifying the elements in the other speech set with the highest degree of proximity to the reference elements into the same category, and obtaining multiple groups of similar speech sets; Performing similarity score processing on the similar speech set to obtain a similarity score matrix; According to the similarity score matrix, clustering is performed on the speech set to obtain speech clusters; Establishing a speaker classification model according to the speech clusters; aligning speech frames in the audio stream with the speaker according to the speaker classification model; The similarity score processing is performed on the similar speech set to obtain a similarity score matrix, including: Performing feature extraction on any two of the similar speech sets to obtain a first feature vector set and a second feature vector set respectively; Inputting the first feature vector set and the second feature vector set into a score calculation model to obtain a similarity score; Performing similarity calculation on the first feature vector set and the second feature vector set to obtain a similarity matrix; A similarity score matrix is ​​obtained according to the similarity score and the similarity matrix.

2. The method according to claim 1, characterized in that After the step of inputting the audio stream into at least two speech segmentation models to obtain speech sets respectively, the method further includes: According to the segmentation duration of each speech segmentation model, respectively performing a corresponding time shift on the segmentation starting point of each speech segmentation model; Inputting the audio stream into each of the time-shifted speech segmentation models to obtain auxiliary speech sets respectively, wherein the auxiliary speech sets correspond one-to-one to the speech sets; The auxiliary speech set is merged with the corresponding speech set to update the speech set.

3. The method according to claim 2, characterized in that The time shift length of the segmentation starting point of the speech segmentation model is half of the corresponding segmentation time length of the speech segmentation model.

4. The method according to claim 1, characterized in that Each of the speech sets includes at least two speech segments; the speech sets are classified to obtain multiple groups of similar speech sets, including: Taking a speech set obtained by a speech segmentation model with the shortest segmentation time among the speech segmentation models as a reference speech set; Calculating in sequence the reference length center distance between each speech segment of the reference speech set and each speech segment of all the speech sets; For any speech segment of the reference speech set, a speech segment with the smallest reference length center distance in each of the speech sets is selected to form a similar speech set.

5. The method according to claim 1, characterized in that The training process of the score calculation model is as follows: Acquire a training audio stream and a speaker label, wherein the speech frames in the training audio stream contain speech from at least one speaker, and the speaker label is used to characterize the speaker to which the speech frames in the training audio stream are aligned; Inputting the training audio stream into at least two of the speech segmentation models to obtain speech training sets respectively; Classifying the speech training set to obtain multiple groups of similar speech training sets; Performing feature extraction on any two of the similar speech training sets to obtain a first feature vector training set and a second feature vector training set respectively; Inputting the first feature vector training set and the second feature vector training set into the score calculation model to obtain a similarity training score; Obtaining a predicted similarity according to the similarity training score, the first feature vector training set, and the second feature vector training set; Obtaining true similarity through similarity calculation according to the speaker label, the training audio stream, the first feature vector training set, and the second feature vector training set; Determining a loss function according to the true similarity and the predicted similarity; The score calculation model is updated according to the loss function.

6. The method according to claim 1, characterized in that The first input of the score calculation model is the first feature vector set, the second input of the score calculation model is the second feature vector set, the output of the score calculation model is the similarity score, the score calculation model includes a difference calculation module and a score calculation module, the difference calculation module includes a first fusion layer, a second fusion layer, at least two parallel first feedforward neural networks and at least two parallel second feedforward neural networks, the first feature vector set is used as the input of the first feedforward neural network, the second input of the score calculation model is used as the input of the second feedforward neural network, the output of the first feedforward neural network is used as the input of the first fusion layer, the output of the second feedforward neural network is used as the input of the second fusion layer, the output of the first fusion layer and the output of the second fusion layer are subtracted as the output of the difference calculation module, the output of the difference calculation module is used as the input of the score calculation module, the output of the score calculation module is used as the output of the score calculation model, the first fusion layer is used to splice the output of the first feedforward neural network, and the second fusion layer is used to splice the output of the second feedforward neural network.

7. A speaker segmentation and clustering device, characterized in that: include: An acquisition unit, configured to acquire an audio stream, wherein the speech frames in the audio stream contain speech from at least one speaker; A segmentation unit, configured to input the audio stream into at least two speech segmentation models to obtain speech sets respectively, wherein the speech segmentation models are configured to perform speech segmentation according to a fixed segmentation duration, and the segmentation durations of any two speech segmentation models are different; A classification unit, used for classifying the speech set to obtain multiple groups of similar speech sets, wherein the classification refers to classifying the elements of the speech set, taking the elements in one of the speech sets as reference elements in turn, judging the degree of proximity between the elements of another speech set and the reference elements, classifying the elements in the other speech set with the highest degree of proximity to the reference elements into the same category, and obtaining multiple groups of similar speech sets; A scoring calculation unit is used to perform similarity scoring processing on the similar speech set to obtain a similarity score matrix; A clustering unit, used for performing clustering processing on the speech set according to the similarity score matrix to obtain speech clusters; A modeling unit, used for establishing a speaker classification model according to the speech clusters; an alignment unit, configured to align speech frames in the audio stream with the speaker according to the speaker classification model; The similarity score processing is performed on the similar speech set to obtain a similarity score matrix, including: Performing feature extraction on any two of the similar speech sets to obtain a first feature vector set and a second feature vector set respectively; Inputting the first feature vector set and the second feature vector set into a score calculation model to obtain a similarity score; Performing similarity calculation on the first feature vector set and the second feature vector set to obtain a similarity matrix; A similarity score matrix is ​​obtained according to the similarity score and the similarity matrix.

8. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speaker segmentation and clustering method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used to execute the speaker segmentation and clustering method according to any one of claims 1 to 6.

Images