Speaker clustering method, apparatus and device
By dividing speech segments into multiple nodes and constructing a community network, and using a community detection algorithm to determine the speaker set, the problem of low clustering accuracy in existing technologies is solved, achieving more efficient speaker recognition and speech segment separation.
Patent Information
- Application Number
- CN202210028998.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-11
- Publication Date
- 2025-12-16
- Estimated Expiration
- 2042-01-11
AI Technical Summary
Existing speaker clustering techniques are prone to misidentifying a single person as multiple categories or incorrectly merging different speakers, resulting in low clustering accuracy.
The speech segments are divided into multiple nodes, and a community network is constructed using a community detection algorithm. Speaker feature similarity is used as the boundary value to determine the speech segment set corresponding to each speaker. Dimensionality reduction and smoothing can be performed optionally, and the modularity increment can be optimized to improve clustering accuracy.
It effectively improves the accuracy of speaker clustering, enabling more accurate identification and separation of different speakers in multi-person speech, and improving the efficiency of speech segment recognition and localization.
Smart Images
Figure CN114495911B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of speech processing, in particular to a speaker clustering method, device and system, and an electronic device. BACKGROUND
[0002] With the wide application of intelligent speech technology in people's daily life, how to more accurately identify the start and end time points of the utterance of different speakers in the multi-person speech interaction process has gradually become a research hotspot as a prerequisite for speech and other backend recognition technologies.
[0003] The main purpose of the speaker diarization technology is to automatically identify the number of speakers in a long audio of multi-person communication (usually a single-channel spoken dialogue voice with more multi-person overlap segments) and detect the start and end time stamps of the speech of each speaker in the audio to solve the problem of "who spoke when", thereby facilitating quick retrieval and positioning of the voice segment of a specific speaker, and serving as the basis for subsequent speech recognition and voiceprint recognition modules and being widely used in voice transcription and indexing applications in conference scenarios.
[0004] The speaker diarization technology mainly adopts a clustering method to cluster a large number of voice segments according to the speaker identity and cluster the segments belonging to the same speaker together. At present, the mainstream speaker clustering technology adopts common clustering methods such as k-means, AHC hierarchical clustering, and spectral clustering. However, in the process of implementing the present application, the inventors found that the above technical solutions at least have the following problems: a single person is easily identified as multiple classes, or different speakers are incorrectly merged, and the accuracy of speaker clustering is low. SUMMARY
[0005] The present application provides a speaker clustering method to solve the problem of low accuracy of speaker clustering in the prior art. The present application further provides a speaker clustering device and system, and an electronic device.
[0006] The present application provides a speaker clustering method, comprising:
[0007] dividing the voice to be processed into a plurality of voice segments;
[0008] obtaining speaker features of the voice segments;
[0009] constructing a community network with the voice segments as nodes and speaker feature similarities as edge values;
[0010] determining the voice segment set corresponding to each speaker according to the community network through a community detection algorithm.
[0011] Optionally, the determining, by the community detection algorithm, the speech segment set corresponding to each speaker according to the community network comprises:
[0012] Removing all edges in the community network, and taking each node of the community network as a community;
[0013] Respectively adding back edges not in the community network into the network, adding one edge at a time, if the edge added into the network connects two different communities, merging the two communities, and determining a modularity increment of a new community division according to the speaker feature similarity, selecting two communities with the largest modularity increment to be merged, until no merging can be found to increase the modularity;
[0014] Selecting, according to the modularity values of various community divisions, a community division with the largest modularity as the speech segment set corresponding to each speaker.
[0015] Optionally, the method further comprises:
[0016] The speaker feature is processed by dimension reduction through a uniform manifold approximation and projection algorithm.
[0017] Optionally, the method further comprises:
[0018] The speaker feature is processed by smoothing.
[0019] Optionally, the obtaining the speaker feature of the speech segment comprises:
[0020] The speaker feature is obtained through a speaker feature recognition network.
[0021] Optionally, the method further comprises:
[0022] The speaker clustering result obtained by the community detection algorithm is displayed, each cluster in the clustering result corresponds to one speaker, and each node in each cluster corresponds to one speech segment;
[0023] The clustering result is updated according to the correction information provided by a user.
[0024] Optionally, the method further comprises:
[0025] The speaking time corresponding to each speaker is determined according to the speech segment set corresponding to each speaker.
[0026] The corresponding speaker and speech segment are displayed according to the target speaking time.
[0027] The application also provides a speaker clustering device, comprising:
[0028] A speech division unit is configured to divide the to-be-processed speech into a plurality of speech segments.
[0029] a speaker feature recognition unit configured to obtain speaker features of the voice segments;
[0030] a community network construction unit configured to construct a community network with the voice segments as nodes and speaker feature similarities as edge values;
[0031] a community detection unit configured to determine voice segment sets corresponding to respective speakers according to the community network by using a community detection algorithm.
[0032] The application also provides an electronic device, comprising:
[0033] a processor and a memory;
[0034] the memory is configured to store a program for implementing the speaker recognition method, and the device is powered on and runs the program of the method by using the processor.
[0035] The application also provides a conference transcription system, comprising:
[0036] a conference terminal configured to collect conference voice and send the conference voice to a server;
[0037] the server is configured to divide the conference voice into a plurality of voice segments, obtain speaker features of the voice segments, construct a community network with the voice segments as nodes and speaker feature similarities as edge values, determine voice segment sets corresponding to respective speakers according to the community network by using a community detection algorithm, and form speech texts corresponding to the respective speakers according to transcription texts of the respective voice segments.
[0038] The application also provides a computer readable storage medium, wherein instructions are stored in the computer readable storage medium, and when the instructions are run on a computer, the computer is caused to execute the various methods.
[0039] The application also provides a computer program product comprising instructions, and when the instructions are run on a computer, the computer is caused to execute the various methods.
[0040] Compared with the prior art, the application has the following advantages:
[0041] The speaker clustering method provided by the application divides the voice to be processed into a plurality of voice segments, obtains speaker features of the voice segments, constructs a community network with the voice segments as nodes and speaker feature similarities as edge values, and determines voice segment sets corresponding to respective speakers according to the community network by using a community detection algorithm. In this way, speaker clustering based on community detection can effectively improve the accuracy of speaker clustering. BRIEF DESCRIPTION OF DRAWINGS
[0042] Figure 1 A flowchart of an embodiment of the speaker clustering method provided in the present application;
[0043] Figure 2 A social network diagram of an embodiment of the speaker clustering method provided in the present application;
[0044] Figure 3 A speaker clustering diagram of an embodiment of the speaker clustering method provided in the present application. DETAILED DESCRIPTION
[0045] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application. However, the present application can be practiced in a large number of other ways not described herein, and it is understood that similar modifications can be made by one skilled in the art without departing from the spirit and scope of the present application, and therefore the present application is not limited to the specific implementations disclosed below.
[0046] In the present application, a speaker clustering method and device, a speaker log system, and an electronic device are provided. In the following embodiments, various schemes are described one by one in detail.
[0047] First Embodiment
[0048] Reference is made to Figure 1 which is a flowchart of the speaker clustering method of the present application. In the present embodiment, the method can include the following steps:
[0049] Step S101: dividing the to-be-processed voice into a plurality of voice segments.
[0050] In this step, the voice segmentation module can remove the non-voice part and divide the input speech into small segments, i.e., a plurality of voice segments. In specific implementation, any cutting method can be used, such as 1.5 seconds to 2 seconds for one segment, and there can be 0.5 seconds to 1 second overlap between adjacent segments.
[0051] Step S103: obtaining the speaker feature of the voice segment.
[0052] The speaker feature can be the voiceprint feature of the speaker, and the method provided in the present embodiment can realize clustering of the voiceprint feature.
[0053] In this step, the feature vector extraction module can extract the feature vector capable of judging the speaker, such as i-vector, d-vector, etc., from the voice segment. In specific implementation, the feature vector extraction module can use a speaker feature recognition network based on a neural network, take the voice segment as the input data of the network, and output the speaker feature through the network. Since the speaker feature vector extraction belongs to a relatively mature prior art, it is not described herein again.
[0054] In this embodiment, the speaker features of multiple speech segments constitute a speaker feature matrix. The rows of this matrix can be feature dimensions, and the columns can be speech segments. For example, the speaker feature corresponding to each speech segment is a 500-dimensional feature vector. If the entire conference speech is divided into 1000 speech segments, it would be a 500*1000 matrix, which can be reduced to a new matrix of, for example, 30*1000.
[0055] Step S105: Construct a community network using speech segments as nodes and speaker feature similarity as boundary values.
[0056] The method provided in this application treats the speaker clustering problem as a community detection problem. The basic definition of a community in a community network is: a set of tightly connected nodes with many internal connections and relatively few external connections. Based on the structured relationships of the network graph, speech segments can be reasonably segmented, i.e., assigned to their respective speakers.
[0057] like Figure 2 As shown, this embodiment constructs a community network using speech segments as nodes and speaker feature similarity as edge values. Speaker feature similarity refers to the speaker feature similarity between two speech segments corresponding to the two nodes connecting the edges. Speaker feature similarity can be the product of the speaker features of two speech segments. In specific implementation, the speaker feature similarity between any two speech segments can be calculated.
[0058] In practical applications, the method may further include a step of dimensionality reduction of the speaker features, such as reducing 500-dimensional speaker features to 30 dimensions. This approach can effectively improve the accuracy and stability of speaker clustering.
[0059] In this embodiment, the speaker features are dimensionality reduced using the Unified Manifold Approximation and Projection (UMAP) algorithm. UMAP dimensionality reduction preserves the global topological structure, that is, it retains a distance relationship between each global class (speaker), thus effectively improving the cohesion of speaker clustering. In specific implementations, other feature dimensionality reduction methods, such as t-SNE, LDA, and PCA, can also be used.
[0060] Step S107: Using a community detection algorithm, determine the set of speech segments corresponding to each speaker based on the community network.
[0061] This step, based on the aforementioned community network, uses a community detection algorithm to cluster speakers across all speech segments, thereby determining the number of speakers and assigning speaker identities to each speech segment. The community detection algorithm can be the Leiden algorithm, etc.
[0062] In one example, step S107 can include the following sub-steps:
[0063] Step S1071: Remove all edges in the social network, and take each node of the social network as a community.
[0064] Step S1073: Add back edges not in the social network into the network one by one, and if the added edge connects two different communities, merge the two communities, and determine the increment of modularity of the new community division according to the speaker feature similarity, select the two communities with the largest increment of modularity to merge, and continue until no merging can be found to increase the modularity.
[0065] The community detection is a method of optimizing the expected value of modularity, and the modularity can be the ratio of the total edge weight in the community to the total edge weight in the network minus an expected value.
[0066] Step S1073: Select the community division with the largest modularity as the speech segment set corresponding to each speaker according to the modularity values corresponding to various community divisions.
[0067] The community division with the largest modularity is the speaker clustering result of all speech segments. In an ideal case, if the entire conference includes three speakers, all speech segments (such as segments 5, 15, 64, etc.) of one speaker constitute a community.
[0068] The optimization goal of the community detection algorithm is to maximize the modularity of the entire data. In a specific implementation, the modularity value can be calculated using the following formula:
[0069]
[0070]
[0071] In the formula, it is assumed that there are n speech segments, A represents the corresponding adjacency matrix. A_ij represents the speaker feature similarity between the speech segment of the i-th node and the speech segment of the j-th node, k_i and k_j are the degrees of the i-th node and the j-th node, respectively, and 2m is the sum of the degrees of the speaker features of all speech segments. Here, the meaning of "degree" is the sum of the products of each speaker feature and all other speaker features.
[0072] In one example, the method can further include the step of smoothing the speaker features. In implementation, the results of clustering can be arranged in the time sequence of the original meeting, and then smoothed. For example, the speaker features of a voice are 1 1 1 1 2 1 2 2 2 2 2, which can be smoothed to 1 1 1 1 1 1 2 2 2 2 2 2. By smoothing the speaker features, the results of clustering can be more readable when presented.
[0073] In one example, the method can further include the steps of presenting the speaker clustering results obtained by the community detection algorithm, each cluster in the clustering results corresponding to a speaker, and each point in each cluster corresponding to a voice segment; and updating the clustering results according to the correction information provided by a user for the clustering results. The correction information can include correction information for the speaker, correction information for the voice segment, or both. Figure 3 The speaker clustering results are shown, each cluster corresponding to a speaker, and each dot corresponding to a voice segment.
[0074] In one example, the method provided by the embodiments of the present application can be applied in a meeting scenario, the voice to be processed includes multi-person meeting voice, the speakers include meeting users, and the voice segment set corresponding to each speaker includes a meeting voice segment set corresponding to each meeting user. Accordingly, the method can further include the steps of determining the speaking time corresponding to each meeting user according to the voice segment set corresponding to each meeting user, and presenting the corresponding meeting user and voice segment according to the target speaking time. The target speaking time can be an arbitrary time point or time period in the meeting specified by a user. By applying the method in a meeting scenario, the voice of each participant in a multi-person meeting can be separated, and the speaking time of different speakers can be distinguished and labeled in the collected voice signal stream, i.e., the identity of each voice segment corresponding to a speaker is detected by indexing time.
[0075] As can be seen from the above embodiments, the speaker clustering method provided by the embodiments of the present application divides the voice to be processed into a plurality of voice segments, obtains speaker features of the voice segments, constructs a community network by taking the voice segments as nodes and the speaker feature similarity as edge values, and determines the voice segment set corresponding to each speaker according to the community network by using a community detection algorithm. By using this processing method, the accuracy of speaker clustering based on community detection can be effectively improved.
[0076] Second Embodiment
[0077] In the above embodiments, a speaker clustering method is provided, and the present application also provides a speaker clustering device corresponding to the method. The device corresponds to the above-mentioned embodiments of the method. Since the device embodiments are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments. The device embodiments described below are only illustrative.
[0078] The present application further provides a speaker clustering device, comprising: a voice division unit, a speaker feature recognition unit, a social network construction unit, and a social group detection unit.
[0079] The voice division unit is configured to divide the voice to be processed into a plurality of voice segments; the speaker feature recognition unit is configured to obtain speaker features of the voice segments; the social network construction unit is configured to construct a social network by taking the voice segments as nodes and taking speaker feature similarities as edge values; and the social group detection unit is configured to determine voice segment sets corresponding to each speaker according to the social network by using a social group detection algorithm.
[0080] In one example, the social group detection unit is specifically configured to remove all edges in the social network and take each node of the social network as a community; add edges not in the social network back to the network one by one, and if an added edge connects two different communities, merge the two communities, and determine a modularity increment of a new community division according to the speaker feature similarities, select two communities with the largest modularity increment to merge, and continue until no merging operation can increase the modularity; and select a community division with the largest modularity as the voice segment set corresponding to each speaker according to modularity values of various community divisions.
[0081] In one example, the device can further comprise a dimension reduction unit configured to perform dimension reduction processing on the speaker features by using a uniform manifold approximation and projection algorithm.
[0082] In one example, the device can further comprise a smoothing processing unit configured to perform smoothing processing on the speaker features.
[0083] In one example, the speaker feature recognition unit is specifically configured to obtain the speaker features by using a speaker feature recognition network.
[0084] In one example, the device can further comprise a speaker clustering result display unit and a clustering result correction unit. The speaker clustering result display unit is configured to display a speaker clustering result obtained by using the social group detection algorithm, wherein each cluster in the clustering result corresponds to a speaker, and each point in each cluster corresponds to a voice segment; and the clustering result correction unit is configured to update the clustering result according to correction information provided by a user for the clustering result.
[0085] In one example, the apparatus can further include a time labeling unit and a speaker retrieval unit. The time labeling unit is configured to determine speaking times of each speaker according to the voice segment set corresponding to each speaker. The speaker retrieval unit is configured to display the corresponding speaker and voice segment according to the target speaking time.
[0086] Third embodiment
[0087] In the above embodiments, a speaker clustering method is provided, and a corresponding electronic device is also provided. The device corresponds to the above-mentioned embodiments of the method. Since the device embodiments are basically similar to the method embodiments, they are described more simply, and the relevant parts are described in the method embodiments. The device embodiments described below are only illustrative.
[0088] An electronic device of the present embodiment includes a processor and a memory. The memory is configured to store a program for implementing a speaker clustering method. After the device is powered on and the program of the method is executed by the processor, the following steps are performed: dividing the voice to be processed into a plurality of voice segments; obtaining speaker features of the voice segments; constructing a community network with voice segments as nodes and speaker feature similarity as edge values; determining a voice segment set corresponding to each speaker according to the community network through a community detection algorithm.
[0089] Fourth embodiment
[0090] Corresponding to the above-mentioned speaker clustering method, the present application also provides a conference transcription system. The present embodiment has the same parts as the first embodiment, and the relevant parts are described in the first embodiment.
[0091] The conference transcription system provided by the present application is used to cluster the text transcribed from the conference voice according to the voice of the speaker, and label each piece of text with the corresponding speaker. The system includes a conference terminal and a server.
[0092] The conference terminal is configured to collect conference voice and send the conference voice to the server. The server is configured to divide the conference voice into a plurality of voice segments, obtain speaker features of the voice segments, construct a community network with voice segments as nodes and speaker feature similarity as edge values, determine a voice segment set corresponding to each speaker according to the community network through a community detection algorithm, and form speaking text corresponding to each speaker according to the transcription text of each voice segment.
[0093] Although the present application is disclosed with reference to the preferred embodiments above, it is to be understood that the application is intended to cover all possible variations and modifications thereof. For example, it is intended that any "means plus function" claims in the application cover the structures and techniques disclosed by this document as well as equivalents thereof. It is intended that any embodiment disclosed in this document covering "comprising", "including", and "having" include additional subject matter unless the context clearly indicates otherwise. Thus, for example, a system that "comprises", "includes", or "has" a list of steps is not necessarily limited to those steps, but can include additional steps not expressly listed or inherent to the system. Similarly, a system that "comprises", "includes", or "has" a list of elements is not necessarily limited to those elements, but can include additional elements not expressly listed or inherent to the system. Further, unless otherwise indicated, the steps of the disclosed embodiments can be performed in any order, and are not limited to the order in which they are described.
[0094] In one typical arrangement, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.
[0095] The memory can include non-persistent memory and / or volatile memory, such as random access memory (RAM) and / or cache memory. The memory is an example of computer readable media.
[0096] 1. Computer readable media includes both non-transitory and transitory media used to store data for access by a computer. Non-transitory computer readable media include, for example, memory devices that include the following media: RAM; ROM; electrically erasable programmable ROM (EEPROM); flash memory or other memory technology; CD-ROM; digital versatile disks (DVD) or other optical storage; magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices; or any other medium that can be used to store information for access by a computer.
[0097] 2. As will be appreciated by one skilled in the art, embodiments of the present application can be provided as a method, system or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable medium(s) having computer usable program code embodied in the medium that can be executed by a computer.
Claims
1. A speaker clustering method, characterized by, The method comprises the following steps: dividing the to-be-processed speech into a plurality of speech segments, wherein the to-be-processed speech comprises a plurality of mixed speech segments of multiple persons; obtaining speaker features of the speech segments; constructing a community network by taking the speech segments as nodes and speaker feature similarities as edge values; performing speaker clustering on the speech segments in the community network according to the speaker feature similarities in the community network by using a community detection algorithm to determine a speech segment set corresponding to each speaker, comprising: removing edges of the community network and taking the nodes of the community network as communities; re-adding edges not added to the community network to the network one by one, merging two communities if an added edge connects the two communities, and determining a modularity increment of a new community division formed according to the speaker feature similarities, and selecting two communities with large modularity increments to be merged until no merging operation can be found to increase the modularity; selecting a community division with large modularity as the speech segment set corresponding to each speaker according to the modularity values corresponding to various community divisions.
2. The method of claim 1, wherein, The method further comprises the following steps: performing dimensionality reduction processing on the speaker features by using a uniform manifold approximation and projection algorithm.
3. The method of claim 1, wherein, The method further comprises the following step: performing smoothing processing on the speaker features.
4. The method of claim 1, wherein, The method of obtaining the speaker features of the speech segments comprises the following step: obtaining the speaker features by using a speaker feature recognition network.
5. The method of claim 1, wherein, The method further comprises the following steps: displaying a speaker clustering result obtained by using the community detection algorithm, wherein each cluster in the clustering result corresponds to a speaker, and each point in each cluster corresponds to a speech segment; updating the clustering result according to modification information provided by a user.
6. The method of claim 1, wherein, The method further comprises the following steps: determining a speaking time corresponding to each speaker according to the speech segment set corresponding to each speaker; displaying a corresponding speaker and a speech segment according to a target speaking time.
7. A speaker clustering apparatus characterized by comprising: The method comprises the following steps: dividing to-be-processed speech into a plurality of speech segments by using a speech division unit, wherein the to-be-processed speech comprises a plurality of mixed speech segments of multiple persons; obtaining speaker features of the speech segments by using a speaker feature recognition unit; constructing a community network by taking the speech segments as nodes and speaker feature similarities as edge values by using a community network construction unit; performing speaker clustering on the speech segments in the community network according to the speaker feature similarities in the community network by using a community detection unit to determine a speech segment set corresponding to each speaker, comprising: removing edges of the community network and taking the nodes of the community network as communities; 8. An electronic device, comprising: re-adding edges not added to the community network to the network one by one, merging two communities if an added edge connects the two communities, and determining a modularity increment of a new community division formed according to the speaker feature similarities, and selecting two communities with large modularity increments to be merged until no merging operation can be found to increase the modularity; and selecting a community division with large modularity as the speech segment set corresponding to each speaker according to the modularity values corresponding to various community divisions. The method comprises the following steps: a processor and a memory. A memory for storing a program implementing the speaker clustering method according to any one of claims 1-6, the device being powered and running the program of the method by the processor.
9. A conference transcription system, characterized by Comprise: A conference terminal for collecting conference speech and sending the conference speech to a server; The server is used for dividing the conference speech into a plurality of speech segments, the conference speech comprising a plurality of mixed speech segments; obtaining speaker features of the speech segments; A community network is constructed with speech segments as nodes and speaker feature similarities as edge values; through a community detection algorithm, speaker clustering is performed on the speech segments in the community network according to the speaker feature similarities in the community network to determine the speech segment set corresponding to each speaker; according to the transcription text of each speech segment, the speech text corresponding to each speaker is formed; wherein, through the community detection algorithm, speaker clustering is performed on the speech segments in the community network according to the speaker feature similarities in the community network to determine the speech segment set corresponding to each speaker, comprising: removing the edges of the community network, and taking the nodes of the community network as communities; the edges not joined in the community network are respectively added back to the network, one edge is added each time, if the edge added to the network connects two different communities, the two communities are merged, and the modularity increment of the new community division is determined according to the speaker feature similarity, the two communities with larger modularity increment are selected for merging, until no merging can be found to increase the modularity; according to the modularity values corresponding to various community divisions, the community division with larger modularity is selected as the speech segment set corresponding to each speaker.