A Speaker Identification Method Based on the Large Model Cam++ of Speech Recognition and Voiceprint Recognition

Abstract

The present application relates to the technical field of speech and voiceprint recognition, and provides a method for distinguishing speakers based on speech recognition and voiceprint recognition big model Cam++, the method comprising: obtaining the start timestamp and end timestamp corresponding to each sentence in the input audio through the speech recognition big model Cam++, and dividing the audio segment corresponding to each sentence according to the start timestamp and the end timestamp; filtering the audio segments whose duration is less than the preset duration threshold; inputting the filtered audio segments into the voiceprint recognition big model Cam++ to obtain the voiceprint feature vector of each audio segment; obtaining the initial K‑means clustering number; initial K‑means clustering, extracting the deviated abnormal feature vector; performing secondary processing on the extracted deviated abnormal feature vector; clustering and merging groups; processing all ungrouped segments. The present application uses an advanced big model to improve the accuracy of the big model's own voiceprint recognition, while improving the granularity of distinguishing speakers, performing multi-layer K‑means distinction, and improving the accuracy of the number of speaker groupings.

Description

Technical Field

[0001] The present application relates to the technical field of speech and voiceprint recognition, and in particular to a method for distinguishing speakers based on a large model Cam++ for speech and voiceprint recognition. Background Art

[0002] Existing audio speaker differentiation solutions mainly use the following technical means to achieve speaker recognition and differentiation in audio:

[0003] Speaker Recognition technology: uses machine learning and deep learning methods to distinguish different speakers by analyzing speaker characteristics of audio signals (such as voice pitch, rhythm, voice fingerprint, etc.). Common models include methods based on convolutional neural networks (CNN), long short-term memory networks (LSTM), or self-attention networks (Transformer).

[0004] Feature extraction: The audio characteristics of the speaker are described by extracting features from the speech signal (such as Mel-frequency cepstral coefficients (MFCC), spectrogram features, etc.). Commonly used audio processing tools include Librosa and Kaldi.

[0005] Clustering algorithms: Apply clustering algorithms (such as K-means, DBSCAN) or Gaussian mixture models (GMM) to classify speech fragments in the audio to achieve speaker separation and recognition.

[0006] Voice Activity Detection (VAD): Use voice activity detection technology to first separate the voice part from the audio and reduce the interference of irrelevant noise. Common tools include WebRTC VAD.

[0007] Deep learning frameworks and tools: Speaker differentiation technology is usually based on development languages ​​such as Python. Common deep learning frameworks include TensorFlow and PyTorch, and audio processing libraries include Kaldi, pyAudio, SpeechBrain, etc.

[0008] However, the above technical solution has the disadvantages of low recognition accuracy and low granularity of distinction. In addition, the traditional K-means algorithm randomly selects the initial clustering center, which leads to different clustering results for different initial center selections. The selection of the initial center has a significant impact on the final clustering result and may cause the algorithm to converge to a local optimal solution rather than a global optimal solution, resulting in poor clustering effect. Summary of the invention

[0009] In order to help solve the above technical problems, this application provides a speaker differentiation method based on the large model Cam++ of speech recognition and voiceprint recognition, which adopts the following technical solutions:

[0010] A method for speaker differentiation based on a large model Cam++ for speech recognition and voiceprint recognition, wherein the method comprises:

[0011] Step S1: obtaining the start timestamp and end timestamp corresponding to each sentence in the input audio through the speech recognition large model Cam++, and dividing the audio segment corresponding to each sentence according to the start timestamp and end timestamp;

[0012] Step S2: extracting audio clips whose duration is less than a preset duration threshold;

[0013] Step S3: input the audio clips after step S2 into the voiceprint recognition large model Cam++ to obtain the voiceprint feature vector of each audio clip;

[0014] Steps S1 to S3 are used to obtain the voiceprint feature vector of each audio clip;

[0015] Step S4: obtaining the initial number of K-means clusters, wherein the step S4 includes calculating the initial number of K-means clusters by using the elbow rule and the silhouette rule, obtaining the number of elbow rule clusters and the number of silhouette rule clusters, and taking the larger value as the initial number of K-means clusters;

[0016] Step S5: initial K-means clustering, extracting the deviated abnormal feature vectors, said step S5 includes performing K-means clustering on the voiceprint feature vectors based on the Euclidean distance according to the number of the initial K-means clusters, obtaining k voiceprint feature vector groups, and at the same time, marking the voiceprint feature vector groups with speaker labels according to the number of clusters k, wherein the speaker label is a number i, 1≤i≤K, i is an integer, and extracting the deviated abnormal feature vectors in each voiceprint feature vector group according to the similarity within the cluster;

[0017] Step S6: performing secondary processing on the extracted deviated abnormal feature vectors through the steps S4 and S5, wherein the step S6 includes reprocessing all the extracted deviated abnormal feature vectors through the steps S4 and S5, and extracting the abnormal feature vectors that are still deviated after the reprocessing;

[0018] Steps S4 to S6 are used to perform K-means clustering on the voiceprint feature vectors based on Euclidean distance, and to extract the deviated abnormal feature vectors;

[0019] Step S7: clustering and merging the groups obtained in step S5 and step S6, wherein step S7 includes performing inter-group similarity judgment on the voiceprint feature vector groups obtained by clustering in step S5 and step S6 according to the difference between clusters, and if the average similarity between two groups is greater than or equal to a preset inter-group similarity threshold, merging the two groups, and merging the two speaker labels corresponding to the two groups into one speaker label;

[0020] Step S8: Processing all ungrouped segments, the step S8 includes marking the speaker labels of the previous segment and the next segment of the segment for all the segments taken out in the step S2 and the step S6,

[0021] If the previous segment is a grouped segment, the speaker label of the segment is used as the previous speaker label. If the previous segment is an ungrouped segment, the search is continued forward until a grouped segment is found and its speaker label is used; or if the next segment is a grouped segment, the speaker label of the segment is used as the next speaker label. If the next segment is an ungrouped segment, the search is continued backward until a grouped segment is found and its speaker label is used.

[0022] Preferably, the elbow rule comprises:

[0023] Step S411: Select the number of clusters and calculate the sum of squares of cluster errors SSE for different cluster numbers k;

[0024] Step S412: draw an elbow diagram, and draw the number of clusters k and the corresponding clustering error sum of squares SSE value in the same diagram;

[0025] Step S413: Select the inflection point where the sum of squared clustering errors (SSE) value drops sharply and then slowly decreases as the optimal clustering number k.

[0026] Preferably, the elbow rule further comprises:

[0027] If there are n data points, each data point x i Belongs to a cluster C k , then the clustering error sum of squares SSE is calculated as follows:

[0028] ;in: is the cluster centroid, C k The center of mass, x i is a data point, x j is a point in the cluster, is the square of the distance from the data point to the centroid of the cluster.

[0029] Preferably, the contour rule comprises:

[0030] Step S421: Calculate the intra-cluster similarity. Calculated by:

[0031] , where C i is the cluster containing data point i, is the distance between data point i and data point j;

[0032] Step S422: Inter-cluster differences Calculated by:

[0033] , where C k is different from C i The clustering of is the data point i and cluster C k The distance to the midpoint j;

[0034] Step S423: Based on the intra-cluster similarity The difference between clusters and , the silhouette coefficient of data point i is calculated by :

[0035] ; Silhouette coefficient The value range is [-1,1]:

[0036] Step S424. Average the silhouette coefficients of all data points to obtain the overall clustering effect in the following manner:

[0037] , where n is the total number of data points, is the silhouette coefficient for each data point.

[0038] Preferably, the step S423 further includes:

[0039] : Indicates that the data point is clustered correctly;

[0040] :It is difficult to determine the ownership of this data point;

[0041] : Indicates that the data point is incorrectly assigned to the current cluster.

[0042] In summary, this application has the following beneficial effects:

[0043] 1. More accurate determination of the number of initial clusters

[0044] The elbow rule and silhouette rule evaluate the clustering effect from the perspectives of the sum of squared clustering errors (SSE) and clustering quality (silhouette coefficient), respectively. The combination of the two can determine the initial number of clusters more comprehensively and accurately.

[0045] Traditional methods for determining the number of clusters often rely on experience or subjective judgment, while the elbow rule and silhouette rule provide objective evaluation criteria and reduce the errors caused by subjectivity.

[0046] 2. Secondary processing of abnormal feature vectors

[0047] By performing secondary processing on the abnormal feature vectors that deviate from the first clustering, the impact of outliers on the clustering results can be further reduced, and the accuracy and stability of clustering can be improved.

[0048] 3. Cluster merging strategy

[0049] Improve the rationality of clustering results: By judging the similarity between groups, clusters that meet the conditions are merged, which can make the clustering results more consistent with the distribution of actual data and reduce the unreasonable clustering results caused by improper selection of the initial number of clusters.

[0050] 4. Strategies for handling ungrouped fragments

[0051] Improve data utilization: By marking the ungrouped segments with the speaker labels of the previous or next grouped segments, the data utilization can be improved and the information loss caused by data loss can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic block diagram of an embodiment of a speaker differentiation method based on the speech recognition and voiceprint recognition large model Cam++ of the present application;

[0053] Figure 2 The figure is a flowchart of an embodiment of a method for speaker differentiation based on the large model Cam++ of speech recognition and voiceprint recognition of the present application. DETAILED DESCRIPTION

[0054] The present application is further described below in conjunction with the accompanying drawings, and the structure and principle of the present application are very clear to people in the field. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] Figure 1 This is a schematic block diagram of an embodiment of a speaker differentiation method based on the speech recognition and voiceprint recognition large model Cam++ of the present application, Figure 2 The figure is a flowchart of an embodiment of a method for speaker differentiation based on the large model Cam++ of speech recognition and voiceprint recognition of the present application.

[0056] The method may include the following steps:

[0057] Step S1: The start timestamp and end timestamp corresponding to each sentence in the audio are input through the speech recognition model, and the audio segment corresponding to each sentence is cut out according to the start timestamp and the end timestamp.

[0058] In step S1, step S1 is used to segment the audio into sentences. For a piece of audio, the start and end timestamps corresponding to each sentence are obtained through a large speech recognition model, and the audio segments corresponding to each sentence are segmented.

[0059] The Cam++ model uses a densely connected time-delay neural network and a context-dependent masking module to achieve efficient speaker feature extraction and pattern recognition. This enables the model to quickly and accurately identify the speaker when processing speech signals. Compared with other mainstream speaker recognition models, the Cam++ model has lower computational complexity and faster inference speed.

[0060] Step S2: Take out the audio clips whose duration is less than the preset duration threshold. In step S2, based on the length of the audio clips, take out the audio clips that are too short. Since these audio clips are too short, the voiceprint features are not obvious and need further processing. In this embodiment, the preset duration threshold can be 1 second, which is limited by the shortest audio length input by the Cam++ model (the standard Cam++ model cannot process audio clips with a duration of less than 1 second).

[0061] Step S3: Input the audio clips after step S2 into the voiceprint recognition model Cam++ to obtain the voiceprint feature vector of each audio clip.

[0062] Step S4: Obtaining the initial K-means cluster number. In step S4, the initial cluster number is calculated by the elbow rule and the silhouette rule to obtain the elbow rule cluster number and the silhouette rule cluster number, and the larger value is taken as the initial cluster number.

[0063] Specifically, the elbow rule is a heuristic method for determining the optimal number of clusters (k) in a clustering algorithm, and is commonly used in K-means clustering. The elbow rule calculates the sum of squared errors (SSE) under different numbers of clusters and plots the relationship between the number of clusters and SSE (sum of squared cluster errors). Ideally, as the number of clusters increases, SSE will gradually decrease, but after a certain point, the reduction becomes insignificant. This "elbow" is the optimal number of clusters.

[0064] The elbow rule includes:

[0065] Step S411: Select the number of clusters and calculate the SSE of different cluster numbers k;

[0066] Step S412: draw an elbow diagram, and draw the number of clusters k and the corresponding SSE value in the same diagram;

[0067] Step S413: Select the inflection point where the SSE value drops sharply and then slowly decreases as the optimal clustering number k.

[0068] If there are n data points, each data point x i Belongs to a cluster C k , then the clustering error sum of squares SSE is calculated as follows:

[0069] ;in: is the cluster centroid, C k The center of mass, x i is a data point, x j is a point in the cluster, is the square of the distance from the data point to the centroid of the cluster.

[0070] The silhouette rule is a metric used to evaluate clustering quality, especially to determine whether a data point belongs to a given cluster. This method combines two metrics: intra-cluster similarity and inter-cluster difference, to measure the clustering quality of each data point and finally calculate the overall clustering effect. The closer the overall silhouette coefficient is to 1, the better the clustering effect.

[0071] Step S421: Calculate the intra-cluster similarity. Calculated by:

[0072] , where Ci is the cluster containing data point i, is the distance between data point i and data point j;

[0073] Step S422: Inter-cluster differences Calculated by:

[0074] , where Ck is a cluster different from Ci, is the distance between data point i and point j in cluster Ck;

[0075] Step S423: Based on the intra-cluster similarity The difference between clusters and , the silhouette coefficient of data point i is calculated by :

[0076] ; Silhouette coefficient The value range is [-1,1].

[0077] : Indicates that the data point is clustered correctly;

[0078] :It is difficult to determine the ownership of this data point;

[0079] : Indicates that the data point is incorrectly assigned to the current cluster.

[0080] Step S424. Average the silhouette coefficients of all data points to obtain the overall clustering effect in the following manner:

[0081] , where n is the total number of data points, is the silhouette coefficient for each data point.

[0082] Step S5: Initial K-means clustering, extracting the deviated abnormal feature vectors. In step S5, the voiceprint feature vectors are clustered by K-means based on the Euclidean distance according to the initial number of clusters, and the deviated abnormal feature vectors in each group are extracted according to the similarity within the cluster. At the same time, according to the number of clusters k, the voiceprint feature vector group is labeled with a speaker, and the speaker label is a number i, 1≤i≤K, and i is an integer.

[0083] Step S6: Perform secondary processing on the extracted deviated abnormal feature vectors through step S4 and step S5. In step S6, all the extracted deviated abnormal feature vectors are processed again through step S4 and step S5, and the still deviated abnormal feature vectors are extracted.

[0084] Step S7: Clustering and merging the groups obtained in step S5 and step S6. In step S7, the groups obtained by clustering in step S5 and step S6 are subjected to inter-group similarity judgment based on the difference between clusters. If the average inter-group similarity is greater than or equal to the preset inter-group similarity threshold, the two groups are merged, and the two speaker labels corresponding to the two groups are merged into one speaker label. In this embodiment, the inter-group similarity threshold can be 0.5, which is the threshold with the best effect selected after multiple experiments based on the experimental data set.

[0085] Step S8: Process all ungrouped segments.

[0086] Specifically, in step S8, for all the segments taken out in step S2 and step S6, the speaker labels of the previous segment and the next segment are marked.

[0087] If the previous segment is a grouped segment, the speaker label of the segment is used as the previous speaker label. If the previous segment is an ungrouped segment, the search is continued forward until a grouped segment is found and its speaker label is used; or if the next segment is a grouped segment, the speaker label of the segment is used as the next speaker label. If the next segment is an ungrouped segment, the search is continued backward until a grouped segment is found and its speaker label is used.

[0088] Specifically, the present application first calculates the initial number of K-means clusters by the elbow rule and the contour rule, obtains the number of elbow rule clusters and the number of contour rule clusters, and takes the larger value as the initial number of K-means clusters. The purpose of selecting a larger value as the initial number of clusters here is to group as many groups as possible. As the number of groups increases, the number of voiceprint feature vectors in each group will decrease, resulting in one actual speaker corresponding to multiple groups, and thus one actual speaker cannot correspond to one group (step S7 solves this problem); on the contrary, if the number of groups is very small, it will lead to the problem that the voices of different people are grouped into one group and cannot be distinguished. Then merge according to the similarity between groups.

[0089] Then, according to the similarity within the cluster, the deviated abnormal feature vectors in each voiceprint feature vector group are extracted, and then the above-mentioned grouping operation is performed on these deviated abnormal feature vectors, and then a batch of still deviated abnormal feature vectors are extracted to improve the accuracy of classification.

[0090] In the first clustering, the existing vectors will be grouped by clustering according to the cluster number K. For these K groups, speaker labels will be added from 1 to K (in random order), which means that the i-th group (1≤i≤K) is the i-th speaker. In other words, for the voiceprint feature vectors that have been grouped, each group corresponds to a speaker label. In step S7, the inter-group similarity is judged based on the difference between clusters. If the average similarity between two groups is greater than or equal to the preset inter-group similarity threshold (that is, one speaker actually corresponds to two groups), the two groups are merged, and the speaker labels of the two groups are merged at the same time.

[0091] Since the difference between the voiceprint feature vector of the abnormal feature vector and other vectors is too large, the reference value of the result is weak when calculating the similarity. If the abnormal feature vector is processed first and manually supplemented with grouping, and then the similarity between groups is calculated, the accuracy of the similarity calculation between groups will be affected. Therefore, only after the inter-group merging is completed, the clustering and grouping of vectors other than the abnormal feature vector is truly completed, and step S8 can be performed to process the abnormal vector. This is the reason why step S7 is placed before step S8.

[0092] After executing step S7, at this time, for the voiceprint feature vectors that have been grouped, each group corresponds to a speaker label; the purpose of this application is to correspond all grouped and ungrouped segments to speaker labels, and for the voiceprint feature vectors that have not yet been grouped (the segments taken out in steps S2 and S6), step S8 will label them.

[0093] In step S8, if the previous segment is a grouped segment, the speaker label of the segment is used as the previous speaker label; if the previous segment is an ungrouped segment, the search is continued forward until a grouped segment is found and its speaker label is used; or if the next segment is a grouped segment, the speaker label of the segment is used as the next speaker label; if the next segment is an ungrouped segment, the search is continued backward until a grouped segment is found and its speaker label is used.

[0094] Generally speaking, the probability that a segment and the segments on both sides belong to the same speaker is relatively high. Although this method may not be 100% accurate, compared to losing the labels of this part of the audio segment, this approach is chosen to ensure the integrity of all segments. If this part of the audio segment is directly lost, the accuracy of distinguishing this part of the audio segment will directly drop to 0, which is definitely the lowest.

Claims

1. A speaker differentiation method based on the large model Cam++ of speech recognition and voiceprint recognition, characterized in that: The method comprises: Step S1: obtaining the start timestamp and end timestamp corresponding to each sentence in the input audio through the speech recognition large model Cam++, and dividing the audio segment corresponding to each sentence according to the start timestamp and end timestamp; Step S2: extracting audio clips whose duration is less than a preset duration threshold; Step S3: input the audio clips after step S2 into the voiceprint recognition large model Cam++ to obtain the voiceprint feature vector of each audio clip; Steps S1 to S3 are used to obtain the voiceprint feature vector of each audio clip; Step S4: obtaining the initial number of K-means clusters, wherein the step S4 includes calculating the initial number of K-means clusters by using the elbow rule and the silhouette rule, obtaining the number of elbow rule clusters and the number of silhouette rule clusters, and taking the larger value as the initial number of K-means clusters; Step S5: initial K-means clustering, extracting the deviated abnormal feature vectors, said step S5 includes performing K-means clustering on the voiceprint feature vectors based on the Euclidean distance according to the initial K-means clustering number, obtaining K voiceprint feature vector groups, and at the same time, marking the voiceprint feature vector groups with speaker labels according to the initial K-means clustering number K, wherein the speaker label is a number i, 1≤i≤K, i is an integer, and extracting the deviated abnormal feature vectors in each voiceprint feature vector group according to the intra-cluster similarity; Step S6: performing secondary processing on the extracted deviated abnormal feature vectors through the steps S4 and S5, wherein the step S6 includes reprocessing all the extracted deviated abnormal feature vectors through the steps S4 and S5, and extracting the abnormal feature vectors that are still deviated after the reprocessing; Steps S4 to S6 are used to perform K-means clustering on the voiceprint feature vectors based on Euclidean distance, and to extract the deviated abnormal feature vectors; Step S7: clustering and merging the groups obtained in step S5 and step S6, wherein step S7 includes performing inter-group similarity judgment on the voiceprint feature vector groups obtained by clustering in step S5 and step S6 according to the difference between clusters, and if the average similarity between two groups is greater than or equal to a preset inter-group similarity threshold, merging the two groups, and merging the two speaker labels corresponding to the two groups into one speaker label; Step S8: Processing all ungrouped segments, the step S8 includes marking the speaker labels of the previous segment and the next segment of the segment for all the segments taken out in the step S2 and the step S6, If the previous segment is a grouped segment, the speaker label of the segment is used as the previous speaker label. If the previous segment is an ungrouped segment, the search is continued forward until a grouped segment is found and its speaker label is used; or if the next segment is a grouped segment, the speaker label of the segment is used as the next speaker label. If the next segment is an ungrouped segment, the search is continued backward until a grouped segment is found and its speaker label is used.

2. The speaker differentiation method based on the speech recognition and voiceprint recognition large model Cam++ according to claim 1 is characterized in that: The elbow rule includes: Step S411: Select the number of clusters and calculate the sum of squares of cluster errors SSE for different numbers of clusters; Step S412: draw an elbow diagram, and draw the number of clusters and the corresponding cluster error sum of squares SSE value in the same diagram; Step S413: Select the inflection point where the sum of squares of clustering errors (SSE) value drops sharply and then slowly decreases as the optimal number of clusters.

3. The speaker differentiation method based on the speech recognition and voiceprint recognition large model Cam++ according to claim 2 is characterized in that: The elbow rule also includes: If there are n data points, each data point x i Belongs to a cluster C k , then the clustering error sum of squares SSE is calculated as follows: ;in: is the cluster centroid, C k The center of mass, x i is a data point, x j is a point in the cluster, is the square of the distance from the data point to the centroid of the cluster.

4. The speaker differentiation method based on the speech recognition and voiceprint recognition large model Cam++ according to claim 3 is characterized in that: The contour rules include: Step S421: Calculate the intra-cluster similarity. Calculated by: , where C i is the cluster containing data point i, is the distance between data point i and data point j; Step S422: Inter-cluster differences Calculated by: , where C k is different from C i The clustering of is the data point i and cluster C k The distance to the midpoint j; Step S423: Based on the intra-cluster similarity The difference between clusters and , the silhouette coefficient of data point i is calculated by : ; Silhouette coefficient The value range is [-1,1]: Step S424. Average the silhouette coefficients of all data points to obtain the overall clustering effect in the following manner: , where n is the total number of data points, is the silhouette coefficient for each data point.

5. The speaker differentiation method based on the speech recognition and voiceprint recognition large model Cam++ according to claim 4 is characterized in that: The step S423 further includes: : Indicates that the data point is clustered correctly; :It is difficult to determine the ownership of this data point; : Indicates that the data point is incorrectly assigned to the current cluster.

Images