A method for generating a speaker diary based on audio-visual fusion clustering

Abstract

The application discloses a speaker diary generation method based on audio-visual fusion clustering, and aims to solve the problem of "who speaks at what time" in a multi-speaker scene. The method is realized through the following steps: first, an overlapping-aware speech segmentation model is used to segment the audio segment, solving the problem of overlapping speech; second, an advanced speaker verification model is used to extract the speaker voiceprint features of each audio segment and a speaker score matrix generated through face tracking and speaker detection; then, through an audio-video joint clustering method, the number of clusters is optimized according to the audio features and visual information, and K-means clustering is used to complete speaker clustering; the experimental results show that the system adopting the method achieves the lowest diary error rate (DER) on the Ego4D validation set.

Description

TECHNICAL FIELD

[0001] The present application belongs to the cross field of speech signal processing and computer vision, and specifically relates to a speaker diarization method based on audio-visual fusion, namely a speaker diarization method based on audio-visual fusion clustering, aiming to identify the identities and time segments of multiple speakers, and suitable for online conference recording, speech verification and human-computer interaction applications. BACKGROUND

[0002] Speaker diarization technology aims to solve the problem of "who speaks at what time" in a multi-speaker scenario, and is an important prerequisite for downstream tasks such as online conference transcription, speaker verification and human-computer interaction. Traditional speaker diarization methods usually adopt a multi-stage processing flow, including speech activity detection, speech segmentation, speaker embedding extraction and clustering. However, these methods perform poorly when faced with complex egocentric video scenarios, mainly due to the following reasons:

[0003] 1. Background noise diversity: There are various background noises and volume changes in egocentric videos, which can lead to errors in speech activity detection and make it difficult to accurately predict the number of speakers.

[0004] 2. Poor reliability of visual information: Existing audio-video joint diarization methods assume that all speakers are visible, but in egocentric videos, speakers may temporarily leave the field of view, affecting the effectiveness of visual information.

[0005] Therefore, existing technologies are difficult to effectively address the speaker diarization problem in egocentric video scenarios. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a speaker diarization method based on audio-visual fusion clustering, which can more accurately generate speaker diarization. The present method can also solve the robustness problem of speaker diarization in egocentric videos in the prior art.

[0007] A speaker diarization method based on audio-visual fusion clustering, the input signal in the present method is a collected video signal, which includes image sequence frames of the video and corresponding audio frames;

[0008] The present method includes the following steps:

[0009] S1. A speaker segmentation model with overlap perception is used to obtain the speech segments of each speaker;

[0010] For the audio frames (i.e. audio signals), a speaker segmentation model with overlap perception is used to detect the speech segments of the speakers; the speech segments also record the start time and end time of the speech segments;

[0011] S2. Extraction of speaker audio embedding:

[0012] sending the speech segments of step 1 activity into the speaker verification model to extract audio embeddings, i.e. the speaker's voiceprint feature vector;

[0013] S3. Active speaker detection:

[0014] processing the video signal by face detection and tracking algorithm and active speaker detection model to get the speech score matrix A[i][j] of each speaker, i.e. visual embedding;

[0015] The speech score matrix A[i][j] represents the probability of the speaker with id i speaking in the jth video frame;

[0016] S4. Audio-visual clustering:

[0017] sending both audio embeddings and visual embeddings into the clustering module for processing;

[0018] The clustering module implements the following steps:

[0019] 4.1. Speaker number prediction: using visual embeddings to calculate the probability p about the total number of speakers number , predicting the potential number of speakers through the probability;

[0020] 4.2. Optimize clustering number: based on optimization algorithm, combining speaker number probability p number and feature value difference vector e gap obtained by audio embedding, determine the best speaker number K av ;

[0021] 4.3. K-means clustering: according to the determined best speaker number K av , apply K-means clustering algorithm to assign audio segments to K av clusters, forming a log;

[0022] The final log result is a list of triplets represented by [speaker id, start time, end time] data structure.

[0023] In step S1, a 5-second sliding window is used, and the audio signal is processed by the overlap perception speaker segmentation model to detect the speech segments of the speaker.

[0024] In step S2, each speech segment obtained in step S1 is taken as the input of the CAM++ speaker verification model [H. Wang, S. Zheng, Y. Chen, L. Cheng, an Q. Chen, “Cam++: A fast and efficient network for speaker verification using context-aware masking,” arXiv preprint arXiv:2303.00332, 2023.] and the output is wherein is the speaker audio embedding of the i-th audio segment, M is the number of speech segments, and d = 192 is the feature dimension.

[0025] In step S3, based on the video signal, a face detection algorithm S3FD [S. Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, and S. Z. Li, “S3fd: Single shot scale-invariant face detector,” in Proceedings of the IEEE international conference on computer vision, 2017, pp. 192-201] is used to detect the speaker faces in the video, and a ResNet algorithm [He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016:770-778.] is used for global tracking of the faces to mark the same speaker at different times in the video with a unique ID, and finally an active speaker detection model (Light-ASD) [J. Liao, H. Duan, K. Feng, W. Zhao, Y. Yang, and L. Chen, “A light weight model for active speaker detection,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 932-22 941.] is used to obtain the speaking probability of each speaker in each video frame, i.e., to obtain the speaking score matrix of each speaker where N is the number of detected speakers, and T is the number of video frames.

[0026] where A[i][j] represents the probability of the speaker with ID i speaking in the jth video frame.

[0027] In step S4.1, the maximum value p of each row of the speech score matrix A obtained in S3 is calculated spk As the probability of each speaker speaking in the video, the speaker number prediction step calculates the probability by the following formula

[0028]

[0029] where p number [n]∈[0,1] represents the probability of a total of n speakers, represents a set of all possible combinations of selecting N individuals from n individuals; by using this probability, the potential number of speakers can be predicted.

[0030] In step S4.2, the optimization algorithm is an adaptive spectral clustering speaker diarization method based on normalized maximum eigenvalue difference;

[0031] According to [T. J. Park, K. J. Han, M. Kumar, S. Narayanan, Auto-tuning spectral clustering for speaker diarization using normalized maximum eigengap, IEEE Signal Processing Letters 27 (2019) 381-385.], the adaptive spectral clustering speaker diarization method based on normalized maximum eigenvalue difference: construct a similarity matrix E sim from the cosine similarity of the speaker audio embedding E of all segments; sim Take the average of E sim and the transpose of E Then, construct the Laplacian matrix L with the matrix , and perform singular value decomposition (SVD) to obtain the eigenvalues Σ and the eigenvectors U, and finally obtain the eigenvalue difference vector e gap from ∑, and the optimization clustering number step determines the optimal number of speakers K u$％e by weighted averaging p gap r and the eigenvalue difference vector e av ; The calculation formula of the optimization algorithm is as follows:

[0032]

[0033] Wherein, the value of lambda is tested by experiment, and the optimal value is 0.7. argmax means finding the index position of the maximum value in the given function or array, and the softmax function is used to map a vector to a probability distribution, so that the output value is between [0, 1], and the sum of all outputs is 1.

[0034] In step S4.3, the K-means clustering step is to determine the optimal number of speakers K av After that, the K-means clustering algorithm is applied to the first K av characteristic vectors of U; through clustering, the audio segments are assigned to K av clusters, that is, to K av different speakers;

[0035] Further, the final speaker log is obtained; the speaker log is embodied in a list of triplets [speaker serial number, start time, end time].

[0036] Compared with voice recognition alone, the fusion of image recognition in the application can obtain a more accurate prediction of the number of clusters (i.e., the number of speakers), which helps the subsequent clustering results.

[0037] Advantages:

[0038] The application discloses a speaker log generation method based on audio-visual fusion clustering, and aims to solve the problem of "who is speaking at what time" in a multi-speaker scene. The method is realized through the following steps: first, an overlap-aware speech segmentation model is used to segment the audio segments, solving the problem of overlapping speech; second, an advanced speaker embedding model is used to extract audio and a speaker score matrix generated by face tracking and speaker detection; then, an audio-video joint clustering method is used to optimize the number of clusters according to audio features and visual information, and K-means clustering is used to complete speaker clustering; the experimental results show that the system using the method achieves the lowest log error rate (DER) on the Ego4D validation set. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a general flowchart of the speaker log generation method based on audio-visual fusion clustering of the application. DETAILED DESCRIPTION

[0040] The application will be further described in detail below in combination with the drawings and specific embodiments:

[0041] Embodiment 1: The flow of the entire method of the speaker log generation method based on audio-visual fusion clustering is as Figure 1 described, and the specific steps are as follows:

[0042] 1. Overlap-aware speaker segmentation

[0043] The present application adopts an advanced overlap-aware speaker segmentation model, A. Plaquet and H. Bredin, "Powerset multi-class cross entropy loss for neural speaker diarization," arXiv preprint arXiv:2310.13025, 2023. The model processes the audio signal through local sliding windows. Specifically, the audio signal is split into windows with a length of 5 seconds and a step of 0.5 seconds, which are used as the input of the model. The output of the model is a set of audio segments, each of which indicates one or more speakers associated with it. It should be particularly pointed out that the overlap-aware speaker segmentation model used in the present application is based on the powerset multi-class formula, which has the advantage of significantly improving performance in overlapping speech scenarios compared to traditional multi-label formulas. The present application inputs the audio signal into the model, and finally obtains the starting time of each speech segment, which is represented by T in the present application.

[0044] 2. Audiovisual speaker embedding generation

[0045] 1) Speaker audio embedding: The speaker audio embedding model used by the present invention is CAM++ [H. Wang, S. Zheng, Y. Chen, L. Cheng, and Q. Chen, “Cam++: A fast and efficient network for speaker verification using context-aware masking,” arXiv preprint arXiv:2303.00332, 2023.], which is a high-efficiency speaker verification network that combines the advantages of ECAPA-TDNN [B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” arXiv preprint arXiv:2005.07143, 2020.] and vanilla TDNN. The architecture uses a densely connected time delay neural network (D-TDNN) [Y.-Q. Yu and W.-J. Li, “Densely connected time delay neural network for speaker verification.” in Interspeech, 2020, pp. 921-925.] as the backbone and incorporates a new context-aware masking (CAM) module. The CAM++ model receives the series of audio segments T obtained in step one as input, and the output is where is the speaker audio embedding of the i-th audio segment, M is the number of speech segments, and d = 192 is the feature dimension.

[0046] 2) Speaker visual embedding: In addition to traditional audio-based features, this invention proposes audiovisual speaker embedding to integrate visual information for subsequent speaker clustering. Inspired by Talknet [R. Tao, Z. Pan, RK Das, X. Qian, MZ Shou, and H. Li, “Is someone speaking? exploring long-term temporal features for audio-visual active speaker detection,” in Proceedings of the 29th ACM international conference on multimedia, 2021, pp. 3927–3935], this invention first performs face tracking, associates the detected face bounding boxes across time, and assigns speaker IDs. Then, this invention uses Light-ASD

[25] to predict the speaking scores of all people in the entire video. After that, this invention can obtain a speaking score matrix. Where N is the number of people detected, and T is the number of video frames.

[0047] 3. Audio and video joint clustering

[0048] Traditional pure audio methods typically use spectral clustering for speaker diarization, which can automatically determine the number of clusters based on eigenvalue differences [TJPark, N. Kanda, D. Dimitriadis, KJHan, S. Watanabe, and S. Narayanan, “A review of speaker diarization: Recent advances with deep learning,” Computer Speech & Language, vol. 72, p. 101317, 2022.]. Simply put, it first calculates the speaker audio embedding E for all segments, then constructs a cosine similarity matrix E. sim Then, then, based on E sim Trimmed version Construct the Laplacian matrix L and perform singular value decomposition (SVD) to obtain the eigenvalues ​​Σ and eigenvectors U. Finally, obtain the eigenvalue difference vector e from Σ. gap .

[0049] However, it is prone to some degree of bias under complex background noise. In this work, the present invention proposes an audiovisual clustering algorithm that combines visual information to achieve more robust clustering. As mentioned in the previous section, the present invention uses Light-ASD to predict the speaking score of all people at different time instants. Then, the highest score of each person at all time instants is taken as their probability of speaking in the video. The present invention represents the speaking probability of the i-th person in the video as p spk [i] which is obtained by the maximum speaking score of this person in all segments. Then, the present invention can calculate a series of probabilities about the total number of speakers As shown below:

[0050]

[0051] where p number [n] ∈ [0, 1] represents the probability of a total of n speakers, represents the set of all possible combinations of selecting N individuals from n individuals.

[0052] Combining p number and e gap The present invention develops an algorithm to determine the optimal number of speakers K av by optimization:

[0053]

[0054] where λ is the weight coefficient.

[0055] Finally, the present invention applies K-means clustering to the top K av feature vectors of U, and assigns the audio segments to K av clusters. Based on the clustering results, the embeddings of the segments belonging to the same cluster are averaged to obtain the representative speaker embedding of this cluster.

[0056] Experimental results

[0057] 1. Dataset

[0058] To evaluate the effectiveness of the present method, experiments are conducted below based on the public dataset EGO4D commonly used for the speaker diarization task from the egocentric perspective. This dataset captures video and audio of daily activities from the participant’s egocentric perspective simultaneously. The Ego4D dataset contains an audiovisual diary benchmark with approximately 50 hours of annotated data to support these tasks, including 389 / 50 / 133 clips, with 34 / 5 / 11 hours of training / training / test videos annotated. Each segment is 5 minutes long. The present invention reports the performance on the audiovisual diary benchmark validation set. See document K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu et al., “Ego4D: Around the world in 3,000 hours of egocentric video,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18995-19012.

[0059] 2. Evaluation metrics

[0060] The present invention uses the well-studied Diarization Error Rate (DER) as an evaluation metric. DER is computed as the sum of the durations of three different errors: False Alarms (FA), Misses (MISS), and Speaker Confusions (CONF), divided by the total duration of speech. A lower DER indicates better performance. The present invention uses the evaluation toolkit of pyannote.audio 2.1 for evaluation. See document H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe,” in 24th INTERSPEECH Conference (INTERSPEECH 2023). ISCA, 2023, pp. 1983-1987.

[0061] The model of the present method is compared to the most recent state-of-the-art methods, as follows:

[0062] Intel [K. Min, “Intel labs at ego4d challenge 2022: A better baseline for audio-visual diarization,” arXiv preprint arXiv:2210.07764, 2022.] improved speech active speaker detection using an improved training scheme and a pre-trained SPELL [K. Min, S. Roy, S. Tripathi, T. Guha, and S. Majumdar, “Learning long-term spatial-temporal graphs for active speaker detection,” in European Conference on Computer Vision. Springer, 2022, pp. 371-387.] model.

[0063] STHG [K. Min, “Sthg: Spatial-temporal heterogeneous graph learning for advanced audio-visual diarization,” arXiv preprint arXiv:2306.10608, 2023.] uses a unified heterogeneous graph learning framework to model all speakers in a video.

[0064] 3D-Speaker [Y. Chen, S. Zheng, H. Wang, L. Cheng, T. Zhu, C. Song, R. Huang, Z. Ma, Q. Chen, S. Zhang et al., “3d-speaker-toolkit: An open source toolkit for multi-modal speaker verification and diarization,” arXiv preprint arXiv:2403.19971, 2024.] is an open source toolkit that combines audio and visual information to achieve speaker identity verification and separation through deep learning methods.

[0065] Pyannote-audio-2.1 [H. Bredin, “pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe,” in 24th INTERSPEECH Conference (INTERSPEECH 2023). ISCA, 2023, pp. 1983-1987.] An end-to-end pipeline design that combines acoustic features and speaker identity information to enable multi-speaker separation in audio.

[0066] Powerset [A. Plaquet and H. Bredin, “Powerset multi-class cross entropy loss for neural speaker diarization,” arXiv preprint arXiv:2310.13025, 2023.] A neural network speaker diarization task that improves classification accuracy in multi-speaker audio separation by encoding all possible combinations of speakers as a set of classes.

[0067] (1) Performance of different methods on the EGO4D (v2) validation set:

[0068] Table 1: Performance comparison of different methods on the EGO4 validation set.

[0069]

[0070]

[0071] While the embodiments of the present application have been described in conjunction with the preferred embodiments of the application, various modifications and variations of the present application will be apparent to those skilled in the art from the teachings herein, and it is intended that the present application encompass such modifications and variations as fall within the scope of the appended claims.

Claims

1. A speaker log generation method based on audiovisual fusion clustering, characterized in that, The input signal in this method is the acquired video signal, which includes the image sequence frames of the video and the corresponding audio frames. This method includes the following steps: S1. Use an overlap-aware speaker segmentation model to obtain the speech segments of each speaker; For the aforementioned audio frame, the speaker's speech segments are detected using an overlap-aware speaker segmentation model; the speech segments also record the start and end times of the speech segments. S2. Extraction of speaker audio embedding: The speech segment obtained in step S1 is fed into the speaker verification model to extract the audio embedding, that is, to extract the speaker's voiceprint feature vector. S3. Active speaker detection: The video signal is processed by face detection and tracking algorithms and active speaker detection model to obtain the speech score matrix A[i][j] of each speaker, which is the visual embedding; The speaking score matrix A[i][j] represents the probability that the speaker with ID i speaks in the j-th video frame; S4. Audiovisual Clustering: Both audio and visual embeddings are fed into the clustering module for processing; The clustering module is implemented using the following steps: 4.

1. Speaker Count Prediction: Calculate the probability p of the total number of speakers using visual embedding. number The probability is used to predict the number of potential speakers; 4.

2. Optimize the number of clusters: Based on the optimization algorithm, combined with the probability p of the number of speakers. number The feature difference vector e obtained from audio embedding gap Determine the optimal number of speakers K av ; 4.

3. K-means clustering: Based on the determined optimal number of speakers K av The K-means clustering algorithm is applied to assign audio segments to K clusters. av Logs are generated from each cluster; The final log result is a list of triples represented by the data structure [speaker ID, start time, end time]; in step 4.1, the maximum value p of each row of the speech score matrix A obtained in S3 is calculated. spk The speaker count prediction step calculates the probability, as the probability that each speaker has spoken in the video, using the following formula. Where p number [n]∈[0,1] represents the probability that there are a total of n speakers. This represents the set of all possible combinations of selecting n individuals from N individuals; by using this probability, the number of potential speakers can be predicted; in step 4.2, the optimization algorithm is an adaptive spectral clustering speaker log method based on the normalized maximum eigenvalue difference; An adaptive spectral clustering speaker log method based on normalized maximum eigenvalue difference is adopted, and a similarity matrix E is constructed using the cosine similarity of the speaker audio embeddings E of all segments. sim E sim With E sim The average of the transposes is used to obtain the average matrix. Then, using the average matrix Construct the Laplacian matrix L, and perform singular value decomposition to obtain eigenvalues ​​Σ and eigenvectors U. Finally, obtain the eigenvalue difference vector e from Σ. gap The step of optimizing the number of clusters is to apply weight coefficient λ to p number and the difference vector of eigenvalues ​​e gap Perform a weighted average to determine the optimal number of speakers K. av The calculation formula for the optimization algorithm is as follows: argmax finds the index of the maximum value in a given function or array. The softmax function maps a vector to a probability distribution such that the output values ​​are between [0,1] and the sum of all outputs is 1.

2. The speaker log generation method based on audiovisual fusion clustering according to claim 1, characterized in that, In step S1, a 5-second sliding window is used to pass the audio signal through an overlap-perception speaker segmentation model to detect the speaker's speech segments.

3. The speaker log generation method based on audiovisual fusion clustering according to claim 1, characterized in that, In step S2, each speech segment obtained in step S1 is used as input to the CAM++ speaker verification model, and the output is... in is the speaker audio embedding of the i-th audio segment, M is the number of speech segments, and d = 192 is the feature dimension.

4. The speaker log generation method based on audiovisual fusion clustering according to claim 1, characterized in that, In step S3, based on the video signal, the S3FD face detection algorithm is used to detect the speaker's face in the video, and the ResNet algorithm is used for global face tracking. The same speaker at different times in the video is labeled with a unique ID. Finally, the active speaker detection model is used to obtain the speaking probability of each speaker in each video frame, that is, to obtain the speaking score matrix of each speaker. Where N is the number of people detected speaking, and T is the number of video frames; Where A[i][j] represents the probability that the speaker with ID i speaks in the j-th video frame.

5. The speaker log generation method based on audiovisual fusion clustering according to claim 1, characterized in that, In step S4.3, the K-means clustering step involves determining the optimal number of speakers K. av After that, for the first K of U av The K-means clustering algorithm is applied to the feature vectors; through clustering, audio segments are assigned to K... av In each cluster, i.e., assigned to K av Two different speakers; This leads to the final speaker log; the speaker log is represented by a list of triplets of [speaker number, start time, end time].

Images