Speaker log generation method based on audio-visual fusion clustering

Abstract

The invention discloses a speaker log generation method based on audio-visual fusion clustering, and aims to solve the problem that who speaks at any time in a multi-speaker scene. The method is realized by the following steps of: firstly, segmenting an audio clip by adopting an overlapping perception voice segmentation model to solve the overlapping voice problem; secondly, extracting voiceprint features of a speaker of each segment of audio and a speaking score matrix generated through face tracking and speaker detection by using an advanced speaker verification model; then, through an audio and video combined clustering method, the clustering number is optimized according to audio features and visual information, and speaker clustering is completed by using K-means clustering; experimental results show that a system adopting the method obtains the lowest log error rate (DER) on an Ego4D verification set.

Description

Technical Field

[0001] The present invention belongs to the intersection of speech signal processing and computer vision, and specifically relates to a speaker logging method based on audio-visual fusion, that is, a speaker log generation method based on audio-visual fusion clustering, which aims to identify the identities and time segments of multiple speakers and is suitable for applications such as online conference recording, voice verification and human-computer interaction. Background Art

[0002] Speaker log technology aims to solve the problem of "who speaks and when" in multi-speaker scenarios. It is an important prerequisite for downstream tasks such as online conference transcription, speaker verification, and human-computer interaction. Traditional speaker log methods usually adopt a multi-stage processing flow, including voice activity detection, voice segmentation, speaker embedding extraction and clustering. However, these methods perform poorly in the face of complex egocentric video scenes. The main reasons include:

[0003] 1. Background noise diversity: There are various background noises and volume changes in the self-centered video, which makes voice activity detection prone to errors and difficult to accurately predict the number of speakers.

[0004] 2. Poor reliability of visual information: Existing audio and video joint logging methods assume that all speakers are visible, but in egocentric videos, the speaker may leave the field of view for a short time, affecting the effectiveness of visual information.

[0005] Therefore, it is difficult for existing technologies to effectively deal with the speaker log problem in egocentric video scenarios. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method for generating speaker logs based on audio-visual fusion clustering, which can generate speaker logs more accurately. The method can also solve the robustness problem of egocentric video speaker logs in the prior art.

[0007] A speaker log generation method based on audio-visual fusion clustering, wherein the input signal in the method is a collected video signal, and the video signal includes a video image sequence frame and a corresponding audio frame;

[0008] The method comprises the following steps:

[0009] S1. Use the overlap-aware speaker segmentation model to obtain the speech segments of each speaker;

[0010] For the audio frame (i.e., audio signal), the speaker's speech segment is detected by using an overlap-aware speaker segmentation model; the speech segment also records the start time and end time of the speech segment;

[0011] S2. Extraction of speaker audio embedding:

[0012] Feed the speech segment of the activity in step 1 into the speaker verification model to extract the audio embedding, that is, extract the speaker's voiceprint feature vector;

[0013] S3. Active speaker detection:

[0014] The video signal is processed by the face detection and tracking algorithm and the active speaker detection model to obtain the speaking score matrix A[i][j] of each speaker, i.e., the visual embedding;

[0015] The speaking score matrix A[i][j] represents the probability that the speaker with id i speaks in the jth video frame;

[0016] S4. Audio-visual clustering:

[0017] Both audio embedding and visual embedding are fed into the clustering module for processing;

[0018] The clustering module is implemented using the following steps:

[0019] 4.1. Speaker Quantity Prediction: Using Visual Embedding to Calculate the Probability p of the Total Number of Speakers number , predicting the number of potential speakers by means of the probabilities;

[0020] 4.2. Optimizing the number of clusters: Based on the optimization algorithm, combined with the probability of the number of speakers p number The eigenvalue difference vector e obtained by audio embedding gap , determine the optimal number of speakers K av ;

[0021] 4.3.K-means clustering: Based on the optimal number of speakers K av , apply K-means clustering algorithm to assign audio clips to K av In the clusters, logs are formed;

[0022] The final log result is a list of triples represented by the [speaker id, start time, end time] data structure.

[0023] In step S1, a 5-second sliding window is used to pass the audio signal through an overlapping perceptual speaker segmentation model to detect the speaker's speech segment.

[0024] In step S2, each speech segment obtained in step S1 is used as the input of the CAM++ speaker verification model [H.Wang, S.Zheng, Y.Chen, L.Cheng, and Q.Chen, “Cam++: Afast and efficient network for speaker verification using context-awaremasking,” arXiv preprint arXiv:2303.00332, 2023.], and the output is in is the speaker audio embedding of the ith audio segment, M is the number of speech segments, and d = 192 is the feature dimension.

[0025] In step S3, based on the video signal, the face detection algorithm S3FD [S. Zhang, X. Zhu, Z. Lei, H. Shi, X. Wang, and S. Z. Li, "S3fd: Single shot scale-invariant face detector," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 192–201] is used to detect the face of the speaker in the video, and the ResNet algorithm [He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.] is used to perform global face tracking, and the same speaker at different times in the video is marked with a unique ID. Finally, the active speaker detection model (Light-ASD) [J. Liao, H. Duan, K. Feng, W. Zhao, Y. Yang, and L. Chen, "A light weight model for active speaker detection,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 22 932–22 941.] The speaking probability of each speaker in each video frame is obtained, that is, the speaking score matrix of each speaker is obtained. Where N is the number of speakers detected, and T is the number of video frames;

[0026] Where A[i][j] represents the probability that the speaker with ID i speaks in the jth video frame.

[0027] In step S4.1, the maximum value p of each row is calculated for the speech score matrix A obtained in S3. spk As the probability that each speaker has spoken in the video, the speaker number prediction step calculates the probability by the following formula

[0028]

[0029] where p number [n]∈[0,1] represents the probability that there are n speakers in total, represents the set of all possible combinations of selecting N individuals from n individuals; by using this probability, the number of potential speakers can be predicted.

[0030] In step S4.2, the optimization algorithm is an adaptive spectral clustering speaker log method based on normalized maximum eigenvalue difference;

[0031] According to [TJPark, KJHan, M.Kumar, S.Narayanan, Auto-tuning spectral clustering for speaker diarization using normalized maximum eigenvalue difference, IEEE Signal Processing Letters 27 (2019) 381–385.] Adaptive spectral clustering speaker log method based on normalized maximum eigenvalue difference: construct the similarity matrix E with the cosine similarity of the speaker audio embedding E of all segments sim ; E sim With E sim The average value matrix is ​​obtained by averaging the transpose of Then, the matrix Construct the Laplace matrix L and perform singular value decomposition (SVD) to obtain the eigenvalue Σ and eigenvector U. Finally, the eigenvalue difference vector e is obtained from Σ. gap The step of optimizing the number of clusters is to use the weight coefficient λ to adjust p" u$％e r and the eigenvalue difference vector e gap Perform weighted averaging to determine the optimal number of speakers K av ; The calculation formula of the optimization algorithm is as follows:

[0032]

[0033] After experimental testing, lambda is 0.7 for the best effect. argmax means finding the index position of the maximum value in a given function or array. The softmax function is used to map a vector to a probability distribution so that the output value is between [0,1] and the sum of all outputs is 1.

[0034] In step S4.3, the K-means clustering step is to determine the optimal number of speakers K av After that, the K before U av The K-means clustering algorithm is applied to the feature vectors; by clustering, the audio clips are assigned to K av clusters, that is, assigned to K av different speakers;

[0035] Then the final speaker log is obtained; the speaker log is represented by a list of triples of [speaker number, start time, end time].

[0036] Compared with single speech recognition, the fusion of image recognition in the present invention can obtain a more accurate prediction of the number of clusters (ie, the number of speakers), which is helpful for subsequent clustering results.

[0037] Beneficial effects:

[0038] The present invention discloses a speaker log generation method based on audio-visual fusion clustering, aiming to solve the problem of "who speaks when" in multi-speaker scenarios. The method is implemented by the following steps: first, an overlap-aware speech segmentation model is used to segment audio clips to solve the problem of overlapping speech; second, an advanced speaker embedding model is used to extract audio and a speaking score matrix generated by face tracking and speaker detection; then, an audio-video joint clustering method is used to optimize the number of clusters according to audio features and visual information, and K-means clustering is used to complete speaker clustering; experimental results show that the system using this method achieves the lowest log error rate (DER) on the Ego4D validation set. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the overall process of the speaker log generation method based on audio-visual fusion clustering of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0041] Example 1: The entire process of the speaker log generation method based on audio-visual fusion clustering is as follows: Figure 1 The specific steps are as follows:

[0042] 1. Overlap-aware speaker segmentation

[0043] The present invention adopts an advanced overlap-aware speaker segmentation model, document A. Plaquet and H. Bredin, "Powerset multi-class cross entropy loss for neural speaker diarization," arXiv preprint arXiv:2310.13025, 2023. The model processes the audio signal through a local sliding window. Specifically, the audio signal is segmented in a manner where each window length is 5 seconds and the step length is 0.5 seconds, which is used as the input of the model. The model outputs a set of audio clips, each of which indicates one or more speakers associated with it. It should be particularly pointed out that the overlap-aware speaker segmentation model used in the present invention is based on the power set multi-class formula, which has the advantage of significantly improving performance in overlapping speech scenarios compared to the traditional multi-label formula. The present invention inputs the audio signal into the model, and finally obtains the start time of each speech segment after processing, which is represented by T in the present invention.

[0044] 2. Audio-visual speaker embedding generation

[0045] 1) Speaker audio embedding: The speaker audio embedding model used in the present invention is CAM++[H.Wang,S.Zheng,Y.Chen,L.Cheng,and Q.Chen,“Cam++:Afast and efficient network for speaker verification using context-awaremasking,”arXiv preprint arXiv:2303.00332,2023.], which is an efficient speaker verification network that combines the advantages of ECAPA-TDNN[B.Desplanques,J.Thienpondt,and K.Demuynck,“Ecapa-tdnn:Emphasized channelattention,propagation and aggregation in tdnn based speaker verification,”arXiv preprint arXiv:2005.07143,2020.] and vanilla TDNN. The architecture uses a densely connected time delay neural network (D-TDNN) [Y.-Q.Yu and W.-J.Li, "Densely connected time delay neural network for speaker verification." in Interspeech, 2020, pp. 921–925.] as the backbone and combines it with a new context-aware masking (CAM) module. The CAM++ model receives as input a sequence of audio segments T obtained in step 1 and outputs in is the speaker audio embedding of the ith audio segment, M is the number of speech segments, and d = 192 is the feature dimension.

[0046] 2) Speaker visual embedding: In addition to traditional audio-based features, the present invention also proposes audiovisual speaker embedding to integrate visual information for subsequent speaker clustering. Inspired by Talknet [R.Tao, Z.Pan, RKDas, X.Qian, MZShou, and H.Li, “Issomeone speaking? exploring long-term temporal features for audio-visual active speaker detection,” in Proceedings of the 29th ACMinternational conference on multimedia, 2021, pp.3927–3935], the present invention first performs face tracking, associates the detected facial bounding boxes across time, and assigns speaker IDs. Then, the present invention uses Light-ASD

[25] to predict the speaking scores of all people in the entire video. Afterwards, the present invention can obtain a speaking score matrix Where N is the number of people detected and T is the number of video frames.

[0047] 3. Audio and Video Joint Clustering

[0048] Traditional pure audio methods usually use spectral clustering for speaker diarization, which can automatically determine the number of clusters based on eigenvalue differences [TJPark, N.Kanda, D.Dimitriadis, KJHan, S.Watanabe, and S.Narayanan, "A review of speaker diarization: Recent advances with deep learning," Computer Speech & Language, vol. 72, p. 101317, 2022.]. Simply put, first calculate the speaker audio embedding E of all segments, construct the cosine similarity and construct the similarity matrix E sim Then, based on E sim A trimmed version of Construct the Laplace matrix L and perform singular value decomposition (SVD) to obtain the eigenvalues ​​Σ and eigenvectors U. Finally, the eigenvalue difference vector e is obtained from Σ gap .

[0049] However, it is prone to a certain degree of deviation under complex background noise. In this work, we propose an audio-visual clustering algorithm that combines visual information to achieve more robust clustering. As described in the previous section, we use Light-ASD to predict the speaking scores of all people at different times. Then, the highest score of each person at all times is used as the probability that they speak in the video. We denote the speaking probability of the i-th person in the video as p spk [i], which is obtained by the maximum speaking score of the person in all segments. The present invention can then calculate a series of probabilities about the total number of speakers As shown below:

[0050]

[0051] Among them, p number [n]∈[0,1] represents the probability that there are n speakers in total, Represents the set of all possible combinations of selecting N individuals from n individuals.

[0052] Combined with p number and e gap The present invention develops a method for determining the optimal number of speakers K by optimization. av The algorithm:

[0053]

[0054] Where λ is the weight coefficient.

[0055] Finally, the present invention has a top K av K-means clustering is applied to the feature vector to assign the audio clips to K av Based on the clustering results, the embeddings of the segments belonging to the same cluster are averaged to obtain the representative speaker embedding of the cluster.

[0056] Experimental Results

[0057] 1. Dataset

[0058] To evaluate the effectiveness of our approach, we perform experiments on EGO4D, a public dataset commonly used for speaker logging tasks from an egocentric perspective. This dataset captures both video and audio of daily activities from the egocentric perspective of the participant. The Ego4D dataset contains an audio-visual logging benchmark with approximately 50 hours of annotated data to support these tasks, including 389 / 50 / 133 clips, annotated with 34 / 5 / 11 hours of train / train / test videos. Each clip is 5 minutes long. We report the performance on the validation set of the audio-visual logging benchmark. See the literature, K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu et al., "Ego4d: Around the world in 3,000 hours of egocentric video," in Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition,2022,pp.18995–19 012.

[0059] 2. Evaluation Metrics

[0060] The present invention uses the well-studied log error rate (DER) as an evaluation metric. DER is calculated as the sum of the durations of three different errors: false alarms (FA), missed detections (MISS), and speaker confusion (CONF), and then divided by the total duration of the speech. Lower DER indicates better performance. The present invention is evaluated using the evaluation toolkit of pyannote.audio 2.1. See the literature, H.Bredin, "pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe," in 24th INTERSPEECH Conference (INTER-SPEECH 2023). ISCA, 2023, pp. 1983–1987. Performance Comparison

[0061] The model of the proposed method is compared with the most advanced methods in recent times, as follows:

[0062] Intel[K.Min,“Intel labs at ego4d challenge 2022:A better baseline for audio-visual diarization,”arXiv preprint arXiv:2210.07764,2022.] improved speech active speaker detection using an improved training scheme and a pre-trained SPELL[K.Min,S.Roy,S.Tripathi,T.Guha,and S.Majumdar,“Learning long-term spatial-temporal graphs for active speaker detection,”inEuropean Conference on Computer Vision.Springer,2022,pp.371–387.]model.

[0063] STHG[K.Min, “Sthg: Spatial-temporal heterogeneous graph learning for advanced audio-visual diarization,” arXiv preprint arXiv:2306.10608, 2023.] uses a unified heterogeneous graph learning framework to model all speakers in a video.

[0064] 3D-Speaker[Y.Chen,S.Zheng,H.Wang,L.Cheng,T.Zhu,C.Song,R.Huang,Z.Ma,Q.Chen,S.Zhang et al.,“3d-speaker-toolkit:An open source toolkit for multi-modal speaker verification and diarization,”arXivpreprint arXiv:2403.19971,2024.] is an open source toolkit that combines audio and visual information to verify and separate speaker identities through deep learning methods.

[0065] Pyannote-audio-2.1[H.Bredin, “pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe,” in 24th INTERSPEECH Conference (INTER-SPEECH 2023). ISCA, 2023, pp. 1983–1987.] An end-to-end pipeline design that combines acoustic features and speaker identity information to achieve multi-speaker separation in audio.

[0066] Powerset[A.Plaquet and H.Bredin, "Powerset multi-class cross entropyloss for neural speaker diarization," arXiv preprint arXiv:2310.13025, 2023.] is used for the task of neural network speaker separation, which improves the classification accuracy in multi-speaker audio separation by encoding all possible combinations of speakers as a set of classes.

[0067] (1) Performance of different methods on the EGO4D (v2) validation set:

[0068] Table 1: Performance comparison of different methods on the EGO4 validation set.

[0069]

[0070]

[0071] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A speaker log generation method based on audio-visual fusion clustering, characterized in that: The input signal in this method is a captured video signal, which includes a video image sequence frame and a corresponding audio frame; The method comprises the following steps: S1. Use the overlap-aware speaker segmentation model to obtain the speech segments of each speaker; For the audio frame (i.e., audio signal), the speaker's speech segment is detected by using an overlap-aware speaker segmentation model; the speech segment also records the start time and end time of the speech segment; S2. Extraction of speaker audio embedding: Feed the speech segment of the activity in step 1 into the speaker verification model to extract the audio embedding, that is, extract the speaker's voiceprint feature vector; S3. Active speaker detection: The video signal is processed by the face detection and tracking algorithm and the active speaker detection model to obtain the speaking score matrix A[i][j] of each speaker, i.e., the visual embedding; The speaking score matrix A[i][j] represents the probability that the speaker with id i speaks in the jth video frame; S4. Audio-visual clustering: Both audio embedding and visual embedding are fed into the clustering module for processing; The clustering module is implemented using the following steps: 4.

1. Speaker Quantity Prediction: Using Visual Embedding to Calculate the Probability p of the Total Number of Speakers number , predicting the number of potential speakers by means of the probabilities; 4.

2. Optimizing the number of clusters: Based on the optimization algorithm, combined with the probability of the number of speakers p number The eigenvalue difference vector e obtained by audio embedding gap , determine the optimal number of speakers K av ; 4.3.K-means clustering: Based on the optimal number of speakers K av , apply K-means clustering algorithm to assign audio clips to K av In the clusters, logs are formed; The final log result is a list of triples represented by the [speaker id, start time, end time] data structure.

2. The speaker log generation method based on audio-visual fusion clustering according to claim 1 is characterized in that: In step S1, a 5-second sliding window is used to pass the audio signal through an overlapping perceptual speaker segmentation model to detect the speaker's speech segment.

3. The speaker log generation method based on audio-visual fusion clustering according to claim 1 is characterized in that: In step S2, each speech segment obtained in step S1 is used as the input of the CAM++ speaker verification model, and the output is in is the speaker audio embedding of the ith audio segment, M is the number of speech segments, and d = 192 is the feature dimension.

4. The speaker log generation method based on audio-visual fusion clustering according to claim 1 is characterized in that: In step S3, based on the video signal, the face detection algorithm S3FD is used to detect the speaker's face in the video, and the ResNet algorithm is used to perform global face tracking, and the same speaker at different times in the video is marked with a unique ID. Finally, the active speaker detection model (Light-ASD) is used to obtain the speaking probability of each speaker in each video frame, that is, the speaking score matrix of each speaker is obtained. Where N is the number of speakers detected, and T is the number of video frames; Where A[i][j] represents the probability that the speaker with ID i speaks in the jth video frame.

5. The speaker log generation method based on audio-visual fusion clustering according to claim 1 is characterized in that: In step S4.1, the maximum value p of each row is calculated for the speech score matrix A obtained in S3. spk As the probability that each speaker has spoken in the video, the speaker number prediction step calculates the probability by the following formula where p number [n]∈[0,1] represents the probability that there are n speakers in total, represents the set of all possible combinations of selecting N individuals from n individuals; by using this probability, the number of potential speakers can be predicted.

6. The speaker log generation method based on audio-visual fusion clustering according to claim 1 is characterized in that: In step S4.2, the optimization algorithm is an adaptive spectral clustering speaker log method based on normalized maximum eigenvalue difference; The similarity matrix E is constructed by using the cosine similarity of the speaker audio embedding E of all segments using the adaptive spectral clustering speaker log method based on the normalized maximum eigenvalue difference. sim ; E sim With E sim The average value matrix is ​​obtained by averaging the transpose of Then, the mean value matrix Construct the Laplace matrix L and perform singular value decomposition (SVD) to obtain the eigenvalue ∑ and eigenvector U. Finally, the eigenvalue difference vector e is obtained from ∑. gap The step of optimizing the number of clusters is to use the weight coefficient λ to adjust p" u$％e r and the eigenvalue difference vector e gap Perform weighted averaging to determine the optimal number of speakers K av ; The calculation formula of the optimization algorithm is as follows: argmax means finding the index position of the maximum value in a given function or array. The softmax function is used to map a vector to a probability distribution so that the output value is between [0,1] and the sum of all outputs is 1.

7. The speaker log generation method based on audio-visual fusion clustering according to claim 6 is characterized in that: In step S4.3, the K-means clustering step is to determine the optimal number of speakers K av After that, the K before U av The K-means clustering algorithm is applied to the feature vectors; through clustering, the audio clips are assigned to K av clusters, that is, assigned to K av different speakers; Then the final speaker log is obtained; the speaker log is represented by a list of triples of [speaker number, start time, end time].

Images