A speaker segmentation and clustering method and device, electronic equipment and storage medium

Abstract

The application provides a speaker segmentation and clustering method and device, electronic equipment and storage medium, comprising: performing speech activity detection processing on the audio to be segmented to determine a plurality of valid speech segments; inputting the plurality of valid speech segments into an embedding extractor without large interval fine-tuning to extract speaker embedding vectors, and outputting an embedding vector sequence; inputting the embedding vector sequence into a hierarchical probabilistic linear discriminant analysis model, clustering each valid speech segment belonging to different speakers based on a variational Bayesian inference algorithm, and outputting a speaker segmentation and clustering result of the audio to be segmented; wherein the hierarchical probabilistic linear discriminant analysis model is a three-level structure of segment level, speaker level and conversation level, and the model is trained based on features extracted by large interval fine-tuning. Thus, the problem of mismatch between training and inference duration can be solved, and the accuracy and robustness of speaker segmentation and clustering can be improved.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to a speaker segmentation and clustering method, apparatus, electronic device, and storage medium. Background Technology

[0002] Speaker segmentation and clustering is an important speech processing technique that aims to automatically analyze a continuous audio stream containing multiple speakers, identify and label "who speaks and when," thereby achieving structured organization of the speech content. This technology is widely used in fields such as meeting recording, broadcast transcription, telephone customer service systems, and intelligent video editing, and is one of the key components in building intelligent voice interaction systems.

[0003] Current mainstream speaker segmentation systems often employ an embedding extraction + clustering framework. Among these, variational Bayesian Hidden Markov Model (LPMM) clustering based on x-vectors and Probabilistic Linear Discriminant Analysis (PLDA) are used as generative models to probabilistically model the embedding vector distributions of different speakers. However, research has revealed that traditional standard PLDA models typically employ a single-layer planar structure, modeling multiple sources of variation such as speaker, channel, and noise within a unified latent space, failing to effectively distinguish variations across different time scales. For example, short-term acoustic fluctuations (such as phoneme variations and prosodic variations) are treated equally with long-term stable speaker identity features, leading to incorrect segmentation in scenarios with rapid speaker switching or short speech segments. Furthermore, environmental differences in cross-session recordings (such as device type, background noise, and room reverberation) are often misclassified as new speakers, causing over-segmentation and severely impacting system robustness. Therefore, a speaker segmentation method is urgently needed to improve the accuracy of speaker segmentation and clustering. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a speaker segmentation and clustering method, apparatus, electronic device, and storage medium. This application introduces a three-level hierarchical PLDA model structure of "segment-speaker-conversation", which can explicitly model the sources of variation at different levels in speech data. Segment-level factors capture short-term acoustic dynamics (such as pronunciation transients and speech rate changes), speaker-level factors represent stable individual speech features across time periods, and conversation-level factors absorb conversation-specific biases caused by recording environment, equipment differences, etc. In addition, this application also adopts an asymmetric training strategy, using features extracted by large-interval fine-tuning to train the PLDA model, while using a basic embedding extractor during inference, effectively solving the problem of mismatch between training and inference time, thereby improving the accuracy and robustness of speaker segmentation and clustering.

[0005] This application provides a speaker segmentation and clustering method, which includes: Speech activity detection processing is performed on the audio to be segmented to identify multiple valid speech segments; Multiple valid speech segments are input into an embedding extractor that has not undergone large-interval fine-tuning to extract speaker embedding vectors, and an embedding vector sequence is output. The embedded vector sequence is input into a pre-trained hierarchical probabilistic linear discriminant analysis model. Based on the variational Bayesian inference algorithm, each effective speech segment is clustered to belong to different speakers, and the speaker segmentation clustering results of the audio to be segmented are output. The hierarchical probabilistic linear discriminant analysis model has a three-level structure of segment level, speaker level, and conversation level, and the model is trained based on features extracted by large-interval fine-tuning.

[0006] In one possible implementation, the variational Bayesian inference algorithm is used to cluster each valid speech segment to different speakers, and the speaker segmentation clustering results of the audio to be segmented are output, including: Based on the hierarchical probabilistic linear discriminant analysis model, the posterior probability of each effective speech segment belonging to different hidden speaker states is calculated by jointly optimizing the conversation-level, speaker-level, and segment-level latent variables through the variational Bayesian inference algorithm. The speaker label for each valid speech segment is determined based on multiple posterior probabilities, and the speaker segmentation and clustering results are output.

[0007] In one possible implementation, the step of jointly optimizing conversation-level, speaker-level, and segment-level latent variables based on the hierarchical probabilistic linear discriminant analysis model using a variational Bayesian inference algorithm to calculate the posterior probability of each effective speech segment belonging to different hidden speaker states includes: Initialize the prior distributions of the conversation-level factor, speaker-level factor, and segment-level factor; Based on the embedded vector sequence of the effective speech segments, a joint variational objective function containing conversation-level, speaker-level, and segment-level latent variables is constructed; The posterior distributions of the conversation-level factors, speaker-level factors, and segment-level factors are updated based on the variational Bayesian inference algorithm. Based on the updated session-level factor, speaker-level factor, and segment-level factor, the posterior probability of each valid speech segment belonging to each hidden speaker state is recalculated. Based on the current latent variable estimates, the spatial matrices and residual covariance matrices of the conversation-level factors, speaker-level factors, and segment-level factors are redefined until the change in the joint variational objective function is less than a preset threshold, at which point the calculation of the posterior probability is stopped.

[0008] In one possible implementation, updating the posterior distributions of the conversation-level factors, speaker-level factors, and segment-level factors based on the variational Bayesian inference algorithm includes: Based on the speech features, speaker-level factors, and segment-level factors of all the effective speech segments in the session, update the posterior distribution of the session-level factors corresponding to the session. The posterior distribution of the corresponding speaker-level factor is updated based on the embedded feature vectors of all the valid speech segments of the speaker, the corresponding conversation-level factor and segment-level factor. The posterior distribution of the segment-level factors of the effective speech segment is updated based on the acoustic features of the effective speech segment and the corresponding conversation-level and speaker-level factors.

[0009] In one possible implementation, the hierarchical probabilistic linear discriminant analysis model is determined through the following steps: The sample embedding feature vectors of a set of long-duration audio samples are organized into a dataset by session, and each sample embedding feature vector is associated with a session index, a potential speaker index, and a speech segment index. A three-layer hierarchical probabilistic linear discriminant analysis model is constructed, wherein the hierarchical probabilistic linear discriminant analysis model includes a global mean vector, a conversation-level subspace matrix, a speaker-level subspace matrix, and a segment-level subspace matrix; The hierarchical probabilistic linear discriminant analysis model is trained based on the sample embedding feature vectors of a set of long-duration audio samples. During the training process, a hierarchical constraint regularization strategy is introduced, and a variational Bayesian inference framework is adopted. The parameters of the hierarchical probabilistic linear discriminant analysis model are iteratively optimized through the expectation-maximization algorithm. The sample embedding feature vectors used to train the hierarchical probabilistic linear discriminant analysis model are extracted by an embedding extractor that has been fine-tuned with a large interval. When the change in the value of the joint variational objective function of the hierarchical probabilistic linear discriminant analysis model is less than a preset threshold, convergence is determined and the trained hierarchical probabilistic linear discriminant analysis model is identified.

[0010] In one possible implementation, the hierarchical constraint regularization strategy includes: Dynamic weight adjustment mechanism and hierarchical constraint regularization mechanism.

[0011] This application embodiment also provides a speaker segmentation and clustering device, the speaker segmentation and clustering device comprising: The semantic activity detection module is used to perform speech activity detection processing on the audio to be segmented and identify multiple valid speech segments. The feature extraction module is used to input multiple effective speech segments into an embedding extractor that has not undergone large-interval fine-tuning to extract speaker embedding vectors and output an embedding vector sequence. The clustering result determination module is used to input the embedded vector sequence into a pre-trained hierarchical probabilistic linear discriminant analysis model, and perform clustering processing on each effective speech segment belonging to different speakers based on the variational Bayesian inference algorithm, and output the speaker segmentation clustering result of the audio to be segmented; wherein, the hierarchical probabilistic linear discriminant analysis model is a three-level structure of segment level, speaker level and conversation level, and the model is trained based on features extracted by large-interval fine-tuning.

[0012] In one possible implementation, the clustering result determination module is used to perform clustering processing based on the variational Bayesian inference algorithm to assign each effective speech segment to a different speaker, and output the speaker segmentation clustering result of the audio to be segmented: Based on the hierarchical probabilistic linear discriminant analysis model, the posterior probability of each effective speech segment belonging to different hidden speaker states is calculated by jointly optimizing the conversation-level, speaker-level, and segment-level latent variables through the variational Bayesian inference algorithm. The speaker label for each valid speech segment is determined based on multiple posterior probabilities, and the speaker segmentation and clustering results are output.

[0013] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the speaker segmentation and clustering method described above are performed.

[0014] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the speaker segmentation and clustering method described above.

[0015] This application provides a speaker segmentation and clustering method, apparatus, electronic device, and storage medium. The speaker segmentation and clustering method includes: performing speech activity detection processing on the audio to be segmented to determine multiple effective speech segments; inputting the multiple effective speech segments into an embedding extractor that has not undergone large-interval fine-tuning to extract speaker embedding vectors and outputting an embedding vector sequence; inputting the embedding vector sequence into a pre-trained hierarchical probabilistic linear discriminant analysis model, and performing clustering processing based on a variational Bayesian inference algorithm to assign each effective speech segment to different speakers, and outputting the speaker segmentation and clustering result of the audio to be segmented; wherein, the hierarchical probabilistic linear discriminant analysis model has a three-level structure of segment level, speaker level, and conversation level, and the model is trained based on features extracted by large-interval fine-tuning. This application introduces a three-tiered hierarchical PLDA model structure of "segment-speaker-conversation," which can explicitly model the sources of variation at different levels in speech data. Segment-level factors capture short-term acoustic dynamics (such as articulation transients and speech rate changes), speaker-level factors represent stable individual speech features across time periods, and conversation-level factors absorb conversation-specific biases caused by recording environment, equipment differences, etc., thereby improving the accuracy of speaker segmentation and clustering. Furthermore, this application employs an asymmetric training strategy, using features extracted with large-interval fine-tuning to train the PLDA model, while using a basic embedding extractor during inference. This effectively solves the problem of mismatch between training and inference time, thereby improving the accuracy and robustness of speaker segmentation and clustering.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a speaker segmentation and clustering method provided in this application embodiment; Figure 2 This is one of the structural schematic diagrams of a speaker segmentation and clustering device provided in the embodiments of this application; Figure 3 A second schematic diagram of a speaker segmentation and clustering device provided in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in the embodiment of the application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0020] First, the applicable scenarios for this application will be introduced. This application can be applied to the field of audio processing technology.

[0021] Research has revealed that traditional PLDA models employ a single-layer planar structure, modeling multiple sources of variation such as speaker, channel, and noise within a unified latent space. This approach fails to effectively distinguish variations across different time scales. For instance, short-term acoustic fluctuations (such as phoneme variations and prosodic variations) are treated equally with long-term stable speaker identity features, leading to incorrect segmentation in scenarios with rapid speaker switching or short speech segments. Furthermore, environmental differences across conversation recordings (such as device type, background noise, and room reverberation) are often misidentified as new speakers, causing over-segmentation and severely impacting system robustness. Therefore, a new speaker segmentation method is urgently needed to improve the accuracy of speaker segmentation and clustering.

[0022] Based on this, the embodiments of this application provide a speaker segmentation and clustering method. This application introduces a three-level hierarchical PLDA model structure of "segment-speaker-conversation", which can explicitly model the sources of variation at different levels in speech data. Segment-level factors capture short-term acoustic dynamics (such as pronunciation transients and speech rate changes), speaker-level factors represent stable individual speech features across time periods, and conversation-level factors absorb conversation-specific biases caused by recording environment, equipment differences, etc., thereby improving the accuracy of speaker segmentation and clustering.

[0023] Please see Figure 1 , Figure 1 This is a flowchart illustrating a speaker segmentation and clustering method provided in an embodiment of this application. Figure 1 As shown in the embodiments of this application, the speaker segmentation and clustering method includes: S101: Perform speech activity detection processing on the audio to be segmented to identify multiple valid speech segments.

[0024] In this step, the input audio to be segmented is preprocessed. The key first step is to perform Voice Activity Detection (VAD) to identify and extract valid segments containing speech content from the original audio stream, and to exclude silence, background noise or other non-speech interference, thereby improving the accuracy and computational efficiency of subsequent feature extraction and clustering analysis.

[0025] In a specific implementation, a pre-trained speech activity detection model is used to classify each frame of the segmented audio, outputting whether the frame belongs to a speech activity region. The speech activity detection model employs a deep neural network-based binary classifier architecture, such as a convolutional neural network (CNN) or a recurrent neural network (LSTM). The input is the Mel-spectrum or filter bank features of each frame, and the output is the posterior probability that the frame belongs to "speech" or "non-speech." This model is trained under supervised conditions on a large-scale speech dataset, effectively distinguishing speech from various background noises (such as air conditioner noise, keyboard clicks, and music), and maintaining good robustness under different signal-to-noise ratios. After obtaining the frame-by-frame speech / non-speech decision results, smoothing and boundary refinement operations are performed. For example, by setting a minimum speech segment length threshold (e.g., 300ms), adjacent short speech segments are merged, and isolated speech blocks with excessively short durations are removed to avoid misclassifying transient noise as speech. Simultaneously, acoustic cues such as the energy envelope change rate and spectral slope can be used to finely adjust the speech start and end points, ensuring the accuracy of speech segment boundaries. After the above VAD processing, the original audio is divided into several valid speech segments.

[0026] S102: Input multiple valid speech segments into an embedding extractor that has not undergone large-interval fine-tuning to extract speaker embedding vectors and output an embedding vector sequence.

[0027] In this step, multiple valid speech segments are input into an embedding extractor that has not undergone large-interval fine-tuning to extract speaker embedding vectors, and the output is an embedding vector sequence.

[0028] Here, the training process of the embedding extractor is as follows: Phase 1: Basic Training: The ResNet34 network is initially trained using a large-scale labeled speaker recognition dataset (e.g., VoxCeleb1+2). This phase employs the standard AAM-Softmax loss function, where the margin parameter is set to a small value (e.g., 0.2), enabling the model to converge quickly and learn basic speaker discrimination capabilities in the early stages of training. Simultaneously, data augmentation techniques, such as velocity perturbation and additive noise injection, are introduced to improve the model's robustness to different recording conditions. The optimizer uses AdamW with an initial learning rate of 1e-3, which is decayed using cosine annealing or Step scheduling until the loss function stabilizes. Phase 2: Large Margin Fine-Tuning (LMF): After the basic model converges, the large margin fine-tuning stage is entered, aiming to further increase the inter-class distance and compress intra-class differences, thereby obtaining more discriminative embedding representations. Key improvements at this stage include: increasing the input audio length from 2 seconds to 6 seconds or longer to simulate the long-duration characteristics of continuous speech in real-world scenarios; increasing the margin parameter in AAM-Softmax to a higher value (e.g., 0.5) to impose stronger classification boundary constraints; disabling data augmentation operations such as velocity perturbations that may cause domain shifts to avoid distribution inconsistencies between training and inference; reducing the learning rate to 1 / 20 of the base stage (e.g., 5e-5) and employing a recurrent learning rate strategy to facilitate fine-tuning.

[0029] In this application, a basic embedding extractor without large-interval fine-tuning is used, i.e., a model that has only completed the first stage of training, and it is applied to embedding extraction for short speech segments (e.g., 1.5 seconds). This "asymmetric training architecture" uses LMF embeddings to train the hierarchical PLDA model during training, and uses the basic embedding extractor in conjunction with the trained PLDA for similarity calculation during inference. This effectively solves the performance degradation problem caused by the mismatch between training and testing time in traditional methods, retaining the discriminative gain brought by large-interval fine-tuning while ensuring feature consistency and robustness on short speech segments. The embedding extractor training method proposed in this application, through a two-stage progressive optimization and asymmetric application mechanism, significantly improves the discriminative ability and deployment stability of embedding vectors in complex speech environments, providing a high-quality and reliable input representation foundation for subsequent speaker clustering based on hierarchical probabilistic linear discriminant analysis models.

[0030] S103: Input the embedded vector sequence into the pre-trained hierarchical probabilistic linear discriminant analysis model, and perform clustering processing on each effective speech segment belonging to different speakers based on the variational Bayesian inference algorithm, and output the speaker segmentation clustering results of the audio to be segmented; wherein, the hierarchical probabilistic linear discriminant analysis model is a three-level structure of segment level, speaker level and conversation level, and the model is trained based on features extracted by large-interval fine-tuning.

[0031] In this step, the embedded vector sequence is input into the hierarchical probabilistic linear discriminant analysis model. The variational Bayesian inference algorithm is used to cluster each effective speech segment to different speakers, and the speaker segmentation clustering results of the audio to be segmented are output.

[0032] In one possible implementation, the variational Bayesian inference algorithm is used to cluster each valid speech segment to different speakers, and the speaker segmentation clustering results of the audio to be segmented are output, including: (1): Based on the hierarchical probabilistic linear discriminant analysis model, the posterior probability of each effective speech segment belonging to different hidden speaker states is calculated by jointly optimizing the latent variables at the conversation level, speaker level and segment level through variational Bayesian inference algorithm.

[0033] Here, based on the hierarchical probabilistic linear discriminant analysis model, the posterior probability of each effective speech segment belonging to different hidden speaker states is calculated by jointly optimizing the latent variables at the conversation level, speaker level, and segment level through the variational Bayesian inference algorithm.

[0034] It's important to clarify that "conversation-level" refers to a complete recorded event or dialogue (e.g., a meeting, a phone call, a radio program), encompassing a set of speech data collected under the same equipment, environment, and channel conditions. The introduction of "conversation-level factors" and a conversation-level subspace matrix specifically captures these non-speaker-related systematic biases. This ensures that when the same speaker appears in different sessions (e.g., different meeting rooms), the model will not incorrectly classify them as multiple speakers due to differences in recording conditions (i.e., preventing "oversegmentation"). "Speaker-level" refers to the identity characteristics of an individual speaker appearing in one or more sessions; it is a semantic entity that remains relatively stable across time periods. Speaker-level factors explicitly model the intrinsic identity characteristics of each speaker and map them to the embedding space through the speaker-level subspace matrix. This allows the same speaker to be correctly clustered into the same category even if their pronunciation varies slightly in different contexts. "Segment-level" refers to the most basic unit of time in audio, typically a short segment after dividing a continuous speech stream into fixed durations (e.g., 1.5 seconds). In hierarchical PLDA, "segment-level factors" are used to capture these local dynamic variations, avoiding misclassification as speaker differences. By introducing independent segment-level subspace matrices and segment-level factors, the model can explicitly absorb short-term fluctuations, improving its robustness to rapidly switching or fragmented speech.

[0035] In one possible implementation, the step of jointly optimizing conversation-level, speaker-level, and segment-level latent variables based on the hierarchical probabilistic linear discriminant analysis model using a variational Bayesian inference algorithm to calculate the posterior probability of each effective speech segment belonging to different hidden speaker states includes: A: Initialize the prior distributions of the conversation-level factors, speaker-level factors, and segment-level factors.

[0036] Here, a dimension is assigned to each session. The vector of conversation-level factors, assigned to each speaker with dimension [missing information]. The speaker-level factor vector is assigned a dimension of for each speech segment. The vector of segment-level factors.

[0037] B: Based on the embedded vector sequence of the effective speech segments, construct a joint variational objective function that includes conversation-level, speaker-level, and segment-level latent variables.

[0038] Here, the joint variational objective function is determined using the following formula. :

[0039] in, Represents observation data, These represent latent variables at the segment level, conversation level, and speaker level, respectively. Indicates model parameters. As a prior distribution, It is a posterior distribution.

[0040] C: The posterior distributions of the conversation-level factors, speaker-level factors, and segment-level factors are updated based on the variational Bayesian inference algorithm.

[0041] Here, the posterior distributions of the conversation-level factors, speaker-level factors, and segment-level factors are updated based on the variational Bayesian inference algorithm.

[0042] In one possible implementation, updating the posterior distributions of the conversation-level factors, speaker-level factors, and segment-level factors based on the variational Bayesian inference algorithm includes: Based on the speech features, speaker-level factors, and segment-level factors of all the valid speech segments within the session, update the posterior distribution of the session-level factors corresponding to the session; based on the embedded feature vectors of all the valid speech segments of the speaker, the corresponding session-level factors, and segment-level factors, update the posterior distribution of the corresponding speaker-level factors; based on the acoustic features of the valid speech segments and the corresponding session-level factors and speaker-level factors, update the posterior distribution of the segment-level factors of the valid speech segments.

[0043] Here, a. Conversation-level factor update: Traverse all conversations and update the conversation-level factors using Bayesian inference based on all speech features, speaker-level factors, and segment-level factors within the conversation; b. Speaker-level factor update: Update the speaker-level factors based on the features of all speech segments of the speaker and the corresponding conversation-level and segment-level factors; c. Segment-level factor update: Update the segment-level factors based on the acoustic features of the segment and the corresponding conversation-level and speaker-level factors; d. Speaker assignment probability update: Recalculate the posterior probability of each speech segment belonging to a different speaker based on the updated three-level factors.

[0044] D: Based on the updated session-level factor, speaker-level factor, and segment-level factor, recalculate the posterior probability of each valid speech segment belonging to each hidden speaker state.

[0045] Here, based on the updated session-level factor, speaker-level factor, and segment-level factor, the posterior probability of each valid speech segment belonging to each hidden speaker state is recalculated.

[0046] E: Based on the current latent variable estimates, redetermine the spatial matrices and residual covariance matrices of the conversation-level factors, speaker-level factors, and segment-level factors until the change in the joint variational objective function is less than a preset threshold, then stop calculating the posterior probability.

[0047] Here, based on the current latent variable estimates, the spatial matrices of the conversation-level factors, speaker-level factors, and segment-level factors, as well as the residual covariance matrix, are redefined until the change in the joint variational objective function is less than a preset threshold, at which point the calculation of the posterior probability stops.

[0048] The above steps are repeated, and the change in the joint variational objective function is monitored. When the improvement between successive iterations is less than a preset threshold, the change is recorded. When the algorithm converges, the calculation of the posterior probability is stopped.

[0049] (2): Determine the speaker label for each valid speech segment based on the multiple posterior probabilities, and output the speaker segmentation and clustering results.

[0050] Here, the speaker label for each valid speech segment is determined based on multiple posterior probabilities, and the speaker segmentation and clustering results are output.

[0051] The system normalizes the posterior probability vector for each row (i.e., each speech segment) and uses the maximum a posteriori criterion to determine the initial speaker label. To eliminate label jumps caused by local noise or blurred boundaries, the system introduces a temporal post-processing mechanism based on the HMM state transition structure, thereby reasonably preserving speaker switching points and suppressing unnecessary jumps. Since the same speaker may appear multiple times in different conversations or long audio clips, and the initial clustering may divide them into clusters with different numbers, cross-segment speaker alignment is required: the cosine similarity or PLDA score between the centers of each cluster is calculated. If the similarity between two clusters is higher than a set threshold, they are determined to be the same speaker, and spectral clustering or agglomerative hierarchical clustering is used to perform secondary aggregation of the clusters to further improve cross-time consistency.

[0052] In one possible implementation, the hierarchical probabilistic linear discriminant analysis model is determined through the following steps: i: Organize the sample embedding feature vectors of a set of long-duration audio samples into a dataset by session, and associate each sample embedding feature vector with the session index, the potential speaker index, and the speech segment index.

[0053] It should be noted that the feature vector embedded in this sample is extracted by an embedding extractor that has been optimized through "large margin fine-tuning". In this way, the PLDA model learns similarity metrics based on highly discriminative embeddings, thereby improving its clustering performance.

[0054] ii: Construct a three-layer hierarchical probabilistic linear discriminant analysis model, wherein the hierarchical probabilistic linear discriminant analysis model includes a global mean vector, a conversation-level subspace matrix, a speaker-level subspace matrix, and a segment-level subspace matrix.

[0055] This application proposes a three-tiered hierarchical PLDA model of "segment-speaker-conversation" to better characterize the variability of speech data at different scales. Its mathematical form is as follows:

[0056] in, Indicates a session index; Indicates the speaker index within the session; Indicates the segment index within the speaker's speech; The global mean vector has dimensions of . ; The session-level subspace matrix has dimensions of . It is used to model session-level effects such as recording environment and channel characteristics; For the first The session-level factors corresponding to each session follow a standard normal distribution. ; The speaker-level subspace matrix has dimensions of This is used to model consistent speaker features across time periods; For the first In the first session The speaker-level factors of each speaker follow a standard normal distribution. ; The segment-level subspace matrix has a dimension of . It is used to model short-term acoustic changes; For the first In the first session The first speaker The segment-level factors of each speech segment follow a standard normal distribution. ; The residual noise term follows a set pattern. Distribution. This three-layer structure can explicitly distinguish recording conditions, speaker characteristics, and short-term acoustic changes, overcoming the shortcomings of traditional single-layer PLDA models in mixed modeling.

[0057] iii: The hierarchical probabilistic linear discriminant analysis model is trained based on the sample embedding feature vectors of a set of long-term audio samples. During the training process, a hierarchical constraint regularization strategy is introduced, and a variational Bayesian inference framework is adopted. The parameters of the hierarchical probabilistic linear discriminant analysis model are iteratively optimized through the expectation-maximization algorithm. The sample embedding feature vectors used to train the hierarchical probabilistic linear discriminant analysis model are extracted by an embedding extractor that has been fine-tuned with a large interval.

[0058] Here, the hierarchical probabilistic linear discriminant analysis model is trained based on the sample embedding feature vectors of a set of long-term audio samples. During the training process, a hierarchical constraint regularization strategy is introduced, and a variational Bayesian inference framework is adopted. The parameters of the hierarchical probabilistic linear discriminant analysis model are iteratively optimized through the expectation-maximization algorithm.

[0059] The sample embedding feature vector used for training is extracted by an embedding extractor that has undergone large-interval fine-tuning, while the embedding extractor used to extract the effective speech segments of the audio to be segmented is an embedding extractor that has not undergone large-interval fine-tuning, so as to realize asymmetric feature modeling using the hierarchical probabilistic linear discriminant analysis model.

[0060] In one possible implementation, the hierarchical constraint regularization strategy includes: a dynamic weight adjustment mechanism and a hierarchical constraint regularization mechanism.

[0061] Among them, the inter-level constraint design includes: variance constraint: requiring the variances of factors at different levels to satisfy a decreasing hierarchical relationship, i.e. .in These represent conversation-level, speaker-level, and segment-level factors, respectively. This constraint reflects the hierarchical characteristics of different levels across varying scales. Orthogonality constraint: This requires that the subspace matrices at different levels remain approximately orthogonal to avoid repeatedly capturing the same variation patterns. This is achieved by minimizing the inner product between the subspace matrices. Sparsity constraint: L1 regularization is introduced into the subspace matrices, prompting the model to learn sparser factor representations, enhancing the model's interpretability and generalization ability.

[0062] Here, the dynamic weight adjustment mechanism adjusts the weights of the loss function of factors at different levels during the training process: In the early stage of training, the weights of segment-level factors are increased to ensure that the model learns basic acoustic features first; in the middle stage of training, the weights of conversation-level, speaker-level and segment-level factors are balanced to promote multi-level collaborative learning; in the later stage of training, the weights of conversation-level and speaker-level factors are enhanced to strengthen the modeling of high-level stable features.

[0063] Among them, this application adopts a hierarchical learning rate scheduling mechanism: (1) a smaller learning rate is used for session-level parameters to ensure global stability; (2) a larger learning rate is used for segment-level parameters to quickly adapt to short-term changes; (3) speaker-level parameters are at an intermediate level to balance stability and adaptability.

[0064] iv: When the change in the value of the joint variational objective function of the hierarchical probabilistic linear discriminant analysis model is less than a preset threshold, convergence is determined and the trained hierarchical probabilistic linear discriminant analysis model is identified.

[0065] This application introduces a three-tiered hierarchical modeling structure of "segment-speaker-conversation". This structure can more accurately separate variability from different sources in speech data: conversation-level factors effectively eliminate differences in recording conditions and environment, ensuring that the vectors of the same speaker remain consistent under different recording environments; speaker-level factors specifically model long-term acoustic characteristics, improving the distinguishability between different speakers; segment-level factors capture short-term acoustic changes, significantly reducing erroneous segmentation within the same speaker. Through multi-scale information fusion, the hierarchical structure's processing capability for short speech segments is also enhanced. Even if the statistical features of a single segment are insufficient, the model can still effectively compensate using conversation-level and speaker-level prior information, thereby improving the segmentation accuracy of short speech segments. Furthermore, this structure explicitly separates conversation-specific variations from speaker-specific features, improving the model's cross-domain generalization ability and maintaining high robustness across datasets and complex scenarios. Under harsh acoustic conditions such as noise and reverberation, hierarchical PLDA exhibits stronger robustness. This is because the conversation-level factors can absorb and explain most of the changes caused by acoustic degradation, effectively protecting speaker-level features from contamination and ensuring the system's stability in complex environments. Meanwhile, the robust PLDA training strategy proposed in this application further enhances the system's reliability. By employing an asymmetric architecture of "non-large-interval fine-tuning embedding extractor + large-interval fine-tuning training of PLDA," the inconsistency between the training and inference phases in traditional methods is effectively eliminated. Under this strategy, performance degradation of short audio segments is completely avoided, while retaining the enhanced discriminative power of PLDA brought about by large-interval fine-tuning training. The model achieves performance gains during the training phase while maintaining inference efficiency.

[0066] This application provides a speaker segmentation and clustering method, which includes: performing speech activity detection processing on the audio to be segmented to determine multiple effective speech segments; inputting the multiple effective speech segments into an embedding extractor that has not undergone large-interval fine-tuning to extract speaker embedding vectors and outputting an embedding vector sequence; inputting the embedding vector sequence into a pre-trained hierarchical probabilistic linear discriminant analysis model, and performing clustering processing based on variational Bayesian inference algorithm to assign each effective speech segment to different speakers, and outputting the speaker segmentation and clustering result of the audio to be segmented; wherein, the hierarchical probabilistic linear discriminant analysis model has a three-level structure of segment level, speaker level, and conversation level, and the model is trained based on features extracted by large-interval fine-tuning. This application introduces a three-tiered hierarchical PLDA model structure of "segment-speaker-conversation," which can explicitly model the sources of variation at different levels in speech data. Segment-level factors capture short-term acoustic dynamics (such as articulation transients and speech rate changes), speaker-level factors represent stable individual speech features across time periods, and conversation-level factors absorb conversation-specific biases caused by recording environment, equipment differences, etc., thereby improving the accuracy of speaker segmentation and clustering. Furthermore, this application employs an asymmetric training strategy, using features extracted with large-interval fine-tuning to train the PLDA model, while using a basic embedding extractor during inference. This effectively solves the problem of mismatch between training and inference time, thereby improving the accuracy and robustness of speaker segmentation and clustering.

[0067] Please see Figure 2 , Figure 3 , Figure 2 This is one of the structural schematic diagrams of a speaker segmentation and clustering device provided in the embodiments of this application; Figure 3 This is a second schematic diagram of a speaker segmentation and clustering device provided in an embodiment of this application. Figure 3 As shown, the speaker segmentation clustering device 200 includes: The semantic activity detection module 210 is used to perform speech activity detection processing on the audio to be segmented and identify multiple valid speech segments; Feature extraction module 220 is used to input multiple effective speech segments into an embedding extractor that has not been fine-tuned by large intervals to extract speaker embedding vectors and output an embedding vector sequence. The clustering result determination module 230 is used to input the embedded vector sequence into a pre-trained hierarchical probabilistic linear discriminant analysis model, and perform clustering processing on each effective speech segment belonging to different speakers based on the variational Bayesian inference algorithm, and output the speaker segmentation clustering result of the audio to be segmented; wherein, the hierarchical probabilistic linear discriminant analysis model is a three-level structure of segment level, speaker level and conversation level, and the model is trained based on features extracted by large-interval fine-tuning.

[0068] Furthermore, the clustering result determination module 230 is used to perform clustering processing based on the variational Bayesian inference algorithm to assign each effective speech segment to different speakers, and output the speaker segmentation clustering result of the audio to be segmented: Based on the hierarchical probabilistic linear discriminant analysis model, the posterior probability of each effective speech segment belonging to different hidden speaker states is calculated by jointly optimizing the conversation-level, speaker-level, and segment-level latent variables through the variational Bayesian inference algorithm. The speaker label for each valid speech segment is determined based on multiple posterior probabilities, and the speaker segmentation and clustering results are output.

[0069] Furthermore, the clustering result determination module 230 is used to jointly optimize the conversation-level, speaker-level, and segment-level latent variables based on the hierarchical probabilistic linear discriminant analysis model using a variational Bayesian inference algorithm, and calculate the posterior probability of each effective speech segment belonging to different hidden speaker states: Initialize the prior distributions of the conversation-level factor, speaker-level factor, and segment-level factor; Based on the embedded vector sequence of the effective speech segments, a joint variational objective function containing conversation-level, speaker-level, and segment-level latent variables is constructed; The posterior distributions of the conversation-level factors, speaker-level factors, and segment-level factors are updated based on the variational Bayesian inference algorithm. Based on the updated session-level factor, speaker-level factor, and segment-level factor, the posterior probability of each valid speech segment belonging to each hidden speaker state is recalculated. Based on the current latent variable estimates, the spatial matrices and residual covariance matrices of the conversation-level factors, speaker-level factors, and segment-level factors are redefined until the change in the joint variational objective function is less than a preset threshold, at which point the calculation of the posterior probability is stopped.

[0070] Furthermore, the clustering result determination module 230 is used to update the posterior distributions of the conversation-level factors, the speaker-level factors, and the segment-level factors based on the variational Bayesian inference algorithm: Based on the speech features, speaker-level factors, and segment-level factors of all the effective speech segments in the session, update the posterior distribution of the session-level factors corresponding to the session. The posterior distribution of the corresponding speaker-level factor is updated based on the embedded feature vectors of all the valid speech segments of the speaker, the corresponding conversation-level factor and segment-level factor. The posterior distribution of the segment-level factors of the effective speech segment is updated based on the acoustic features of the effective speech segment and the corresponding conversation-level and speaker-level factors.

[0071] Furthermore, such as Figure 3 As shown, the speaker segmentation and clustering device 200 also includes a model training module 240, which determines the hierarchical probabilistic linear discriminant analysis model through the following steps: The sample embedding feature vectors of a set of long-duration audio samples are organized into a dataset by session, and each sample embedding feature vector is associated with a session index, a potential speaker index, and a speech segment index. A three-layer hierarchical probabilistic linear discriminant analysis model is constructed, wherein the hierarchical probabilistic linear discriminant analysis model includes a global mean vector, a conversation-level subspace matrix, a speaker-level subspace matrix, and a segment-level subspace matrix; The hierarchical probabilistic linear discriminant analysis model is trained based on the sample embedding feature vectors of a set of long-duration audio samples. During the training process, a hierarchical constraint regularization strategy is introduced, and a variational Bayesian inference framework is adopted. The parameters of the hierarchical probabilistic linear discriminant analysis model are iteratively optimized through the expectation-maximization algorithm. The sample embedding feature vectors used to train the hierarchical probabilistic linear discriminant analysis model are extracted by an embedding extractor that has been fine-tuned with a large interval. When the change in the value of the joint variational objective function of the hierarchical probabilistic linear discriminant analysis model is less than a preset threshold, convergence is determined and the trained hierarchical probabilistic linear discriminant analysis model is identified.

[0072] This application provides a speaker segmentation and clustering device, comprising: a semantic activity detection module for performing speech activity detection processing on the audio to be segmented to determine multiple effective speech segments; a feature extraction module for inputting the multiple effective speech segments into an embedding extractor that has not undergone large-interval fine-tuning to extract speaker embedding vectors and outputting an embedding vector sequence; and a clustering result determination module for inputting the embedding vector sequence into a pre-trained hierarchical probabilistic linear discriminant analysis model, performing clustering processing based on a variational Bayesian inference algorithm to assign each effective speech segment to a different speaker, and outputting the speaker segmentation and clustering result of the audio to be segmented; wherein the hierarchical probabilistic linear discriminant analysis model has a three-level structure of segment-level, speaker-level, and conversation-level, and the model is trained based on features extracted by large-interval fine-tuning. This application introduces a three-level hierarchical PLDA model structure of "segment-speaker-conversation", which can explicitly model the sources of variation at different levels in speech data. Segment-level factors capture short-term acoustic dynamics (such as pronunciation transients and speech rate changes), speaker-level factors represent stable individual speech features across time periods, and conversation-level factors absorb conversation-specific biases caused by differences in recording environment and equipment, thereby improving the accuracy of speaker segmentation and clustering.

[0073] Please see Figure 4 , Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0074] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the speaker segmentation and clustering method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0075] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the speaker segmentation and clustering method in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0076] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0080] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speaker segmentation and clustering method, characterized in that, The speaker segmentation and clustering method includes: Speech activity detection processing is performed on the audio to be segmented to identify multiple valid speech segments; Multiple valid speech segments are input into an embedding extractor that has not undergone large-interval fine-tuning to extract speaker embedding vectors, and an embedding vector sequence is output. The embedded vector sequence is input into a pre-trained hierarchical probabilistic linear discriminant analysis model. Based on the variational Bayesian inference algorithm, each effective speech segment is clustered to belong to different speakers, and the speaker segmentation clustering results of the audio to be segmented are output. The hierarchical probabilistic linear discriminant analysis model has a three-level structure of segment level, speaker level, and conversation level, and the model is trained based on features extracted by large-interval fine-tuning.

2. The speaker segmentation and clustering method according to claim 1, characterized in that, The variational Bayesian inference algorithm is used to cluster each effective speech segment to different speakers, and the speaker segmentation clustering results of the audio to be segmented are output, including: Based on the hierarchical probabilistic linear discriminant analysis model, the posterior probability of each effective speech segment belonging to different hidden speaker states is calculated by jointly optimizing the conversation-level, speaker-level, and segment-level latent variables through the variational Bayesian inference algorithm. The speaker label for each valid speech segment is determined based on multiple posterior probabilities, and the speaker segmentation and clustering results are output.

3. The speaker segmentation and clustering method according to claim 2, characterized in that, The hierarchical probabilistic linear discriminant analysis model, based on which the session-level, speaker-level, and segment-level latent variables are jointly optimized using a variational Bayesian inference algorithm, calculates the posterior probability of each effective speech segment belonging to different hidden speaker states, including: Initialize the prior distributions of the conversation-level factor, speaker-level factor, and segment-level factor; Based on the embedded vector sequence of the effective speech segments, a joint variational objective function containing conversation-level, speaker-level, and segment-level latent variables is constructed; The posterior distributions of the conversation-level factors, speaker-level factors, and segment-level factors are updated based on the variational Bayesian inference algorithm. Based on the updated session-level factor, speaker-level factor, and segment-level factor, the posterior probability of each valid speech segment belonging to each hidden speaker state is recalculated. Based on the current latent variable estimates, the spatial matrices and residual covariance matrices of the conversation-level factors, speaker-level factors, and segment-level factors are redefined until the change in the joint variational objective function is less than a preset threshold, at which point the calculation of the posterior probability is stopped.

4. The speaker segmentation and clustering method according to claim 3, characterized in that, The update of the posterior distributions of the conversation-level factors, speaker-level factors, and segment-level factors based on the variational Bayesian inference algorithm includes: Based on the speech features, speaker-level factors, and segment-level factors of all the effective speech segments in the session, update the posterior distribution of the session-level factors corresponding to the session. The posterior distribution of the corresponding speaker-level factor is updated based on the embedded feature vectors of all the valid speech segments of the speaker, the corresponding conversation-level factor and segment-level factor. The posterior distribution of the segment-level factors of the effective speech segment is updated based on the acoustic features of the effective speech segment and the corresponding conversation-level and speaker-level factors.

5. The speaker segmentation and clustering method according to claim 1, characterized in that, The hierarchical probabilistic linear discriminant analysis model is determined through the following steps: The sample embedding feature vectors of a set of long-duration audio samples are organized into a dataset by session, and each sample embedding feature vector is associated with a session index, a potential speaker index, and a speech segment index. A three-layer hierarchical probabilistic linear discriminant analysis model is constructed, wherein the hierarchical probabilistic linear discriminant analysis model includes a global mean vector, a conversation-level subspace matrix, a speaker-level subspace matrix, and a segment-level subspace matrix; The hierarchical probabilistic linear discriminant analysis model is trained based on the sample embedding feature vectors of a set of long-duration audio samples. During the training process, a hierarchical constraint regularization strategy is introduced, and a variational Bayesian inference framework is adopted. The parameters of the hierarchical probabilistic linear discriminant analysis model are iteratively optimized through the expectation-maximization algorithm. The sample embedding feature vectors used to train the hierarchical probabilistic linear discriminant analysis model are extracted by an embedding extractor that has been fine-tuned with a large interval. When the change in the value of the joint variational objective function of the hierarchical probabilistic linear discriminant analysis model is less than a preset threshold, convergence is determined and the trained hierarchical probabilistic linear discriminant analysis model is identified.

6. The speaker segmentation and clustering method according to claim 5, characterized in that, The hierarchical constraint regularization strategy includes: Dynamic weight adjustment mechanism and hierarchical constraint regularization mechanism.

7. A speaker segmentation and clustering device, characterized in that, The speaker segmentation and clustering device includes: The semantic activity detection module is used to perform speech activity detection processing on the audio to be segmented and identify multiple valid speech segments. The feature extraction module is used to input multiple effective speech segments into an embedding extractor that has not undergone large-interval fine-tuning to extract speaker embedding vectors and output an embedding vector sequence. The clustering result determination module is used to input the embedded vector sequence into a pre-trained hierarchical probabilistic linear discriminant analysis model, and perform clustering processing on each effective speech segment belonging to different speakers based on the variational Bayesian inference algorithm, and output the speaker segmentation clustering result of the audio to be segmented; wherein, the hierarchical probabilistic linear discriminant analysis model is a three-level structure of segment level, speaker level and conversation level, and the model is trained based on features extracted by large-interval fine-tuning.

8. The speaker segmentation and clustering device according to claim 7, characterized in that, The clustering result determination module is used to perform clustering processing based on the variational Bayesian inference algorithm to assign each effective speech segment to a different speaker, and outputs the speaker segmentation clustering result of the audio to be segmented: Based on the hierarchical probabilistic linear discriminant analysis model, the posterior probability of each effective speech segment belonging to different hidden speaker states is calculated by jointly optimizing the conversation-level, speaker-level, and segment-level latent variables through the variational Bayesian inference algorithm. The speaker label for each valid speech segment is determined based on multiple posterior probabilities, and the speaker segmentation and clustering results are output.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the speaker segmentation clustering method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the speaker segmentation and clustering method as described in any one of claims 1 to 6.

Images