On-line speaker diarization method and system based on memory bank and weighted random delay

Abstract

The present application relates to the technical field of online speaker diarization, and provides an online speaker diarization method and system based on a memory bank and weighted random delay to solve the problem that full-amount memory attention mechanisms often introduce a large amount of irrelevant speaker or background noise interference, which not only increases the computing burden, but also may reduce the discrimination accuracy of the model. The online speaker diarization method based on the memory bank and the weighted random delay comprises the following steps: obtaining an input acoustic feature sequence with a given length; and converting the input acoustic feature sequence into a speaker activity probability sequence by using an online speaker diarization model. The online speaker diarization model comprises a causal embedded encoder, a convolutional lookahead layer and a memory-aware online attractor decoder which are connected in series, and can make full use of long-time future information and memory bank information, thereby improving the recognition ability for unobserved speakers.

Description

Technical Field

[0001] This invention relates to the field of online speaker logging technology, and in particular to an online speaker logging method and system based on a memory bank and weighted random delay. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Speaker logging represents a significant research direction in the field of speech processing. Its main applications include automatic speech recognition (ASR), speaker indexing, speaker identification, real-time captioning, and audio analysis. Speaker logging is a crucial and indispensable preprocessing step in these applications. With the explosive growth of applications such as remote video conferencing, real-time live captioning, medical consultation recording, and intelligent voice assistants, traditional offline processing methods, due to their high latency, can no longer meet the demands of modern human-computer interaction. Therefore, online speaker logging has become a research hotspot in both academia and industry. With the evolution of deep learning technology, end-to-end neural speaker logging methods based on permutation invariance training have achieved breakthrough progress. The EEND model can directly process overlapping speech, significantly outperforming traditional step-by-step cascaded systems based on "speech activity detection-embedding extraction-clustering" in terms of accuracy. However, despite the excellent performance of the EEND method in offline systems, transferring it to online scenarios with strict causal constraints still faces many unresolved theoretical and engineering challenges.

[0004] The lack of speaker generalization ability is a major bottleneck restricting the practical application of online systems: existing EEND models are usually optimized on closed training sets, tending to overfit the speaker voiceprint features in the training data. In the online processing of long-duration conference audio, the model can only access a limited number of current and historical audio segments, and the cumulative effect of error becomes increasingly significant with long-term continuous inference. Since conferences often last for several hours and involve multiple people frequently taking turns speaking, when encountering unfamiliar speakers not present in the training set, or when a speaker's acoustic features change due to emotional changes, physical distance movement, and device drift caused by long-term recording, the model struggles to extract sufficiently discriminative embedding vectors within a short instantaneous window due to the lack of global feature references. This lack of ability to model unseen speakers in long-term sequences makes the system prone to speaker label confusion, frequent jumps, or identity loss when processing complex conference streams, seriously affecting the stability of real-time logs for long conferences.

[0005] There is a rigid trade-off between latency and performance: to compensate for the lack of future information in online inputs, mainstream methods typically introduce a fixed lookahead window to capture some future context. However, this design leads to a strong coupling between model parameters and a specific latency length: high-latency models trained for high-precision scenarios cannot be applied to low-latency scenarios with stringent real-time requirements, and vice versa. In practical deployments, facing network jitter or different business needs, it is often necessary to maintain multiple models with different latency versions, significantly increasing training costs. Furthermore, to address the problem of insufficient local information, some works have attempted to introduce a memory to store historical information. However, full-scale memory attention mechanisms often introduce a large amount of interference from irrelevant speakers or background noise, not only increasing the computational burden but also potentially reducing the model's discrimination accuracy. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides an online speaker log method and system based on a memory bank and weighted random delay.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides an online speaker log method based on a memory bank and weighted random delay.

[0008] In one or more embodiments, an online speaker logging method based on a memory bank and weighted random delay is provided, including: Obtain the input acoustic feature sequence of a given length; The input acoustic feature sequence is converted into a sequence of speaker activity probabilities using an online speaker log model; wherein the online speaker log model includes a causal embedding encoder, a convolutional lookahead layer, and a memory-aware online attractor decoder connected in sequence. The causal embedding encoder is used to encode the input acoustic feature sequence into several frame embedding representations for each speaker; the convolutional lookahead layer is obtained by adaptive training with weighted random delay and is used to convolve the current frame embedding representation of each speaker with the set future frame embedding representation to obtain the enhanced embedding representation of each speaker. The memory-aware online attractor decoder is used for: Based on the enhanced embedding representation of each speaker, the attractors of the same speaker in previous frames and the attractors of other speakers in the current frame are considered to generate spatiotemporal features of each speaker and map them to generate internal memory features. The enhanced embedding representations of each speaker are converted into query vectors. Memory units are selected based on their similarity to memory units in the pre-trained memory bank, and then weighted summation is performed to obtain external memory features. After concatenating the internal features with the external memory features, the corresponding attractors are mapped and their inner products are calculated with the corresponding speaker embedding representations to obtain the speech activity probabilities of each speaker.

[0009] As one implementation method, the causal embedding encoder uses a Retention mechanism instead of a masked self-attention mechanism; the Retention mechanism removes the Softmax function on the attention value, making the calculation of attention a recursive method.

[0010] In one implementation, the memory-aware online attractor decoder includes a streaming context branch and an external memory interaction branch, used to generate internal memory features and external memory features, respectively.

[0011] In one implementation, the streaming context branch is used to convert the speaker embedding representation into corresponding query vectors, key vectors and value vectors, and update the state over time to obtain the speaker temporal features. The speaker temporal features are used to calculate the attention weight matrix between speakers through a cross-attractor self-attention mechanism to obtain the speaker spatial features. The speaker temporal features and spatial features are linearly superimposed and mapped to obtain the internal features.

[0012] In one implementation, the external memory interaction branch is used to perform similarity matching between the query vectors transformed from the speaker embedding representations obtained from the streaming context branch and the memory bank of the pre-trained data, and to select a number of memory units; then, the selected memory units are weighted and summed to obtain the external memory features.

[0013] As one implementation method, in the process of adaptively training the convolutional look-ahead layer using weighted random delay, a sampling probability dynamic adjustment mechanism is adopted to determine the sampling probability as the training progress progresses. The sampling probability dynamic adjustment mechanism is formed by linear interpolation and mixing of two sets of basic weight distributions.

[0014] As one implementation method, the basic weights of the dynamic adjustment sampling probability mechanism include: baseline weights. and linear bias weights ; The baseline weights follow a uniform distribution; all optional delay values ​​are assigned equal weights. ; The linear bias weights follow a distribution that increases linearly with the delay value. The weights are set in relation to the delay magnitude. It is directly proportional, and the calculation formula is: ; Minimum delay; Final sampling weights The linear combination of the two sets of weights mentioned above is determined by the training progress. Controlling the mixing ratio: ; exist Within the range; The calculated weights, after being normalized, are used as probability parameters for a multinomial distribution for sampling, thus obtaining the sampling probability. : ; This is the set of delayed candidates.

[0015] In one or more embodiments, an online speaker logging system based on a memory bank and weighted random delay includes: The input acoustic feature sequence acquisition module is used to acquire an input acoustic feature sequence of a given length. The speaker activity probability sequence conversion module is used to convert the input acoustic feature sequence into speaker activity probability sequences using an online speaker log model; wherein, the online speaker log model includes a causal embedding encoder, a convolutional lookahead layer and a memory-aware online attractor decoder connected in series. The causal embedding encoder is used to encode the input acoustic feature sequence into several frame embedding representations for each speaker; the convolutional lookahead layer is obtained by adaptive training with weighted random delay and is used to convolve the current frame embedding representation of each speaker with the set future frame embedding representation to obtain the enhanced embedding representation of each speaker. The memory-aware online attractor decoder is used for: Based on the enhanced embedding representation of each speaker, the attractors of the same speaker in previous frames and the attractors of other speakers in the current frame are considered to generate spatiotemporal features of each speaker and map them to generate internal memory features. The enhanced embedding representations of each speaker are converted into query vectors. Memory units are selected based on their similarity to memory units in the pre-trained memory bank, and then weighted summation is performed to obtain external memory features. After concatenating the internal features with the external memory features, the corresponding attractors are mapped and their inner products are calculated with the corresponding speaker embedding representations to obtain the speech activity probabilities of each speaker.

[0016] A third aspect of the present invention provides a computer-readable storage medium.

[0017] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the online speaker log method based on a memory bank and weighted random delay as described above.

[0018] A fourth aspect of the present invention provides an electronic device.

[0019] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the online speaker log method based on a memory bank and weighted random delay as described above.

[0020] Compared with the prior art, the beneficial effects of the present invention are: The online speaker log model of this invention includes a causal embedding encoder, a convolutional lookahead layer, and a memory-aware online attractor decoder connected in series. The causal embedding encoder encodes the input acoustic feature sequence into several frame embedding representations for each speaker. The convolutional lookahead layer is adaptively trained using weighted random delays, convolving the current frame embedding representation of each speaker with a predetermined future frame embedding representation to obtain enhanced embedding representations for each speaker. Finally, the memory-aware online attractor decoder considers attractors from previous frames of the same speaker and attractors from other speakers in the current frame to generate internal memory features. The enhanced embedding representations of each speaker are converted into query vectors, and then external memory features are obtained by weighted summation of memory units selected from a pre-trained memory bank. Furthermore, corresponding attractors are formed through concatenation mapping and their inner product is calculated with the corresponding speaker embedding representation to obtain the speech activity probability of each speaker. This fully utilizes long-term future information and memory bank information, improving the ability to identify speakers not yet seen. Attached Figure Description

[0021] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0022] Figure 1 This is a schematic diagram of the online speaker log method based on memory bank and weighted random delay according to an embodiment of the present invention; Figure 2 This describes the correlation between system latency and the performance metric DER under different training strategies in this embodiment of the invention. Figure 3 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used in the embodiments of the invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0026] Terminology Explanation: Speaker digitization refers to dividing an audio recording into segments based on the speaker, like keeping a log, and marking the time information of each speaker's speech and the length of the corresponding segment. In other words, it solves the problem of who spoke and when, without requiring any prior information about the speaker's identity.

[0027] To address the limitation on the number of speakers, several extended architectures of EEND have emerged. Horiguchi et al. proposed an EEND extension with an encoder-decoder-based attractor, known as the EDA architecture. This method applies an LSTM-based encoder-decoder to the output of EEND to generate multiple attractors. Attractors are continuously generated until their probability of existence falls below a threshold. Each attractor is then multiplied by an embedding generated from EEND to compute the probability of speech activity for each speaker. EEND-EDA was evaluated on datasets with a large number of speakers, showing better performance than clustering-based baseline systems. Subsequently, an online speaker log model for long-duration conference processing, LS-EEND, was proposed, which achieves improvements in both inference latency and accuracy by replacing the original attention module with a retention module.

[0028] EEND architecture: End-to-End Neural Speaker Diarization architecture, proposed by Fujita et al. in 2019, laid the foundation for end-to-end speaker logging technology. It executes the entire speaker logging process based on a single neural network. The input to EEND is a sequence of acoustic features of length T. The output of networks like MFCC is a sequence of speaker labels, indicating the probability of each speaker speaking at each time frame. Initially, EEND was implemented using bidirectional long short-term memory (BSSM). After achieving good system performance, it was extended to networks based on self-attention.

[0029] The advantages of EEND are as follows: First, the EEND architecture is well-suited for handling overlapping speech. Second, the network directly optimizes the system's output accuracy through a loss function. It can be retrained on real data (i.e., non-synthetic data) with only a reference speaker label, which is often difficult to achieve in previous work. The limitations of EEND are as follows: First, the model architecture limits the maximum number of speakers the model can handle. Second, EEND consists of BLSTM or self-attention-based neural networks, making online processing difficult. Empirical evidence suggests that EEND tends to overfit the distribution of the training data.

[0030] Online speaker digitization (OPD) aims to solve the problem of who speaks and when in real-time audio streaming scenarios such as teleconferences and live video broadcasts, and is a core technology for backend implementation of meeting transcription and speech recognition. However, existing technologies perform poorly when processing long-duration conference audio online. To address this issue, this invention proposes an OPD method based on a memory bank and weighted random delay. First, by introducing a memory module pre-trained on a large-scale external dataset, a rich prior knowledge space of speakers is provided to the system, significantly enhancing the model's ability to generalize and recognize speakers not yet seen. Second, a weighted random delay training mechanism is designed. By dynamically sampling within a preset delay range and adjusting the weight distribution according to the training progress, a single model can handle recognition accuracy under different real-time scenarios, greatly improving the system's flexibility and reducing training costs. Finally, to address the noise problem in the memory module, a dynamic Top-K sparse attention mechanism is proposed. By dynamically selecting the most relevant feature basis vectors during training, redundant interference is effectively suppressed, further reducing the system's speaker error rate while optimizing computational efficiency.

[0031] Figure 1 A schematic diagram of the online speaker log method based on memory bank and weighted random delay according to an embodiment of the present invention is provided.

[0032] according to Figure 1 The online speaker log method based on memory and weighted random delay in this embodiment may include the following steps: Step 1: Obtain the input acoustic feature sequence of a given length.

[0033] The speaker logging task is defined as estimating the activity states of multiple speakers based on an input audio sequence. Given a length of... Input acoustic feature sequence ,in, for The feature vector at each time point. The system's goal is to output the corresponding speaker activity probability sequence. .in, Indicates at time The probability of each speaker being active. The maximum number of speakers set for the system.

[0034] Step 2: Use an online speaker log model to convert the input acoustic feature sequence into a sequence of speaker activity probabilities; wherein, the online speaker log model includes a causal embedding encoder, a convolutional lookahead layer and a memory-aware online attractor decoder connected in series.

[0035] Causal embedding encoders are used to encode input acoustic feature sequences into several frames of embedding representations for each speaker.

[0036] In most speaker embedding networks, self-attention-based encoders are typically used to learn long-term dependencies. However, for streaming log tasks, the system needs to process very long audio streams continuously (e.g., conferences lasting several hours), which presents a significant computational challenge due to the quadratic time complexity of self-attention. This embodiment of the Causal Embedding Encoder uses a Retention mechanism instead of a masked self-attention mechanism. The Retention mechanism removes the Softmax function from the attention values, making the calculation of attention recursive and possessing linear complexity relative to the time step.

[0037] The role of the causal embedding encoder is to map the raw acoustic features into a high-dimensional speaker embedding representation. To meet the needs of long audio streaming processing, this invention employs an encoder based on the Conformer structure combined with a Retention mechanism.

[0038] Input features First, the features are passed through a downsampling layer and linear projection to transform them into initial hidden layer features. Then, these features are fed into the encoder, which is composed of... It consists of stacked Conformer blocks. Unlike traditional self-attention mechanisms, the Conformer blocks use a retention mechanism.

[0039] The parallel representation formula for the Retention mechanism is as follows: ; ; ; ; in, and These represent matrix transpose and element-wise product, respectively. These represent the query vector, key vector, and value vector, respectively. It is a calculation The trainable matrix. Represents the exponentially decaying term, whose elements are defined as follows: (only in) If true, then true; otherwise, true (0). , and They are embedded together as relative positions. and All are constant real numbers, among which , ; It is a decay factor used to control the rate of forgetting historical information. The Rention mechanism can be re-formulated recursively: ; ; in They represent the time steps respectively The query vector, key vector, and value vector. yes The state matrix at each moment is updated recursively along the time dimension and stores all key context information up to the current moment.

[0040] Parallel representations are suitable for efficient training, while recursive representations with linear time complexity can be used for inference. The speaker logging task learns the speaker's voiceprint, which is a time-invariant signal over long time periods (e.g., throughout the entire processed audio stream). To maintain the consistency of the speaker embedding throughout the audio stream, a decay factor is applied. Set to 1 to avoid information decay. Furthermore, similar to previous studies on speaker logging methods, it was found that removing the position embedding term had no significant impact on model performance.

[0041] The convolutional look-ahead layer is trained using weighted random delay training. It is used to convolve the current frame embedding representation of each speaker with the set future frame embedding representation to obtain the enhanced embedding representation of each speaker.

[0042] For most online / streaming applications, short response times are acceptable. Utilizing some future frames can significantly improve the quality of speaker embeddings, especially in challenging scenarios where new speakers appear in only a few frames. Therefore, a look-ahead mechanism is implemented using one-dimensional convolutions in the temporal dimension to leverage a small number of future frames and make more confident decisions. The latency (defined as the number of future frames utilized) is determined by the convolution kernel and padding size. Finally, the embeddings are normalized using the L2 norm to enhance their stability.

[0043] In streaming inference, the kernel size of the convolutional lookout layer directly determines the system's latency and performance. To enable a single model to adapt to various application scenarios ranging from low to high latency, a progress-based linear hybrid sampling strategy is introduced instead of using a fixed convolutional kernel during the training of the convolutional lookout layer.

[0044] In the process of adaptively training the convolutional look-ahead layer using weighted random delay, a sampling probability dynamic adjustment mechanism is adopted to determine the sampling probability as the training progress progresses. This dynamic adjustment mechanism is formed by linear interpolation and mixing of two sets of basic weight distributions.

[0045] During training, the convolutional layers at the end of the causal embedding encoder, in each training step, extract latency candidates from a pre-defined set. Randomly sample a delay value The model is based on this delay value. Dynamically truncate the convolutional kernel weights to simulate different look-ahead windows. This is equivalent to data augmentation on the delay dimension, forcing the model to learn feature representations of contexts of different lengths.

[0046] To balance the model's performance under different latency conditions, this invention designs a method that adapts to the training progress. (scope The mechanism for dynamically adjusting the sampling probability; the basic weights of the dynamic sampling probability adjustment mechanism include: baseline weight. and linear bias weights ; The baseline weights follow a uniform distribution; all optional delay values ​​are assigned equal weights. This represents "unbiased" random selection, ensuring that the model does not forget its ability to handle low-latency scenarios at any stage.

[0047] The linear bias weights follow a distribution that increases linearly with the delay value. The weights are set in relation to the delay magnitude. It is directly proportional, and the calculation formula is: ; This represents the minimum latency; under this distribution, the larger the latency value, the higher its sampling weight. This indicates an emphasis on "high-latency, long-context" tasks, guiding the model to focus more on long-contextual information scenarios in the later stages of training.

[0048] Final sampling weights The linear combination of the two sets of weights mentioned above is determined by the training progress. Controlling the mixing ratio: ; exist Within the range; The calculated weights, after being normalized, are used as probability parameters for a multinomial distribution for sampling, thus obtaining the sampling probability. : ; This is the set of delayed candidates.

[0049] Early training ( ): The mixed weights are mainly composed of The dominant sampling probability is approximately uniformly distributed. The model extensively explores all possible delay settings, establishing adaptability to different viewpoints.

[0050] Later stages of training ( ): Mixed weights gradually towards The sampling probabilities exhibit a linear, step-like distribution where "the greater the latency, the higher the probability." In this case, the model allocates more training resources to training with large, high-latency convolutional kernels to improve the upper limit of model performance, while retaining a certain number of low-latency samples (with non-zero weights) to prevent performance degradation due to low latency.

[0051] By employing this smooth linear transition strategy, the embodiments of the present invention effectively solve the training balance problem in multi-latency scenarios, and avoid the model from falling into a local optimum of a single latency mode too early.

[0052] In this embodiment, unlike block-level online logging methods that generate block-level attractors to log the entire block, this system designs an online attractor decoder to extract attractors frame by frame. This method eliminates the need for buffering, allowing the model to automatically select and utilize previous information, such as representative acoustic features and attractors of previously appearing speakers. The decoder takes speaker embeddings as input and aims to detect new speakers immediately when they appear, and collects more speaker embeddings over time to update the attractors of existing speakers. Unlike sequence generation tasks such as speech recognition or machine translation, multiple attractors in speaker logging tasks do not follow some ordered semantic dependency, making parallel processing more efficient. Therefore, this embodiment proposes a non-autoregressive attractor decoder to generate multiple attractors simultaneously.

[0053] The input to the memory-aware online attractor decoder is: In each frame, the speaker embedding They are fed into the decoder to generate or update the corresponding attractors. All Each attractor shares the same repeated embedding as its input source. To differentiate the inputs of multiple attractors, the system first copies the speaker embedding. Next, location embeddings (PEs) are generated along the speaker dimension. Next, the speaker embedding is concatenated with the PE along the feature dimension. A linear layer is then applied to obtain the feature dimension. .

[0054] Memory-aware online attractor decoder is used for: Based on the enhanced embedding representation of each speaker, the attractors of the same speaker in previous frames and the attractors of other speakers in the current frame are considered to generate spatiotemporal features of each speaker and map them to generate internal memory features. The enhanced embedding representations of each speaker are converted into query vectors. Memory units are selected based on their similarity to memory units in the pre-trained memory bank, and then weighted summation is performed to obtain external memory features. After concatenating the internal features with the external memory features, the corresponding attractors are mapped and their inner products are calculated with the corresponding speaker embedding representations to obtain the speech activity probabilities of each speaker.

[0055] The memory-aware online attractor decoder includes a streaming context branch and an external memory interaction branch, which are used to generate internal memory features and external memory features, respectively.

[0056] The streaming context branch is used to convert the speaker embedding representation into the corresponding query vector, key vector and value vector, and update the state over time to obtain the speaker temporal features. The speaker temporal features are used to calculate the attention weight matrix between speakers through the cross-attractor self-attention mechanism, and then obtain the speaker spatial features. The speaker temporal features and spatial features are linearly superimposed and mapped to obtain the internal features.

[0057] Streaming context branching: In order to derive the first The speaker in the first Frame attractor In addition to the first In addition to speaker embedding in frame extraction, two other information sources need to be considered: (1) Previous frames of the same speaker attractor ,in This helps maintain the temporal consistency of attractors for a particular speaker. An along-frame retention module is designed to reference previous frames. For the same reason, in retention... Set to 1 and remove the position embedding item.

[0058] (2) No. Other speakers in the frame attractor ,in By utilizing information from other attractors, the distance between attractors can be increased, thereby better distinguishing different speakers. To this end, a cross-attractor self-attention module following the standard self-attention mechanism is designed, followed by a feedforward module. Decoders are cascaded. Each decoder block, the final branch outputs the internal memory branch. .

[0059] The specific calculation method is as follows: The input to this branch is defined as the initialized attractor query tensor. Let the first... The input to the layer decoder is... For the first layer, At that moment The focus is on ,in The preset maximum number of speakers (including invalid bits). This is the feature dimension. Then, the features are updated through the three cascaded sub-modules described above: Along-Frame Retention Module: For the The speaker's attractor slot at time Corresponding features Calculate features The corresponding query vector. Key vector Value vector : ; in, , and These are the corresponding query weight, key weight, and value weight, respectively. State update and output (recursive form): ; ; in, For the first The speaker The state matrix at each moment; It is the attenuation factor. For the first The speaker The state matrix at each moment; for Transpose of; This is the output of the Retention module along the frame; after passing through residual connections and layer normalization (LayerNorm), the output of this step is denoted as the temporal feature. .

[0060] Cross-Attractor Self-Attention Module: Input is time features of Time slice ; ; , and Time features of Time slice The corresponding query, key, and value; , and Corresponding , and The weight.

[0061] Calculate the attention weight matrix between speakers : ; ; in, for Transpose of; for The dimension; This is the output of the cross-attractor self-attention module; this process allows each attractor to aggregate information from other attractors based on relevance (e.g., suppressing similar invalid slots), and the output, after residual connections and layer normalization, is denoted as spatial features. ; Feedforward Neural Network (Feed Forward Module): Input spatial features : intermediate features ; The final output is: ; in, and It is a linear mapping function; For activation functions; This is the normalization function; As an intermediate feature; This is a characteristic of internal memory.

[0062] The external memory interaction branch is used to perform similarity matching between the query vectors transformed from the speaker embedding representations obtained from the streaming context branch and the memory bank of the pre-trained data, and to select a number of memory units; then the selected memory units are weighted and summed to obtain the external memory features.

[0063] External Memory Interaction Branch: This branch introduces a memory bank based on pre-trained data. ,in This represents the number of memory units. The calculation process is as follows: Similarity calculation: Calculate the current query vector using an additive attention mechanism. (Output from branch one) and memory bank Each unit Matching score : ; in These are linear projection parameters; for The transpose of .

[0064] Top-K Sparse Selection: To reduce interference from irrelevant features, the system implements a Top-K selection strategy. Based on a set ratio coefficient... Only the highest scorer is retained. One memory unit, and the fractions of the remaining units are set to negative infinity; This indicates rounding down to the nearest integer.

[0065] Weighted aggregation: Softmax normalization is applied to the filtered scores to obtain the weights. The external memory features are obtained by weighted summation of the memory units. : ; in, is the weight of the k-th memory unit.

[0066] Feature fusion: combining internal features External features The features are concatenated along the feature dimension to obtain the concatenated features. Mapping back to the original dimensions through a linear projection layer. To generate the final attractor : ; ; in, This is a concatenation function; It is a linear layer function.

[0067] Decoder Output: The decoder output vector is then L2-norm normalized to have unit length, serving as the final attractor. Speaker activity is computed by calculating the inner product between the frame-by-frame attractor and the speaker embedding: ; in This represents the element-wise Sigmoid function; for Transpose of; This indicates that the speaker is embedded.

[0068] In this embodiment, the memory-aware online attractor decoder is trained using the Dynamic Top-K Sparse Attention Training method: During the interaction between the memory-aware decoder and the external memory bank, if full attention is always used in the computation, irrelevant background audio can interfere with the feature extraction of the current speaker. Furthermore, introducing extremely sparse Top-K filtering too early may lead to model convergence difficulties due to unstable gradients in the early stages of training. To address this, this invention designs a dynamic Top-K training strategy based on step-wise exponential decay. This strategy guides the model to achieve a smooth transition from global correlation to local focus by controlling the sparsity parameter k-ratio (i.e., the proportion of retained memory units).

[0069] Decay formula definition: Define the current training round as... The formula for calculating the sparsity ratio k-ratio is as follows: ; in: The base number 0.9 indicates the decay rate.

[0070] Period 5: This indicates that the k-ratio decreases by 10% every 5 epochs. Lower limit 0.1: This sets a hard threshold to ensure that at least 10% of the most relevant memory units are retained for computation, preventing excessive information loss. This indicates rounding down to the nearest integer.

[0071] Early training phase: In the initial stage of training ( When the k-ratio approaches 1.0, the Top-K mechanism degenerates into full attention. This allows the model to fully utilize the gradient information of the entire memory, quickly establishing a connection between the feature space and the vectors in the memory, ensuring the stability of the optimization.

[0072] In the later stages of training: As the number of training epochs increases, the k-ratio decreases exponentially. The model is forced to gradually ignore those "noise" basis vectors with low similarity, focusing only on the top K most relevant speaker features. This forces the model to learn more discriminative feature representations in later stages, effectively suppressing false alarms and speaker confusion in long meeting scenarios. Through this strategy, the model can improve the overall performance of the system without introducing additional hyperparameters.

[0073] Table 1. Analysis of experimental results;

[0074] The experiment used the AMI dataset, and the evaluation metric was the speaker log error rate (DER).

[0075] SSep + VAD E2E (Speech Sepration and voice activity detection End-to-End model) is a multi-task system that integrates speech separation, voice activity detection and speaker logging. It uses speech separation technology to guide speaker logging, especially for the overlapping speech problem that frequently occurs in conference scenarios (such as the AMI dataset).

[0076] LS-EEND (Long-Form Streaming End-to-End Neural Diarization) proposes a frame-by-frame streaming end-to-end speaker log model that breaks through the limitations of previous block processing and caching mechanisms. It can directly process audio streams with extremely long durations and a large and variable number of speakers, and is also the baseline model used in this invention.

[0077] The Core Samples model is based on the VBX algorithm, which performs well in offline tasks. By introducing a dynamic caching module and a clustering matching strategy, it solves the problems of memory limitations and accuracy degradation in online processing of long audio files.

[0078] The librispeech Memory and AMI Memory are memory libraries that utilize two different speech datasets. Librispeech is a large-scale, publicly available English corpus containing approximately 1000 hours of speech data, with audio clips mostly consisting of clean speech segments between 10 and 30 seconds. The AMI dataset, on the other hand, is a corpus containing 171 long-duration conferences, totaling approximately 100 hours. Its audio segments are longer, ranging from 30 to 60 minutes, and include realistic noise such as reverberation and background noise.

[0079] As shown in Table 1, the experimental test results on the AMI dataset demonstrate that the DER of the model was reduced to some extent after introducing a pre-training memory bank and a Top-K sparse attention mechanism. The following is a detailed analysis of the experimental data: Overall performance: Compared to earlier online separation methods, the baseline model LS-EEND performed better, controlling the error rate on the development set to 20.97%. The method proposed in this invention, by combining an external memory bank and a Top-K mechanism, further reduced the DER to below 20%, verifying the effectiveness of this scheme in assisting to improve recognition accuracy.

[0080] Performance using a non-same-origin memory (Librispeech Memory): When the model uses a memory built from the external Librispeech dataset, the system's DER decreases to 19.66%, a 1.31% reduction from the LS-EEND baseline. Librispeech primarily consists of clean, single-speaker speech, which differs significantly in acoustic features from AMI (Audio Mixed Instructions) scenarios, which include reverberation, ambient noise, and multi-person conversations. This result demonstrates that even with cross-domain data distributions, the Top-K mechanism can filter out mismatched speaker information to some extent, helping the model extract useful speaker features and exhibiting good generalization adaptability.

[0081] Performance using the AMI Memory: The model performs best when the memory is built using data consistent with the test scenario (i.e., using the AMI's own training data). The performance DER further decreases to 18.90%. This aligns with our intended design: the AMI memory contains speaker cluster centers that better reflect the target test environment (e.g., specific conference room acoustics and equipment noise). Under these conditions, the Top-K mechanism can more accurately retrieve relevant speaker feature basis vectors, effectively reducing irrelevant interference information in feature interaction calculations.

[0082] The experimental data above demonstrate that the Top-K sparse attention mechanism proposed in this invention, combined with a pre-trained memory bank, can effectively assist streaming speaker log systems. Whether in cross-domain or homogeneous data scenarios, this method can suppress interfering features and reduce speaker error rates to a certain extent, demonstrating good practical application feasibility.

[0083] The online speaker log model in this embodiment of the invention is trained using a multi-stage progressive length training method. Addressing the "length extrapolation" problem in long conference audio processing—that is, the model cannot directly generalize to long audio after training on short audio—this invention adopts the multi-stage progressive training strategy proposed by LS-EEND.

[0084] This invention employs the concept of Curriculum Learning, dividing the training process into four consecutive stages. By progressively doubling the chunk size of the training audio segments, it guides the model to adapt to long-term temporal dependencies. The specific implementation steps are as follows: Initial Phase: Training is first conducted on short audio segments (e.g., 200 seconds). This phase utilizes a large amount of simulated data to enable the model to quickly learn basic speaker discrimination and short-term clustering features.

[0085] Length doubling adaptation: In the subsequent fine-tuning stage, this invention adjusts the length of the input audio slices according to... The sequence is multiplied four times consecutively.

[0086] Chunk-wise Recurrent Paradigm: To handle sequences up to 1600 seconds long with limited GPU memory, a chunk-wise recurrent paradigm of the Retention mechanism was enabled during training. This means that computation is performed in parallel within each long slice, while the recurrent state is passed between slices. This ensures that gradients can propagate effectively across slice boundaries, enabling the model to learn to capture speaker dependencies spanning tens of minutes.

[0087] Experiments show that this progressive strategy from easy to difficult can effectively avoid the convergence difficulties caused by training directly on long sequences, and ensure that the model maintains the consistency of speaker identity when processing hour-long conference recordings.

[0088] This invention constructs an external memory enhancement architecture based on a pre-trained knowledge base: it introduces a static memory base pre-trained using large-scale heterogeneous data, which contains a general speaker feature manifold. During online inference, a cross-attention mechanism maps currently ambiguous speech features to a high-confidence memory space, enhancing the model's ability to identify speakers not yet seen.

[0089] This invention proposes a weighted random delay training mechanism: treating "inference delay" as a variable training parameter, randomly sampling the size of the lookahead layer convolution kernel during training, and combining it with a dynamic weight adjustment strategy, the model is forced to adapt to low latency while making full use of long-term future information, thus enabling deployment in multiple latency scenarios with a single training.

[0090] This invention designs a dynamic Top-K sparse attention mechanism. To address the noise problem in memory retrieval, a sparse selection strategy involving annealing during training is proposed. This mechanism forces the model to focus only on the few most relevant memory basis vectors, effectively achieving feature denoising and reducing computational overhead while improving performance.

[0091] Figure 2 The correlation between system latency and performance metric DER under different training strategies is shown. The comparison curves in the figure reveal that the weighted random latency training strategy exhibits the following two main characteristics in model performance: First, significant performance improvements were achieved across most latency settings. In the low to medium latency range, the DER of the weighted random latency strategy was consistently lower than that of the original baseline model and the unweighted uniform random training model, with the most significant advantage observed at extremely low latency. This demonstrates that this dynamic training mechanism not only effectively improves the model's adaptability to different latency perspectives but also maintains good recognition accuracy in mainstream real-time scenarios.

[0092] Secondly, there is a certain degree of performance loss at higher latency. When the latency is set high, although the weighted random latency model still outperforms the original baseline, its performance DER increases compared to the model specifically trained for fixed latency with a memory bank, resulting in some performance loss. This is because a single model needs to consider the latency range across the entire frequency band during training, making reasonable performance concessions in high-latency scenarios.

[0093] In one or more embodiments, an online speaker logging system based on a memory bank and weighted random delay is also provided, which can be implemented in software. The online speaker logging system based on a memory bank and weighted random delay includes the following software modules: The input acoustic feature sequence acquisition module is used to acquire an input acoustic feature sequence of a given length. The speaker activity probability sequence conversion module is used to convert the input acoustic feature sequence into speaker activity probability sequences using an online speaker log model; wherein, the online speaker log model includes a causal embedding encoder, a convolutional lookahead layer and a memory-aware online attractor decoder connected in series. The causal embedding encoder is used to encode the input acoustic feature sequence into several frame embedding representations for each speaker; the convolutional lookahead layer is obtained by adaptive training with weighted random delay and is used to convolve the current frame embedding representation of each speaker with the set future frame embedding representation to obtain the enhanced embedding representation of each speaker. The memory-aware online attractor decoder is used for: Based on the enhanced embedding representation of each speaker, the attractors of the same speaker in previous frames and the attractors of other speakers in the current frame are considered to generate spatiotemporal features of each speaker and map them to generate internal memory features. The enhanced embedding representations of each speaker are converted into query vectors. Memory units are selected based on their similarity to memory units in the pre-trained memory bank, and then weighted summation is performed to obtain external memory features. After concatenating the internal features with the external memory features, the corresponding attractors are mapped and their inner products are calculated with the corresponding speaker embedding representations to obtain the speech activity probabilities of each speaker.

[0094] It should be noted that each module in the online speaker log system based on memory and weighted random delay in this embodiment corresponds one-to-one with each step in the online speaker log method based on memory and weighted random delay in the above embodiment, and their specific implementation processes are the same, so they will not be repeated here.

[0095] The structure of the electronic device according to an embodiment of the present invention will be described in detail below. Figure 3 This is a schematic diagram of the composition structure of an electronic device provided in an embodiment of the present invention. It can be understood that... Figure 3 The diagram shows only an exemplary structure of the electronic device, not the entire structure. Some or all of the structures shown may be implemented as needed.

[0096] The electronic device provided in this embodiment of the invention includes: at least one processor 301, a memory 302, a user interface 303, and at least one network interface 304. The various components of the online speaker log system based on a memory bank and weighted random delay are coupled together via a bus system 305. It is understood that the bus system 305 is used to implement communication between these components. In addition to a data bus, the bus system 305 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 3 The general designated all buses as Bus System 305.

[0097] The user interface 303 may include a monitor, keyboard, mouse, trackball, click wheel, buttons, touchpad, or touch screen.

[0098] It is understood that memory 302 can be volatile memory or non-volatile memory, or both. In this embodiment of the invention, memory 302 is capable of storing data to support the operation of the terminal. Examples of this data include any computer programs used to operate on the terminal, such as operating systems and applications. The operating system includes various system programs, such as framework layers, core library layers, driver layers, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications.

[0099] In some embodiments, the online speaker logging system based on a memory bank and weighted random delay provided in this invention can be implemented using a combination of hardware and software. As an example, the online speaker logging system based on a memory bank and weighted random delay provided in this invention can be a processor in the form of a hardware decoding processor, programmed to execute the online speaker logging method based on a memory bank and weighted random delay provided in this invention. For example, the processor in the form of a hardware decoding processor can employ one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0100] As an example, processor 301 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0101] As an example of the hardware implementation of the online speaker logging system based on memory bank and weighted random delay provided in this embodiment of the invention, the device provided in this embodiment of the invention can be directly executed by a processor 301 in the form of a hardware decoding processor. For example, it can be executed by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components to implement the online speaker logging method based on memory bank and weighted random delay provided in this embodiment of the invention.

[0102] The memory 302 in this embodiment of the invention is used to store various types of data to support the operation of an online speaker log system based on a memory bank and weighted random delay, or to store data for execution. Figure 1 The program code for the method shown. Examples of this data include: any executable instructions for operating on an online speaker logging system based on a memory and weighted random delay, such as executable instructions that can be included in a program implementing the online speaker logging method based on a memory and weighted random delay of embodiments of the present invention.

[0103] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including functions for executing... Figure 1 The program code for the method shown. In such an embodiment, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by the central processing unit, it performs the various functions defined in the apparatus of this application.

[0104] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0105] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for online speaker logging based on a memory bank and weighted random delay, characterized in that, include: Obtain the input acoustic feature sequence of a given length; The input acoustic feature sequence is converted into a sequence of speaker activity probabilities using an online speaker log model; wherein the online speaker log model includes a causal embedding encoder, a convolutional lookahead layer, and a memory-aware online attractor decoder connected in sequence. The causal embedding encoder is used to encode the input acoustic feature sequence into several frame embedding representations for each speaker; the convolutional lookahead layer is obtained by adaptive training with weighted random delay and is used to convolve the current frame embedding representation of each speaker with the set future frame embedding representation to obtain the enhanced embedding representation of each speaker. The memory-aware online attractor decoder includes a streaming context branch and an external memory interaction branch; The streaming context branch is based on the enhanced embedding representation of each speaker, taking into account the attractors of the same speaker in previous frames and the attractors of other speakers in the current frame, and generates spatiotemporal features of each speaker accordingly and maps them to generate internal memory features. The external memory interaction branch converts the enhanced embedding representations of each speaker into query vectors, filters memory units based on their similarity to memory units in the pre-trained memory bank, and then obtains the external memory features by weighted summation. The memory-aware online attractor decoder is also used to concatenate internal features with external memory features, map them to form corresponding attractors, and perform inner product calculation with the corresponding speaker embedding representation to obtain the speech activity probability of each speaker; In the process of adaptively training the convolutional look-out layer using weighted random delay, a sampling probability dynamic adjustment mechanism is adopted to determine the sampling probability as the training progress progresses. This dynamic adjustment mechanism is formed by linear interpolation and mixing of two sets of basic weight distributions. The basic weights of the dynamic sampling probability adjustment mechanism include: baseline weights. and linear bias weights ; The baseline weights follow a uniform distribution; all optional delay values ​​are assigned equal weights. ; The linear bias weights follow a distribution that increases linearly with the delay value. The weights are set in relation to the delay magnitude. It is directly proportional, and the calculation formula is: ; Minimum delay; Final sampling weights The weights are a linear combination of the two sets mentioned above, determined by the training progress. Controlling the mixing ratio: ; exist Within the range; The calculated weights, after being normalized, are used as probability parameters for a multinomial distribution for sampling, thus obtaining the sampling probability. : ; This is the set of delayed candidates.

2. The online speaker log method based on memory and weighted random delay as described in claim 1, characterized in that, The causal embedding encoder uses a Retention mechanism instead of a masked self-attention mechanism; the Retention mechanism removes the Softmax function from the attention value, making the calculation of attention a recursive process.

3. The online speaker log method based on memory and weighted random delay as described in claim 1, characterized in that, The streaming context branch is used to convert the speaker embedding representation into corresponding query vectors, key vectors and value vectors, and update the state over time to obtain the temporal features of each speaker. The temporal features of each speaker are used to calculate the attention weight matrix between speakers through a cross-attractor self-attention mechanism, thereby obtaining the spatial features of each speaker. The temporal features and spatial features of each speaker are linearly superimposed and mapped to obtain the internal features.

4. The online speaker log method based on memory and weighted random delay as described in claim 1, characterized in that, The external memory interaction branch is used to perform similarity matching between the query vectors transformed from the speaker embedding representations obtained from the streaming context branch and the memory bank of the pre-trained data, and to select a number of memory units; then, the selected memory units are weighted and summed to obtain the external memory features.

5. An online speaker log system based on a memory bank and weighted random delay, characterized in that, The online speaker logging method based on a memory bank and weighted random delay as described in any one of claims 1-4 includes: The input acoustic feature sequence acquisition module is used to acquire an input acoustic feature sequence of a given length. The speaker activity probability sequence conversion module is used to convert the input acoustic feature sequence into speaker activity probability sequences using an online speaker log model; wherein, the online speaker log model includes a causal embedding encoder, a convolutional lookahead layer and a memory-aware online attractor decoder connected in series. The causal embedding encoder is used to encode the input acoustic feature sequence into several frame embedding representations for each speaker; the convolutional lookahead layer is obtained by adaptive training with weighted random delay and is used to convolve the current frame embedding representation of each speaker with the set future frame embedding representation to obtain the enhanced embedding representation of each speaker. The memory-aware online attractor decoder is used for: Based on the enhanced embedding representation of each speaker, the attractors of the same speaker in previous frames and the attractors of other speakers in the current frame are considered to generate spatiotemporal features of each speaker and map them to generate internal memory features. The enhanced embedding representations of each speaker are converted into query vectors. Memory units are selected based on their similarity to memory units in the pre-trained memory bank, and then weighted summation is performed to obtain external memory features. After concatenating the internal features with the external memory features, the corresponding attractors are mapped and their inner products are calculated with the corresponding speaker embedding representations to obtain the speech activity probabilities of each speaker.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the online speaker log method based on a memory bank and weighted random delay as described in any one of claims 1-4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the online speaker log method based on memory bank and weighted random delay as described in any one of claims 1-4.

Images