Overlapped speech detection method, device, equipment and storage medium

Abstract

The present application belongs to the technical field of data processing, and discloses an overlapping speech detection method, device, equipment and storage medium, the method comprising: generating a continuous masking weight sequence according to a speech activity probability sequence corresponding to an acoustic feature sequence of a to-be-detected speech; performing weighted processing on a first feature sequence according to the continuous masking weight sequence to obtain a masking feature sequence; maintaining a speaker prototype feature set of speaker prototype feature vectors; calculating a similarity value of a frame-level speaker feature of a current frame and each prototype feature vector as an auxiliary discrimination basis; combining the auxiliary discrimination basis to analyze the masking feature sequence, frame by frame, to determine whether each speech frame of the to-be-detected speech is multi-person overlapping speech, and output an overlapping speech probability sequence. The present application can be applied to financial technology, medical health and other business management systems, and solves the technical problems of frequent jumping of detection results at the start and end of speech and misjudgment of single high-energy speech as overlapping in the overlapping speech detection mode.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology and is applied in the fields of financial technology and healthcare. In particular, it relates to an overlapping speech detection method, device, equipment and storage medium. Background Technology

[0002] Overlapped Speech Detection (OSD) aims to determine whether a single person or multiple people are speaking at each moment in a speech segment, playing a crucial role in downstream applications such as speech separation, transcription, speaker logging, and dialogue analysis.

[0003] Voice Activity Detection (VAD) is a technique that determines the presence of human voices frame by frame in an audio stream. Its basic principle is to extract the acoustic features (such as spectral energy, Mel-frequency cepstral coefficients, etc.) of each audio frame, and then use a statistical model or neural network to output a decision for each frame, identifying whether it is a "speech frame" or a "silence frame." VAD is the front-end module of a speech processing system, and its accuracy directly affects the performance of all subsequent processing stages.

[0004] Existing overlapping speech detection methods have two core shortcomings.

[0005] First, hard decision-making at boundaries leads to detection jitter. Traditional methods process speech activity detection and overlapping speech detection in parallel, using a hard masking method with either 0 or 1 at the start and end boundaries of speech segments. Features of silent frames are directly set to zero, while speech frame features are fully preserved. The principle of hard masking is to set a fixed threshold. When the probability of speech output by the VAD is higher than the threshold, the masking weight is 1 (fully preserved); when it is lower than the threshold, the masking weight is 0 (completely discarded). This binarization operation lacks a transition at the boundaries. Once the VAD makes a decision deviation near the boundary due to noise interference or weak signal, the features fed into the overlapping detection model will undergo abrupt changes between adjacent frames, causing the detection results to frequently jump at the start and end of speech. In compliance checks of dual recording (audio and video recording) in banks for fintech, this jitter makes it impossible to accurately define the start and end times of overlapping segments, making it difficult for reviewers to confirm the specific duration of illegal interruptions. In remote medical consultations, boundary jitter causes the automatic segmentation of doctor-patient dialogue to become fragmented, making it impossible to accurately extract complete speech segments in subsequent consultation behavior analysis.

[0006] Secondly, the lack of speaker perception makes it easy for high-energy single-speaker speech to be misjudged as overlapping. Existing methods rely solely on acoustic features such as spectral energy for judgment. Acoustic features are mathematical representations of the physical properties of audio signals, such as spectrum, energy, and fundamental frequency, primarily used to characterize the "content" and "loudness" of sound, rather than "who is speaking." When a single speaker is emotionally agitated and raises their volume, their acoustic spectrum exhibits high-energy, complex distribution characteristics highly similar to overlapping speech from multiple speakers, because both show simultaneous enhancement of multiple frequency components in the frequency domain. Models that rely solely on acoustic features cannot distinguish between "one person speaking loudly" and "two people speaking simultaneously," as there is no essential difference in their spectral morphology. In banking and insurance sales scenarios, when salespeople passionately explain product terms, the acoustic features are characterized by continuous high energy and rich harmonic structures, often misjudged by existing systems as customer interruptions or two people talking over each other, resulting in a high false alarm rate during compliance reviews. In online medical consultations, patients' voices may tremble or increase in volume due to pain or emotional excitement, which may be misjudged as overlapping, distorting the doctor's communication quality assessment results and affecting the fairness of medical service evaluation.

[0007] In summary, current overlapping speech detection methods suffer from two major pain points: boundary jitter and single-speaker / overlap confusion. Summary of the Invention

[0008] This invention provides an overlapping speech detection method, apparatus, device, and storage medium, which can solve the technical problems in existing overlapping speech detection methods, such as frequent jumps in detection results at the start and end of speech and the easy misjudgment of high-energy single-person speech as overlapping.

[0009] In a first aspect, the present invention provides an overlapping speech detection method, comprising: Obtain the acoustic feature sequence and frame-level speaker feature sequence of the speech to be detected; Speech activity detection is performed based on the acoustic feature sequence to obtain a speech activity probability sequence; Generate a continuous masking weight sequence based on the speech activity probability sequence; The first feature sequence is weighted according to the continuous masking weight sequence to obtain a masking feature sequence, wherein the first feature sequence is the acoustic feature sequence or a fused feature sequence obtained by fusing the acoustic feature sequence with the frame-level speaker feature sequence; Maintain a speaker prototype feature set that contains at least one speaker prototype feature vector; Calculate the similarity value between the frame-level speaker features of the current frame and each prototype feature vector in the speaker prototype feature set, as an auxiliary discrimination criterion; By combining the aforementioned auxiliary discrimination criteria with the analysis of the masking feature sequence, the system determines frame by frame whether each speech frame of the speech to be detected is a single person's speech or multiple people's overlapping speech, and outputs an overlapping speech probability sequence.

[0010] In a second aspect, the present invention provides an overlapping speech detection device, comprising: The acquisition module is used to acquire the acoustic feature sequence and frame-level speaker feature sequence of the speech to be detected; The detection module is used to detect speech activity based on the acoustic feature sequence to obtain a speech activity probability sequence; The weight generation module is used to generate a continuous masking weight sequence based on the speech activity probability sequence; The feature weighting module is used to weight the first feature sequence according to the continuous masking weight sequence to obtain a masking feature sequence, wherein the first feature sequence is the acoustic feature sequence or a fused feature sequence obtained by fusing the acoustic feature sequence with the frame-level speaker feature sequence; The feature maintenance module is used to maintain a speaker prototype feature set containing at least one speaker prototype feature vector; The similarity calculation module is used to calculate the similarity value between the frame-level speaker features of the current frame and each of the prototype feature vectors in the speaker prototype feature set, as an auxiliary discrimination basis; The judgment module is used to analyze the masking feature sequence in conjunction with the auxiliary discrimination criteria, determine frame by frame whether each speech frame of the speech to be detected is single-person speech or multiple-person overlapping speech, and output the overlapping speech probability sequence.

[0011] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described overlapping speech detection method.

[0012] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned overlapping speech detection method.

[0013] Understandably, traditional methods use a hard masking approach that is either 0 or 1, meaning that the features of silent frames are directly set to zero while the features of speech frames are fully preserved. This lack of transition at the boundaries means that VAD decision bias can directly lead to abrupt feature changes and cause detection jitter.

[0014] This invention provides a fundamental solution. First, speech activity detection is performed to obtain a speech activity probability sequence. Unlike traditional hard decision methods, this step outputs not 0 / 1 labels, but a continuous probability value between 0 and 1 for each frame, preserving complete information about the possibility of speech presence. A continuous masking weight sequence is then generated based on this probability sequence, mapping the VAD probability to masking weights that also take continuous values ​​between 0 and 1. This continuous masking weight sequence is then used to weight the first feature sequence, resulting in a masked feature sequence. Silent frames, due to their low VAD probability, are assigned weights approaching 0, significantly suppressing their corresponding acoustic features, but not completely reducing them to zero. Speech frames, due to their high VAD probability, are assigned weights approaching 1, preserving their features. At the boundary between silence and speech, the weights smoothly increase or decrease with the VAD probability, preventing abrupt changes in the feature vector. Thus, the problem of "fixed threshold binarization leading to abrupt changes in boundary features" in the background technology is solved.

[0015] The background section points out that existing methods rely solely on acoustic features such as spectral energy to determine overlap. These acoustic features primarily characterize the "content" and "loudness" of the sound, rather than "who is speaking." When a single person speaks loudly and emotionally, their spectral morphology is highly similar to that of overlapping speech from multiple people, both exhibiting high energy and complex frequency distributions, making it impossible to distinguish them based on acoustic features alone.

[0016] The present invention maintains a speaker prototype feature set, which contains at least one speaker prototype feature vector, used to store and track representative voiceprint features of different speakers during dialogue. The similarity value between the frame-level speaker features of the current frame and each prototype feature vector in the speaker prototype feature set is further calculated as an auxiliary discrimination criterion. This auxiliary discrimination criterion is then combined with a masking feature sequence to jointly determine single-speaker or overlapping speech frame by frame.

[0017] The key to this approach lies in the fact that the speaker prototype feature vector stores identity information—"who is speaking"—rather than energy information—"how loud they are speaking." When a single person is emotionally agitated and speaks loudly, although the acoustic features exhibit high energy and a complex spectrum, their frame-level speaker features still maintain a high similarity to an existing prototype in the speaker prototype feature set. This is because the voiceprint characteristics of loud and soft speaking (depending on the physiological structure of the vocal organs) are inherently stable and do not change with volume. This high similarity value serves as an auxiliary discrimination criterion, providing a crucial disambiguation signal for overlap detection. When the model receives the information that "the speaker in the current frame highly matches a known speaker A," it can infer that speaker A is still speaking alone, just louder, thus avoiding misjudgment as overlap. Conversely, when multiple people overlap, the frame-level speaker features of the mixed speech will not match any known prototype in the feature set, resulting in a low similarity value. Based on this, the model judges that overlap has indeed occurred. Thus, the problem of "misjudgment caused by the similarity of the spectra of high-energy single-person speech and overlapping multi-person speech" in the background technology is solved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an overlapping speech detection method according to an embodiment of the present invention.

[0020] Figure 2 yes Figure 1 A flowchart of step S110.

[0021] Figure 3 This is another flowchart illustrating the overlapping speech detection method in one embodiment of the present invention.

[0022] Figure 4 yes Figure 1 A flowchart of step S160.

[0023] Figure 5 yes Figure 1 A flowchart of step S180.

[0024] Figure 6 This is a schematic diagram of an overlapping speech detection device according to an embodiment of the present invention.

[0025] Figure 7 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention.

[0026] Figure 8 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Figure 1 A flowchart of the overlapping speech detection method provided in the embodiments of the present invention is shown below. Figure 1 As shown, the overlapping speech detection method provided in this embodiment of the invention includes the following steps.

[0029] Step S110: Obtain the acoustic feature sequence and frame-level speaker feature sequence of the speech to be detected; Specifically, in this step, the voice signal to be detected can be received and preprocessed (such as uniform sampling rate and frame windowing).

[0030] Furthermore, when acquiring acoustic feature sequences, the following methods can be used: (1) extracting high-level contextual representations using a self-supervised pre-trained model; (2) employing traditional acoustic feature extractors, such as Mel frequency cepstral coefficients or filter bank features, and performing time-series modeling through several convolutional layers or recurrent networks; (3) directly using the original waveform as input and extracting features through a one-dimensional convolutional network. The resulting acoustic feature sequence is a set of frame-level vectors, with each frame vector characterizing the acoustic properties such as spectrum and energy at that moment.

[0031] Furthermore, when obtaining frame-level speaker feature sequences, the following methods can be used: (1) Use a pre-trained speaker recognition model to extract a fixed-dimensional embedding vector from the acoustic features of each frame; (2) Use a joint training method to add a speaker feature extraction branch inside the overlap detection model to generate frame-level speaker representations in an end-to-end manner; (3) Use the pooling or transformation of the acoustic feature sequence directly as a preliminary speaker-related representation.

[0032] As a concrete example, in the field of fintech, feature extraction can be performed on dual-channel or single-channel recordings in the analysis of bank intelligent customer service call logs to obtain frame-level acoustic features representing customers and customer service representatives, as well as their respective speaker feature vectors, providing a foundation for subsequent analysis.

[0033] As another concrete example, in the healthcare field, the voice features of doctors and patients can be extracted from recordings of doctor-patient conversations on online consultation platforms. Doctors and patients may be in different acoustic environments (e.g., quiet on the doctor's end, and noisy in the patient's home), and robust feature extraction ensures that subsequent detection is not affected by environmental differences.

[0034] In some embodiments of the present invention, such as Figure 2 As shown, the frame-level speaker feature sequence of the speech to be detected is obtained, including: Step S111: The acoustic front-end features of the speech to be detected are processed by a pre-trained speaker model to obtain an initial frame-level speaker representation. Specifically, in this step, for the speech to be detected, low-level acoustic features can first be extracted through acoustic front-end processing. Acoustic front-end processing involves segmenting the audio waveform into frames (e.g., frame length 25 milliseconds, frame shift 10 milliseconds), windowing, and then extracting the filter bank features or Mel-frequency cepstral coefficients for each frame. The resulting acoustic front-end feature sequence is denoted as... ,in Total number of frames, per frame It is a feature vector.

[0035] Subsequently, the acoustic front-end feature sequence The input is fed into a pre-trained speaker model. This pre-trained speaker model can be any neural network trained on a large-scale speaker recognition task, such as an x-vector model based on a Time-Delay Neural Network (TDNN), a speaker embedding model based on a Residual Network (ResNet), or a speaker feature extractor based on Emphasized Channel Attention, Propagation, and Aggregation in Time-Delay Neural Network (ECAPA-TDNN). The speaker model processes the acoustic front-end features of each frame and outputs the initial frame-level speaker representation corresponding to that frame. The resulting sequence of initial frame-level speaker representations is denoted as... Each frame This is a fixed-dimensional vector that encodes the speaker's timbre, prosody, and other identity-related information corresponding to the audio frame. Because the pre-trained speaker model is trained on large-scale, multi-speaker data, its extracted representations have good speaker discriminative and generalization abilities, and can extract meaningful identity features even from speakers not seen during training.

[0036] As a concrete example in the fintech field, a bank conducted compliance analysis on audio recordings of wealth management product sales. The recordings alternated between the voices of the wealth management manager and the customer. The system extracted 40-dimensional filter bank features from the dual-recorded audio as an acoustic front-end feature sequence. Then, these features are fed frame by frame into a pre-trained speaker model based on the ECAPA-TDNN architecture. This model has been pre-trained on tens of thousands of hours of telephone speech data and exhibits good adaptability to telephone channels and financial terminology scenarios. For each frame of the financial manager's speech, the model outputs an initial frame-level speaker representation. It is clustered in a relatively compact region in the vector space; for each frame of the client's speech, the representation is... They then cluster in another region. The two types of representations are well separable in the vector space, providing high-quality speaker identity cues for subsequent cross-attention fusion.

[0037] As another concrete example in the healthcare field, in remote rehabilitation guidance voice conversations, rehabilitation therapists and patients communicate one-on-one through an online platform. The system extracts Mel-Frequency Cepstral Coefficients (MFCC) features from the recordings as an acoustic front-end feature sequence. Since patients may be in non-professional acoustic environments such as their homes, background noise (such as television sounds or family members walking around) is common. The pre-trained speaker model outputs an initial frame-level speaker representation for each frame. Thanks to the robustness of the pre-trained model, even under low signal-to-noise ratio conditions, the representations of different frames from the same speaker maintain a high degree of consistency, while the speaker representations of noisy frames exhibit a random and scattered pattern due to the lack of a stable voiceprint structure, which is convenient for subsequent modules to distinguish and utilize.

[0038] Step S112: Using the acoustic feature sequence as the query and the initial frame-level speaker representation as the key and value, frame-level feature fusion is performed through a cross-attention mechanism to obtain the frame-level speaker feature sequence and form the fused feature sequence.

[0039] Specifically, in this step, the initial frame-level speaker representation sequence obtained in step S111 can be used. Through a cross-attention module, and with the acoustic feature sequence (denoted as...) ) Perform frame-level fusion.

[0040] Specifically, the cross-attention mechanism operates as follows: For each frame, the acoustic feature vector of that frame is used as the basis for the cross-attention mechanism. As the query vector, the initial frame-level speaker representation of that frame. The attention output is calculated using the query vector and key vector as key and value vectors. The calculation process is as follows: first, the attention weights are calculated from the query vector and key vector, and then the attention weights are applied to the value vector to obtain the speaker attention output vector. Since cross-attention is performed at the frame-by-frame level, that is, the acoustic features of each frame only interact with the speaker representation in the same frame, this operation does not introduce cross-frame contextual dependencies, ensuring frame-level independence and low latency characteristics of the processing.

[0041] Then, the speaker attention output vector of the cross-attention output is compared with the original acoustic feature vector. The features are fused (e.g., element-wise addition or concatenation along the feature dimensions) to form the fused feature vector for that frame. The fused feature sequence consists of the fused feature vectors from all frames. Simultaneously, the vector sequence output by cross-attention is the final frame-level speaker feature sequence.

[0042] Through the cross-attention mechanism, acoustic features can selectively integrate speaker identity information based on their own content. In frames where the acoustic features themselves do not carry obvious speaker-discriminating information (such as voiceless consonants and fricatives), cross-attention automatically reduces its dependence on speaker representation, thereby avoiding the introduction of irrelevant interference.

[0043] As a concrete example, in the fintech field, in the aforementioned analysis of bank wealth management product sales recordings, regarding the wealth management manager's remarks at the beginning... Frame, acoustic features High-level representations extracted by the self-supervised encoder, initial frame-level speaker representations This is the output of ECAPA-TDNN. The cross-attention module uses... For query, Using these as keys and values, we calculate which speaker identity information in the frame is most relevant to the current acoustic content. Since the acoustic content of this frame happens to be the financial manager's unique speech rate and tone pattern, the cross-attention weights will focus on... Based on the corresponding dimension, generate a speaker attention output that highly matches the acoustic content. Then, compare this output with... The resulting fusion feature retains the original acoustic content information such as spectrum and energy, while also embedding the identity marker that "this is a financial manager speaking." For frames containing customer questions, the same process integrates the customer's identity information into the corresponding fusion feature. In this way, segments corresponding to different speakers in the fusion feature sequence carry different identity markers, providing clear speaker differentiation cues for subsequent overlapping speech detection.

[0044] As another concrete example in the healthcare field, in the aforementioned remote rehabilitation guidance scenario, the rehabilitation therapist is demonstrating and explaining movements clearly and at a moderate pace. Simultaneously, the patient occasionally offers brief responses such as "I understand" or "Is that so?". Regarding the therapist's speech... Frame, acoustic features The spectral structure of its clear speech was encoded, and the initial frame-level speaker representation was obtained. The therapist's voiceprint characteristics were encoded. Cross-attention was used to... For querying, from Extracting voiceprint information that matches the spectral structure, generating speaker attention output and... Fusion. For frames of patient responses, the fusion features carry the patient's voiceprint signature. When two people's voices overlap in a frame, the initial frame-level speaker representation for that frame is... The process itself is rather chaotic, and the cross-attention mechanism still relies on... When performing a query, the attention output is also chaotic, which contrasts sharply with the clear identity markers in single-person frames, providing additional clues for overlap detection.

[0045] Understandably, the two steps of "extracting initial representations from a pre-trained speaker model" and "frame-level fusion with cross-attention" achieve precise frame-level alignment between acoustic features and speaker identity information. The pre-trained speaker model provides highly discriminative initial speaker representations for each frame, ensuring the quality and robustness of speaker information. The cross-attention mechanism uses frame-level acoustic features as queries, extracting the most relevant identity information from speaker representations within the same frame for fusion. This results in a fused feature sequence that retains complete acoustic content while clearly labeling the speaker identity attributes of each frame. This frame-level fusion method avoids the identity information averaging problem caused by global speaker embedding, enabling accurate identity labeling for each frame in dialogue scenarios with frequent and rapid speaker switching. This provides more discriminative input features for subsequent VAD and OSD decoders, thereby improving the overall accuracy of the overlapping speech detection method.

[0046] Step S120: Detect speech activity based on the acoustic feature sequence to obtain a speech activity probability sequence; Specifically, in this step, the acoustic feature sequence obtained in S110 can be input into a speech activity detection (VAD) module, which outputs the speech presence probability (a continuous value between 0 and 1) frame by frame. The VAD module can be implemented as follows: (1) a neural network-based classifier (such as a multilayer perceptron, temporal convolutional network, or Transformer); (2) an algorithm based on statistical signal processing (such as a dual-threshold method based on energy and zero-crossing rate, or spectral subtraction posterior probability calculation based on noise estimation); (3) end-to-end joint training, where the VAD part shares some weights with the subsequent network. The output probability sequence retains the smooth transition information of speech activity, rather than hard 0 / 1 labels.

[0047] As a concrete example, in the field of fintech, in dual-recorded audio for insurance sales, the VAD module outputs a probability of 0.92 for frames where the salesperson explains the terms, a probability of 0.88 for frames where the customer briefly responds, and a probability of 0.05 for gaps in silence between both parties, forming a continuous probability curve.

[0048] As another concrete example, in the healthcare field, during remote consultation recordings, when multiple doctors are discussing, VAD will assign a high probability to frames with speech and a low probability to device noise or paper-turning sounds, filtering out non-speech interference.

[0049] In some embodiments of the present invention, such as Figure 3 As shown, after step S120 and before step S140, the following steps are also included: Step S130: Perform probability domain calibration on the speech activity probability sequence by temperature scaling and / or Platt calibration to obtain a calibrated speech activity probability sequence used to generate the continuous masking weight sequence.

[0050] Specifically, in this step, the original speech activity probability sequence can be denoted as... Each frame The calibration process can be performed using one or a combination of the following methods: Temperature scaling: scaling the original probability By temperature parameter The calibrated probability is obtained by transforming the scale function. Temperature parameters Controlling the degree of centralization or smoothness of the probability distribution, The original distribution remains unchanged. This causes the probability to shrink towards 0.5, reducing the risk of overconfidence. This polarizes the probability towards 0 or 1, enhancing confidence. The temperature parameter can be determined through validation set optimization.

[0051] Platt calibration: By learning a logistic regression model, the probability scores of the original VAD output are mapped to calibrated probabilities. ,in The logit value output by the original VAD model (and) (They can be converted to each other using the logit function). and These are learnable scalar parameters. This method can directly correct for probability bias caused by differences in the distribution of training data and actual deployment data.

[0052] After the above calibration, the resulting speech activity probability sequence The original probability sequence will be replaced as the input for generating the continuous masking weight sequence.

[0053] As a concrete example in the fintech field, an insurance company uses the same overlapping speech detection model in its quality inspection system, but the dual-recorded audio comes from different recording devices (such as the counter microphone array and the mobile app). Due to the differences in the pickup characteristics of different devices, the original probability distribution output by the VAD module is significantly different: the counter equipment has low noise, and the VAD probability is generally higher (even in silent segments, it is 0.1-0.2); the mobile app has complex environmental noise, and the VAD probability confidence is relatively low. If directly fed into the sigmoid masking module, the same adaptive threshold θ will produce inconsistent masking effects on different devices. By adding temperature scaling calibration (T=1.3) after VAD, the probability distribution on the app is smoothed, reducing the difference between it and the counter equipment distribution, making the calibrated probabilities comparable on different devices, thereby ensuring the threshold is correct when generating masking weights later. Its versatility.

[0054] As another concrete example, in the healthcare field, in remote auscultation and consultation systems, doctors use professional conference microphones while patients use their mobile phones' built-in microphones, resulting in a significant difference in the signal-to-noise ratio between the two recordings. The VAD module often outputs a low probability of patient speech activity (e.g., only 0.3-0.5 for a soft response), while the probability for doctor speech is often high. Directly using the raw probabilities to generate masking weights may lead to excessive suppression of patient speech, resulting in the loss of crucial descriptions of the patient's condition. By introducing Platt calibration after VAD, two logistic regression models are learned using a small amount of labeled data to adapt the probability mappings for the doctor and patient respectively. This reasonably increases the low probability for the patient, ensuring that the calibrated probability accurately reflects the true level of speech activity and that subsequent continuous masking treats both parties' speech fairly.

[0055] Understandably, in this embodiment, by inserting a probability domain calibration step after speech activity detection and before the generation of continuous masking weights, the differences in VAD probability output distribution caused by different acoustic acquisition environments (such as different devices, different room reverberation, and different signal-to-noise ratios) can be effectively eliminated, making the calibrated probability sequence comparable across scenarios. This provides a standardized input for subsequently generating continuous masking weights using unified adaptive threshold and gating steepness parameters, avoiding masking inconsistencies caused by probability distribution drift, and improving the robustness and generalization ability of the overall overlapping speech detection method in diverse deployment environments.

[0056] Step S140: Generate a continuous masking weight sequence based on the speech activity probability sequence; Specifically, in this step, a continuous masking weight sequence can be generated based on the probability sequence of S120 using a mapping function. This mapping function can be any differentiable function that smoothly maps a [0,1] input to a [0,1] output, such as the sigmoid function, the softplus function, a combination of a step function and a linear function, or implemented using a learnable lookup table. The key is that the weight values ​​monotonically increase with the VAD probability, and the transition is smooth, avoiding binary jumps. This mapping can include adjustable parameters (such as steepness and offset thresholds), but the specific methods for adjusting these parameters fall under the category of weighted weights and are not limited here.

[0057] As a concrete example in the fintech field, in the background noise analysis of remote account opening video calls, when a customer reads a statement, the VAD probability may fluctuate slightly due to breathing pauses. The continuous masking weight will smoothly follow the fluctuations and will not produce weight abrupt changes at the critical point of silence / voice.

[0058] As another concrete example, in the healthcare field, during a mental health counseling conversation, the client's tone might be low or interspersed with sobs, and the probability of VAD (Voice-Activated Disorder) might be at a moderate level. Continuous masking weights would then assign an appropriate weight value, allowing these low-confidence speech information portions to be passed to subsequent modules, avoiding the loss of crucial emotional expression due to hard segmentation.

[0059] In some embodiments of the present invention, step S140 includes: calculating the masking weight of each frame in the continuous masking weight sequence according to the sigmoid function.

[0060] The input to the sigmoid function is the product of the gating steepness parameter and the difference between the current frame speech activity probability and the adaptive threshold, and the masking weight ranges from 0 to 1.

[0061] Specifically, in this embodiment, for the current frame Let its voice activity probability be... The preset gating steepness parameter is Adaptive threshold is The masking weight w_t for the current frame is then calculated using the following formula: ; in, For the sigmoid function. Gated kurtosis parameter. Probability of speech activity in the current frame and adaptive threshold The product of the differences As input to the sigmoid function. Since the output range of the sigmoid function is the open interval (0,1), the calculated masking weights are... The value of is between 0 and 1. The above calculation is then repeated for each frame to obtain a continuous masking weight sequence.

[0062] As a concrete example in the fintech field, in the call recording analysis of a bank's intelligent customer service system, an adaptive threshold was set. Gating steepness The probability of speech activity in a given frame. When the sigmoid value is 0.92, the input to the sigmoid function is The masking weights for this frame are calculated as follows: A value close to 1 indicates that almost all features of the frame are preserved. When another frame is during a short pause in a conversation between a customer service representative and a customer, When the value drops to 0.35, the input to the sigmoid function is... The masking weights are calculated as follows: This indicates that the features of that frame are partially suppressed. The weight values ​​vary with... It changes continuously with the changes and will not suddenly jump at a certain critical point.

[0063] As another concrete example, in the healthcare field, during telemedicine consultation recordings, the microphone worn by doctors may intermittently pick up ambient noise, causing the probability of speech activity in certain frames to fluctuate at a moderate level. Let an adaptive threshold be set... Gating steepness When a certain frame When the sigmoid value is 0.55, the input to the sigmoid function is The masking weight is ;when When it rises to 0.78, the input is... The masking weight is The weights transitioned smoothly from approximately 0.6 to approximately 0.9 without any binary jumps, ensuring the continuity of the doctor's speech segment features.

[0064] Understandably, this embodiment generates continuous masking weights using the sigmoid function, ensuring that the masking weights smoothly and monotonically increase with the change in the probability of speech activity, forming a continuous transition zone between silence and speech segments, thus avoiding the weight jumps at the boundaries of traditional hard-decision masking. Furthermore, the output of the sigmoid function naturally falls between 0 and 1, eliminating the need for additional truncation or normalization operations, simplifying the calculation process and ensuring numerical stability.

[0065] Step S150: The first feature sequence is weighted according to the continuous masking weight sequence to obtain the masking feature sequence.

[0066] Wherein, the first feature sequence is the acoustic feature sequence or a fused feature sequence obtained by fusing the acoustic feature sequence with the frame-level speaker feature sequence.

[0067] Specifically, in this step, the weight sequence generated in S140 is multiplied and weighted frame-by-frame on the first feature sequence. Here, the "first feature sequence" can be the acoustic feature sequence from S110, or a fused feature sequence resulting from the fusion of the acoustic and speaker feature sequences. If acoustic features are selected, only the acoustic representation is masked, preserving the complete speaker information flow; if fused features are selected, the joint representation incorporating speaker identity is masked, allowing subsequent processing to suppress both acoustic and speaker information in silent segments. After weighting, the feature vector of the silent segment approaches zero, speech segment features are preserved, and the transition at the boundaries is smooth.

[0068] As a concrete example, in the field of fintech, when analyzing recordings of bank wealth management risk disclosures, the masked feature sequence clearly shows the features of the wealth manager's statements and the customer's questions, while the features of the periods when the two are silent and thinking are set to near zero, effectively reducing the interference of background noise and silent frames on overlap detection.

[0069] As another concrete example, in the field of healthcare, in the recording of multidisciplinary consultations in the emergency department, masking operations suppress the features of silent segments such as the electrical noise of equipment and the sound of keyboard typing when doctors are not speaking, so that the subsequent model only focuses on the area containing human voices, thereby improving the efficiency and accuracy of overlap detection.

[0070] Step S160: Maintain a speaker prototype feature set containing at least one speaker prototype feature vector; Specifically, in this step, during audio processing, a speaker prototype feature set is dynamically constructed and updated. Initially, this set can be empty. As speech frames are received, the system aggregates prototype vectors from frame-level speaker features based on specific criteria (such as high VAD probability and stable speaker features) and stores them in the set. The generation methods for prototype vectors include, but are not limited to: averaging features from multiple frames of the same speaker, exponential moving average, or selecting representative vectors through clustering algorithms. The set maintenance mechanism may include strategies for adding, merging, and aging / deleting prototypes.

[0071] As a concrete example in the fintech field, in the recording of securities trading calls, two speaker prototypes, "Trader A" and "Client B," were gradually established through continuous listening. When Trader A switches headset microphones, their voice characteristics may change slightly, and the system can adaptively update its prototype vector to maintain the robustness of the representation.

[0072] As another concrete example, in the field of healthcare, during long-term chronic disease management follow-up calls, the doctor's voiceprint is relatively stable for each call, and the system can maintain a long-term prototype for the attending physician. However, since each follow-up patient is different, the system will quickly build a prototype of the new patient at the beginning of the conversation and remove it from the feature set after the call ends to save storage space.

[0073] In some embodiments of the present invention, such as Figure 4 As shown, step S160 includes the following steps.

[0074] Step S161: Obtain the speech activity probability and the overlapping speech probability of each speech frame of the speech to be detected; Specifically, in this step, during the processing of the speech to be detected, for each frame... The system acquires two probability values ​​simultaneously. The first is the probability of voice activity. The first is from the speech activity detection step, indicating the probability that human voices are present in the frame. The second is the probability of overlapping speech. The overlapping speech probability sequence, derived from the overlapping speech detection step, indicates the likelihood that multiple people are speaking simultaneously in that frame. These two probability values ​​together form the basis for selecting high-confidence single-person speech frames.

[0075] As a concrete example in the fintech field, a securities firm performs real-time analysis of recorded conversations between investment advisors and clients. At frame 150, the probability of voice activity in that frame is analyzed. A value of 0.96 indicates that someone is almost certainly speaking in this frame. Simultaneously, the probability of overlapping speech in this frame... A value of 0.04 indicates that the system judges the frame to be highly unlikely to be overlapping speech. These two probability values ​​together suggest that frame 150 is very likely a single person speaking clearly. In another frame (frame 320), It is 0.91, but A value of 0.87 indicates that although someone was speaking in the frame, it was likely two people speaking simultaneously. The combination of the two probability values ​​provided a precise basis for subsequent screening.

[0076] As another concrete example, in the healthcare field, a telemedicine platform analyzes online consultation recordings between doctors and patients. At frame 80, the probability of voice activity in that frame is... The probability of overlapping speech is 0.89. The value is 0.06. This frame corresponds to a doctor asking a patient individually, "How has your sleep been lately?" The system simultaneously detects a high VAD probability and a low OSD probability, indicating that this frame contains clear and stable monologue. In the subsequent 200th frame, It is 0.93, but The probability was 0.78, corresponding to an overlap between the patient's urgent need to provide additional symptom information and the doctor's subsequent questions. By acquiring both probabilities simultaneously, the system can accurately distinguish between single-person segments and overlapping segments on the timeline.

[0077] Step S162: Frames with a voice activity probability higher than a first threshold and an overlap voice probability lower than a second threshold are identified as high-confidence single-person voice frames. Specifically, in this step, two thresholds can be set, namely the first threshold. Second threshold For each frame, determine whether it simultaneously satisfies two conditions: the probability of speech activity. Above the first threshold (This indicates that the frame has a sufficiently high confidence level in the presence of speech), and the probability of overlapping speech is... Below the second threshold (This indicates that the frame has a sufficiently low confidence level for overlapping speech). Frames that meet both of these conditions are identified as high-confidence single-speaker speech frames. In these frames, the system is very certain that only one speaker is clearly speaking, therefore the corresponding frame-level speaker features are high-quality and uncontaminated, suitable for updating the speaker prototype feature set. Frames that do not meet the conditions (such as silent frames, low-confidence speech frames, or high-confidence overlapping frames) will be skipped and will not participate in subsequent prototype update operations, thus preventing noise or mixed speech from contaminating the prototype.

[0078] As a concrete example, in the fintech field, in the aforementioned securities call scenario, a first threshold can be set. Second threshold Frame 150 (Above 0.8) and (Below 0.2), therefore identified as a high-confidence single-speaker speech frame, its frame-level speaker features will be used to update the prototype. Frame 320 (Above 0.8) but (Much higher than 0.2), does not meet the second condition, and is excluded from the update candidate. Frame 80 (Below 0.8), even Very low thresholds are also excluded. Through this dual-threshold screening, only truly high-quality single-person voice frames can enter the prototype update process, ensuring the purity of the speaker prototype.

[0079] As another concrete example, in the healthcare field, in the aforementioned remote consultation scenario, the system is also set up... , Frame 80 (Doctor's inquiry) and The 200th frame (doctor-patient overlap) was identified as a high-confidence single-person speech frame. Frame 350 corresponds to a patient whose voice is weak due to coughing. It is only 0.55, although The value is 0.03, but because the VAD probability is below the threshold, this frame will not be used for prototype updates. This design avoids mixing low-quality features into the prototype due to weak or unclear patient voices, ensuring the accuracy and representativeness of doctor and patient prototypes.

[0080] Step S163: Pool the high-confidence single-speaker speech frame and the frame-level speaker features in the neighborhood of the high-confidence single-speaker speech frame to obtain aggregated speaker features; Specifically, in this step, for the high-confidence single-speaker speech frame determined in step S162, the frame-level speaker features of this single frame are not directly used for prototype update. Instead, the frame-level speaker features of this frame and several frames in its neighborhood are used for pooling. The neighborhood range can be the K frames before and after the current frame (e.g., 3 frames before and after). The frames in the neighborhood also need to meet certain quality conditions (e.g., the VAD probability is higher than a certain low threshold) to avoid introducing obviously silent frames into the pooling. The pooling operation can be average pooling (averaging the feature vectors of all frames in the neighborhood dimension-wise), max pooling, or attention-weighted pooling, etc. The result after pooling is the aggregated speaker features. Through neighborhood pooling, the instantaneous fluctuations or noise interference that may exist in the single-frame feature extraction can be smoothed, making the obtained aggregated speaker features more stable and representative, and better reflecting the speaker's voiceprint characteristics at that moment.

[0081] As a concrete example in the fintech field, in the aforementioned securities call, frame 150 was identified as a high-confidence single-speaker voice frame. The system took frame-level speaker features from frames 147 to 153 (three frames before and after) and performed average pooling. Of these seven frames, except for frame 152, which had a slightly lower VAD probability due to instantaneous background noise, the other six frames were high-quality voice frames. The average value of these seven frame-level speaker feature vectors (each 256-dimensional) was calculated dimension by dimension to obtain a 256-dimensional aggregated speaker feature vector. Compared to using only the single-frame features of frame 150, the feature fluctuations caused by noise in frame 152 were smoothed and diluted by the normal features of the surrounding six frames in the pooled aggregated features. The final aggregated features more accurately reflected the stable voiceprint characteristics of the current speaker (such as an investment advisor).

[0082] As another concrete example in the healthcare field, in the aforementioned remote consultation, frame 80 (the doctor's inquiry) was identified as a high-confidence single-speaker voice frame. The system performs average pooling on the frame-level speaker features from frames 77 to 83. During the doctor's statement, "How have you been sleeping lately?", there are natural fluctuations in tone and speech rate, resulting in slight fluctuations in the single-frame features. Through pooling, the features of these seven frames are smoothly integrated, and the resulting aggregated speaker features effectively remove the instantaneous fluctuations of single frames while preserving the doctor's core voiceprint information, providing a more robust feature representation for subsequent prototype matching and updates.

[0083] Step S164: Calculate the similarity between the aggregated speaker features and each of the prototype feature vectors in the speaker prototype feature set; Specifically, in this step, the aggregated speaker features obtained in step S163 are compared with all currently stored prototype feature vectors in the speaker prototype feature set, and their similarity is calculated one by one. The similarity can be calculated using cosine similarity, negative exponent of Euclidean distance, or dot product, etc. For each prototype vector in the prototype feature set, a similarity value is calculated. Finally, the highest similarity is found from all similarity values ​​and denoted as . And record the prototype vector corresponding to the highest similarity.

[0084] As a concrete example in the fintech field, in the aforementioned securities call, the prototype feature set already stores two prototype vectors: prototype A corresponds to the investment advisor (256 dimensions), and prototype B corresponds to the client (256 dimensions). The aggregated speaker features obtained in step S163 (from the neighborhood of frame 150) are used to calculate cosine similarity with prototype A and prototype B respectively. The cosine similarity with prototype A is 0.94, and the cosine similarity with prototype B is 0.21. (Highest similarity...) This corresponds to prototype A. This indicates that the speaker (investment advisor) in the current frame is a high match with the stored "investment advisor" prototype.

[0085] As another concrete example in the healthcare field, in the aforementioned remote consultation, the prototype feature set already stores "doctor" and "patient" prototypes. Calculating the cosine similarity between the aggregated speaker features of frame 80 and these two prototypes yields a similarity of 0.91 with the doctor prototype and 0.12 with the patient prototype. (Highest similarity score...) This corresponds to the doctor prototype. This verifies that the doctor is indeed speaking in the current frame, consistent with the records in the prototype library.

[0086] Step S165: If the highest similarity is greater than or equal to a preset matching threshold, the aggregated speaker features are updated to the prototype feature vector corresponding to the highest similarity using a moving average method. Specifically, in this step, when When the current frame's speaker highly matches a speaker in the existing prototype feature set, it is considered to be the same speaker. In this case, instead of directly replacing the prototype with aggregated features, a moving average approach is used for updating. The update formula is: ; in, For the updated prototype vector, This is the currently stored prototype vector. The aggregated speaker features obtained in step three, The preset moving average coefficient . The closer the value is to 1, the slower and more stable the prototype update; The closer the value is to 0, the more sensitive and faster the prototype update is. The moving average update method allows the prototype vector to continuously absorb the feature changes of the same speaker at different time periods (such as timbre fine-tuning due to fatigue or emotional changes), while maintaining overall stability and not being disturbed by the instantaneous abnormal features of a certain frame.

[0087] As a concrete example, in the fintech sector, in the aforementioned securities call, Preset matching threshold ,satisfy Update prototype A (investment advisor) using a moving average method. Let... Aggregation features The current value of prototype A is the 256-dimensional vector obtained in step S163. After the update, the new prototype... After this update, prototype A shifts slightly towards the feature direction of the current frame, incorporating the investment advisor's latest voiceprint information from the current call. As the call progresses, the investment advisor's voice may undergo subtle changes due to prolonged speaking; the moving average mechanism automatically tracks these slow changes, ensuring the prototype always maintains an accurate representation of the speaker's current state.

[0088] As another specific example, in the field of healthcare, in the aforementioned remote consultations, Matching threshold The conditions are met. Update the doctor prototype using a moving average method. Let... The updated doctor prototype is 0.9. Old prototype + 0.1 Aggregation Features. In remote consultation scenarios, doctors may see multiple patients in a single morning, their voices gradually shifting from loud and clear to slightly hoarse. A moving average mechanism can adapt to this fatigue-induced timbre shift in a slow and smooth manner, ensuring the doctor's prototype remains accurate and doesn't deviate increasingly from the current frame's features due to gradual changes in voice.

[0089] Step S166: If the highest similarity is less than the matching threshold, then a prototype feature vector is added to the speaker prototype feature set, using the aggregated speaker features as the initial value.

[0090] Specifically, in this step when When the current speaker does not match any of the known speakers in the existing prototype feature set, it is inferred that a new speaker has appeared. At this point, using the aggregated speaker features obtained in step S163 as the initial value, a new prototype feature vector is added to the speaker prototype feature set. The newly added prototype vector is the aggregated speaker feature itself. Subsequent high-confidence single-person speech frames of this new speaker will continue to update this prototype using a moving average method, making it increasingly stable.

[0091] As a concrete example in the fintech field, in the aforementioned securities call, suppose that halfway through the call, the customer hands the phone to their spouse for confirmation. When the spouse speaks clearly for the first time, the system pools the neighborhood of that frame to obtain aggregated speaker features. The similarity of this aggregated feature with existing prototypes A (investment advisor) and B (customer) is then calculated to obtain... The value is only 0.28, below the matching threshold of 0.7. Using this aggregated feature as the initial value, the system adds a new prototype C to the prototype feature set, representing "customer's spouse". Subsequently, as the customer's spouse continues to speak, their aggregated features will match prototype C and be progressively updated and optimized. In this way, the prototype feature set dynamically expands to cover all speakers involved in the session.

[0092] As another concrete example in the healthcare field, during the aforementioned remote consultation, suppose that a patient's young child suddenly interrupts to ask a question. When the child speaks clearly for the first time, the similarity between their aggregated speaker features and the existing prototypes (doctor and patient prototypes) is only 0.19, below the matching threshold of 0.7. The system adds a new prototype C with this aggregated feature as the initial value. In subsequent conversations, the system can identify and track the new speaker's speech through prototype C, providing accurate speaker differentiation for possible subsequent family health management records.

[0093] Understandably, this embodiment achieves accurate and robust dynamic maintenance of the speaker prototype feature set through the organic coordination of four stages: dual-threshold screening, neighborhood pooling, similarity matching, and branch update. Dual-threshold screening ensures that only high-quality, non-overlapping single-speaker speech frames participate in prototype updates, preventing noise and overlapping frames from contaminating the prototype and guaranteeing the purity of the prototype vectors. Neighborhood pooling smooths the instantaneous fluctuations of single-frame features by utilizing temporal context information, resulting in aggregated speaker features that are more stable and representative than single-frame features. The similarity matching mechanism enables the system to accurately determine whether the current speaker is a known speaker or a new speaker, and adopts different update strategies for the two cases. For known speakers, a smooth update is achieved through moving averages, maintaining prototype stability while tracking slow changes in voiceprints; for new speakers, new prototypes are created promptly, achieving dynamic expansion of the prototype set. This maintenance mechanism allows the speaker prototype feature set to continuously self-optimize during the conversation, providing increasingly accurate and comprehensive speaker identity references for auxiliary discrimination calculations, thereby continuously improving the accuracy and robustness of overlapping speech detection.

[0094] Step S170: Calculate the similarity value between the frame-level speaker features of the current frame and each prototype feature vector in the speaker prototype feature set, as an auxiliary discrimination criterion; Specifically, in this step, for each frame being processed, the similarity between its frame-level speaker features and all prototype vectors in the feature set is calculated. Similarity calculation methods include, but are not limited to: cosine similarity, negative exponential transformation of Euclidean distance, dot product, or calculating a matching score using a small neural network. The final output auxiliary discrimination criterion can be the similarity value with the most similar prototype, a similarity vector, or a statistical measure of the similarity value (such as the maximum or average value). This criterion reflects the degree of association between the speaker's identity in the current frame and known speakers.

[0095] As a concrete example in the fintech field, during video interviews with robo-advisors, the speaker's features in the current frame are compared with stored customer and advisor prototypes. If the similarity to the customer prototype is as high as 0.95, the system knows that the current speaker is a customer, and vice versa. This provides crucial clues for speaker separation in overlap detection.

[0096] As another concrete example, in the field of healthcare, in the assessment of doctor-patient dialogue in psychiatry, doctors guide patients to recall by asking questions. If the current frame has a low similarity to both the doctor prototype and the patient prototype (e.g., 0.3 and 0.4 respectively) and complex acoustic features, it strongly suggests that the two are speaking in an overlapping manner. This is an important indicator for diagnosing communication patterns.

[0097] Step S180: Analyze the masking feature sequence in conjunction with the auxiliary discrimination criteria, determine frame by frame whether each speech frame of the speech to be detected is single-person speech or multi-person overlapping speech, and output the overlapping speech probability sequence.

[0098] Specifically, the masking feature sequence generated in S150 is fused with the auxiliary discrimination criteria generated in S170. The fusion method can be feature concatenation, weighted summation, gated fusion, or attention fusion. The fused features are input to the overlapping speech detection decoder, which can employ various sequence modeling architectures, such as recurrent neural networks, temporal convolutional networks, and Transformers, to output a continuous overlapping speech probability for each frame. The output probability represents the likelihood that the frame contains overlapping speech from multiple speakers, and can be further processed (such as thresholding and smoothing filtering) to obtain the final overlapping segments.

[0099] As a concrete example in the fintech field, during compliance audits of credit card telemarketing, the system outputs the overlap probability frame by frame. When a marketer interrupts a customer, the overlap probability quickly rises from 0.1 to 0.85. This period is recorded as a potential violation overlap, and the corresponding speaker prototypes are labeled as the marketer and the customer, making it easier for auditors to identify the responsible party.

[0100] As another concrete example in the healthcare field, during social communication assessments of children with autism, therapists intentionally create overlapping conversations to observe the child's responses. The system outputs a sequence of overlap probabilities for the entire conversation, precisely quantifying the duration and frequency of overlaps, providing objective behavioral data for clinical assessment.

[0101] Understandably, traditional methods use a hard masking approach that is either 0 or 1, meaning that the features of silent frames are directly set to zero while the features of speech frames are fully preserved. This lack of transition at the boundaries means that VAD decision bias can directly lead to abrupt feature changes and cause detection jitter.

[0102] This invention provides a fundamental solution. First, speech activity detection is performed to obtain a speech activity probability sequence. Unlike traditional hard decision methods, this step outputs not 0 / 1 labels, but a continuous probability value between 0 and 1 for each frame, preserving complete information about the possibility of speech presence. A continuous masking weight sequence is then generated based on this probability sequence, mapping the VAD probability to masking weights that also take continuous values ​​between 0 and 1. This continuous masking weight sequence is then used to weight the first feature sequence to obtain a masked feature sequence. Silent frames, due to their low VAD probability, are assigned weights approaching 0, significantly suppressing their corresponding acoustic features, but not completely reducing them to zero. Speech frames, due to their high VAD probability, are assigned weights approaching 1, preserving their features. At the boundary between silence and speech, the weights smoothly increase or decrease with the VAD probability, preventing abrupt changes in the feature vector. Thus, the problem of "abrupt boundary features caused by fixed threshold binarization" in the background technology is solved.

[0103] The background section points out that existing methods rely solely on acoustic features such as spectral energy to determine overlap. These acoustic features primarily characterize the "content" and "loudness" of the sound, rather than "who is speaking." When a single person speaks loudly and emotionally, their spectral morphology is highly similar to that of overlapping speech from multiple people, both exhibiting high energy and complex frequency distributions, making it impossible to distinguish them based on acoustic features alone.

[0104] This invention maintains a speaker prototype feature set, which contains at least one speaker prototype feature vector, used to store and track representative voiceprint features of different speakers during dialogue. The similarity value between the frame-level speaker features of the current frame and each prototype feature vector in the speaker prototype feature set is further calculated as an auxiliary discrimination criterion. This auxiliary discrimination criterion is then combined with a masking feature sequence to jointly determine single-speaker or overlapping speech frame by frame.

[0105] The key to this approach lies in the fact that the speaker prototype feature vector stores identity information—"who is speaking"—rather than energy information—"how loud they are speaking." When a single person is emotionally agitated and speaks loudly, although the acoustic features exhibit high energy and a complex spectrum, their frame-level speaker features still maintain a high similarity to an existing prototype in the speaker prototype feature set. This is because the voiceprint characteristics of loud and soft speaking (depending on the physiological structure of the vocal organs) are inherently stable and do not change with volume. This high similarity value serves as an auxiliary discrimination criterion, providing a crucial disambiguation signal for overlap detection. When the model receives the information that "the speaker in the current frame highly matches a known speaker A," it can infer that speaker A is still speaking alone, just louder, thus avoiding misjudgment as overlap. Conversely, when multiple people overlap, the frame-level speaker features of the mixed speech will not match any known prototype in the feature set, resulting in a low similarity value. Based on this, the model judges that overlap has indeed occurred. Thus, the problem of "misjudgment caused by the similarity of the spectra of high-energy single-person speech and overlapping multi-person speech" in the background technology is solved.

[0106] In some embodiments of the present invention, such as Figure 5 As shown, step S180 includes the following steps.

[0107] Step S181: Calculate the information entropy of the frame-level speaker features of the current frame; Specifically, in this step, for the current frame Obtain the extracted frame-level speaker feature vector, denoted as... , dimension First, apply a softmax function to the feature vector to transform it into a probability distribution, i.e., for the feature vector... Each dimension has a normalized value. ,in Then calculate the information entropy of this probability distribution. Summing across all dimensions. Information entropy. The value of reflects the degree of concentration or dispersion of activation within the speaker feature vector of the current frame. The smaller the value, the more concentrated the energy of the feature vector is in a few dimensions, and the clearer and more definite the speaker's features are; The larger the value, the more dispersed the energy of the feature vector is across multiple dimensions, indicating that the speaker's features are more chaotic and uncertain.

[0108] As a concrete example in the fintech field, in the telephone sales monitoring of a bank's intelligent risk control system, the system analyzes a recorded call. In one frame, the salesperson is clearly and steadily introducing the product terms. After softmax normalization, the frame-level speaker feature vector shows energy highly concentrated in a few dimensions (e.g., the cumulative probability of the first three largest dimensions exceeds 0.8), and the information entropy is calculated. The level is relatively low. In another frame, a salesperson and a customer are having a heated argument; their voices appear simultaneously and intertwine. The speaker feature vectors in this frame show relatively uniform activation across all dimensions, with dispersed energy. The calculated information entropy is... The entropy level is relatively high. This significant difference in entropy values ​​provides crucial clues for subsequent judgments.

[0109] As another concrete example, in the healthcare field, during a remote rehabilitation guidance session, a therapist is instructing a patient on breathing exercises. While the patient individually follows the instructions to take deep breaths and count, their voice may tremble slightly due to physical activity, but the speaker's feature vector remains relatively concentrated, allowing for the calculation of information entropy. When a therapist notices a patient's improper movements and immediately corrects them verbally, creating an overlapping segment where both speakers are talking simultaneously, the mixed voices cause all dimensions of the speaker's feature vector to be activated simultaneously, increasing information entropy. It jumps to 0.85. This change in entropy accurately captures the characteristic chaotic state at the time of overlap.

[0110] Step S182: The masking feature sequence, the similarity value, and the information entropy are concatenated to obtain the concatenated features; Specifically, in this step, after completing the entropy calculation in step S181, for the current frame... The system has obtained three features: (1) the masking feature vector, denoted as Its dimensions (2) Similarity value, denoted as , is a scalar; (3) The information entropy calculated in step S181 is denoted as This is also a scalar. Concatenating the three along the feature dimension yields the concatenated feature vector. . Specifically, The semicolon ";" indicates a vector concatenation operation. The dimension is This splicing feature simultaneously encodes the acoustic environment information of the current frame (the effective speech acoustic features retained after VAD masking), the speaker identity matching information (the degree of similarity with the known prototype), and the uncertainty information of the speaker features (the degree of feature confusion), forming a comprehensive representation with complementary information.

[0111] As another concrete example, in the fintech field, during a compliance check of an investment advisor's call at a securities firm, the system processes a particular frame. The masking feature vector of this frame... (256-dimensional) Contains masked acoustic spectrum information; similarity value = 0.89, a single value, indicating a high degree of match between this frame and the "investment advisor" prototype; Information entropy This is also a single numerical value. The three elements are concatenated to form a 258-dimensional concatenated feature vector. This vector carries three signals simultaneously: "acoustic features", "high identity matching degree" and "feature stability", and is sent to the subsequent decoder.

[0112] As another concrete example, in the healthcare field, during the analysis of audio recordings from a remote multidisciplinary consultation, the system processes an overlapping suspected frame. This frame masks the feature vector. (256-dimensional) From the mixed acoustic signal after VAD soft masking; similarity value This indicates that the frame does not match any known doctor prototype in the feature set; information entropy This indicates that the speaker features in this frame have a high degree of confusion. The concatenated vector forms a 258-dimensional vector. The vector carries signals that are: "complex acoustic features", "low identity matching degree" and "high feature confusion degree", which together indicate overlapping speech.

[0113] Step S183: Based on the splicing features, determine frame by frame whether each speech frame is a single person's speech or multiple people's overlapping speech.

[0114] Specifically, in this step, the obtained concatenated feature vectors can be... The input is fed into the overlapping speech detection decoder. This decoder can be any neural network architecture suitable for sequence modeling, such as a recurrent neural network, a temporal convolutional network, or a Transformer. The decoder reads... Based on the mapping relationship learned internally, it outputs a probability value between 0 and 1 for the current frame. This indicates the possibility that the frame contains overlapping speech from multiple people. The closer the value is to 1, the more likely the frame is to contain overlapping speech from multiple people; The closer the value is to 0, the more likely the frame is to be a single-person speech sequence. The decoder processes all frames sequentially, ultimately outputting a complete sequence of overlapping speech probabilities. .

[0115] As a concrete example in the fintech field, in the aforementioned investment advisor call scenario, feature splicing... (High similarity, low entropy) samples are fed into the decoder, which outputs the overlap probability. It was determined to be a single-person speech. In another frame, splicing features... (Low similarity, high entropy) is fed into the decoder, and the decoder outputs... The frame was identified as overlapping speech and marked as a violation point where the customer interrupted the consultant, for use in subsequent compliance review processes.

[0116] As another concrete example, in the healthcare field, within the aforementioned multi-disciplinary team (MDT) consultation scenario, feature splicing... (High similarity, low entropy) is fed into the decoder, and the decoder outputs... This is determined to be a single doctor speaking alone. When two doctors disagree on a treatment plan and simultaneously state their views, the splicing feature becomes... (Low similarity, high entropy), decoder output The system identifies overlapping speech as the cause of the overlap. Based on this, the system records the dialogue segment corresponding to the overlap, providing a time-based reference for subsequent meeting minutes compilation and decision disagreement analysis.

[0117] Understandably, this embodiment achieves a discriminative capability beyond simple acoustic analysis by concatenating information entropy with masking features and similarity values. Information entropy, starting from the internal structure of the speaker's feature vector, quantifies the "degree of disorder" of the speaker's information in the current frame. This indicator is highly correlated with "whether it is multi-person overlapping speech." When information entropy is added as an independent feature, the decoder can use complementary information from three dimensions—acoustic features (masking features), identity matching information (similarity values), and feature disorder (information entropy)—to perform comprehensive reasoning. This effectively distinguishes between "single-person high-energy speech" (low entropy, high similarity) and "multi-person real overlapping speech" (high entropy, low similarity), significantly reducing the misjudgment rate in these two easily confused scenarios and improving the detection accuracy of real overlapping speech.

[0118] In some embodiments of the present invention, the overlapping speech detection method is performed using a neural network model obtained through cross-domain adaptive training, the training process of which includes: Step S210: The neural network model is pre-trained using a simulation dataset. Each simulated audio sample in the simulation dataset includes single-person speaking segments and multiple-person overlapping speaking segments, and the time proportion corresponding to the number of each speaker covers multiple distribution ratios. The simulation dataset also covers multiple noise types, multiple signal-to-noise ratios, and multiple reverberation conditions. Specifically, in this step, a large-scale simulation dataset can be constructed for model pre-training. The simulation dataset is constructed by collecting a large number of clean speech samples from different speakers and recording or collecting various types of background noise samples. Then, the speech samples from different speakers are programmatically mixed and superimposed according to preset rules to generate simulated conference audio.

[0119] When generating simulated audio, the number of speakers is configured to cover multiple proportional distributions. Specifically, each simulated audio clip includes single-speaker segments, two-speaker overlapping segments, and three- or more-speaker overlapping segments, with different proportions of segments in the total duration combined according to different distribution ratios. For example, in the first set of simulated data, single-speaker segments account for 50%, two-speaker overlapping segments for 30%, and three- or more-speaker overlapping segments for 20%; in the second set, single-speaker segments account for 30%, two-speaker overlapping segments for 50%, and three- or more-speaker overlapping segments for 20%; in the third set, single-speaker segments account for 70%, two-speaker overlapping segments for 20%, and three- or more-speaker overlapping segments for 10%, and so on. This construction method, covering multiple distribution ratios, allows the model to be exposed to diverse overlap density scenarios during the pre-training stage, avoiding overfitting to a specific overlap ratio.

[0120] Furthermore, the simulation dataset covers a variety of noise types (such as white noise, pink noise, street noise, office air conditioning noise, and background voices in a coffee shop), various signal-to-noise ratios (such as -5 dB, 0 dB, 5 dB, 10 dB, 15 dB, and 20 dB), and various reverberation conditions (such as no reverberation, reverberation in a small room, reverberation in a medium-sized conference room, and reverberation in a large lecture hall). Through these multi-dimensional data enhancements, the simulation dataset can simulate extremely rich acoustic scenarios.

[0121] Furthermore, the neural network model is pre-trained using the constructed simulation dataset. The model's backbone network (including the acoustic feature encoder, speaker feature extractor, VAD decoder, cross-attention module, OSD decoder, etc.) learns from a randomly initialized state during this stage. Pre-training employs standard supervised learning methods, with loss functions including the mean squared error loss of VAD and OSD, aiming to simultaneously improve the accuracy of speech activity detection and overlapping speech detection.

[0122] As a concrete example, in the fintech field, a fintech company developed a voice quality inspection system for remote bank account opening video verification. During the model pre-training phase, the company utilized internally accumulated anonymized voice data and publicly available voice datasets to construct a simulation dataset covering various financial business scenarios. In the simulation audio, single-person speaking segments simulated scenarios such as a customer reading an account opening statement alone, or a customer service representative asking for identity information alone; overlapping segments between two people simulated scenarios where the customer and customer simultaneously answered questions while the customer service representative confirmed information; and overlapping segments between three people simulated scenarios where the customer, customer service, and risk control personnel communicated together. Different overlap ratios covered various dialogue modes, ranging from "customer service-led, brief customer responses" to "frequent interaction between customer and customer service." Noise types included common bank branch noises such as queuing calls, ATM machine operation, and teller cash counting sounds, with signal-to-noise ratios ranging from noisy lobby environments (5 dB) to quiet VIP rooms (25 dB). After pre-training on this simulation dataset, the model mastered the basic ability to recognize overlapping speech in financial scenarios.

[0123] As another concrete example, in the healthcare field, an internet healthcare platform introduced overlapping speech detection functionality into its online consultation system to analyze the quality of doctor-patient communication. During the pre-training phase, a simulation dataset covering various medical consultation scenarios was constructed. In the simulated audio, single-person segments simulated doctors asking about medical history and patients describing symptoms; two-person overlapping segments simulated common consultation interaction patterns such as doctors interrupting patients to ask follow-up questions and patients eagerly providing additional information; and three-person overlapping segments simulated scenarios where doctors, patients, and their families discuss treatment plans together. Different overlap ratios covered various consultation communication styles, including "doctor-led," "patient-centered," and "family-involved." Noise types included background noise in hospital waiting areas, instrument prompts in examination rooms, and telephone ringing, with a signal-to-noise ratio ranging from quiet specialist clinics (20 dB) to noisy emergency triage areas (8 dB). Reverberation conditions simulated the acoustic characteristics of small examination rooms, medium-sized examination rooms, and large waiting halls. After pre-training on this dataset, the model possessed the basic ability to recognize overlapping speech in medical scenarios.

[0124] Step S220: Fine-tune the pre-trained neural network model using a real meeting scenario dataset. During the fine-tuning process, optimize the adaptive parameters involved in generating the continuous masking weight sequence. Oversample overlapping speech frames in the real meeting scenario dataset during fine-tuning, and dynamically adjust the intensity of oversampling according to the fine-tuning process. Specifically, in this step, after the pre-training in step S210 is completed, the model has basic VAD and OSD capabilities. However, due to the unavoidable domain differences between simulation data and real data (such as natural pause patterns, emotional changes, and non-verbal sounds in real dialogues), it is necessary to fine-tune the model using a real meeting scenario dataset.

[0125] The real-world meeting scenario dataset consists of actual recorded multi-person dialogue audio, such as internal company meeting recordings, project discussion audio, and group interview recordings, with frame-level VAD and OSD annotations. During fine-tuning, all model parameters continue to participate in updates, but with a smaller learning rate.

[0126] During fine-tuning, the focus is on optimizing the adaptive parameters involved in generating the continuous masking weight sequence. These parameters are the gating kurtosis parameters. and adaptive threshold During the pre-training phase, and It may be set to a general default value (such as...) , When fine-tuning on real data, the model automatically adjusts based on the probability distribution characteristics of VAD in real-world scenarios. and The value of is chosen to match the transition characteristics of the masking weights with the statistical patterns of speech activity in real data. For example, if pauses between speakers are frequent and brief in real-world scenarios, It may be fine-tuned to a lower value to avoid misinterpreting short pauses as silent segments.

[0127] Simultaneously, overlapping speech frames in the real-world meeting scenario dataset are oversampled during fine-tuning. In real-world meeting data, the number of overlapping speech frames is typically far less than that of single-person speech frames (e.g., single-person frames account for more than 80%, while overlapping frames account for less than 20%). This class imbalance can lead to insufficient attention paid to overlapping frames during model fine-tuning. The specific oversampling operation is as follows: in each training cycle, overlapping speech frames are repeatedly sampled to make their number roughly equal to that of single-person speech frames. Furthermore, the intensity of oversampling is dynamically adjusted according to the fine-tuning process. In the early stages of fine-tuning, the oversampling intensity is high (e.g., repeating overlapping frames to the same number as single-person frames) to quickly correct the model's bias towards single-person speech. As fine-tuning progresses, the oversampling intensity gradually decreases (e.g., adjusting the number of overlapping frames to 60% or 80% of the number of single-person frames), allowing the model to encounter data closer to the true distribution in the later stages of training, thereby achieving a balance between maintaining class balance and fitting the true distribution.

[0128] As a concrete example in the fintech field, in the aforementioned remote bank account opening scenario, after the fintech company completed model pre-training, it collected 500 hours of real remote face-to-face interview recording data. A professional annotation team completed frame-level VAD and OSD annotations, constructing a real-world meeting scenario dataset. During fine-tuning, the model discovered that the probability distribution of customer and customer service voice activities in real face-to-face interview scenarios differed from the simulation data. In real interviews, customer service representatives' voices were more formal and continuous, while customers' voices were shorter and more intermittent. This difference was addressed by optimizing the gating steepness. and adaptive threshold The model increases the sensitivity of the masking weights to short speech responses, ensuring that brief customer responses such as "yes" or "correct" are not overly suppressed by masking. Meanwhile, since overlapping segments account for approximately 12% of the real data (primarily occurring when customers preemptively answer while verifying information), far lower than single-person segments, oversampling of overlapping frames was performed during fine-tuning. The initial oversampling rate was 8 times (making the number of overlapping frames close to 1:1 with the number of single-person frames), gradually reduced to 3 times, and finally reduced to no oversampling. This strategy ensures the model maintains high sensitivity to real-world short-term overlapping and interruption phenomena.

[0129] As another concrete example, in the healthcare field, in the aforementioned online medical consultation scenario, the platform collected 300 hours of real online consultation recordings, covering multiple departments such as internal medicine, pediatrics, and psychological counseling, and completed frame-level annotation. During fine-tuning, the model found significant differences in dialogue patterns across different departments. In internal medicine consultations, doctors led the questioning, and patients passively answered, with less overlap; in psychological counseling, clients experienced greater emotional fluctuations, resulting in a higher frequency of overlap. By optimizing adaptive parameters, the model can flexibly adjust masking responses in different departmental dialogue patterns. Simultaneously, when oversampling overlapping frames, the initial oversampling rate was differentiated based on the overlap frequency of each department: in the psychological counseling data, overlapping frames already accounted for approximately 25%, so the initial oversampling rate was set to 4 times; in the internal medicine consultation data, overlapping frames accounted for only about 6%, so the initial oversampling rate was set to 16 times. This department-specific differentiated oversampling strategy enables the model to effectively recognize overlapping speech in different departmental scenarios.

[0130] Step S230: In the deployment environment, the neural network model is adapted using unlabeled scene data. During the adaptation process, the parameters of the backbone network are fixed, the parameters of the probability calibration layer and the statistics of the speaker prototype feature set are updated, and the consistency of the output results of the same speech under different data augmentation methods is used as a regularization constraint to optimize the adaptation effect.

[0131] Specifically, in this step, after fine-tuning in step S220, the model possesses good detection capabilities in real-world meeting scenarios. However, there may still be domain shifts between the actual deployment environment and the training data (e.g., different recording devices, different room acoustics, different regional accents, etc.). Therefore, after deploying the model to a specific target environment, lightweight adaptation is performed using unlabeled scene data collected in that environment.

[0132] During the adaptation process, the backbone network parameters of the fixed model (including all core network layers such as acoustic feature encoder, speaker feature extractor, VAD decoder, cross attention module, OSD decoder, etc.) are updated only two types of parameters: one is the probability calibration layer parameters, namely the temperature scaling parameter T or Platt calibration parameters a and b; the other is the statistics of the speaker prototype feature set, namely the values ​​of each prototype feature vector.

[0133] The probability calibration layer parameters are updated by using unlabeled data to find calibration parameters that best match the VAD probability distribution with the expected distribution (such as the empirical value of the proportion of speech frames in the actual deployment environment). The statistics of the speaker prototype feature set are updated by directly using frames with high VAD probabilities in the unlabeled data and updating the prototype feature vector through clustering or moving average to adapt it to the speaker group characteristics in the current deployment environment.

[0134] Simultaneously, a consistency regularization constraint is introduced to optimize the adaptive effect. Specifically, for the same unlabeled audio segment, different data augmentation methods are applied (e.g., adding different levels of Gaussian noise, changing playback speed, adjusting volume), resulting in multiple augmented versions. These augmented versions are then input into the model to obtain their respective outputs (VAD probability sequences and OSD probability sequences). Consistency regularization requires that the outputs of the same audio segment under different augmentation methods should remain consistent. This consistency requirement serves as a constraint, guiding the update direction of calibration layer parameters and prototype statistics. This constraint ensures that the adaptive adjustments learned by the model are based on the stable characteristics of the audio content itself, rather than overfitting to specific augmentation methods.

[0135] As a concrete example in the fintech field, a bank deployed this overlapping speech detection system to the wealth management sales areas of its branches nationwide. Although the second phase involved fine-tuning using real face-to-face interview data from the head office, differences existed in the brands of recording equipment, pickup distances, and room sizes across different branches. After deployment in each branch, the system underwent adaptive testing using unlabeled dual-recorded audio accumulated within a week at each branch. During this adaptation process, the backbone network parameters were frozen, and only the temperature scaling parameter T and the prototype feature vectors of wealth management managers and customers frequently encountered at that branch were updated. Consistency regularization constraints ensured that the model maintained consistent output even when faced with slightly different echo characteristics across different branches. After adaptation, the model's VAD probability calibration for the specific recording environment of that branch was more accurate, and the distinguishability of speaker prototypes was also higher.

[0136] As another concrete example, in the healthcare field, a telemedicine platform deployed its overlapping speech detection system to remote consultation rooms in different partner hospitals. The microphone models, room sizes, and participating physician groups varied across these hospitals. After deployment in each hospital, the system adaptively utilized unlabeled consultation recordings accumulated at that hospital. During adaptation, the model backbone remained fixed, and only the probabilistic calibration layer parameters and the prototype feature vectors of commonly participating physicians in that hospital were updated. By applying consistency regularization constraints, the model maintained consistent VAD and OSD outputs under different levels of audio enhancement processing (such as simulating different microphone gains), thereby learning robust calibration parameters insensitive to specific device characteristics. After adaptation, the system was better able to adapt to the specific acoustic environment of each hospital, resulting in more accurate detection of overlapping discussion segments between doctors within that hospital.

[0137] Understandably, this embodiment systematically addresses the domain shift problem faced by deep learning speech models during cross-domain deployment through a three-stage training strategy: "simulation pre-training – real-world fine-tuning – unlabeled adaptive training." The first stage utilizes simulated datasets covering various overlap ratios, noise types, signal-to-noise ratios, and reverberation conditions for pre-training, enabling the model to achieve good initial generalization capabilities under diverse acoustic scenarios. The second stage fine-tunes on real data and introduces dynamic oversampling, mitigating the class imbalance problem caused by the scarcity of overlapping frames in real-world scenarios. Simultaneously, it optimizes adaptive parameters to make masking weights more closely match the distribution of real data. The third stage, combining unlabeled adaptive training with consistency regularization, allows the model to adapt to the specific acoustic characteristics of the deployment environment without requiring new labeled data, reducing performance degradation when deployed across devices, rooms, and speaker groups. This three-stage training gradually bridges the distribution differences between simulation and reality, and between the training and deployment environments, significantly improving the robustness and transferability of the overlapping speech detection method in practical applications.

[0138] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0139] In one embodiment, an overlapping speech detection device is provided, which corresponds one-to-one with the overlapping speech detection method described in the above embodiments. For example... Figure 6 As shown, the overlapping speech detection device includes an acquisition module 610, a detection module 620, a weight generation module 630, a feature weighting module 640, a feature maintenance module 650, a similarity calculation module 660, and a judgment module 670. Detailed descriptions of each functional module are as follows: The acquisition module 610 is used to acquire the acoustic feature sequence and frame-level speaker feature sequence of the speech to be detected; Detection module 620 is used to detect speech activity based on the acoustic feature sequence to obtain a speech activity probability sequence; The weight generation module 630 is used to generate a continuous masking weight sequence based on the speech activity probability sequence. The feature weighting module 640 is used to perform weighting processing on the first feature sequence according to the continuous masking weight sequence to obtain a masking feature sequence, wherein the first feature sequence is the acoustic feature sequence or a fused feature sequence obtained by fusing the acoustic feature sequence with the frame-level speaker feature sequence; The feature maintenance module 650 is used to maintain a speaker prototype feature set containing at least one speaker prototype feature vector; The similarity calculation module 660 is used to calculate the similarity value between the frame-level speaker features of the current frame and each of the prototype feature vectors in the speaker prototype feature set, as an auxiliary discrimination basis; The judgment module 670 is used to analyze the masking feature sequence in conjunction with the auxiliary judgment criteria, judge each speech frame of the speech to be detected as either single-person speech or multi-person overlapping speech, and output the overlapping speech probability sequence.

[0140] In one embodiment, the feature weighting module 640 is specifically used for: The masking weight of each frame in the continuous masking weight sequence is calculated according to the sigmoid function, wherein the input of the sigmoid function is the product of the gating steepness parameter and the difference between the current frame speech activity probability and the adaptive threshold, and the masking weight ranges from 0 to 1.

[0141] In one embodiment, the detection module 620 is further configured to: The speech activity probability sequence is probabilistically calibrated by temperature scaling and / or Platt calibration to obtain a calibrated speech activity probability sequence used to generate the continuous masking weight sequence.

[0142] In one embodiment, the determination module 670 is specifically used for: Calculate the information entropy of the frame-level speaker features in the current frame; The masking feature sequence, the similarity value, and the information entropy are concatenated to obtain the concatenated feature; Based on the splicing features, each audio frame is determined to be either single-person speech or overlapping speech by multiple people.

[0143] In one embodiment, the acquisition module 610 is further configured to: The acoustic front-end features of the speech to be detected are processed by a pre-trained speaker model to obtain an initial frame-level speaker representation. Using the acoustic feature sequence as the query and the initial frame-level speaker representation as the key and value, frame-level feature fusion is performed through a cross-attention mechanism to obtain the frame-level speaker feature sequence and form the fused feature sequence.

[0144] In one embodiment, the overlapping speech detection device further includes a training module 680, which is specifically used for: The neural network model is pre-trained using a simulation dataset. Each simulated audio sample in the simulation dataset includes single-person speaking segments and multiple-person overlapping speaking segments, and the time proportion corresponding to the number of each speaker covers multiple distribution ratios. The simulation dataset also covers multiple noise types, multiple signal-to-noise ratios, and multiple reverberation conditions. The pre-trained neural network model is fine-tuned using a real meeting scenario dataset. During the fine-tuning process, the adaptive parameters involved in generating the continuous masking weight sequence are optimized. Oversampling is performed on overlapping speech frames in the real meeting scenario dataset during fine-tuning, and the intensity of oversampling is dynamically adjusted according to the fine-tuning process. In the deployment environment, the neural network model is adapted using unlabeled scene data. During the adaptation process, the parameters of the backbone network are fixed, the parameters of the probability calibration layer and the statistics of the speaker prototype feature set are updated, and the consistency of the output results of the same speech under different data augmentation methods is used as a regularization constraint to optimize the adaptation effect.

[0145] In one embodiment, the feature maintenance module 650 is specifically used for: Obtain the speech activity probability and the overlapping speech probability of each speech frame of the speech to be detected; Frames with a voice activity probability higher than a first threshold and an overlap voice probability lower than a second threshold are identified as high-confidence single-person voice frames. Pooling is performed on the high-confidence single-speaker speech frame and the frame-level speaker features in the neighborhood of the high-confidence single-speaker speech frame to obtain aggregated speaker features; Calculate the similarity between the aggregated speaker features and each of the prototype feature vectors in the speaker prototype feature set; If the highest similarity is greater than or equal to the preset matching threshold, the aggregated speaker features are updated to the prototype feature vector corresponding to the highest similarity using a moving average method. If the highest similarity is less than the matching threshold, then a prototype feature vector is added to the speaker prototype feature set, using the aggregated speaker features as the initial value.

[0146] Understandably, traditional methods use a hard masking approach that is either 0 or 1, meaning that the features of silent frames are directly set to zero while the features of speech frames are fully preserved. This lack of transition at the boundaries means that VAD decision bias can directly lead to abrupt feature changes and cause detection jitter.

[0147] This invention provides a fundamental solution. First, speech activity detection is performed to obtain a speech activity probability sequence. Unlike traditional hard decision methods, this step outputs not 0 / 1 labels, but a continuous probability value between 0 and 1 for each frame, preserving complete information about the possibility of speech presence. A continuous masking weight sequence is then generated based on this probability sequence, mapping the VAD probability to masking weights that also take continuous values ​​between 0 and 1. This continuous masking weight sequence is then used to weight the first feature sequence, resulting in a masked feature sequence. Silent frames, due to their low VAD probability, are assigned weights approaching 0, significantly suppressing their corresponding acoustic features, but not completely reducing them to zero. Speech frames, due to their high VAD probability, are assigned weights approaching 1, preserving their features. At the boundary between silence and speech, the weights smoothly increase or decrease with the VAD probability, preventing abrupt changes in the feature vector. Thus, the problem of "fixed threshold binarization leading to abrupt changes in boundary features" in the background technology is solved.

[0148] The background section points out that existing methods rely solely on acoustic features such as spectral energy to determine overlap. These acoustic features primarily characterize the "content" and "loudness" of the sound, rather than "who is speaking." When a single person speaks loudly and emotionally, their spectral morphology is highly similar to that of overlapping speech from multiple people, both exhibiting high energy and complex frequency distributions, making it impossible to distinguish them based on acoustic features alone.

[0149] The present invention maintains a speaker prototype feature set, which contains at least one speaker prototype feature vector, used to store and track representative voiceprint features of different speakers during dialogue. The similarity value between the frame-level speaker features of the current frame and each prototype feature vector in the speaker prototype feature set is further calculated as an auxiliary discrimination criterion. This auxiliary discrimination criterion is then combined with a masking feature sequence to jointly determine single-speaker or overlapping speech frame by frame.

[0150] The key to this approach lies in the fact that the speaker prototype feature vector stores identity information—"who is speaking"—rather than energy information—"how loud they are speaking." When a single person is emotionally agitated and speaks loudly, although the acoustic features exhibit high energy and a complex spectrum, their frame-level speaker features still maintain a high similarity to an existing prototype in the speaker prototype feature set. This is because the voiceprint characteristics of loud and soft speaking (depending on the physiological structure of the vocal organs) are inherently stable and do not change with volume. This high similarity value serves as an auxiliary discrimination criterion, providing a crucial disambiguation signal for overlap detection. When the model receives the information that "the speaker in the current frame highly matches a known speaker A," it can infer that speaker A is still speaking alone, just louder, thus avoiding misjudgment as overlap. Conversely, when multiple people overlap, the frame-level speaker features of the mixed speech will not match any known prototype in the feature set, resulting in a low similarity value. Based on this, the model judges that overlap has indeed occurred. Thus, the problem of "misjudgment caused by the similarity of the spectra of high-energy single-person speech and overlapping multi-person speech" in the background technology is solved.

[0151] Based on the above overlapping speech detection methods, such as Figure 7 As shown in the diagram, this embodiment of the invention also provides a structural schematic of an apparatus for overlapping speech detection method steps. The apparatus includes a processor 71 and a memory 72 coupled to the processor 71. The memory 72 stores a computer program, which, when executed by the processor 71, causes the processor 71 to perform the steps of the overlapping speech detection method described in the above embodiment.

[0152] For further details regarding the implementation of the above-described overlapping speech detection method steps by the processor 71 in the device, please refer to the description of the overlapping speech detection method provided in the above-described embodiments of the invention, which will not be repeated here.

[0153] The processor 71 can also be called a CPU (Central Processing Unit). The processor 71 may be an integrated circuit chip with signal processing capabilities. The processor 71 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or the processor 71 can be any conventional processor.

[0154] like Figure 8 As shown in the diagram, this embodiment of the invention also provides a schematic diagram of a computer-readable storage medium, on which a readable computer program 81 is stored. The computer program 81 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in various embodiments of the invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks or optical disks, ROM (Read-Only Memory), RAM (Random Access Memory), or terminal devices such as computers, servers, mobile phones, and tablets.

[0155] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, apparatuses, or modules, and may be electrical, mechanical, or other forms.

[0156] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0157] Furthermore, the functional modules in the various embodiments of this invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0158] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0159] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium (e.g., SSD (solid state disk)).

[0160] The technical solution provided by the present invention has been described in detail above. Specific examples have been used to illustrate the principle and implementation of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation and application scope based on the idea of ​​the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0161] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, optical storage, etc.) containing computer-usable program code.

[0162] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0166] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for detecting overlapping speech, characterized in that, include: Obtain the acoustic feature sequence and frame-level speaker feature sequence of the speech to be detected; Speech activity detection is performed based on the acoustic feature sequence to obtain a speech activity probability sequence; Generate a continuous masking weight sequence based on the speech activity probability sequence; The first feature sequence is weighted according to the continuous masking weight sequence to obtain a masking feature sequence, wherein the first feature sequence is the acoustic feature sequence or a fused feature sequence obtained by fusing the acoustic feature sequence with the frame-level speaker feature sequence; Maintain a speaker prototype feature set that contains at least one speaker prototype feature vector; Calculate the similarity value between the frame-level speaker features of the current frame and each prototype feature vector in the speaker prototype feature set, as an auxiliary discrimination criterion; By combining the aforementioned auxiliary discrimination criteria with the analysis of the masking feature sequence, the system determines frame by frame whether each speech frame of the speech to be detected is a single person's speech or multiple people's overlapping speech, and outputs an overlapping speech probability sequence.

2. The overlapping speech detection method according to claim 1, characterized in that, The step of generating a continuous masking weight sequence based on the speech activity probability sequence includes: The masking weight of each frame in the continuous masking weight sequence is calculated according to the sigmoid function, wherein the input of the sigmoid function is the product of the gating steepness parameter and the difference between the current frame speech activity probability and the adaptive threshold, and the masking weight ranges from 0 to 1.

3. The overlapping speech detection method according to claim 2, characterized in that, After performing speech activity detection based on the acoustic feature sequence to obtain a speech activity probability sequence, and before generating a continuous masking weight sequence based on the speech activity probability sequence, the method further includes: The speech activity probability sequence is probabilistically calibrated by temperature scaling and / or Platt calibration to obtain a calibrated speech activity probability sequence used to generate the continuous masking weight sequence.

4. The overlapping speech detection method according to claim 1, characterized in that, The step involves analyzing the masking feature sequence based on the auxiliary discrimination criteria, determining frame by frame whether each speech frame of the speech to be detected is single-person speech or multi-person overlapping speech, and outputting an overlapping speech probability sequence, including: Calculate the information entropy of the frame-level speaker features in the current frame; The masking feature sequence, the similarity value, and the information entropy are concatenated to obtain the concatenated feature; Based on the splicing features, each audio frame is determined to be either single-person speech or overlapping speech by multiple people.

5. The overlapping speech detection method according to claim 1, characterized in that, Obtain the frame-level speaker feature sequence of the speech to be detected, including: The acoustic front-end features of the speech to be detected are processed by a pre-trained speaker model to obtain an initial frame-level speaker representation. Using the acoustic feature sequence as the query and the initial frame-level speaker representation as the key and value, frame-level feature fusion is performed through a cross-attention mechanism to obtain the frame-level speaker feature sequence and form the fused feature sequence.

6. The overlapping speech detection method according to any one of claims 1 to 5, characterized in that, The overlapping speech detection method is executed through a neural network model obtained through cross-domain adaptive training. The training process of the neural network model includes: The neural network model is pre-trained using a simulation dataset. Each simulated audio sample in the simulation dataset includes single-person speaking segments and multiple-person overlapping speaking segments, and the time proportion corresponding to the number of each speaker covers multiple distribution ratios. The simulation dataset also covers multiple noise types, multiple signal-to-noise ratios, and multiple reverberation conditions. The pre-trained neural network model is fine-tuned using a real meeting scenario dataset. During the fine-tuning process, the adaptive parameters involved in generating the continuous masking weight sequence are optimized. Oversampling is performed on overlapping speech frames in the real meeting scenario dataset during fine-tuning, and the intensity of oversampling is dynamically adjusted according to the fine-tuning process. In the deployment environment, the neural network model is adapted using unlabeled scene data. During the adaptation process, the parameters of the backbone network are fixed, the parameters of the probability calibration layer and the statistics of the speaker prototype feature set are updated, and the consistency of the output results of the same speech under different data augmentation methods is used as a regularization constraint to optimize the adaptation effect.

7. The overlapping speech detection method according to claim 1, characterized in that, The maintenance of the speaker prototype feature set, which contains at least one speaker prototype feature vector, includes: Obtain the speech activity probability and the overlapping speech probability of each speech frame of the speech to be detected; Frames with a voice activity probability higher than a first threshold and an overlap voice probability lower than a second threshold are identified as high-confidence single-person voice frames. Pooling is performed on the high-confidence single-speaker speech frame and the frame-level speaker features in the neighborhood of the high-confidence single-speaker speech frame to obtain aggregated speaker features; Calculate the similarity between the aggregated speaker features and each of the prototype feature vectors in the speaker prototype feature set; If the highest similarity is greater than or equal to the preset matching threshold, the aggregated speaker features are updated to the prototype feature vector corresponding to the highest similarity using a moving average method. If the highest similarity is less than the matching threshold, then a prototype feature vector is added to the speaker prototype feature set, using the aggregated speaker features as the initial value.

8. An overlapping speech detection device, characterized in that, include: The acquisition module is used to acquire the acoustic feature sequence and frame-level speaker feature sequence of the speech to be detected; The detection module is used to detect speech activity based on the acoustic feature sequence to obtain a speech activity probability sequence; The weight generation module is used to generate a continuous masking weight sequence based on the speech activity probability sequence; The feature weighting module is used to weight the first feature sequence according to the continuous masking weight sequence to obtain a masking feature sequence, wherein the first feature sequence is the acoustic feature sequence or a fused feature sequence obtained by fusing the acoustic feature sequence with the frame-level speaker feature sequence; The feature maintenance module is used to maintain a speaker prototype feature set containing at least one speaker prototype feature vector; The similarity calculation module is used to calculate the similarity value between the frame-level speaker features of the current frame and each of the prototype feature vectors in the speaker prototype feature set, as an auxiliary discrimination basis; The judgment module is used to analyze the masking feature sequence in conjunction with the auxiliary discrimination criteria, determine frame by frame whether each speech frame of the speech to be detected is single-person speech or multiple-person overlapping speech, and output the overlapping speech probability sequence.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the overlapping speech detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the overlapping speech detection method as described in any one of claims 1 to 7.

Images