A voice overlap detection method and device, electronic equipment and storage medium

Abstract

The present application relates to a kind of speech overlap detection method, device, electronic equipment and storage medium, comprising: constructing the training set containing multiple morphological overlap samples, development set and test set, extract log-Mel feature;Multi-task acoustic detection model is constructed, and through the pre-training speech encoder of freezing, Adapter module and Decoder module extract timing correlation representation, after being pooled, overlap classification main branch, auxiliary branch and the multiple auxiliary task branches such as the number of speakers, overlap time length, signal-to-noise ratio between speakers are set on shared window level representation, and multi-task joint training is carried out;Using development set calibrates decision threshold under false alarm rate constraint;To be detected long audio sliding window segmentation and window inference are carried out, and based on main auxiliary branch fusion score and decision threshold, output detection result.The present application can significantly improve the accuracy and robustness of overlap speech detection, effectively control false alarm rate, meet the delivery requirements of actual business scenarios.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and more specifically, to a speech overlap detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] Overlapping speech detection is a technique used to determine whether two or more speakers are speaking simultaneously in an audio clip. This technology is widely used in scenarios such as meeting transcription, customer service quality inspection, speaker logging, automatic speech recognition preprocessing, audio quality inspection, and multi-speaker speech analysis. In these applications, failure to accurately identify overlapping speech can easily lead to errors in speech recognition results, confusion in speaker attribution, missing meeting minutes, or inaccurate quality inspection results.

[0003] Existing overlapping speech detection technologies mainly fall into the following categories: The first category is based on energy, endpoint detection, or speech activity detection rules, which judges suspected overlapping regions by changes in audio energy, speech segment boundaries, or speech activity states; the second category is indirect judgment methods that rely on speaker separation or speaker log systems, which infer whether there is overlap by identifying the temporal relationship of different speakers' speech activities; the third category is a binary classification method based on neural networks, which divides the audio into fixed-length windows, inputs them into an acoustic model for overlapping / non-overlapping classification, and outputs the detection results through a fixed threshold.

[0004] The existing methods mentioned above still have shortcomings in practical applications: In boundary scenarios such as rapid alternation between two speakers, extremely short pauses, and adjacent speech that does not actually overlap, it is easy to misjudge "non-overlapping" speech as "overlapping" speech; the cost of annotating real overlapping speech is high, and the annotation boundary is unstable, making it difficult for training data to fully cover complex situations such as different gender combinations, overlap duration, signal-to-noise ratio between speakers, background noise, and long and short speech embeddings; existing neural network methods usually only use a single label of "whether it overlaps" for training, lacking explicit learning of acoustic structural information such as the number of speakers, overlap duration, and energy relationship between speakers; at the same time, fixed thresholds have poor transferability across different data domains and business scenarios, which can easily cause fluctuations in the false alarm rate, and long audio file-level judgment may also be misjudged as overlap due to a small number of abnormal windows, making it difficult to meet the requirements of low false alarms and stable delivery in actual business.

[0005] Therefore, there is an urgent need for an overlapping speech detection system and method that can construct training samples covering complex scenarios, jointly learn multiple acoustic structural information related to overlapping speech, and perform threshold calibration according to development sets or business requirements, thereby improving the accuracy, robustness and deliverability of overlapping speech detection. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a speech overlap detection method, device, electronic device and storage medium to address the above-mentioned deficiencies of the prior art.

[0007] The technical solution adopted by this invention to solve its technical problem is as follows: In a first aspect, the present invention provides a speech overlap detection method, comprising the following steps: Step 1: Obtain raw audio data and construct training, development, and test sets from real speech segments, speaker information, background noise, and real labeled audio. Resample the raw audio data and the audio data used for training into mono audio with a uniform sampling rate, and crop or fill it into a fixed-length window, extracting the log-Mel acoustic features of the window. Step 2: Construct a multi-task acoustic detection model, which includes a pre-trained speech encoder, an adapter module, a decoder module, and a pooling unit connected in sequence; The pre-trained speech encoder is used to encode the log-Mel acoustic features, and the first vectorized implicit representation feature representing the audio content is output. The first vectorized implicit representation feature is processed using the Adapter module to output a second vectorized implicit representation feature representing speech overlap information; The Decoder module is used to process the second vectorized implicit representation feature to output a third vectorized implicit representation feature that represents speech overlap information and preserves temporal relationships. The third vectorized implicit representation feature is pooled using the pooling unit to obtain a window-level representation; multiple task branches are set on the window-level representation, and the task branches include at least an overlapping classification main branch and an overlapping classification auxiliary branch; The multi-task acoustic detection model is trained using the training set with multi-task loss weighting. Step 3: Use the development set to scan the candidate thresholds, calculate the false alarm rate for each candidate threshold, select the optimal threshold from the candidate thresholds whose false alarm rate does not exceed the preset upper limit, and use the optimal threshold as the decision threshold; Step 4: Obtain the long audio to be detected, and divide the long audio to be detected into N sub-audio segments according to the specified window length and the specified step size; Step 5: Input the N sub-audio segments sequentially into the trained multi-task acoustic detection model, calculate the overlap score of each sub-audio segment based on the output of the main overlap classification branch and the auxiliary overlap classification branch, use the judgment threshold to determine whether each sub-audio segment overlaps or not, and output the speech overlap detection result of the long audio segment to be detected.

[0008] In the speech overlap detection method of the present invention, in step two, the pre-trained speech encoder is Whisper-large-v3, the adapter module is a Conformer network structure, the decoder module is a Transformer network structure, and the pooling unit is a multi-head attention pooling unit; the first vectorized implicit representation feature, the second vectorized implicit representation feature, and the third vectorized implicit representation feature all maintain the same temporal relationship as the original audio.

[0009] The speech overlap detection method of this invention, wherein in step two, the multiple task branches further include a speaker count branch, an overlap duration branch, and an inter-speaker signal-to-noise ratio branch; the main overlap classification branch uses the BCEWithLogits loss function, and the supervision target is the overlap / non-overlap label; the auxiliary overlap classification branch uses the cross-entropy loss function, and the supervision target is the overlap / non-overlap binary classification label; the speaker count branch uses the cross-entropy loss function, and the supervision target is the speaker count category label; the overlap duration branch uses the SmoothL1 loss function, and the supervision target is the overlap duration within the window; the inter-speaker signal-to-noise ratio branch uses the SmoothL1 loss function, and the supervision target is the inter-speaker signal-to-noise ratio value; for non-overlapping samples, the supervision target of the inter-speaker signal-to-noise ratio branch is set to the sentinel value.

[0010] In the speech overlap detection method of this invention, in step two, the total loss function of multi-task loss weighted training is expressed as: + + +

[0011] in, The BCEWithLogits loss for the overlapping classification main branch, The cross-entropy loss is the value of the overlapping classification auxiliary branch. The cross-entropy loss is the number of speakers branch. The SmoothL1 loss for the overlapping duration branch, The SmoothL1 loss is the signal-to-noise ratio branch between the speakers; , , , and These are the weighting coefficients corresponding to each type of loss.

[0012] The speech overlap detection method of the present invention, wherein the specific method for constructing the training set, development set and test set in step one includes: obtaining a speaker speech segment pool based on the speech activity detection results, dividing the training set, development set and test set according to the speaker, so that the speech segments of the same speaker do not appear in any two sets of the training set, development set and test set at the same time; The following training samples are constructed based on the speaker's speech segment pool: A fixed-length window is extracted from a single speaker's speech segment, or speech segments shorter than the fixed-length window are placed at the left, right, or middle of the window and padded with zeros to construct a single-speaker speech negative sample. Two speech segments from different speakers are placed sequentially in the same window, with a gap in between and zero overlap duration, to construct a two-speaker non-overlapping interval negative sample. By superimposing the speech segments of two different speakers in a random alignment manner, and ensuring that the overlap duration is not less than a preset threshold, a positive sample of full-length overlap between the two speakers is constructed. A long speech segment and a short speech segment are superimposed at the endpoints of the window to construct a positive sample with overlapping long and short endpoints. By embedding a short speech segment into a long speech segment to form a local overlap, a positive sample of long-short embedding overlap is constructed.

[0013] In the speech overlap detection method of the present invention, in step one, the overlap duration of each positive sample is controlled to be greater than or equal to 0.5 seconds, the overlap duration of each negative sample is controlled to be 0 seconds, and ambiguous samples with an overlap duration between 0 and 0.5 seconds are discarded; in the process of constructing training samples, the gender combination, background noise signal-to-noise ratio, and inter-speaker signal-to-noise ratio are controlled, and the metadata of each training sample is recorded, wherein the metadata includes at least the number of speakers, overlap duration, gender combination, and inter-speaker signal-to-noise ratio.

[0014] The speech overlap detection method of the present invention, wherein in step five, the overlap score is calculated as follows: a sigmoid operation is performed on the output of the main branch of the overlap classification to obtain a first overlap probability; a softmax operation is performed on the output of the auxiliary branch of the overlap classification and the probability value corresponding to the overlap category is taken to obtain a second overlap probability; the smaller value between the first overlap probability and the second overlap probability is taken as the overlap score; for multiple prediction results within the same time period, when the proportion of predictions that are predicted to be overlaps to the total number of predictions exceeds a threshold, the time period is determined to be overlapping speech, and a file-level overlap determination result is output.

[0015] Secondly, the present invention also provides an apparatus for detecting overlapping speech, wherein the detection apparatus comprises: The data construction and preprocessing module is used to acquire raw audio data, construct training sets, development sets, and test sets from real speech segments, speaker information, background noise, and real labeled audio; resample the raw audio data and the audio data used for training into mono audio with a uniform sampling rate, and crop or fill it into a fixed-length window, and extract the log-Mel acoustic features of the window; A model building and training module is used to construct a multi-task acoustic detection model. The model includes a pre-trained speech encoder, an adapter module, a decoder module, and a pooling unit connected in sequence. The pre-trained speech encoder encodes the log-Mel acoustic features, outputting a first vectorized implicit representation feature representing the audio content. The adapter module processes the first vectorized implicit representation feature, outputting a second vectorized implicit representation feature representing speech overlap information. The decoder module processes the second vectorized implicit representation feature, outputting a third vectorized implicit representation feature representing speech overlap information while preserving temporal relationships. The pooling unit pools the third vectorized implicit representation feature to obtain a window-level representation. Multiple task branches are set on the window-level representation, including at least a main overlap classification branch supervised by the BCEWithLogits method and an auxiliary overlap classification branch supervised by the CrossEntropy method. The multi-task acoustic detection model is trained using the training set with multi-task loss weighting. The threshold calibration module is used to scan the candidate thresholds using the development set, calculate the false alarm rate under each candidate threshold, and select the optimal threshold as the judgment threshold among the candidate thresholds whose false alarm rate does not exceed a preset upper limit. The sliding window segmentation module is used to acquire the long audio to be detected and segment the long audio to be detected into N sub-audio segments according to a specified window length and a specified step size; The result output module is used to input the N sub-audio files sequentially into the trained multi-task acoustic detection model, calculate the overlap score of each sub-audio file based on the output of the main overlap classification branch and the auxiliary overlap classification branch of the multi-task acoustic detection model, use the judgment threshold to determine whether each sub-audio file overlaps or not, and output the speech overlap detection result of the long audio file to be detected.

[0016] Thirdly, the present invention also provides an electronic device, comprising: One or more processors; and Memory, used to store one or more computer programs; When the one or more computer programs are executed by the one or more processors, the one or more processors implement the speech overlap detection method as described above.

[0017] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program is executed by a processor to perform the speech overlap detection method as described in any of the preceding claims.

[0018] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs multiple forms of negative samples, including single-speaker negative samples and two-speaker non-overlapping interval negative samples. It specifically designs dedicated negative samples for boundary scenarios such as rapid alternation between two speakers, extremely short pauses, and adjacent but non-overlapping speech, enabling the model to learn the subtle differences between "two-speaker non-overlapping" and "two-speaker overlapping" speech. Simultaneously, by constructing two-speaker full-length overlapping positive samples, long-short endpoint overlapping positive samples, and long-short embedding overlapping positive samples, it covers various overlap patterns in real-world scenarios, effectively enhancing the model's ability to distinguish between difficult negative samples and boundary samples. Compared to existing methods that only use simple positive and negative samples, this invention significantly reduces the false alarm rate in boundary scenarios. 2. This invention sets up five task branches on the shared window-level representation: the main branch for overlap classification (BCEWithLogits), the auxiliary branch for overlap classification (CrossEntropy), the branch for the number of speakers (CrossEntropy), the branch for overlap duration (SmoothL1), and the branch for signal-to-noise ratio between speakers (SmoothL1). Through multi-task loss weighted joint training, the model not only learns the discriminative information of "whether there is overlap" during training, but also simultaneously learns multi-dimensional acoustic structural information such as the number of speakers, overlap duration, and energy relationships between speakers. These auxiliary tasks provide rich supervision signals for the shared representation layer, guiding the model to learn more discriminative and robust feature representations. Existing technologies usually only use a single label of "whether there is overlap" for training, lacking explicit learning of acoustic structural information. This invention effectively makes up for this deficiency through multi-task learning, fundamentally reducing misjudgments caused by single-label training. 3. This invention employs an overlapping sliding window approach to segment the long audio file to be detected, ensuring that the same time period is covered by multiple adjacent windows, thus enabling multi-view prediction. During the file-level fusion stage, for multiple prediction results within the same time period, only when the proportion of predictions indicating overlap exceeds a predetermined threshold is that the time period determined to be overlapping speech. This strategy effectively avoids the problem of the entire file being misjudged due to a small number of abnormal windows being mistakenly identified as overlapping, significantly improving the stability and reliability of long audio file-level detection results, and meeting the requirements for stable delivery in real-world business scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort: Figure 1 This is a flowchart of a speech overlap detection method according to Embodiment 1 of the present invention.

[0020] Figure 2 yes Figure 1 A schematic diagram of a speech overlap detection method.

[0021] Figure 3 yes Figure 1 A diagram illustrating multiple task branches.

[0022] Figure 4 This is a schematic diagram of a speech overlap detection device according to Embodiment 2 of the present invention.

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0024] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0025] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0026] "Multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0027] Furthermore, the terms indicating orientation, such as "up, down, front, back, left, right, upper end, lower end, longitudinal," etc., are all based on the posture and position of the device or equipment described in this solution during normal use.

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0029] Example 1: like Figure 1-2 As shown, this invention provides a speech overlap detection method based on structured mixed data and a multi-task acoustic model, which includes the following steps: Step 1: Data Construction and Preprocessing.

[0030] Raw audio data is acquired, and training, development, and test sets are constructed from real speech segments, speaker information, background noise, and real labeled audio. Specifically, a speaker speech segment pool is first obtained based on the speech activity detection results. The training, development, and test sets are then divided according to the speaker to ensure that speech segments of the same speaker do not appear in any two sets of the training, development, and test sets simultaneously, thereby avoiding the impact of speaker information leakage on model evaluation.

[0031] The original audio data and the audio data used for training are resampled into mono audio with a uniform sampling rate. Preferably, the sampling rate is 16kHz. The resampled audio is then cropped or padded into a fixed-length window, and the log-Mel acoustic features of the window are extracted. Preferably, the fixed-length window is 2 seconds.

[0032] When constructing the training set, development set, and test set, the following five types of training samples are constructed based on the speaker's speech segment pool: (1) Single-person speech negative samples: a fixed-length window is extracted from the speech segment of a single speaker, or speech segments with a length less than the fixed-length window are placed at the left, right or middle of the window and padded with zeros. Such samples do not contain any overlapping speech and are used to train the model to recognize non-overlapping scenes.

[0033] (2) Two-person non-overlapping interval negative samples: Two different speakers’ speech segments are placed sequentially in the same window, with an interval in between and zero overlap. In this type of sample, the two speakers speak one after the other but there is no temporal overlap. This is used to enhance the model’s ability to distinguish boundary scenarios such as “rapid rotation and extremely short pauses”.

[0034] (3) Two-person full-length overlapping positive samples: The speech segments of two different speakers are superimposed in a random alignment manner so that the overlap duration is not less than a preset threshold. In this type of sample, the two speakers overlap extensively within the window and are used to train the model to recognize typical overlapping speech scenarios. Preferably, the preset threshold is 0.5 seconds.

[0035] (4) Long and short endpoint overlap positive sample: A long speech segment and a short speech segment are superimposed at the window endpoints to form an endpoint overlap, which is used to simulate the brief overlap that occurs when one speaker finishes speaking and another speaker begins speaking in a real-world scenario.

[0036] (5) Long and short embedded overlapping positive samples: A short speech segment is embedded inside a long speech segment to form a local overlap, which is used to simulate the situation in real-world scenarios where another speaker briefly interrupts to speak while one speaker is speaking.

[0037] To improve the stability of model training and the clarity of labels, the overlap time of each positive sample is controlled to be greater than or equal to 0.5 seconds, and the overlap time of each negative sample is controlled to 0 seconds. Blurred samples with an overlap time between 0 and 0.5 seconds are discarded to avoid label noise interfering with model training.

[0038] During the construction of training samples, the gender combination, overlap duration, background noise signal-to-noise ratio (SNR), and inter-speaker SNR are controlled to cover diverse acoustic scenarios and improve the model's robustness to different acoustic conditions. Optimal parameters include: a window length of 2 seconds, a minimum speech duration of 0.5 seconds, a minimum overlap duration of 0.5 seconds, a background noise SNR controlled between 15dB and 30dB, and an inter-speaker SNR controlled between 0dB and 15dB. Furthermore, metadata for each training sample is recorded, including at least the number of speakers, overlap duration, gender combination, and inter-speaker SNR, which serves as supervision signals in subsequent multi-task learning.

[0039] Step 2: Construction and training of multi-task acoustic detection model.

[0040] A multi-task acoustic detection model is constructed, comprising a pre-trained speech encoder, an adapter module, a decoder module, and a pooling unit connected in sequence.

[0041] The pre-trained speech encoder is used to encode the log-Mel acoustic features, outputting a first vectorized implicit representation feature representing the audio content. Preferably, the pre-trained speech encoder can be Whisper-large-v3, with its parameters frozen during training and used only as a feature extractor to avoid pre-trained knowledge degradation due to insufficient training data. The pre-trained speech encoder employs a short-window encoding method to avoid redundant padding problems caused by excessively long input audio.

[0042] The first vectorized implicit representation feature is processed using the Adapter module to output a second vectorized implicit representation feature representing speech overlap information. Preferably, the Adapter module can be a Conformer network structure in the prior art. Conformer combines the advantages of convolutional neural networks and self-attention mechanisms, and can capture global contextual dependencies while modeling local temporal features, further enhancing discriminative information related to speech overlap.

[0043] The Decoder module processes the second vectorized implicit representation feature to output a third vectorized implicit representation feature that characterizes speech overlap information and preserves temporal relationships. Preferably, the Decoder module can be a Transformer network structure from the prior art, which models long-range temporal relationships within a window through a self-attention mechanism. It should be noted that the first, second, and third vectorized implicit representation features all maintain the same temporal relationship as the original audio, thereby preserving the temporal location information of overlap.

[0044] The pooling unit is used to pool the third vectorized implicit representation feature to obtain a window-level representation. Preferably, the pooling unit is a multi-head attention pooling unit, which uses multiple attention heads or multiple query vectors to perform weighted pooling on the third vectorized implicit representation feature in the time dimension, aggregating the variable-length temporal features into a fixed-dimensional window-level representation. In another embodiment, the pooling unit can also be implemented using mean pooling.

[0045] Multiple task branches are set on the window-level representation. These task branches include at least a main overlap classification branch (label 1) supervised by the BCEWithLogits method and an auxiliary overlap classification branch (label 2) supervised by the CrossEntropy method. In this embodiment, the multiple task branches also include a speaker count branch, an overlap duration branch, and a speaker signal-to-noise ratio branch. The specific settings of each branch are as follows: Figure 3 As shown; The overlapping classification main branch adopts the BCEWithLogits loss function, and the supervision target is overlapping / non-overlapping labels. This branch directly outputs the log odds without sigmoid transformation. During training, it calculates the binary cross-entropy loss with the binary labels. During inference, it obtains the first overlap probability after sigmoid transformation, which serves as the main basis for overlap determination. The overlapping classification auxiliary branch adopts the cross-entropy loss function, and the supervision target is the overlapping / non-overlapping binary classification label. This branch treats overlapping and non-overlapping as two independent categories and outputs the binary classification probability distribution after softmax transformation. During inference, the probability value corresponding to the overlapping category is taken as the second overlapping probability, forming a complementary constraint with the main branch, and jointly reducing the risk of false alarms caused by single-head anomalies.

[0046] The speaker count branch uses the cross-entropy loss function, and the supervision target is the speaker count category label, which is used to help the model learn the number of speakers within the window.

[0047] The overlap duration branch uses the SmoothL1 loss function, and the supervision target is the overlap duration within the window. The SmoothL1 loss has a smooth gradient when the error is small and a stable gradient when the error is large, making it suitable for regression tasks.

[0048] The inter-speaker signal-to-noise ratio branch uses the SmoothL1 loss function, and the supervision target is the inter-speaker signal-to-noise ratio value. For non-overlapping samples, since there are no two speakers speaking at the same time, the supervision target of this branch is set to the Sentinel Value, for example, 60dB, which means that the regression target is not applicable to the sample. During loss calculation, the sample corresponding to the Sentinel Value does not generate effective gradient backpropagation.

[0049] Furthermore, the multi-task acoustic detection model is trained using the training set with multi-task loss weighting. The total loss function for multi-task loss weighting training is expressed as: + + +

[0050] Wherein, L1 is the BCEWithLogits loss of the main branch of overlapping classification, L2 is the cross-entropy loss of the auxiliary branch of overlapping classification, L3 is the cross-entropy loss of the speaker count branch, L4 is the SmoothL1 loss of the overlapping duration branch, and L5 is the SmoothL1 loss of the inter-speaker signal-to-noise ratio branch; α1, α2, α3, α4, and α5 are the weight coefficients corresponding to each loss. Preferably, each weight coefficient can be set to an equal value of 0.2, so that each task has the same contribution proportion in training. In other embodiments, the weight coefficients can also be adjusted according to the importance or convergence speed of each task. During training, the output layer parameters of the Adapter module, Decoder module, pooling unit, and each task branch in the model are updated through the backpropagation algorithm, while the pre-trained speech encoder remains frozen.

[0051] Step 3: Threshold calibration.

[0052] The candidate thresholds are scanned using the development set, and the false positive rate (FPR) is calculated for each candidate threshold. The optimal threshold is selected from the candidate thresholds whose false positive rate does not exceed a preset upper limit, and the optimal threshold is used as the decision threshold.

[0053] Specifically, for each candidate threshold, the development set samples are judged for overlap / non-overlap based on the threshold, and compared with the true labels to calculate true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), and then the accuracy, precision, recall, F1 score, and false alarm rate are calculated. Among all candidate thresholds that satisfy the condition that the false alarm rate does not exceed the preset business upper limit, the threshold with the highest F1 score is selected as the decision threshold; if there is no candidate threshold with a false alarm rate not exceeding the preset upper limit, the threshold with the lowest false alarm rate and a relatively high F1 score is selected as the decision threshold, as a fallback strategy to ensure that the model can output usable detection results in any data domain.

[0054] Step 4: Slide window to divide.

[0055] A long audio file to be detected is acquired, and the long audio file is divided into N sub-audio files by a sliding window according to a specified window length and a specified step size. Adjacent sub-audio files overlap in time to achieve multi-view prediction for the same time period. Preferably, the specified window length is 2 seconds and the specified step size is 0.1 seconds.

[0056] Step 5: Reasoning and Output of Results.

[0057] The N sub-audio files are sequentially input into the trained multi-task acoustic detection model to obtain the inference result for each sub-audio file.

[0058] The overlap score is calculated as follows: a sigmoid operation is performed on the logarithmic probability output of the main branch of the overlap classification to obtain a first overlap probability P1; a softmax operation is performed on the logarithmic probability output of the auxiliary branch of the overlap classification and the probability value corresponding to the overlap category is taken to obtain a second overlap probability P2; the smaller value between the first overlap probability P1 and the second overlap probability P2 is taken as the overlap score of the sub-audio, score = min(P1, P2). By taking the smaller value of the two, the final score is higher only when both the main branch and the auxiliary branch tend to overlap, thereby effectively reducing the risk of false alarms caused by single-head anomalies.

[0059] The overlap score is compared with the determination threshold: if the overlap score is greater than or equal to the determination threshold, the sub-audio is determined to be overlapping speech; otherwise, it is determined to be non-overlapping speech.

[0060] For multiple prediction results within the same time period (because the sliding window step size is smaller than the window length, the same time period may be covered by multiple adjacent windows), when the proportion of predictions that are predicted to overlap exceeds a predetermined proportion threshold, the time period is determined to be overlapping speech. Preferably, the predetermined proportion threshold is 50%. Finally, window-level overlap detection results and / or file-level overlap determination results are output for use by downstream tasks (such as meeting transcription, speaker logs, speech recognition front-end processing, etc.).

[0061] In summary, this invention has made systematic innovations in all aspects from data construction, model design, training strategy, threshold calibration to inference fusion. It effectively solves a series of technical problems in the prior art, such as high false alarm rate in boundary scenes, insufficient coverage of complex acoustic scenes, lack of training information for single labels, poor transferability of fixed thresholds, and insufficient stability of long audio detection. It has achieved significant technical progress and has extremely high practical value and commercial application prospects.

[0062] Example 2: The present invention also provides a speech overlap detection device, such as... Figure 4 As shown, the detection device includes a data construction and preprocessing module 11, a model construction and training module 12, a threshold calibration module 13, a sliding window segmentation module 14, and a result output module 15.

[0063] The data construction and preprocessing module 11 is used to acquire raw audio data and construct training, development, and test sets from real speech segments, speaker information, background noise, and real labeled audio. It resamples the raw audio data and the audio data used for training into mono audio with a uniform sampling rate, and then clips or fills the data into fixed-length windows, extracting the log-Mel acoustic features of the windows. The specific data processing method of this module is consistent with step one of Embodiment 1, and will not be repeated here.

[0064] The model building and training module 12 is used to build a multi-task acoustic detection model. The model includes a pre-trained speech encoder 121, an adapter module 122, a decoder module 123, and a pooling unit 124 connected in sequence. The pre-trained speech encoder 121 encodes the log-Mel acoustic features, outputting a first vectorized implicit representation feature representing the audio content. The adapter module 122 processes the first vectorized implicit representation feature, outputting a second vectorized implicit representation feature representing speech overlap information. The decoder module 123 processes the second vectorized implicit representation feature, outputting a third vectorized implicit representation feature representing speech overlap information and preserving temporal relationships. The pooling unit 124 pools the third vectorized implicit representation feature to obtain a window-level representation. Multiple task branches are set on the window-level representation, including at least a main overlap classification branch supervised by the BCEWithLogits method and an auxiliary overlap classification branch supervised by the CrossEntropy method. The multi-task acoustic detection model is trained using the training set with multi-task loss weighting. The specific model structure and training method of this module are the same as step two of Example 1, and will not be repeated here.

[0065] The threshold calibration module 13 is used to scan candidate thresholds using the development set, calculate the false alarm rate for each candidate threshold, and select the optimal threshold as the decision threshold from among the candidate thresholds whose false alarm rate does not exceed a preset upper limit; if there is no candidate threshold that meets the false alarm rate constraint, the threshold with the lowest false alarm rate and the highest F1 score is selected as the decision threshold. The specific calibration method of this module is the same as step three of embodiment 1, and will not be repeated here.

[0066] The sliding window segmentation module 14 is used to acquire the long audio to be detected and segment the long audio to be detected into N sub-audio segments according to a specified window length and a specified step size. Preferably, the specified window length is 2 seconds and the specified step size is 0.1 seconds.

[0067] The result output module 15 is used to sequentially input the N sub-audio files into the trained multi-task acoustic detection model, calculate the overlap score of each sub-audio file based on the output of the main overlap classification branch and the auxiliary overlap classification branch, and use the judgment threshold to determine whether each sub-audio file overlaps or not. For multiple prediction results within the same time period, when the proportion of predictions that are predicted as overlapping exceeds a predetermined proportion threshold (preferably 50%), the time period is determined to be overlapping speech. The module outputs the window-level overlap detection result and / or file-level overlap judgment result of the long audio file to be detected. The specific reasoning method of this module is consistent with step five in embodiment 1, and will not be repeated here.

[0068] Example 3: like Figure 5 The present invention also provides an electronic device, including one or more processors 20 and a memory 21.

[0069] The memory 21 is used to store one or more computer programs 22, which are designed to be executed by the one or more processors 20. When the one or more computer programs are executed by the one or more processors 20, the one or more processors 20 implement the speech overlap detection method as described in Embodiment 1.

[0070] Preferably, the processor 20 may be a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices. The memory 21 may be a read-only memory (ROM), random access memory (RAM), flash memory, a hard disk, a solid-state drive, or other forms of computer-readable storage media.

[0071] Those skilled in the art will understand that the electronic device can be a server, computer, embedded device, mobile terminal, or any other device with computing and processing capabilities, and the present invention does not limit this.

[0072] Example 4: The present invention also provides a computer-readable storage medium on which a computer program is stored.

[0073] When the computer program is run by the processor, it executes the speech overlap detection method as described in Example 1.

[0074] Preferably, the computer-readable storage medium may be a read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk, flash memory, USB flash drive, portable hard disk, magneto-optical disk, or any other medium that can be used to store computer program code. When the computer program is executed by the processor, it implements all or part of the steps in the above method embodiments.

[0075] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, ROM, RAM, etc.

[0076] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A speech overlap detection method, characterized in that, Includes the following steps: Step 1: Obtain raw audio data and construct training, development, and test sets from real speech segments, speaker information, background noise, and real labeled audio. Resample the raw audio data and the audio data used for training into mono audio with a uniform sampling rate, and crop or fill it into a fixed-length window, extracting the log-Mel acoustic features of the window. Step 2: Construct a multi-task acoustic detection model, which includes a pre-trained speech encoder, an adapter module, a decoder module, and a pooling unit connected in sequence; The pre-trained speech encoder is used to encode the log-Mel acoustic features, and the first vectorized implicit representation feature representing the audio content is output. The first vectorized implicit representation feature is processed using the Adapter module to output a second vectorized implicit representation feature representing speech overlap information; The Decoder module is used to process the second vectorized implicit representation feature to output a third vectorized implicit representation feature that represents speech overlap information and preserves temporal relationships. The third vectorized implicit representation feature is pooled using the pooling unit to obtain a window-level representation; multiple task branches are set on the window-level representation, and the task branches include at least an overlapping classification main branch and an overlapping classification auxiliary branch; The multi-task acoustic detection model is trained using the training set with multi-task loss weighting. Step 3: Use the development set to scan the candidate thresholds, calculate the false alarm rate for each candidate threshold, select the optimal threshold from the candidate thresholds whose false alarm rate does not exceed the preset upper limit, and use the optimal threshold as the decision threshold; Step 4: Obtain the long audio to be detected, and divide the long audio to be detected into N sub-audio segments according to the specified window length and the specified step size; Step 5: Input the N sub-audio segments sequentially into the trained multi-task acoustic detection model, calculate the overlap score of each sub-audio segment based on the output of the main overlap classification branch and the auxiliary overlap classification branch, use the judgment threshold to determine whether each sub-audio segment overlaps or not, and output the speech overlap detection result of the long audio segment to be detected.

2. The speech overlap detection method according to claim 1, characterized in that, In step two, the pre-trained speech encoder is Whisper-large-v3, the Adapter module is a Conformer network structure, the Decoder module is a Transformer network structure, and the pooling unit is a multi-head attention pooling unit; the first vectorized implicit representation feature, the second vectorized implicit representation feature, and the third vectorized implicit representation feature all maintain the same temporal relationship as the original audio.

3. The speech overlap detection method according to claim 1 or 2, characterized in that, In step two, the multiple task branches also include a speaker count branch, an overlap duration branch, and an inter-speaker signal-to-noise ratio branch. The main overlap classification branch uses the BCEWithLogits loss function, and the supervision target is the overlap / non-overlap label. The auxiliary overlap classification branch uses the cross-entropy loss function, and the supervision target is the overlap / non-overlap binary classification label. The speaker count branch uses the cross-entropy loss function, and the supervision target is the speaker count category label. The overlap duration branch uses the SmoothL1 loss function, and the supervision target is the overlap duration within the window. The inter-speaker signal-to-noise ratio branch uses the SmoothL1 loss function, and the supervision target is the inter-speaker signal-to-noise ratio value. For non-overlapping samples, the supervision target of the inter-speaker signal-to-noise ratio branch is set to the sentinel value.

4. The speech overlap detection method according to claim 3, characterized in that, In step two, the total loss function for multi-task loss weighted training is expressed as: + + + in, The BCEWithLogits loss for the overlapping classification main branch, The cross-entropy loss is the value of the overlapping classification auxiliary branch. The cross-entropy loss is the number of speakers branch. The SmoothL1 loss for the overlapping duration branch, The SmoothL1 loss is the signal-to-noise ratio branch between the speakers; , , , and These are the weighting coefficients corresponding to each type of loss.

5. The speech overlap detection method according to any one of claims 1, 2, and 4, characterized in that, In step one, the specific method for constructing the training set, development set, and test set includes: obtaining a speaker's speech segment pool based on the speech activity detection results, dividing the training set, development set, and test set according to the speaker, so that the speech segments of the same speaker do not appear in any two sets of the training set, development set, and test set at the same time; The following training samples are constructed based on the speaker's speech segment pool: A fixed-length window is extracted from a single speaker's speech segment, or speech segments shorter than the fixed-length window are placed at the left, right, or middle of the window and padded with zeros to construct a single-speaker speech negative sample. Two speech segments from different speakers are placed sequentially in the same window, with a gap in between and zero overlap duration, to construct a two-speaker non-overlapping interval negative sample. By superimposing the speech segments of two different speakers in a random alignment manner, and ensuring that the overlap duration is not less than a preset threshold, a positive sample of full-length overlap between the two speakers is constructed. A long speech segment and a short speech segment are superimposed at the endpoints of the window to construct a positive sample with overlapping long and short endpoints. By embedding a short speech segment into a long speech segment to form a local overlap, a positive sample of long-short embedding overlap is constructed.

6. The speech overlap detection method according to claim 5, characterized in that, In step one, the overlap duration of each positive sample is controlled to be greater than or equal to 0.5 seconds, the overlap duration of each negative sample is controlled to be 0 seconds, and ambiguous samples with an overlap duration between 0 and 0.5 seconds are discarded; during the construction of training samples, the gender combination, background noise signal-to-noise ratio and inter-speaker signal-to-noise ratio are controlled, and the metadata of each training sample is recorded, the metadata including at least the number of speakers, overlap duration, gender combination and inter-speaker signal-to-noise ratio.

7. The speech overlap detection method according to claim 1, characterized in that, In step five, the overlap score is calculated as follows: a sigmoid operation is performed on the output of the main branch of the overlap classification to obtain the first overlap probability; a softmax operation is performed on the output of the auxiliary branch of the overlap classification and the probability value corresponding to the overlap category is taken to obtain the second overlap probability; the smaller value between the first overlap probability and the second overlap probability is taken as the overlap score; for multiple prediction results within the same time period, when the proportion of predictions that are predicted to be overlaps to the total number of predictions exceeds a threshold, the time period is determined to be overlapping speech, and a file-level overlap determination result is output.

8. A device for detecting overlapping speech, characterized in that, The detection device includes: The data construction and preprocessing module is used to acquire raw audio data, construct training sets, development sets, and test sets from real speech segments, speaker information, background noise, and real labeled audio; resample the raw audio data and the audio data used for training into mono audio with a uniform sampling rate, and crop or fill it into a fixed-length window, and extract the log-Mel acoustic features of the window; A model building and training module is used to construct a multi-task acoustic detection model. The model includes a pre-trained speech encoder, an adapter module, a decoder module, and a pooling unit connected in sequence. The pre-trained speech encoder encodes the log-Mel acoustic features, outputting a first vectorized implicit representation feature representing the audio content. The adapter module processes the first vectorized implicit representation feature, outputting a second vectorized implicit representation feature representing speech overlap information. The decoder module processes the second vectorized implicit representation feature, outputting a third vectorized implicit representation feature representing speech overlap information while preserving temporal relationships. The pooling unit pools the third vectorized implicit representation feature to obtain a window-level representation. Multiple task branches are set on the window-level representation, including at least a main overlap classification branch supervised by the BCEWithLogits method and an auxiliary overlap classification branch supervised by the CrossEntropy method. The multi-task acoustic detection model is trained using the training set with multi-task loss weighting. The threshold calibration module is used to scan the candidate thresholds using the development set, calculate the false alarm rate under each candidate threshold, and select the optimal threshold as the judgment threshold among the candidate thresholds whose false alarm rate does not exceed a preset upper limit. The sliding window segmentation module is used to acquire the long audio to be detected and segment the long audio to be detected into N sub-audio segments according to a specified window length and a specified step size; The result output module is used to input the N sub-audio files sequentially into the trained multi-task acoustic detection model, calculate the overlap score of each sub-audio file based on the output of the main overlap classification branch and the auxiliary overlap classification branch of the multi-task acoustic detection model, use the judgment threshold to determine whether each sub-audio file overlaps or not, and output the speech overlap detection result of the long audio file to be detected.

9. An electronic device, characterized in that, include: One or more processors; as well as Memory, used to store one or more computer programs; When the one or more computer programs are executed by the one or more processors, the one or more processors implement the speech overlap detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program is executed by the processor to perform the speech overlap detection method as described in any one of claims 1 to 7.

Images