A role separation-based sales voice dialogue speaker segmentation and labeling method

Abstract

The present application relates to the technical field of speech processing, in particular to a sales speech dialogue speaker segmentation and marking method based on role separation, comprising the following steps: generating a time-stamped speech segment sequence, synchronously extracting a voiceprint feature vector of each speech segment, starting a silence intention analysis engine, predicting the attribution role of the current silence according to the semantic content of the speech segments before and after the silence, generating a virtual voiceprint feature vector and inserting it into the speech segment sequence; performing role clustering on the voiceprint feature vector, splitting to generate a new speaker cluster; when detecting that there is speech overlap or audio loss in the speech segment, triggering mixed speech separation and generation compensation. The present application improves the abnormal recognition capability in the scene of sales role transformation and sound line disguise, reduces the risk of sales and customer speech confusion, and ensures the coherent and accurate role marking of each speech segment.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a method for speaker segmentation and tagging in sales speech dialogue based on role separation. Background Technology

[0002] In sales-related voice dialogue scenarios, role separation and speaker tagging are fundamental tasks for achieving voice quality inspection, customer profiling, and sales behavior analysis. Especially in call centers, intelligent outbound calling, and voice customer service systems, accurately distinguishing the voice segments belonging to "sales personnel" and "customers" is crucial for subsequent script evaluation, service compliance analysis, and intent recognition.

[0003] Existing speaker separation techniques mostly rely on voiceprint clustering or end-to-end speech segmentation models, which have the following problems and shortcomings: poor adaptability to complex phenomena such as silence, overlap, and relay speech in sales scenarios; traditional speech activity detection algorithms often have difficulty distinguishing silence from background noise (such as keyboard sounds, paper turning), which can easily lead to misjudgment of speech segment boundaries; at the same time, sales scenarios often involve "cross-role relay speech" phenomena such as sales personnel changing and multiple people collaborating to answer questions, and existing methods lack a role attribution inference mechanism for the context of silent segments, making it difficult to handle speaker label breakpoints.

[0004] The inability to perceive semantic information leads to the inability or incorrect binding of role labels after speaker clustering. Conventional voiceprint clustering methods divide speaker clusters based solely on the similarity of speech features, failing to determine whether a particular cluster belongs to "salesperson" or "customer." Some studies bind labels using fixed rules or the role of the first speech, but this is prone to misjudgment when there are role changes, role impersonation, or customer questions. Summary of the Invention

[0005] This invention provides a sales voice dialogue speaker segmentation and labeling method based on role separation, which is a complete voice dialogue role separation and labeling scheme that takes into account speaker clustering accuracy, role recognition accuracy and anomaly handling capabilities, so as to improve the structured processing capability and usability of sales voice data.

[0006] A method for speaker segmentation and tagging in sales voice dialogue based on role separation, comprising the following steps:

[0007] S1: Perform speech activity detection and initial segmentation on the input audio stream, generate a speech segment sequence with timestamps, and simultaneously extract the voiceprint feature vector of each speech segment; when the continuous silence duration exceeds the first threshold, start the silence intent analysis engine, predict the role to which the current silence belongs based on the semantic content of the speech segments before and after the silence segment, generate a virtual voiceprint feature vector and insert it into the speech segment sequence.

[0008] S2: Based on dynamic adaptive clustering, the voiceprint feature vector is clustered into roles to generate at least two speaker clusters; during the clustering process, the voiceprint feature mutation points are monitored in real time. If the voiceprint similarity of adjacent speech segments within a single speaker cluster drops sharply, a new speaker cluster is generated by splitting the cluster; the speech segments including preset sales terms are identified by the sales feature word detection engine, and the current speaker cluster is bound to the sales role label; otherwise, it is marked as the customer role label; for the newly generated speaker clusters, cross-turn role rebinding is initiated: if the speech segments in the new speaker cluster have semantic coherence with the historical sales role speech, the sales role label is inherited.

[0009] S3: When speech overlap or audio loss is detected in a speech segment, mixed speech separation and generative compensation are triggered.

[0010] By separating the sound sources from the overlapping speech segments, a clean single-role sub-segment is obtained;

[0011] For missing audio segments, context-sensitive speech generation is used to complete the content;

[0012] Insert the processed sub-segments into the original speech segment sequence according to the role labels output by S2, and output the complete dialogue text stream with role labels.

[0013] Optionally, the speech activity detection and initial segmentation specifically include using dual-channel parallel processing of the input audio stream:

[0014] Channel 1: Perform coarse endpoint detection based on short-time energy and zero-crossing rate to generate candidate speech segment boundaries;

[0015] Channel 2: Detects non-speech noise segments, including breathing sounds and keyboard typing sounds, using a pre-trained convolutional neural network;

[0016] By fusing the results from both channels, the boundaries of valid speech segments are confirmed if and only if channel one is determined to be a speech segment and channel two is determined to be non-noise. Based on the boundaries of valid speech segments, a speech segment sequence with timestamps is generated.

[0017] Optionally, the extraction of the voiceprint feature vector includes extracting a deep voiceprint feature vector for each speech segment sequence, inputting the Mel spectrogram of the speech segment sequence into the ResNet-34 voiceprint encoder, outputting a multidimensional vector and performing L2 normalization.

[0018] Optionally, S1 further includes calculating the time interval between two adjacent speech segment sequences. If the time interval exceeds a preset first threshold, it is determined that there is a semantically assigned silent segment, and the silent intent analysis engine is activated, specifically including:

[0019] Extract the transcribed text of the speech segment before and after the silence segment, and use them as the semantic context input before and after the silence, respectively.

[0020] Silent Segment Role Analysis: The silent intent analysis engine matches static rule templates to infer whether the silence is a customer thinking scenario or a sales pause scenario. If neither static rule template matches, the two segments of text are input into the semantic classification model for prediction to determine the probability that the silent segment belongs to the customer or sales role.

[0021] Virtual voiceprint feature vector generation: Based on the analyzed silent role attribution, construct a virtual voiceprint vector representing the silent segment: if it belongs to the customer, then the voiceprint of the subsequent speech segment is the main component, supplemented by a perturbation term to enhance naturalness; if it belongs to the sales, then the voiceprint of the preceding speech segment is the main component, and random interference is introduced to adapt to the sales alternation situation.

[0022] The time interval and virtual voiceprint features are combined into a new speech segment object, the state is marked as "to be clustered", and then inserted into the original speech segment sequence.

[0023] Optionally, the dynamic adaptive clustering includes inputting the voiceprint feature vector set into an improved DBSCAN clustering algorithm. The improved DBSCAN clustering algorithm includes setting a dynamic neighborhood radius and a minimum number of samples. The clustering results generate speaker clusters. For each speaker cluster in the clustering results, its internal speech segments are arranged in chronological order. Adjacent speech segments within the speaker cluster are monitored and voiceprint similarity is calculated based on cosine similarity. If the voiceprint similarity meets the dual criteria of sudden drop, a new speaker cluster is generated by splitting, and the remaining original speech clusters after the new speaker cluster is separated are retained. The dual criteria include that the voiceprint similarity is lower than a second threshold and that the similarity change is significant.

[0024] Optionally, the sales feature word detection engine includes extracting all speech segments included in each remaining primitive speaker cluster, inputting them into the sales feature word detection engine for term statistics, calculating the term cumulative score, and when the term cumulative score exceeds a third threshold, binding the current remaining primitive speaker cluster as a sales role label; otherwise, marking it as a customer role label.

[0025] Optionally, S2 further includes calculating a semantic coherence score. Based on the new speaker cluster, all speech segments included in it are extracted, and speech recognition is performed on all speech segments to obtain the corresponding text content. Historical sales speech segments in the speaker cluster that have been bound to the sales role are extracted, and semantic vector encoding is performed. The texts of the new cluster and the historical sales speech segments are respectively input into the sales-specific embedding model to extract segment-level semantic vector representations, and the embedding vector of each speech segment in the sales semantic space is obtained. Based on the embedding vector, the semantic coherence score between the new speech cluster and the historical sales speech is calculated as a basis for measuring the degree of semantic coherence. If the semantic coherence score exceeds the fourth threshold, it is considered that the current new speech cluster and the historical sales role have a semantic continuity relationship, thereby inheriting the sales role label. At the same time, terms that frequently appear in the new cluster but are not yet in the sales lexicon can be dynamically added to the sales feature word list to support future recognition.

[0026] Optionally, the speech overlap is determined by the dual fundamental frequency phenomenon in the spectrum; the audio missing is determined based on the signal-to-noise ratio and duration, and after detecting the anomaly, the abnormal audio segment and its adjacent speech segments are extracted.

[0027] Optionally, in S3, when the abnormal audio segment is determined to be a speech overlap segment, mixed speech separation processing is performed. Specifically, this includes determining the number of separation channels based on the number of role labels output by the role separation, converting the abnormal audio segment into a spectrogram, inputting it into the trained role perception separation network, the role perception separation network combining the role label information to output multiple separation masks, and extracting the clean speech segment of the corresponding role based on each separation mask to form multiple sub-segments with role labels.

[0028] In S3, when an abnormal audio segment is determined to be a missing audio segment, generative audio compensation is performed. Specifically, this includes determining the voice role of the compensation segment based on the role labels of its adjacent segments, extracting the context text before and after the missing segment as the generation input, and inputting it into the voice synthesis model controlled by the role. The speech rate, pitch and voice are controlled by combining the voice features of the role to generate the complete speech segment.

[0029] Optionally, S3 further includes:

[0030] After processing overlapping speech segments and missing audio segments, the separated or generated sub-segments are evenly divided into time intervals and aligned according to the time range of the original abnormal audio segments. Role labels are assigned, the original abnormal audio segments are removed from the speech segment sequence, and replaced with the aligned sub-segments to reconstruct a new speech segment sequence.

[0031] The beneficial effects of this invention are:

[0032] This invention constructs a complete speaker segmentation process consisting of speech activity detection, voiceprint clustering, role recognition, and role rebinding. Combined with a silent intent analysis engine and a virtual voiceprint generation mechanism, it can accurately determine the affiliation of silent segments and include them in role clusters. At the same time, it adopts a dynamic adaptive clustering algorithm and dual-threshold mutation monitoring to improve the ability to identify anomalies in scenarios such as sales role changes and voice disguise, reduce the risk of confusion between sales and customer voices, and ensure that the role labeling of each speech segment is coherent and accurate.

[0033] This invention addresses the common issues of speech overlap and audio gaps in real-world sales conversations by designing a role-aware sound source separation network and a role-constrained generative audio compensation mechanism. The former guides vocal tract separation by incorporating output role labels, ensuring each output segment accurately matches the original role. The latter, through dynamic control of speech rate, fundamental frequency, and prosodic parameters, and the introduction of voiceprint similarity verification, achieves consistent speech segment completion, significantly improving the annotability and data quality in abnormal dialogue scenarios. It is applicable to various scenarios such as automated quality inspection and corpus construction. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0036] Figure 2 This is a schematic diagram of silent segment role prediction according to an embodiment of the present invention. Detailed Implementation

[0037] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. For some well-known technologies, those skilled in the art may also use other alternative methods to implement the invention. Moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0038] like Figures 1-2 As shown, a sales speech dialogue speaker segmentation and tagging method based on role separation includes the following steps:

[0039] S1: Perform speech activity detection and initial segmentation on the input audio stream, generate a sequence of speech segments with timestamps, and simultaneously extract the voiceprint feature vector of each speech segment; when the continuous silence duration exceeds the first threshold, start the silence intent analysis engine, predict the role to which the current silence belongs based on the semantic content of the speech segments before and after the silence segment, generate a virtual voiceprint feature vector and insert it into the speech segment sequence.

[0040] S11, Speech Activity Detection and Initial Segmentation: A dual-channel parallel processing mechanism is used to perform interference-resistant speech activity detection on the input audio stream. The steps are as follows:

[0041] Channel 1 (Energy / Zero Crossing Rate Detection): Based on short-time energy and zero crossing rate calculation, coarsely locate the boundaries of candidate speech segments;

[0042] Channel 2 (CNN Noise Detection): Uses a pre-trained convolutional neural network (CNN) to identify non-speech noise segments, including breathing sounds, keyboard typing sounds, and paper turning sounds.

[0043] Boundary fusion judgment logic: if and only if the following conditions are met:

[0044] Channel 1 determines that the current frame is a speech segment;

[0045] Channel 2 determines that the current frame is a non-noise segment;

[0046] This confirms that the segment is a valid audio segment.

[0047] Output format: Generate a sequence of timestamped speech segments based on boundaries:

[0048] Seg = {seg1, seg2, ..., seg} i}; where each seg i Corresponding timestamp

[0049] More specifically:

[0050] (I) Channel 1: Candidate speech segment detection based on short-time energy and zero-crossing rate:

[0051] Short-time energy measures the overall amplitude intensity within an audio frame; speech segments typically have high energy.

[0052] Zero-crossing rate measures the number of times a signal crosses a zero point within a certain period of time, reflecting the frequency variation characteristics of the signal.

[0053] Vowel part: Low ZCR;

[0054] Consonant / noise segment: High ZCR;

[0055] Silent: Low energy + extremely low ZCR.

[0056] The calculation process divides the input audio into short frames of fixed length (25ms window, 10ms step), and performs the following operations for each frame:

[0057] (a) Short-time energy calculation: Where x[n+m] represents the signal sampling at the m-th sampling point within the frame length M, based on the starting point of the n-th frame. The underlying signal still originates from the original input signal x[n]. E[n] is the short-time energy value of the n-th frame, n is the current frame index, m represents the sample index within the current frame, and M is the number of samples per frame (i.e., the frame length, typically 400–512 sampling points). 2 [n+m] represents the square of the signal at the m-th sampling point within the n-th frame, used to calculate the short-time energy of that frame.

[0058] (b) Zero-crossing rate calculation:

[0059] Where Z[n] is the number of zero crossings in the nth frame, sign(·) is the sign function, outputting ±1, taking +1 when x≥0, and taking -1 when x<0, and x[n+m-1] represents the signal value of the previous sampling point adjacent to x[n+m], used to calculate the zero crossing rate (to determine whether the signal crosses the zero point).

[0060] Logical logic for coarse judgment of speech segment boundaries:

[0061] For each frame: Set the energy threshold θ E and zero-crossing threshold θ Z , can be set as follows:

[0062] θ E =0.2×max(E[n]), which is the maximum value;

[0063] θ Z = 0.5 × mean(Z[n]), which is the average value;

[0064] Judgment rules:

[0065] Continuous candidate speech frames are merged into segments (segments). i .

[0066] (II) Channel Two: Non-speech noise segment recognition based on CNN:

[0067] Identify the following typical non-speech segments:

[0068] Breathing sounds (with periodic, weak, high-frequency characteristics);

[0069] Keyboard typing sound (instantaneous impact + high frequency);

[0070] The sound of paper turning (continuous random friction spectrum);

[0071] Other background sounds (not speech but not silent).

[0072] Input features: Extract Mel spectrograms from audio frames, for example:

[0073] Frame length: 25ms, frame shift: 10ms;

[0074] Mel bandwidth: 64 or 128;

[0075] Each segment lasts 1 second and is converted into an image of size (T,F) = (100,64).

[0076] CNN Model Structure: A lightweight convolutional neural network (CNN) structure is used to extract features and classify the input Mel-ray spectrogram. The network structure is as follows:

[0077] First, input a single-channel Mel spectrogram with dimensions of 100×64×1, representing the time-frequency characteristics of 1 second of audio.

[0078] First convolutional block: The input passes through a two-dimensional convolutional layer with 32 3×3 convolutional kernels to extract primary local features. The ReLU activation function is then applied to enhance non-linear expressive power. Subsequently, it is downsampled through a 2×2 max pooling layer to reduce the feature map size and extract the backbone features.

[0079] The second convolutional block: The feature map enters a 64-channel 3×3 convolutional layer to further deepen the feature representation, followed by ReLU activation and then downsampling through a 2×2 max pooling layer.

[0080] The third convolutional block further encodes the deep speech pattern using 128 3×3 convolutional kernels and activates it using the ReLU function. Then, a global average pooling layer is used to compress the spatial dimension to obtain the global semantic vector.

[0081] Fully connected layer: First, the pooled vector is fed into a 64-dimensional fully connected layer, and a Dropout regularization layer is added to prevent overfitting. Finally, it is passed through a Softmax fully connected layer with an output dimension of 4, and the output is a probability distribution of one of the following four categories: normal speech segment, breathing sound, keyboard typing sound, and paper turning sound.

[0082] Each audio segment is input into a CNN, which outputs a set of probability distributions:

[0083] P = [p] 语音段 ,p 呼吸声 ,p 键盘声 ,p 纸张声 If the highest probability corresponds to a non-speech class (e.g., max(P) = p);键盘声 If the current segment is identified as a noise segment, then the current segment is determined to be a noise segment.

[0084] (III) Fusion Logic: Final Speech Segment Confirmation Conditions:

[0085] frame n ∈ Valid speech segments frame n If it is judged as speech by channel one, it is judged as non-noise by channel two.

[0086] All consecutive frames that meet the conditions constitute a seg i The final output sequence is: Seg = {seg1, seg2, ..., seg...} i}

[0087] S12, Voiceprint Feature Extraction: Extracting each speech segment (segment) i Extracting the deep voiceprint feature vector V i :

[0088] Input preprocessing: seg i The audio is converted into a Mel spectrogram. Each speech segment is converted into a corresponding Mel spectrogram, i.e., a Short Time Fourier Transform (STFT) is performed on the original audio, and mapped to a perceptually equivalent Mel frequency scale to form a two-dimensional time-frequency feature image. This image reflects the distribution characteristics of speech energy over time and frequency, which is convenient for subsequent deep network recognition.

[0089] Feature Encoding: The generated Mel spectrogram is input into a pre-trained residual neural network (ResNet-34) to extract its high-dimensional speaker signature feature representation. This network structure possesses a residual connection mechanism, which can effectively capture the speaker's unique speech texture and acoustic patterns, represented as follows:

[0090] V i =L2Norm(ResNet34(seg) i )),V i ∈R 128 Here, ResNet34 represents the 34-layer residual network used for speaker extraction, and L2Norm represents the L2 normalization operation, which normalizes the feature vector output by ResNet-34 using the L2 norm to compress vector scale differences and enhance the relative distance representation ability during clustering. The final generated speaker features are 128-dimensional normalized vectors, representing the unique representation of the current speech segment in the speaker space.

[0091] S13, Silent Intent Triggering and Virtual Vector Generation:

[0092] S131, Silence detection logic: For any adjacent speech segment seg i ,seg i+1Calculate the interval duration:

[0093] When Δt i ≥T silence When this is triggered, silent intent analysis is performed, where T silence This indicates the silent trigger threshold (first threshold), with a default value of 3 seconds.

[0094] In actual sales conversations, most normal pauses in speech (such as thinking, tone changes, customer hesitation, etc.) usually end between 1.0 and 2.5 seconds. Silence exceeding 3 seconds usually indicates a clear semantic break or a waiting role switch, for example:

[0095] The customer is waiting for a sales response;

[0096] Sales are suspended in preparation for transfer to another party.

[0097] The conversation was interrupted or the system froze.

[0098] Therefore, using 3 seconds as the first threshold can both exclude short pauses in the natural rhythm of speech and effectively capture silent intervals with role affiliation, achieving a balance between accuracy and robustness.

[0099] S132, Semantic Analysis Process: When two adjacent speech segments (seg) are detected... i with seg i+1 There are obvious silence intervals between them, suggesting that these silences may have semantic attribution meanings, such as customer reflection or a sales pause. To determine whether this silence should be attributed to the sales or customer role, a silence intent analysis engine is introduced for attribution inference, extracting the text content of the speech segments before and after the silence segment. i ,text i+1 ), and then send it to the silent intent analysis engine, as follows:

[0100] Extracting semantic content: Extracting the two speech segments seg before and after the silent segment. i seg i+1 Each of the two processes performs speech-to-text conversion to obtain the corresponding text. i ,text i+1 .

[0101] Rule-based matching: If a specific sentence structure or keyword combination exists between the preceding and following sentences, role determination is performed directly. For example:

[0102] The current voice segment ends with "What do you think?" and the next segment begins with "I'm still thinking about it" → This indicates that the customer is thinking, and silence belongs to the customer.

[0103] The current paragraph reads "Please wait," and the next paragraph reads "Let an expert answer" → This indicates that sales have been paused and the call has been transferred; silence is attributed to sales.

[0104] If the above combination of rules is not met, then the text will be... i ,text i+1 The input is concatenated into a pre-trained semantic binary classification model (BERT), and the output probability is used to determine whether the silent person is more likely to belong to the sales or customer role.

[0105] The judgment logic expression of the silent intent analysis engine is as follows:

[0106] if(“How do you feel?”∈text) i )∧(“I'm still thinking”∈text i+1 ) role = "customer";

[0107] elif(“Please wait”∈text) i )∧(“Let the experts answer”∈text i+1 ) role = "Sales";

[0108] else role = BERT_Classifier(text i ,text i+1 );

[0109] Among them, BERT_Classifier(text i ,text i+1 This is a semantic binary classification model trained based on semantic coherence and context, whose output is a role probability label. The input to the semantic binary classification model is:

[0110] text i The audio segment text before silence;

[0111] text i+1 The text of the speech segment after silence;

[0112] The two are concatenated into a single sentence pair input;

[0113] Output binary classification results: P customer or P sales, take the label with the highest probability as the prediction result;

[0114] This model is built upon BERT's semantic understanding capabilities, and its specific structure is as follows:

[0115] The main body of the pre-trained model: BERT-Base-Chinese pre-trained using Chinese characters;

[0116] Input is [CLS]texti [SEP]text i+1 [SEP];

[0117] The output is a vector (sentence pair semantic representation) at the [CLS] position.

[0118] Classifier Header: The [CLS] vector is processed using a 1-2 layer fully connected network, outputting 2D Softmax probabilities.

[0119] Training data construction: Extract samples of "before and after silence" from real sales conversations, manually or by rules label the roles associated with the silence segments, and format them as (text... i ,text i+1 Sample pairs (label); category labels: 0 for customers, 1 for sales, using the standard cross-entropy loss function.

[0120] S133, Virtual Vector Generation Logic:

[0121] If role="customer", then:

[0122] Inheriting the customer's voiceprint as the primary source: using the next voice segment V i+1 The principal component is the speech segment spoken by the customer after a period of silence.

[0123] Adding a small amount of noise: by adding Gaussian noise Simulate natural speech fluctuations to avoid complete vector duplication that could affect clustering discrimination.

[0124] This construction method simulates the natural situation of "a customer continuing to speak after remaining silent", making silent segments more likely to be classified as customer roles during clustering.

[0125] If role = "sales", then:

[0126] Inheriting the previous sales voiceprint as the main feature: the previous voice segment V i Using this as the primary feature source, a scenario is simulated where "sales are paused and then the salesperson or other sales personnel speak."

[0127] Add stronger random perturbation: superimpose uniformly distributed vectors in the interval [-1, 1]. Simulate the uncertainty of "changing sales personnel" or role succession.

[0128] The system helps maintain a certain degree of vector continuity when sales roles change (such as transfers or multiple sales personnel) and guides the clustering system to reasonably classify silent segments into "sales clusters".

[0129] in, This indicates that the mean is 0 and the variance is σ.2 Gaussian noise vector, and V i Same dimension, represents a uniform random vector in the interval [-1,1], used to simulate unstructured speaker feature perturbations, and role represents the silent attribution role.

[0130] S134, Insert a silent virtual segment: Insert a new virtual speech segment into the original speech segment sequence Seg:

[0131]

[0132] V virtual It is a virtual voiceprint feature vector generated for this silent segment. This indicates the start and end timestamps of the silent segment on the timeline. The time interval from the end time of the previous voice segment to the start time of the next voice segment is the time range in which the silence occurs. The segment to be clustered indicates that it has not yet participated in role clustering. The clustering module needs to determine the assigned role label (sales or customer) in subsequent steps. This is an initialization status flag used to control the processing logic of subsequent processes.

[0133] S2: Based on dynamic adaptive clustering, the voiceprint feature vector is clustered into roles to generate at least two speaker clusters; during the clustering process, the voiceprint feature mutation points are monitored in real time. If the voiceprint similarity of adjacent speech segments within a single speaker cluster drops sharply, a new speaker cluster is generated by splitting the cluster; the speech segments including preset sales terms are identified by the sales feature word detection engine, and the current speaker cluster is bound to the sales role label; otherwise, it is marked as the customer role label; for the newly generated speaker clusters, cross-turn role rebinding is initiated: if the speech segments in the new speaker cluster have semantic coherence with the historical sales role speech, the sales role label is inherited.

[0134] S21, Dynamic Adaptive Clustering and Mutation Monitoring:

[0135] The voiceprint feature vector set V = {V1, V2, ..., V} is used to define the voiceprint feature vector set V = {V1, V2, ..., V}. n}; (including virtual vectors generated from silent segments) Input the improved DBSCAN clustering algorithm, using the following dynamic parameter settings:

[0136] Dynamic neighborhood radius setting:

[0137] Among them, SNR currentThe signal-to-noise ratio of the current speech segment is represented by α, which is the baseline clustering radius. The default value is 0.35. After L2 normalization of the voiceprint features, the cosine distance is equivalent to the Euclidean distance. The distance between vectors ranges from [0,2]. The voiceprint distance of the same speaker is mostly concentrated between 0.2 and 0.4. 0.35 is used as the initial neighborhood radius, which can effectively cover most of the normal variations of the same speaker, while excluding most cross-speaker vectors.

[0138] Minimum neighborhood sample size min Pts = 2;

[0139] After initializing the clustering parameters neighborhood radius and minimum neighborhood sample number, for each voiceprint vector V i , using its corresponding neighborhood radius ∈ i Calculate its neighborhood: N i ={V j ∈V∣dist(V i V j )≤∈ i If the number of samples in the neighborhood is |N i |≥minPts, then V i Each core point V is identified as a core point. i Randomly select an unclustered point V from its neighborhood. j Incorporate it into the current cluster C k Continue with V j The neighborhood of the path is expanded by density until the path can no longer be expanded. This process is repeated to generate an initial set of speaker clusters {C1, C2, ..., C}. k};

[0140] Speaker cluster intra-speaker similarity mutation monitoring: For each speaker cluster C in the clustering results k Arrange its internal speech segments in chronological order, and denote them as:

[0141] Calculate the cosine similarity between adjacent voiceprint vectors sequentially: This represents the i-th vector in cluster C, sorted by time. This represents the (i+1)th vector in cluster C, sorted by time.

[0142] If the following conditions for a sudden drop are met:

[0143] sim i <β (β is the second threshold, with a default value of 0.6);

[0144] |sim i -sim i-1 |>0.3;

[0145] The current position is determined to be a voiceprint mutation point, and the original cluster C is moved to this position. k It is split into two clusters, denoted as C. k and New clusters Marked as pending role rebinding state for subsequent semantic coherence determination.

[0146] In voiceprint-based clustering scenarios, the cosine similarity between adjacent speech segments of the same speaker is usually higher than 0.7 to 0.8. If the similarity is lower than 0.6, it means that there are significant changes in voice, speaking style or identity, which may be different speakers. Therefore, setting 0.6 as the lower bound for split determination can better detect situations such as sales spoofing and turn-taking.

[0147] |sim i -sim i-1 | Used to determine the degree of abrupt change in the similarity between adjacent speech segments. The similarity change rate of normal speech segments is usually in the range of ±0.1 to 0.2. If the change exceeds 0.3, it indicates that there is a sharp jump in the vocal characteristics between the speakers, which meets the definition of "abrupt point".

[0148] S22, Sales Role Binding: For each cluster of speakers generated by the clustering C k Perform the following steps in sequence:

[0149] S221, extract all the speech segments (seg) it contains. j ∈C k ;

[0150] S222, Perform speech recognition on each speech segment and extract the text content. j ;

[0151] S223, all text is fed into the sales feature word detection engine for term statistics. The total score (cumulative term score) is defined as:

[0152] Among them, C k For the current speaker cluster, containing several speech segments, S k ={seg1,seg2,…,seg m} represents the cluster C k The set of speech segments in the ASR (segmentation) is represented by T = {"discount", "package", "limited time"}, which represents a pre-defined vocabulary of sales feature words. j ) indicates the segment seg j The text content after automatic speech recognition, conf j ∈[0,1] represents the corresponding speech recognition system's seg j Confidence score for text recognition This is an indicator function; if word w appears in the text... j If it is in the middle, it is 1; otherwise, it is 0.

[0153] If the cumulative term score satisfies: term score >γ, then the speaker cluster C k Bind to the sales role label, otherwise mark it as the customer role label. γ is the third threshold, with a default value of 3.5.

[0154] S23, Cross-Take Role Rebinding: For the new speaker clusters generated after the aforementioned mutation and split. Execution of role inheritance judgment process:

[0155] S231, Extract all speech segments from the new cluster.

[0156] S232, Voice segments for all historical sales roles and Semantic vectors are calculated for the speech segments using a sales-specific embedding model φ(·);

[0157] S233, Calculate the semantic coherence score:

[0158] Where φ(·) represents the BERT embedding model specifically for sales scenarios, N is the number of historical sales voice segments, and cos(·,·) is the vector cosine similarity function.

[0159] The embedding model φ(·) is a text semantic encoder optimized for sales contexts, used to transform speech segments in sales dialogues into dense semantic vectors. This model is based on the pre-trained language model BERT and is specifically adapted to sales discourse features through domain fine-tuning, including product terminology, promotional phrases, and pricing responses. The general-purpose pre-trained Chinese language model BERT-base-chinese is used as the backbone of the semantic encoder.

[0160] Training corpus construction:

[0161] Collect a large number of labeled sales dialogues and construct semantically similar and dissimilar pairs:

[0162] Similar examples: sales opening and subsequent product description;

[0163] Dissimilar examples: sales pitches and expressions of customer resistance;

[0164] Used to train semantic contrast loss.

[0165] The optimization objective is to minimize the embedding distance between semantically similar segments, maximize the distance between semantically unrelated segments, and enhance the model's semantic discrimination ability in sales discourse.

[0166] The semantic vector calculation process for historical sales voice segments and new cluster voice segments is as follows:

[0167] 1. Text extraction and preprocessing: Extracting and preprocessing historical sales audio segments. Perform speech recognition and extract text. For new splitting clusters All audio segments Similarly, perform speech recognition to obtain text. Perform standardized preprocessing on the text (word segmentation, stop word removal, and standardized terminology).

[0168] 2. Semantic Vector Generation: Input all text pairs into the sales embedding model φ(·) and generate semantic vectors for each pair.

[0169] For each generate

[0170] For each generate

[0171] Using [CLS] vectors as segment-level semantic vector representations ensures that the semantic granularity is consistent with that of "dialogue fragments".

[0172] 3. Calculate the cosine similarity between all historical sales segments and new cluster segments, construct a cross-cluster semantic similarity matrix, and average all pairwise similarities to obtain the semantic coherence score S. cont .

[0173] S234, if S cont >δ;then will The sales role label is inherited, and its representative high-frequency terms are dynamically added to the sales feature word library (such as "technical parameters", "configuration requirements", etc.). δ represents the similarity threshold (fourth threshold) for semantic coherence determination, and δ∈[0.8,0.9] with a default value of 0.85.

[0174] If the value is set too low (e.g., <0.75), it can easily lead to semantically unrelated class clusters being mistakenly inherited by roles.

[0175] If the value is set too high (e.g., >0.9), some reasonable sales role continuation will be missed.

[0176] S3: When speech overlap or audio missing is detected in a speech segment, mixed speech separation and generative compensation are triggered, including source separation of speech overlap segments to obtain pure single-role sub-segments, and speech generation and completion of audio missing segments using context-related information.

[0177] Insert the processed sub-segments into the original speech segment sequence according to the role labels output by S2, and output the complete dialogue text stream with role labels.

[0178] S31, Anomaly Detection and Triggering Mechanism: Real-time monitoring of speech segment sequences; triggering the anomaly handling unit when any of the following conditions are met:

[0179] Condition 1 (overlapping speech detection): The presence of two fundamental frequency components in the speech segment spectrum is identified as speech overlap;

[0180] Condition 2 (Missing Segment Detection): If the signal-to-noise ratio of a speech segment is less than 10dB and the duration exceeds 0.5 seconds, it is determined to be an audio missing segment.

[0181] After detecting an anomaly, extract the corresponding abnormal speech segment (seg). err , and its adjacent preceding and following segments seg prev and seg next .

[0182] The fundamental frequency (F0) of human speech is usually in the range of 80 to 300 Hz. At the same time, each speaker generally produces only one main fundamental frequency. If two obvious fundamental frequency peaks are detected simultaneously in the spectral analysis of a certain speech segment, and they are within the reasonable range of human voice fundamental frequency, it means that two speakers are speaking in the same time period, that is, speech overlap has occurred.

[0183] Dual fundamental frequency component detection includes: performing a short-time Fourier transform (STFT) on the target speech segment, extracting the time-frequency spectrum, using a fundamental frequency estimation algorithm based on the autocorrelation function (ACF) to extract the fundamental frequency distribution of each frame, determining whether there are two or more stable fundamental frequency trajectories, and if two obvious fundamental frequency peaks with sufficient spacing repeatedly appear in multiple adjacent frames and these two fundamental frequency trajectories persist throughout the entire speech segment, it is determined that "dual fundamental frequency components exist", the current speech segment is marked as an overlapping speech segment, and it is handed over to the sound source separation module for processing.

[0184] In real-world scenarios, issues such as channel jitter, equipment damage, and user microphone obstruction often lead to localized distortion or loss of speech segments. When the signal-to-noise ratio (SNR) of a speech segment is below 10dB and lasts for more than 0.5 seconds, it usually indicates that the main frequency energy of the speech is significantly submerged in background noise or that the effective content of the entire speech segment is missing, leaving only system noise or breathing sounds.

[0185] Setting 10dB as the lower limit of SNR is based on the acceptable range of speech intelligibility.

[0186] Setting 0.5 seconds as the minimum duration is to avoid misjudging instantaneous background noise;

[0187] This threshold combination can effectively distinguish between normal speech pauses and actual audio loss.

[0188] S32, Mixed speech separation (for overlapping speech): Perform role-label-based source separation, with the following processing logic:

[0189] Input: Abnormal audio segment seg err +S2 outputs a set of character tags {role1,role2,…,role} N};

[0190] Perform a short-time Fourier transform on the audio segment to obtain the spectrum Spec = STFT(seg). err );

[0191] Input the spectrogram into the RoleConditionedSepNet role-aware separation network to generate a set of separation masks: Mask = RoleConditionedSepNet(Spec,M); where N is the number of role types in the current dialogue (e.g., salesperson, customer, virtual), and the number of masks is the same as the number of roles.

[0192] For each channel i, use the corresponding mask to recover clean audio: sub i =ISTFT(Spec·Mask) i ); will each sub i Its corresponding role tag i Binding forms a new subset of speech segments, and ISTFT stands for Inverse Short Time Fourier Transform.

[0193] The Role-Aware Separation Network (RCS-Net) is used in speech overlap scenarios to guide a neural network to separate the vocal channels of mixed speech according to the identified role labels, such as "customer," "salesperson," and "virtual," and output a clean speech mask for each role. The structure is as follows:

[0194] Input components:

[0195] Spectrum input: A complex spectral tensor generated by short-time Fourier transform, representing the time-frequency characteristics of the mixed speech segments;

[0196] Character embedding vector group: The voiceprint cluster center corresponding to each character output by the previous clustering is used as a fixed character representation vector.

[0197] Network structure hierarchical design:

[0198] The encoder part (multi-layer conditional convolution) is a multi-layer two-dimensional convolutional neural network. The input of each layer includes spectrogram features and character embeddings. In each layer, the character embedding vector is injected into the spectrogram processing flow through attention mechanism or feature fusion. The feature map output by each layer retains frequency, time and interaction characteristics between characters.

[0199] Character channel splitting: Based on the number of input characters, the encoder output channel is divided into multiple corresponding sub-channels. Each sub-channel corresponds to one character and serves as the sound source feature sub-map for that character.

[0200] Mask prediction layer: Apply the Sigmoid activation function to each sub-channel, outputting a mask image in the [0,1] interval. The size of each mask image is the same as the original spectrum. Figure 1 "Zhi" indicates the percentage of energy retained by the character at each time-frequency point.

[0201] The process for generating the separation mask is as follows:

[0202] S321. Short-Time Fourier Transform (STFT) Processing: Perform STFT operation on the detected overlapping speech segments to obtain a spectrogram tensor. This tensor retains the time frame, frequency channel, and complex amplitude information, which serves as the main input for separation.

[0203] S322. Role Embedding Vector Construction: Extract role labels (such as "sales" and "customer") involved in the current speech segment from S2, find the voiceprint cluster center corresponding to each role, and form an embedding vector group. Each vector is a role representation of fixed length.

[0204] S323. Feature Fusion and Network Input Assembly: Combine and encode the spectrogram with each role embedding using one of the following methods:

[0205] Conditional convolution method: The embedding vector is expanded into a channel vector of the same size as the spectrogram and concatenated into each convolutional layer;

[0206] Attention fusion method: After the convolution output, an embedding vector is introduced to participate in the channel weighting adjustment, guiding the network to focus on specific sound source features.

[0207] S324. Multi-role mask output: The network outputs multiple separate mask images through each sub-channel. Each mask image is multiplied point by point with the original spectrogram to obtain the spectral representation of the corresponding role.

[0208] S325. Inverse Transform Reconstruction of Speech: Perform inverse STFT on the spectrogram of each role to reconstruct a clean audio segment. Each segment is bound to its role label to form a list of speech segments after role separation.

[0209] S33, Generative Audio Compensation (for Missing Segments): For missing segments... errThe steps for generating voice for role constraints are as follows:

[0210] S331. Retrieve the role tags corresponding to the preceding and following voice segments from S2. R , as a role constraint for the segment to be generated;

[0211] S332. Extract the contextual speech text 2 seconds before and after the missing segment as the generation input;

[0212] S333. Input an end-to-end speech synthesis model based on the VITS architecture and inject role embedding vectors to control speech style;

[0213] S334. Model output completes the speech segment seg gen After generation, voiceprint features are extracted to obtain V. gen ;

[0214] S335. Cluster center vector C of this role role Calculate similarity: cos(V) gen C role ≥0.7;

[0215] If the similarity constraint is not met, regenerate until it meets the requirements;

[0216] S336. Insert the generated segment into the original speech sequence and mark it as "compensation generation" type.

[0217] S34, Sequence Reconstruction and Output: For the set of segments generated by separation or compensation:

[0218] Sub = {sub1, sub2, ..., sub} m}; Perform timeline reconstruction processing. Based on the start and end timestamps of the original exception segment [t s ,t e ], for each sub-segment k Distribute evenly aligned time intervals:

[0219] This indicates the start time t of the original abnormal speech segment. s Adding the relative position ratio of this segment in the total duration, we obtain the start time of the current k-th segment. This ratio represents the position of the k-th segment after dividing the abnormal segment duration into m equal segments.

[0220] This means adding the (k+1)th equal division position to the start time of the abnormal segment, and this means the end time of the kth sub-segment, which extends from position k to position k+1.

[0221] Each sub-segment inherits the role labels output in S2, forming a complete annotated speech segment.

[0222] Finally, the abnormal segment seg err Replace with the aligned set of segments to obtain a new sequence of speech segments:

[0223] new segments =original seq .replace(seg err ,{sub1,…,sub m}); indicates that the abnormal segment seg in the original speech segment sequence will be removed. err Replace with several segments sub1, ..., sub2 generated through separation or compensation. m This forms a new sequence of speech segments. segments ;

[0224] Where m represents the total number of sub-speech segments generated after the abnormal speech segments (such as overlapping or missing segments) are processed, that is, the final number of speech segments to be inserted, and k represents the index of the kth sub-speech segment currently being processed, with a value range of kb = 0, 1, ..., mb-1.

[0225] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0226] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A role separation based sales voice dialogue speaker segmentation and labeling method, characterized in that, Includes the following steps: S1: Perform speech activity detection and initial segmentation on the input audio stream, generate a sequence of speech segments with timestamps, and simultaneously extract the speaker feature vector of each speech segment; When the duration of continuous silence exceeds the first threshold, the silence intent analysis engine is activated. Based on the semantic content of the speech segments before and after the silence segment, the role to which the current silence belongs is predicted, a virtual voiceprint feature vector is generated, and inserted into the speech segment sequence. S2: Based on dynamic adaptive clustering, the voiceprint feature vector is clustered into roles to generate at least two speaker clusters; the dynamic adaptive clustering includes inputting the voiceprint feature vector set into an improved DBSCAN clustering algorithm. The improved DBSCAN clustering algorithm includes setting a dynamic neighborhood radius and a minimum number of samples. The clustering results generate speaker clusters. For each speaker cluster in the clustering results, its internal speech segments are arranged in chronological order. Adjacent speech segments within the speaker cluster are monitored and voiceprint similarity is calculated based on cosine similarity. If the voiceprint similarity of adjacent speech segments within a single speaker cluster meets the dual judgment conditions of a sudden drop in similarity, a new speaker cluster is generated by splitting the new speaker cluster, and the remaining original speech clusters after the new speaker cluster is separated are retained. The dual judgment conditions of a sudden drop in similarity include voiceprint similarity being lower than a second threshold and a significant change in similarity. The sales feature word detection engine identifies speech segments containing preset sales terms and binds the current speaker cluster to the sales role label; otherwise, it marks it as the customer role label. For the newly generated speaker cluster, cross-turn role rebinding is initiated: if the speech segment in the new speaker cluster has semantic coherence with the speech of the historical sales role, the sales role label is inherited. S3: When speech overlap or audio loss is detected in a speech segment, mixed speech separation and generative compensation are triggered. By separating the sound sources from the overlapping speech segments, a clean single-role sub-segment is obtained; For missing audio segments, context-sensitive speech generation is used to complete the content; Insert the processed sub-segments into the original speech segment sequence according to the role labels output by S2, and output the complete dialogue text stream with role labels.

2. The method of claim 1, wherein the method further comprises: The speech activity detection and initial segmentation specifically includes dual-channel parallel processing of the input audio stream: Channel 1: Perform coarse endpoint detection based on short-time energy and zero-crossing rate to generate candidate speech segment boundaries; Channel 2: Detects non-speech noise segments, including breathing sounds and keyboard typing sounds, using a pre-trained convolutional neural network; By fusing the results from both channels, the boundaries of valid speech segments are confirmed if and only if channel one is determined to be a speech segment and channel two is determined to be non-noise. Based on the boundaries of valid speech segments, a speech segment sequence with timestamps is generated.

3. The method of claim 1, wherein the method further comprises: The extraction of the voiceprint feature vector includes extracting a deep voiceprint feature vector for each speech segment sequence, inputting the Mel spectrogram of the speech segment sequence into the ResNet-34 voiceprint encoder, outputting a multidimensional vector and performing L2 normalization.

4. The method of claim 1, wherein the method further comprises: S1 further includes calculating the time interval between two adjacent speech segment sequences. If the time interval exceeds a preset first threshold, it is determined that there is a semantically assigned silent segment, and the silent intent analysis engine is activated, specifically including: Extract the transcribed text of the speech segment before and after the silence segment, and use them as the semantic context input before and after the silence, respectively. Silent Segment Role Analysis: The silent intent analysis engine matches static rule templates to infer whether the silence is a customer thinking scenario or a sales pause scenario. If neither static rule template matches, the two text segments before and after are input into the semantic classification model for prediction to determine the probability that the silent segment belongs to the customer or sales role. Virtual voiceprint feature vector generation: Based on the analyzed silent role attribution, construct a virtual voiceprint vector representing the silent segment: if it belongs to the customer, then the voiceprint of the subsequent speech segment is the main component, supplemented by a perturbation term to enhance naturalness; if it belongs to the sales, then the voiceprint of the preceding speech segment is the main component, and random interference is introduced to adapt to the sales alternation situation. The time interval and virtual voiceprint features are combined into a new speech segment object, the state is marked as "to be clustered", and then inserted into the original speech segment sequence.

5. The method of claim 1, wherein the method further comprises: The sales feature word detection engine includes extracting all speech segments included in each remaining primitive speaker cluster, inputting them into the sales feature word detection engine for term statistics, calculating the term cumulative score, and when the term cumulative score exceeds a third threshold, binding the current remaining primitive speaker cluster as a sales role label; otherwise, marking it as a customer role label.

6. The method of claim 5, wherein the method further comprises: S2 further includes calculating a semantic coherence score. Based on the new speaker cluster, all speech segments included in it are extracted, and speech recognition is performed on all speech segments to obtain the corresponding text content. Historical sales speech segments in the speaker cluster that has been bound to the sales role are extracted, and semantic vector encoding is performed. The texts of the new cluster and the historical sales speech segments are respectively input into the sales-specific embedding model to extract segment-level semantic vector representations and obtain the embedding vector of each speech segment in the sales semantic space. Based on the embedding vector, the semantic coherence score between the new speech cluster and the historical sales speech is calculated as the basis for measuring the degree of semantic coherence. If the semantic coherence score exceeds the fourth threshold, it is considered that the current new speech cluster and the historical sales role have a semantic continuity relationship, thereby inheriting the sales role label.

7. The method of claim 1, wherein the method further comprises: The speech overlap is determined by the dual fundamental frequency phenomenon in the spectrum; the audio missing is determined based on the signal-to-noise ratio and duration. After detecting the anomaly, the abnormal audio segment and its adjacent speech segments are extracted.

8. The method of claim 1, wherein the method further comprises: In S3, when the abnormal audio segment is determined to be a speech overlap segment, mixed speech separation processing is performed. Specifically, this includes determining the number of separation channels based on the number of role labels output by the role separation, converting the abnormal audio segment into a spectrogram, inputting it into the trained role perception separation network, and the role perception separation network combining the role label information to output multiple separation masks. Based on each separation mask, the clean speech segment of the corresponding role is extracted to form multiple sub-segments with role labels. In S3, when an abnormal audio segment is determined to be an audio missing segment, generative audio compensation is performed. Specifically, this includes determining the voice role of the compensation segment based on the role labels of its adjacent segments, extracting the context text before and after the missing segment as the generation input, and inputting it into the voice synthesis model controlled by the role. The speech rate, pitch and voice are controlled by combining the voice features of the role to generate the complete speech segment.

9. The method of claim 8, wherein the method further comprises: S3 further includes: After processing overlapping speech segments and missing audio segments, the separated or generated sub-segments are evenly divided into time intervals and aligned according to the time range of the original abnormal audio segments. Role labels are assigned, the original abnormal audio segments are removed from the speech segment sequence, and replaced with the aligned sub-segments to reconstruct a new speech segment sequence.

Images