A multi-person overlapping speech real-time voiceprint recognition method and system

Abstract

The present application relates to the field of voiceprint recognition and speech transcription, and discloses a multi-person overlapping speech real-time voiceprint recognition method and system. The method comprises: acquiring multi-source audio streams and performing standardization and frame processing to generate a streaming audio frame sequence; based on the sequence, through a loop buffering, noise reduction, endpoint detection and overlap detection model processing, an overlapping interval annotation is obtained; then task assembly, speaker separation, track numbering, voiceprint feature extraction and identity determination are performed to generate a track identity binding structure; finally, speech transcription, segment splicing and conflict clipping are performed, and a voiceprint template library is updated. The present application realizes real-time separation and identity recognition of multi-person overlapping speech, effectively improving the accuracy and robustness of voiceprint recognition in complex scenarios.

Description

Technical Field

[0001] This invention relates to the fields of voiceprint recognition and speech transcription, and in particular to a method and system for real-time voiceprint recognition of multiple overlapping voices. Background Technology

[0002] With the deep penetration of voice interaction technology into smart homes, remote conferencing, and smart terminal control, users' functional requirements for voice systems have expanded from single-speaker scenarios to complex scenarios involving multiple speakers, multiple devices, and multiple environments. Therefore, the collaborative application of voiceprint recognition and speech separation, as well as automatic speech recognition, has become a core requirement. However, existing technical solutions still face multiple bottlenecks in practical implementation. For example, existing voiceprint recognition systems are mostly designed for single devices, standard Mandarin, and near-field speech environments, making it difficult to be compatible with multi-device input, multiple dialects, and mixed near-field and far-field conditions in real-world scenarios. Furthermore, overlapping speech is a common phenomenon in multi-speaker natural dialogue scenarios, but existing technical solutions generally assume that "only one speaker is speaking at a time," failing to construct an effective mechanism for handling overlapping speech. Simultaneously, when voiceprint recognition needs to work in conjunction with modules such as audio acquisition, speech separation, and ASR transcription, there are issues with data format incompatibility and disjointed processing rhythms. This makes it difficult to meet the low-latency requirements of real-time voice interaction.

[0003] Furthermore, in the field of voiceprint recognition and speech transcription, existing solutions for multi-source device audio stream acquisition configuration structures and multi-source device audio stream sets typically revolve around the links of multi-source device audio stream access and preprocessing, audio frame extraction and filtering, overlapping speech interval determination, speaker separation and track organization, voiceprint feature extraction and similarity determination, speech transcription and database writing, etc. These solutions have limitations such as inconsistent device identifier mapping and input channel binding, difficulties in timestamp alignment due to unified sampling rate and unified audio channels, and difficulty in maintaining consistency between track identity binding and structured output data structure in overlapping speech scenarios. Existing methods mostly rely on fixed-duration frame segmentation of the audio frames to be processed, combined with noise reduction and endpoint detection to extract the effective speech frame set. Then, a sliding window traversal overlap detection model is called on the effective speech frame set to form an overlap interval labeling structure. Under the constraints of the overlap interval labeling structure and interval splitting rules, separation model inference and track numbering are performed. Under the constraints of multiple overlapping voices and multiple source device inputs, it is easy to encounter situations where the timestamp index of the overlap interval boundary and the streaming audio frame sequence structure is difficult to align stably, and the single speaker audio track set and track identity binding structure are difficult to maintain a continuous association during the generation and updating process. It is difficult to meet the stable implementation of extracting track audio segments from the track identity binding structure and performing speech transcription model inference, segment timestamp binding, concatenation of adjacent segments of the same speaker, pruning of conflicting segments and writing to the database. For the joint processing of multi-source device audio stream acquisition configuration structure with multi-source device audio stream set, speech effective frame set and overlapping interval annotation structure, single speaker audio track set and track identity binding structure, existing technologies generally lack a consistent constraint expression and recording mechanism in stages such as circular queue enqueue and dequeue, interval processing task assembly, cosine similarity threshold determination operation and template library update instruction structure generation. It is difficult to form a consistent process from acquisition of multi-source device audio stream to alignment, determination and database writing. As a result, the consistency of speaker text segment set construction and structured output data structure database writing is easily affected, which in turn has an adverse effect on continuous processing and traceability. If the relevant limitations can be alleviated, it will be beneficial to the continuous database writing and standardized generation of template library update instruction structure in the audio stream processing process. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a real-time speakerprint recognition method for multi-person overlapping speech, comprising:

[0005] The multi-source device audio stream acquisition configuration structure and multi-source device audio stream set are obtained, and pulse code modulation encoding is unified and fixed duration framing is performed to generate a streaming audio frame sequence structure.

[0006] Based on the streaming audio frame sequence structure, the overlapping interval labeling structure is obtained by performing circular queue enqueue and dequeue, spectral subtraction and noise reduction, dual-threshold endpoint detection and sliding window traversal overlap detection model call processing.

[0007] Based on the overlapping interval labeling structure, the interval processing task is assembled, the separation model reasoning and track numbering are processed, and voiceprint feature extraction and cosine similarity threshold determination are performed to generate the track identity binding structure.

[0008] Based on the track identity binding structure, the speech transcription model inference is performed, adjacent segments of the same speaker are spliced ​​together, and a template library update instruction structure is generated.

[0009] Furthermore, the process of unifying pulse code modulation encoding and fixed-duration framing also includes:

[0010] The pulse code modulation encoding unified processing includes reading the sampling bit width rules and byte order rules from the multi-source device audio stream acquisition configuration structure, performing bit width conversion and amplitude normalization processing, and completing overflow clipping and silence segment preservation during the bit width conversion process, and performing fixed duration frame segmentation processing. The fixed duration frame segmentation processing includes reading the fixed duration frame segmentation rules from the multi-source device audio stream acquisition configuration structure, sliding and segmenting the standard audio stream according to the frame length and frame shift to generate audio frames, and assigning frame sequence numbers and timestamps. When the length of the standard audio stream is less than the frame length, padding or delayed output is performed according to the frame boundary padding strategy to generate a streaming audio frame sequence structure.

[0011] Furthermore, the process of enqueuing and dequeuing a circular queue also includes:

[0012] The circular queue enqueue and dequeue processing includes creating a corresponding queue instance for each internal device identifier index and performing bounded storage and sequential scheduling. When the queue level reaches a preset upper limit, overflow handling is performed. Overflow handling includes discarding the oldest audio frame, delaying the enqueueing of new audio frames, or marking new audio frames with gaps.

[0013] Furthermore, the spectral reduction and noise reduction process also includes:

[0014] The spectral subtraction noise reduction process includes a noise reduction process based on noise estimation and noise subtraction, maintains a noise baseline cache which is aggregated from recent silent candidate audio frames, and updates the noise baseline cache on a rolling basis according to a preset update cycle when the background noise drifts slowly.

[0015] Furthermore, the process of calling the dual-threshold endpoint detection and sliding window traversal overlap detection model to obtain the overlapping interval annotation structure also includes:

[0016] The dual-threshold endpoint detection processing includes calculating and smoothing frame-level energy statistics for each noise-reduced frequency frame, then comparing them with the start and end thresholds to determine the speech segment state. A continuous decision window and shortest speech segment constraint are introduced to handle short-term jitter and impulse noise triggering. A sliding window traversal overlap detection model invocation process is executed, which involves selecting consecutive valid frame segments on the time axis according to the window length and window step size, extracting acoustic features, and calling the overlap detection model to output an overlap probability sequence. Then, threshold judgment is used to generate overlapping and clean intervals. When the set of valid speech frames contains a gap marker field, the gap position is used as a candidate point for the interval boundary, resulting in an overlapping interval labeling structure.

[0017] Furthermore, the process of assembling interval processing tasks also includes:

[0018] The interval processing task assembly process includes performing consistency verification on interval boundaries and converting interval-level descriptions into executable task entities. The task entity includes a task identifier, interval type marker, interval start and end timestamps, frame index range, input segment reference, task priority, and bypass marker. According to the interval splitting rules, a splitting determination is performed on each interval, and a task priority is assigned. The assignment criteria include interval duration, average interval overlap probability, and task waiting time.

[0019] Furthermore, the process of separating model inference from orbital numbering also includes:

[0020] The separation model inference and track numbering process includes using the SepFormer Separation Transformer model to perform speaker separation inference on the input segments of overlapping intervals to obtain a set of candidate track segments, and assigning track numbers to each candidate single speaker audio segment to build an interval track number table. When the number of candidate track segments is inconsistent with the upper limit of the number of tracks configured in the interval splitting rules, the segments are pruned according to energy sorting and silence ratio filtering rules.

[0021] Furthermore, the process of performing voiceprint feature extraction and cosine similarity threshold determination also includes:

[0022] The voiceprint feature extraction and cosine similarity threshold determination operations include using the 3DSpeaker ERes2Net voiceprint feature extraction network to extract voiceprint feature vectors from track audio segments, performing cosine similarity calculation and threshold determination with template voiceprint features in the voiceprint template library, and when there are track interruption markers in the track audio segments, they are divided into multiple sub-segments according to the interruption boundaries, and the track voiceprint feature vectors are extracted separately and the mean is aggregated to generate track aggregated voiceprint features, generating a track identity binding structure.

[0023] Furthermore, the process of performing speech-to-text model inference and concatenating adjacent segments with the same speaker also includes:

[0024] The speech-to-text model inference process includes automatically performing speech recognition inference on the track audio segment to obtain the text content and confidence marker, and completing the segment timestamp binding. When the track audio segment has a track interruption marker or gap marker field, the segment to be transcribed is forcibly segmented at the gap boundary.

[0025] The process of splicing adjacent segments with the same speaker and pruning conflicting segments includes splicing segments with the same speaker identifier under the same session identifier and adjacent segments with timestamp intervals less than a preset merging interval, and pruning conflicting segments with overlapping or intersecting timestamp intervals. The pruning process includes conflict interval location, boundary segmentation, and rewriting of retained segments.

[0026] Furthermore, a real-time speakerprint recognition system for multi-person overlapping speech, applied to any of the methods described above, includes:

[0027] A multi-source device audio stream standardization module is used to receive the multi-source device audio stream acquisition configuration structure and the multi-source device audio stream set;

[0028] The effective speech frame set and overlapping interval labeling module is used to extract the audio frames to be processed from the streaming audio frame sequence structure;

[0029] The single-speaker audio track set generation module is used to receive the overlapping interval annotation structure and interval splitting rules;

[0030] The track identity binding structure generation module is used to perform voiceprint feature extraction and cosine similarity threshold determination operations on a single speaker's audio track set;

[0031] The structured output module is used to extract track audio segments from the track identity binding structure;

[0032] The database write module is used to perform database write operations and generate template library update instruction structures.

[0033] The key innovations of this invention include:

[0034] (1) The multi-source device audio stream acquisition configuration structure and the multi-source device audio stream set are uniformly incorporated into device identification mapping, input channel binding, sampling rate unification, channel unification, pulse code modulation encoding unification and fixed duration frame processing to form a standard audio stream and streaming audio frame sequence structure, which serves as the input object for subsequent circular queue enqueue and dequeue and extraction of audio frames to be processed.

[0035] (2) Based on the streaming audio frame sequence structure, the audio frames to be processed are processed by circular queue enqueue and dequeue, spectral subtraction and noise reduction, dual threshold endpoint detection and sliding window traversal overlap detection model call processing, to obtain the set of effective speech frames and the overlapping interval labeling structure. Under the constraints of the overlapping interval labeling structure and interval splitting rules, the interval processing task assembly, separation model reasoning and track numbering processing are completed to form a single speaker audio track set.

[0036] (3) Perform voiceprint feature extraction and cosine similarity threshold determination on the audio track set of a single speaker to generate a track identity binding structure. The track identity binding structure is then used in the speech transcription model inference, segment timestamp binding, adjacent segment splicing of the same speaker, conflict segment pruning and database writing operations to construct the speaker text segment set and structured output data structure. At the same time, a template library update instruction structure is generated.

[0037] The following are its main beneficial effects:

[0038] (1) In view of the fact that existing solutions are prone to inconsistent device identifier mapping and input channel binding in multi-source device input scenarios, and that it is difficult to maintain the continuity of the same processing object after the sampling rate is unified and the audio channel is unified, this invention organizes the multi-source device audio stream set in a unified manner through standard audio stream and streaming audio frame sequence structure, so that the audio frames to be processed have a consistent object reference basis in the circular queue enqueue and dequeue process and subsequent processing, supporting the generation and calling of subsequent speech effective frame set and overlapping interval labeling structure.

[0039] (2) In view of the problem that existing solutions rely on a single path to process overlapping speech scenarios, which can lead to unclear processing boundaries between overlapping and clean intervals and difficulty in maintaining consistency between processing objects and time, this invention forms an interval-level description of the audio frames to be processed by using a set of effective speech frames and an overlapping interval labeling structure. Under the constraints of interval splitting rules, the interval processing task assembly, separation model inference and track numbering are completed, so that the single speaker audio track set and the overlapping interval labeling structure maintain a corresponding relationship in the same processing flow, and meet the input consistency required for the subsequent track identity binding structure generation.

[0040] (3) In view of the problem that existing solutions are prone to unstable binding of speaker identity and track audio segments under overlapping speech conditions, which affects the continuous construction and storage of speaker text segment set and structured output data structure, this invention generates track identity binding structure through voiceprint feature extraction and cosine similarity threshold determination operation, and makes track identity binding structure run through the inference of associated speech transcription model and segment timestamp binding, as well as the splicing of adjacent segments of the same speaker and the trimming of conflicting segments and storage operation. At the same time, the identity determination input consistency of subsequent processing rounds is maintained through template library update instruction structure, thereby forming a verifiable structured output data structure and storage record for multi-person overlapping speech processing flow. Attached Figure Description

[0041] Figure 1 An overall architecture diagram of a real-time voiceprint recognition system for multiple overlapping voices provided in this application embodiment;

[0042] Figure 2 A flowchart illustrating a real-time voiceprint recognition method for multi-person overlapping speech provided in an embodiment of this application;

[0043] Figure 3 This is a structural block diagram of a real-time voiceprint recognition system for multiple overlapping voices provided in an embodiment of this application. Detailed Implementation

[0044] Figure 1 This application provides an overall architecture diagram of a real-time voiceprint recognition system for multi-person overlapping speech, as illustrated in the embodiments of this application. Figure 1 As shown, the architecture logically presents a top-down, data-flow-driven linear processing pipeline. It consists of four core functional modules arranged sequentially: a multi-source audio acquisition and standardization processing module, an overlapping speech detection and directional separation module, a speaker feature extraction and template matching module, and a speech transcription and structured integration module. These modules are coupled through clearly defined interfaces and data structures. The output of one module serves as the input of the next, and loose coupling is achieved through asynchronous message queues and buffering mechanisms. This ensures the system's real-time performance, robustness, and scalability when processing high-concurrency, continuous audio streams. The following sections will describe the definition, function, detailed implementation methods of each module, and the connections and collaboration processes between them.

[0045] 1. Multi-Source Audio Acquisition and Standardization Processing Module. Definition and Function: This module serves as the system's data input and preprocessing front-end. Its core responsibility is to be compatible with and manage raw audio streams from different physical devices and formats, transforming them into a unified, clean, and structured audio frame sequence within the system, laying the foundation for subsequent advanced processing. It solves the problem of accessing multi-source heterogeneous audio data and ensures the quality of upstream data.

[0046] Detailed implementation method:

[0047] Multi-source acquisition framework: Employs a PyAudio-based streaming acquisition scheme. During system initialization, the "Multi-source Device Audio Stream Acquisition Configuration Structure" is read. This structure defines the device identifier, expected sampling rate, number of channels, hardware interface type (e.g., USB, Bluetooth, 3.5mm analog input), and acquisition priority for each audio source (e.g., microphone array, telephone line, conference system output). The system creates an independent acquisition thread (sub-thread) for each active audio source, managed by a main monitoring thread. Each acquisition sub-thread, according to its configuration, calls the corresponding driver adapter to open the audio stream and sets a fixed circular buffer size (e.g., 1024 sampling points). Audio data is continuously fed into the thread's private circular buffer queue in a streaming manner.

[0048] Asynchronous Coordination and Buffering: The main monitoring thread is responsible for heartbeat detection, device anomaly alerts (such as disconnection or silence), and dynamic parameter adjustments (such as dynamically adjusting the buffer size based on network latency). The acquisition sub-thread accumulates approximately 2 seconds of audio data (or reaches a preset queue level), packaging it into an "audio data block" and attaching metadata such as the source device identifier index, global timestamp, and sequence number. This block is then pushed to a global, thread-safe "raw audio frame buffer queue." This two-level buffering design of "private cache + global queue" effectively isolates the rate differences between different acquisition sources, prevents data loss, and provides a stable data supply for subsequent processing.

[0049] Standardized preprocessing pipeline: The system retrieves audio data blocks from the "raw audio frame buffer queue" and performs standardized pipeline processing sequentially.

[0050] Format standardization: Check and standardize the sampling rate (using an anti-aliasing resampling algorithm), number of channels (by selecting the main channel or weighted merging of multiple channels into a mono), and quantization bit width (e.g., standardizing to 16-bit PCM). Any parameters that do not conform to the target format will trigger corresponding real-time conversion.

[0051] Real-time noise reduction: An improved spectral subtraction noise reduction algorithm is applied. The system maintains a dynamic "noise baseline cache," which is updated by continuously analyzing the statistical characteristics (such as the spectral mean) of audio frames marked as "silence candidates" (preliminarily determined by an energy threshold). For each audio frame, the current noise spectrum is estimated and subtracted in real time, and the processed frames are smoothed at the boundaries to reduce "musical noise." When a slow change in background noise characteristics is detected, the noise baseline is updated continuously at a certain learning rate; when encountering strong impact noise, updates are temporarily suppressed to avoid contaminating the noise model.

[0052] Endpoint detection: A dual-threshold (start threshold and end threshold) endpoint detection algorithm is employed. Short-time energy is calculated for each denoised audio frame. After smoothing and filtering, this energy is compared with the start threshold to trigger the start of a speech segment. Within the speech segment, it is compared with a lower end threshold, and the segment ends only after a certain number of frames have elapsed. This process combines constraints of "shortest speech duration" and "shortest silence duration," effectively filtering out brief non-speech segments such as coughs and tapping, as well as silence gaps, and accurately marking the start and end points of valid speech activity.

[0053] Output: The processed audio data is reorganized into a structured "streaming audio frame sequence structure". This structure contains not only the clean audio frame data, but also the timestamp of each frame, the device index, and the "voice / mute" label marked by the endpoint detection module. This structure is the basic data unit for all subsequent module processing.

[0054] Inter-module connectivity: This module connects to downstream systems by outputting a "streaming audio frame sequence structure." This structure is written to a shared data area called a "preprocessing frame queue." The overlapping speech detection and direction separation module, acting as a consumer, reads data sequentially from this queue. The queue's capacity limits and blocking / non-blocking read strategies serve as flow control and backpressure protection, ensuring that memory overflow does not occur due to data backlog during peak system processing times.

[0055] 2. Overlapping Speech Detection and Directional Separation Module. Definition and Function: This module is the core component for processing overlapping speech. Its function is to intelligently identify "overlapping intervals" where multiple people are speaking simultaneously from a continuous audio stream, and separate these mixed speech signals into independent, single-speaker audio tracks. It directly determines the system's ability to handle complex dialogue scenarios.

[0056] Detailed implementation method:

[0057] Overlap Detection: The system employs a pre-trained overlapping speech detection model (e.g., a model trained based on the pyannote.audio framework). During processing, based on the valid speech segments in the "streaming audio frame sequence structure" output by the previous module, a sliding window is traversed along the time axis. Each window is typically 0.5 to 2 seconds long, with a step size of half the window. For each audio segment within a window, its acoustic features (e.g., Mel spectrograms) are extracted and input into the overlap detection model. The model outputs a probability curve for overlapping speech at each time point within the window. The system applies a threshold to this probability curve (e.g., regions where the probability consistently exceeds 0.6) and, combined with the window sliding, ultimately generates an "overlap interval labeling structure." This structure accurately records which time periods in the entire audio stream are "clean speech" (single speaker) and which are "overlapping speech" (multiple speakers simultaneously), and includes a timestamp and overlap confidence for each interval.

[0058] Task Assembly and Scheduling: Based on the "overlapping interval labeling structure," the system creates "interval processing tasks." Each task includes an interval timestamp, a reference to the corresponding original audio data, and the interval type (overlapping / pure). The system maintains a task priority queue, with priorities dynamically calculated based on factors such as interval length, overlap confidence, and task waiting time. For interval tasks marked as "pure," a "bypass marker" is added, allowing them to skip the separation step and directly enter the voiceprint recognition stage, thus saving computational resources.

[0059] Targeted speech separation: For tasks involving overlapping speech, the system invokes a deep learning-based speech separation model, such as SepFormer (Separation Transformer). The specific process is as follows:

[0060] Data preparation: Extract audio segments corresponding to overlapping regions from the original audio and perform necessary preprocessing (such as length normalization and boundary windowing smoothing).

[0061] Model Inference: The processed audio segments are input into the SepFormer model. This model converts the audio into a time-frequency domain representation using an encoder, and models intra-band and cross-band dependencies using stacked Intra-Transformer and Inter-Transformer layers to learn speaker-related masks. Finally, the mask is applied to the mixed speech representation using a decoder to reconstruct multiple independent time-domain audio signals, each of which is predicted to correspond to a speaker.

[0062] Track post-processing: The audio signals separated by the model are assigned a "track number". The system maintains a "track number table" for each interval task, recording the number of each track, the start and end timestamps (inherited from the original interval and possibly fine-tuned for boundary smoothing), and a reference to the separated audio segment. If the number of separated tracks exceeds the preset limit for the number of speakers (for example, the model outputs 3 tracks but the system only tracks 2 speakers), the system will prune the tracks least likely to be valid speech based on heuristic rules such as track energy level and silence ratio, and record this pruning operation for auditing purposes.

[0063] Exception handling: If the separation model inference fails or returns an abnormal result, the system can automatically fall back to the "bypass" mode, output the entire overlapping interval as a single unseparated track, and mark it as low confidence to ensure that the process is not interrupted.

[0064] Inter-module connectivity: This module's input comes from the "preprocessed frame queue." Its output is a "tracked audio segment set" and an associated "track metadata table." These outputs are organized into an intermediate form of the "track identity binding structure" and sent to the next shared data area—the "separation result queue"—waiting to be consumed by the voiceprint feature extraction module. Clean intervals and separated single-speaker tracks are managed uniformly within this structure.

[0065] 3. Voiceprint Feature Extraction and Template Matching Module. Definition and Function: This module is responsible for identifying each individual speaker's audio track. Its core function is to extract unique "voiceprint features" from the audio, and then compare these features with a pre-registered voiceprint template library to determine the speaker's identity (ID) or mark them as unknown. It is a key technology for achieving "who is speaking" recognition.

[0066] Detailed implementation method:

[0067] Voiceprint Feature Extraction: The system employs advanced voiceprint embedding models, such as ERes2Net in 3DSpeaker. For each input audio track (whether from a clean region or a separated track), preprocessing is first performed: this may include secondary refinement by speech activity detection (VAD) to remove residual micro-silences within the track, and amplitude normalization. Subsequently, the audio segment is input into the ERes2Net network. This network, by combining residual connections and multi-scale feature extraction (through the introduction of dilated convolutions, etc.), can robustly extract a highly recognizable, fixed-dimensional voiceprint feature vector (e.g., 256-dimensional) from the audio. This vector is called the "voiceprint embedding" of the speech segment.

[0068] Feature post-processing: For longer audio tracks, they can be divided into multiple fixed-length sub-segments, and features can be extracted from each segment before averaging and aggregation to obtain more stable and representative track-level voiceprint features. If the track has interruptions (marked by the front end), features of the sub-segments before and after the interruption are extracted separately before aggregation.

[0069] Template Library Management: The system maintains a dynamic "voiceprint template library." During the registration phase, multiple segments (e.g., 3-5 segments) of clean speech from the target speaker are collected. Voiceprint features are extracted from each segment, and the average vector is calculated as the speaker's registration template, which is then bound to a unique ID and stored. The template library supports efficient similarity retrieval, addition, deletion, and modification operations, typically implemented using a vector database (such as Redis or Faiss) to support rapid comparison of large-scale templates.

[0070] Cosine similarity matching: During recognition, the cosine similarity between the voiceprint feature vector of the track to be recognized and all template vectors in the template library is calculated. The cosine similarity value ranges from [-1, 1], with values ​​closer to 1 indicating greater similarity. The system will then identify the template with the highest similarity and its score.

[0071] Threshold Determination and Decision: An acceptance threshold is set (e.g., 0.5). If the highest similarity score exceeds this threshold, the track is determined to belong to the speaker with the corresponding ID, and the ID and score are output. If the highest score is below the threshold, the track is determined to be "Unknown," indicating that it may come from an unregistered speaker or that the match confidence is insufficient. The system records the characteristics of "Unknown" tracks, which can be used for subsequent clustering or as candidates for new user registration.

[0072] Inter-module connectivity: This module takes track audio data from the "Separation Results Queue" as input. The output is a "Track-Identity" binding list, where each track is assigned its most likely speaker ID (or "Unknown") and a confidence score. This result is formatted into a "Preliminary Identity Binding Structure" and written to the "Identity Recognition Results Queue" for use by the final integration module.

[0073] 4. Speech Transcription and Structured Integration Module. Definition and Function: This module is the terminal of the pipeline, responsible for converting the separated and identified speech tracks into readable and searchable text information, and aligning, splicing, and structured outputting all information (who, when, and what was said). It ultimately delivers the core value of the system's processing.

[0074] Detailed implementation method:

[0075] Automatic Speech Recognition (ASR) Transcription: For each audio track that has been initially assigned an identity, an automatic speech recognition engine is invoked to transcribe it. The system can employ an advanced end-to-end ASR model, such as Paraformer. This model directly models the audio waveform, outputs the corresponding text sequence, and typically also outputs timestamp information (the start and end times of each word or phoneme). The transcription process includes acoustic feature extraction, encoder-decoder modeling, and language model-based decoding. The final result is the transcribed text corresponding to that track, along with its fine-grained time alignment information.

[0076] Speaker-Segment Concatenation: Because the original speech stream is segmented into multiple intervals by steps such as breakpoint detection and overlap detection, a single continuous speech from the same speaker may be divided into multiple adjacent "clean intervals" or "separated tracks." This module needs to perform a "segment concatenation" operation. The rule is: for text segments with the same speaker ID and that are temporally adjacent (with an interval less than a preset threshold, such as 200 milliseconds), their text content is concatenated, their timestamp ranges are merged (based on the start time of the first segment and the end time of the last segment), and the average voiceprint confidence can be calculated.

[0077] Conflict Handling and Structuring: In rare cases, minor overlaps or conflicts may occur in timestamps (e.g., due to separation or endpoint detection errors). The system performs conflict resolution, such as based on confidence levels or simple boundary pruning rules, to ensure that each time segment in the final output is occupied by only one speaker. Simultaneously, the module labels each output segment with its source attributes, such as whether it was obtained by separating overlapping intervals.

[0078] Structured Output and Persistence: All integrated information—including speaker ID, transcribed text, timestamps (start and end) of the text, audio source identifier of the segment, whether it comes from an overlapping region, voiceprint matching confidence, ASR transcription confidence, etc.—is organized into a structured data object (such as JSON format). This structured result can be directly pushed to upper-layer applications via API, or serialized and written to a file (such as TXT, JSONL) or stored in a database (such as MySQL, Elasticsearch) for subsequent querying, retrieval, and statistical analysis.

[0079] Template library update feedback: The module can also generate "template library update instructions" based on the processing results. For example, for a high-quality matching "Unknown" fragment, it can suggest adding its features to the template library as a new user registration; for low-confidence matches, it can trigger a re-evaluation or update of the template. This forms a closed-loop learning system.

[0080] Inter-module connection: This module's input comes from the "Preliminary Identity Binding Structure" in the "Identity Recognition Result Queue" and the corresponding raw track audio data (for ASR). It serves as the endpoint of the entire processing pipeline, and its output is the system's final product—a structured speech-to-text document. The entire data flow is now complete.

[0081] Overall system connection and operation process:

[0082] The entire system operates on a "pipeline" and "producer-consumer" model. The four main modules can be deployed as independent services or threads, connected via highly reliable message queues (such as Redis Streams or RabbitMQ) or in-memory queues (such as Python's `queue.Queue`). Control flow and data flow are separated; modules exchange predefined data structures only through queues, resulting in low coupling. Upon system startup, all queues and modules are initialized. Audio data flows in from the acquisition module, sequentially passing through overlap detection, voiceprint recognition, and speech-to-text modules, ultimately forming structured output. Each module can utilize multi-threading / process parallel processing of multiple data units to improve throughput. The monitoring system tracks queue lengths and module processing latency, enabling dynamic load balancing and anomaly alerts. This architecture ensures the system can stably handle complex dialogue scenarios with significant overlap from multiple audio sources in a streaming, low-latency manner.

[0083] In a preferred embodiment, referring to Figure 2 This is a flowchart illustrating a real-time voiceprint recognition method for multiple overlapping speech provided in an embodiment of the present invention. The process may include at least steps S100-S400:

[0084] S100: Obtain the multi-source device audio stream acquisition configuration structure and multi-source device audio stream set, perform pulse code modulation encoding unification and fixed duration framing processing, and generate a streaming audio frame sequence structure.

[0085] S200, based on the streaming audio frame sequence structure, performs circular queue enqueue and dequeue, spectral subtraction and noise reduction, dual-threshold endpoint detection and sliding window traversal overlap detection model call processing to obtain the overlapping interval labeling structure;

[0086] S300: Based on the overlapping interval labeling structure, perform interval processing task assembly, separation model inference and track numbering processing, and perform voiceprint feature extraction and cosine similarity threshold determination operations to generate track identity binding structure.

[0087] S400, based on the track identity binding structure, performs speech transcription model inference, concatenates adjacent segments with the same speaker, and generates a template library update instruction structure.

[0088] S100: Obtain the multi-source device audio stream acquisition configuration structure and multi-source device audio stream set, perform pulse code modulation encoding unification and fixed duration framing processing, and generate a streaming audio frame sequence structure.

[0089] The standardized input sources for the multi-source device audio stream include a multi-source device audio stream acquisition configuration structure and a multi-source device audio stream set. The multi-source device audio stream acquisition configuration structure describes the access and acquisition constraints of each acquisition terminal, and includes at least device identifier, input channel identifier, channel mode, sampling rate rules, sampling bit width rules, clock reference description, access weighting information, buffering and packet loss handling strategies, and abnormal event log entries. The multi-source device audio stream set carries continuous audio data segments from different acquisition terminals and their arrival order markers. Each audio data segment carries an acquisition terminal source marker and a local time reference marker. Specifically, the system reads the multi-source device audio stream acquisition configuration structure, parses the correspondence between device identifiers and input channel identifiers, and sends the access weighting information to the acquisition terminal access component to complete channel establishment. When multiple input channels are exposed at the same acquisition terminal, input channel binding is completed using a joint determination method of channel priority and channel availability, with the determination based on the channel quality log entries defined in the buffering and packet loss handling strategies. Furthermore, when the device identifier reported by the acquisition terminal is missing, duplicated, or does not meet the preset specifications, the system writes the event into the abnormal event log item and marks the corresponding audio data segment as a source to be reviewed, so as to avoid mixing with the subsequently bound channels.

[0090] The device identifier mapping is used to convert external device identifiers into internal device identifier indexes. The mapping process is implemented using a mapping table, which consists of device identifiers, internal device identifier indexes, input channel identifiers, mapping status, and the most recent heartbeat time. Specifically, when establishing an input channel binding, the system first searches the mapping table. If a matching entry exists and the mapping status is valid, the internal device identifier index is directly reused and the most recent heartbeat time is updated. If no matching entry exists, a new entry is generated based on the device identifier, and an internal device identifier index is assigned. Simultaneously, the mapping status is set to pending activation, and it is updated to valid after the first stable audio data segment is output at the acquisition end. Furthermore, when multiple conflicting entries are found in the mapping table, the system resolves the conflicts based on the most recent heartbeat time and channel availability, and writes the resolution process and the removed entries into the exception event log for subsequent database tracing. The input channel binding is executed after mapping is completed, and the binding result is written to the source tag field of the multi-source device audio stream set, so that each audio data segment carries an internal device identifier index and an input channel identifier for subsequent unified processing link reference.

[0091] After completing the device identifier mapping and input channel binding, the system performs unified sampling rate processing on the multi-source device audio stream set. Unified sampling rate is used to convert audio data segments from different acquisition terminals and with different sampling rate rules to the same target sampling rate rule. The conversion process employs a resampling processing link combining interpolation and decimation, consisting of three parts: anti-aliasing filtering, sampling point reconstruction, and boundary sample point joining. Specifically, the system reads the target sampling rate rule and the current sampling rate rule of each acquisition terminal from the multi-source device audio stream acquisition configuration structure, calculates resampling parameters for each audio data segment, and performs anti-aliasing filtering. When an audio data segment crosses a boundary, causing an incomplete filtering window, the system performs sample point completion or delayed merging according to the boundary joining rules in the buffering and packet loss handling strategy, and records the boundary joining type and corresponding segment index in the abnormal event log. Furthermore, when a sampling rate rule mutation event is detected, the system marks the mapping status of that acquisition terminal as pending review in the mapping table and writes the mutation event into the abnormal event log. Subsequent resampling is still performed according to the target sampling rate rule to avoid interrupting the continuous processing link.

[0092] Channel unification processing is performed after the sampling rate is unified, and it is used to convert audio data segments with different channel modes into a unified channel mode. The channel mode includes at least mono and multi-channel descriptions, and the channel unification processing link includes channel selection, channel merging, and channel order specification. Specifically, the system reads the target channel mode and channel order specification from the multi-source device audio stream acquisition configuration structure, performs channel order specification correction on the multi-channel audio data segments, and then extracts reference channels or performs channel merging according to the channel selection rules. Channel merging is implemented using a weighted merging method, and the weighting coefficients are derived from the noise level and signal strength markers maintained in the channel quality record. Furthermore, when there are missing channels or inconsistent channel lengths in the multi-channel audio data segments, the system aligns and fills them according to the boundary connection rules, and writes the missing channel marker into the abnormal event record. Subsequent outputs still maintain the target channel mode to avoid channel mode drift affecting the downstream overlapping interval determination.

[0093] The Pulse Code Modulation (PCM) encoding unification process is performed after sampling rate and channel unification. It unifies audio data segments to the target bit width and target byte order of Pulse Code Modulation (PCM) encoding. Specifically, the system reads the sampling bit width and byte order rules from the multi-source device audio stream acquisition configuration structure, performs bit width conversion and amplitude normalization on each audio data segment, and completes overflow trimming and silence segment preservation during the bit width conversion process. When non-linear compression encoding residue or header tag mismatch is detected in the input audio data segment, the system marks the segment as an encoding anomaly and writes it into the anomaly event log. Simultaneously, the system still completes the conversion and output according to the target PCM encoding rules to avoid link interruption. Further, after the encoding unification process is completed, the system aggregates the continuously output audio sampling points according to the internal device identifier index and input channel identifier to form a continuously readable target encoding sequence. This sequence is registered as the output field name "Standard Audio Stream," which carries the unified sampling rate rules, channel mode, PCM encoding rules, and time base marker.

[0094] Fixed-duration framing is performed after the standard audio stream is generated. It divides the standard audio stream into fixed-duration audio frames and forms a stream-processable sequence structure. Specifically, the system reads the fixed-duration framing rules from the multi-source device audio stream acquisition configuration structure. These rules include frame length, frame shift, whether overlapping framing is enabled, frame boundary padding strategy, and frame sequence number start rules. The system generates audio frames by sliding and framing the standard audio stream according to the frame length and frame shift, and assigns a frame sequence number and timestamp to each audio frame. The timestamp is generated by combining a time reference marker and a sample point count, and together with the internal device identifier index and input channel identifier, constitutes a frame-level source identifier. Further, when the standard audio stream length is insufficient for the frame length, the system performs padding or delayed output according to the frame boundary padding strategy, and writes the padding type, delay count, and corresponding frame sequence number into the exception event log. When packet loss causes discontinuous timestamps, the system inserts gap markers during timestamp generation and writes them into the exception event log for subsequent overlapping interval labeling stage interval boundary judgment.

[0095] Regarding output products and destinations, after completing fixed-duration frame processing, the system registers the sequence composed of audio frames, frame numbers, timestamps, internal device identifier indexes, and input channel identifiers as the output field name "Streaming Audio Frame Sequence Structure." This streaming audio frame sequence structure is then written into the streaming delivery buffer for subsequent steps (S200) to extract audio frames to be processed from the streaming audio frame sequence structure and perform circular queue enqueue / dequeue, spectral subtraction and noise reduction, dual-threshold endpoint detection, and sliding window traversal overlap detection model calls. Simultaneously, the standard audio stream, as the output field name "Standard Audio Stream," is synchronously written to the persistent cache or log medium. The unified sampling rate rules, unified channel modes, unified PCM encoding rules, and abnormal event records carried by the standard audio stream are used as source consistency references for subsequent steps (S300) when performing voiceprint feature extraction and similarity determination on a single speaker's audio track set. These records are also included with the structured output data structure during the write-to-database process in S400, forming a continuous link across the main steps.

[0096] In summary, the technical effects of this step are as follows: a unified entry point for multi-source input is achieved through device identification mapping and input channel binding; audio format specifications are achieved through unified sampling rate, unified audio channels, and unified pulse code modulation encoding; and a streaming audio frame sequence structure is formed through fixed-duration framing, thereby supporting the subsequent S200 to directly process frame-level data.

[0097] S200, based on the streaming audio frame sequence structure, performs circular queue enqueue and dequeue, spectral subtraction and noise reduction, dual-threshold endpoint detection and sliding window traversal overlap detection model call processing to obtain the overlapping interval labeling structure;

[0098] The input source for this step is the streaming audio frame sequence structure output by S100. This streaming audio frame sequence structure includes at least audio frames, frame numbers, timestamps, internal device identifier indexes, and input channel identifiers, and can be associated with frame-level tagging information of the abnormal event record items in S100. Specifically, the system extracts the audio frames to be processed from the streaming audio frame sequence structure according to the timestamp order. First, it performs a circular queue enqueue and dequeue operation to complete frame-level buffering and scheduling. Then, it sequentially performs spectral subtraction and noise reduction, dual-threshold endpoint detection, and sliding window traversal overlap detection model call processing on the dequeue sequence. Finally, it forms the output field name speech valid frame set and the output field name overlap interval labeling structure, where the overlap interval labeling structure will be called as the S310 processing input of S300.

[0099] The circular queue is a frame buffer structure with its head and tail connected, used for bounded storage and sequential scheduling of continuously arriving audio frames. Its structure includes at least queue capacity, a queue head pointer, a queue tail pointer, a queue level, an enqueue count, and a dequeue count. Specifically, the system establishes a corresponding queue instance for each internal device identifier index and input channel identifier. The system writes the audio frames to be processed extracted from the streaming audio frame sequence structure to the position pointed to by the queue tail pointer and advances the queue tail pointer to complete the enqueue. Simultaneously, it reads the audio frames pointed to by the queue head pointer and advances the queue head pointer to complete the dequeue. The queue level, enqueue count, and dequeue count are updated after each enqueue and dequeue, forming a traceable frame order state. Furthermore, when the queue level reaches a preset upper limit, the system performs overflow handling. Overflow handling includes at least one of the following: discarding the oldest audio frame, delaying the enqueue of newly arrived audio frames, and marking newly arrived audio frames with a gap mark. The overflow type, the corresponding frame sequence number range, and the timestamp range are written to the exception event log. Furthermore, when a timestamp reversal, frame sequence number jump, or inconsistency between the internal device identifier index and the input channel identifier is detected, the system isolates and enqueues the abnormal frame, writes the isolation reason and the associated frame index into the abnormal event record, and keeps the queue head and tail advancement rules unchanged to avoid disorder in the subsequent processing links.

[0100] After completing the enqueue and dequeue operations in the circular queue, the system extracts the audio frames to be processed from the dequeue sequence and performs spectral subtraction and noise reduction processing to obtain de-noised frequency frames. The spectral subtraction and noise reduction is a noise reduction process based on noise estimation and noise subtraction, which includes at least noise baseline establishment, noise baseline update, subtraction calculation, and boundary smoothing. Specifically, the system maintains a noise baseline cache during the dequeue process. The noise baseline cache is obtained by aggregating recent silent candidate audio frames. The determination of silent candidate audio frames can come from queue gap markers, low-energy initial screening markers, or silent prompt markers in the abnormal event record. The system extracts noise statistics from silent candidate audio frames and updates the noise baseline cache. Then, it performs noise subtraction calculation on the audio frames to be processed to obtain de-noised frequency frames and performs smoothing processing at the frame boundaries of the de-noised frequency frames to avoid abrupt changes. Furthermore, when background noise drifts slowly, the system updates the noise baseline cache on a rolling basis according to a preset update cycle; when sudden noise causes abnormal noise statistics, the system suppresses the noise baseline update within that window and writes the suppression event and associated frame sequence number into the abnormal event record, so that subsequent endpoint detection continues to run in the continuous frame stream.

[0101] After obtaining the noise-reduced frequency frames, the system performs dual-threshold endpoint detection to obtain speech segment boundaries and silence removal markers, thus forming the selection criteria for the set of valid speech frames. The dual-threshold endpoint detection is a speech start-end determination process based on start and end thresholds, including at least frame-level energy statistics, smoothing calculation, start trigger determination, end trigger determination, and a duration determination window. Specifically, the system calculates and smooths the frame-level energy statistics for each noise-reduced frequency frame, then compares the smoothed statistics with the start threshold. If the start threshold trigger condition is met, the system enters the speech segment state. In the speech segment state, the statistics are compared with the end threshold. If the end threshold trigger condition is met and a preset number of frames are continuously satisfied, the system exits the speech segment state. Furthermore, the system introduces a duration determination window and a shortest speech segment constraint to perform backtracking processing for short-term jitter and impulse noise triggers. During backtracking, the corresponding frame number range is marked as a silence removal marker and written into the abnormal event log. Furthermore, when the audio frame corresponding to the input channel identifier has a gap mark or an isolation queuing mark, the system performs a conservative judgment strategy for the segment, aligns and converges the endpoint boundaries on both sides of the gap, and records the convergence strategy type to avoid multiple back-and-forth switching at the endpoint boundaries at the gap.

[0102] After endpoint detection is completed, the system aggregates the noise-reduced frequency frames that are in the speech segment state and not covered by the silence removal mark to form a set of valid speech frames. This set of valid speech frames is the data structure output from this step, and includes at least valid speech frames, corresponding frame numbers, corresponding timestamps, internal device identifier indexes, input channel identifiers, and silence removal marks. Specifically, the system maintains a monotonically increasing timestamp check during frame aggregation. When a discontinuous timestamp is detected, the system writes a gap marker field into the set of valid speech frames and writes the gap position and length into the exception event record. Furthermore, when multiple valid frames with parallel input channel identifiers exist within the same timestamp window, the system groups them by internal device identifier index and then merges and sorts them by input channel identifier, forming a unified timeline view for subsequent sliding window traversal, ensuring that the overlap detection model can reference a consistent timestamp sequence.

[0103] After obtaining the set of valid speech frames, the system executes a sliding window traversal overlap detection model call to generate an overlap interval labeling structure. The sliding window traversal is a method of selecting consecutive valid frame segments by scrolling along the time axis according to the window length and window step size. The window length, window step size, overlap probability threshold, shortest overlap interval constraint, and interval merging interval constitute the overlap detection parameter set. Specifically, the system sorts the set of valid speech frames by timestamp, combines several consecutive valid speech frames into a window input segment, and extracts the acoustic features required by the overlap detection model from the window input segment. The acoustic features include frequency band energy distribution features and temporal variation features aggregated by frame, and retain the window start and end timestamps, internal device identifier index, and input channel identifier during the extraction process. Subsequently, the system calls the overlap detection model to output the overlap probability for each window and generates an overlap probability sequence, with a one-to-one correspondence between the overlap probability sequence and the window start and end timestamps. Furthermore, the system performs threshold determination and interval generation on the overlap probability sequence, mapping consecutive windows that meet the overlap probability threshold as overlapping intervals and consecutive windows that do not meet the overlap probability threshold as clean intervals. Intervals with lengths less than the shortest overlapping interval constraint are processed by either backtracking or merging. When the interval between adjacent intervals is less than the interval merging interval, the system performs interval merging and writes the merging position and reason to the exception event log. Furthermore, when the set of valid speech frames contains a gap marker field, the system uses the gap position as a candidate interval boundary point during the interval generation stage. Intervals on both sides of the gap are generated separately, and gap segmentation markers are recorded in the overlapping interval annotation structure to avoid timestamp mismatches caused by window splicing across gaps.

[0104] Regarding output products and destinations, the system registers the set of valid speech frames as the output field name "Set of Valid Speech Frames" and retains it in the processing cache of this step for extraction of window input segments when the sliding window traversal overlap detection model is invoked. Simultaneously, its frame sequence number and timestamp index information are associated with the abnormal event record for subsequent database tracing. The system assembles the overlapping interval, clean interval, interval start and end timestamps, internal device identifier index, input channel identifier, and overlap probability sequence into an overlapping interval labeling structure and registers it as the output field name "Overlapping Interval Labeling Structure." This overlapping interval labeling structure is obtained by S310 of S300 during cross-main step connections and used to assemble the interval processing task, thereby entering the separation model inference and track numbering processing link.

[0105] In summary, the technical effects of this step are as follows: frame-level buffering and sequential scheduling are achieved through circular queue enqueue and dequeue; a set of effective speech frames is formed through spectral subtraction and noise reduction and dual-threshold endpoint detection; and the overlapping interval labeling structure is obtained by calling the overlapping detection model through sliding window traversal, providing directly referable interval-level input for subsequent interval splitting and separation processing.

[0106] S300: Based on the overlapping interval labeling structure, perform interval processing task assembly, separation model inference and track numbering processing, and perform voiceprint feature extraction and cosine similarity threshold determination operations to generate track identity binding structure.

[0107] The input sources for this step include the overlapping interval annotation structure of the S200 output field name and the preset interval splitting rules, and are associated with the standard audio stream and streaming audio frame sequence structure of the S100 output field name. The overlapping interval annotation structure is used to describe the start and end timestamp ranges of overlapping intervals and clean intervals and their corresponding overlap probability sequences. The interval splitting rules are used to describe the judgment conditions and task assembly constraints for different interval types to enter different processing links. Specifically, after reading the overlapping interval annotation structure, the system first performs consistency verification on the interval boundaries. The consistency verification includes interval start and end timestamp monotonicity verification, interval interval verification, and interval coverage relationship verification. During the verification process, the interval boundaries are aligned and mapped with the timestamp indices in the streaming audio frame sequence structure to obtain the mapping relationship from interval to frame index. When there is a timestamp gap mark or the interval boundary falls within the gap range, the system marks the interval as a gap-sensitive interval and writes it into the abnormal event record. At the same time, the frame index is corrected according to the interval boundary nearest mapping principle to avoid cross-gap splicing in subsequent segment extraction.

[0108] After the interval boundaries are aligned and mapped, the system performs interval processing task assembly. Interval processing task assembly refers to the process of converting interval-level descriptions into executable task entities. Each task entity includes at least a task identifier, interval type flag, interval start and end timestamps, frame index range, input segment reference, task priority, and bypass flag. The input segment reference points to the corresponding audio segment in the standard audio stream or streaming audio frame sequence structure. Specifically, the system performs a splitting determination for each interval according to interval splitting rules. The splitting determination considers both the interval type flag and the threshold discrimination result of the overlap probability sequence. When an interval is determined to be an overlapping interval, an overlapping interval task is generated and the bypass flag is cleared. When an interval is determined to be a clean interval, a clean interval task is generated and the bypass flag is written, allowing the clean interval task to directly reference the input segment without entering the separation model inference. Furthermore, the system assigns task priorities based on interval duration, the average interval overlap probability, and task waiting time. When an overflow handling record appears in the circular queue, the priority of the associated interval tasks is downgraded or rearranged, and the reason for the rearrangement is written to the exception event record, ensuring traceability of task scheduling.

[0109] After generating the interval tasks, the system performs separation model inference and track numbering processing on the overlapping interval tasks. Separation model inference refers to the process of performing speaker separation inference on the input segments of the overlapping intervals. The separation model is the SepFormer Separation Transformer model inference module, whose input is the audio segment referenced by the overlapping interval task, and whose output is multiple candidate single-speaker audio segments. Specifically, the system extracts continuous sampling segments corresponding to the start and end timestamps of the overlapping interval from the standard audio stream to form segments to be separated, performs length normalization and boundary smoothing on the segments to be separated, and then sends them to the separation model to complete the inference to obtain a set of candidate track segments. When the number of candidate track segments is inconsistent with the upper limit of the number of tracks configured in the interval splitting rules, the system prunes the candidate track segment set according to energy sorting and silence ratio filtering rules to ensure that the number of output tracks meets the upper limit constraint, and writes the pruning strategy and the index of the pruned track into the exception event log. Further, when the separation model inference returns an empty result or returns a segment with an abnormal length, the system switches the overlapping interval task to a bypass flag state and outputs a single-track audio segment, while recording the inference failure flag and the associated task identifier to avoid subsequent link interruption.

[0110] Track numbering is performed after the separation model inference. Track numbering refers to the process of assigning a track number to each candidate single-speaker audio segment and constructing track-level metadata. Specifically, the system maintains a track number table for each interval task. The interval track number table contains at least the track number, interval start and end timestamps, track segment references, track timestamps, and source indexes. The source index includes an internal device identifier index and an input channel identifier. The system writes the candidate track segment set into the track number table one by one and generates track timestamps. The track timestamps are generated by the interval start and end timestamps and the track segment sampling point count. When there is a clean interval task, the system directly writes the interval input segment into the track number table and assigns a single track number based on the bypass marker. Furthermore, if there is a slight overlap in timestamps between adjacent interval tasks, the system performs overlap pruning according to the interval boundary verification results to ensure that the track segment references are not repeatedly covered on the timeline, and writes the pruning boundary into the abnormal event log for subsequent verification when S400 performs splicing of adjacent segments with the same speaker.

[0111] After completing the track numbering process, the system aggregates the track number table to generate a list of output fields and a speaker audio track set. A single speaker audio track set refers to a track-level audio segment set structure organized according to track numbers, containing at least the track number, track audio segment, track timestamp, internal device identifier index, and input channel identifier, while retaining the interval task identifier and abnormal event record reference. Specifically, the system sorts multiple track audio segments corresponding to the same track number by timestamp and performs segment boundary alignment and silence boundary clipping to obtain a continuous sequence of track audio segments, which is then written into the single speaker audio track set. When a gap marker field appears in the track audio segment sequence, the system records the gap position as a track interruption marker and writes it into the abnormal event record, enabling subsequent voiceprint feature extraction to segment at the interruption boundary. Furthermore, the system writes the single speaker audio track set into a track cache pool and retains a reverse index from the track number to the interval task identifier, facilitating subsequent tracing of which interval task the track originated from.

[0112] After generating a set of single-speaker audio tracks, the system performs voiceprint feature extraction and cosine similarity thresholding on the single-speaker audio track set, generating an output field name track identity binding structure. Voiceprint feature extraction refers to the reasoning process of extracting speaker embedding vectors from track audio segments. The system uses the 3DSpeaker ERes2Net voiceprint feature extraction network inference module, which segments the track audio segments into speech segments, normalizes amplitude, and reduces segment length before inputting it into the network, outputting track voiceprint feature vectors and binding them to track numbers for storage. When a track audio segment has a track interruption marker, the system divides it into multiple sub-segments according to the interruption boundary, extracts track voiceprint feature vectors for each sub-segment, and performs aggregation at the track layer to obtain track aggregated voiceprint features. The track aggregated voiceprint features are generated by averaging the sub-segment voiceprint feature vectors, and the index range of the sub-segments participating in the aggregation is recorded. The cosine similarity threshold determination operation refers to the process of calculating the cosine similarity between the track-aggregated voiceprint features and the template voiceprint features in the voiceprint template library structure, and then performing threshold discrimination. The voiceprint template library structure is a template storage and indexing structure, which at least includes speaker identifier, template voiceprint features, template update timestamp, template status, and threshold parameter set. The system reads the voiceprint template library structure through the cache interface and calculates the track similarity score sequence one by one, and then completes the maximum score selection and threshold discrimination according to the threshold parameter set. When the maximum score is less than the threshold, the track is marked as unknown and the candidate score sequence is recorded. When the maximum score is not less than the threshold, the corresponding speaker identifier is written into the track identity binding record. Furthermore, when the voiceprint template library structure reading fails or the template status is frozen, the system marks the track as unknown and writes it into the abnormal event record, while retaining the track-aggregated voiceprint features for the S400 database writing stage association record to avoid the identity binding link being interrupted.

[0113] The track identity binding structure is generated after voiceprint feature extraction and cosine similarity threshold determination. The track identity binding structure includes at least the track number, speaker identifier, similarity score, "unknown" marker, track timestamp, internal device identifier index, and input channel identifier, and is associated with track audio segment references in the single speaker audio track set. Specifically, the system associates and assembles the identity binding record of each track with the corresponding track audio segment reference to form a track identity binding structure that can be directly read by subsequent links, and registers this track identity binding structure as the output field name "Track Identity Binding Structure" for this step. In terms of output destination, the single-speaker audio track set and track identity binding structure are synchronously written into the processing cache and published downstream. The track identity binding structure will be used by S400 to extract track audio segments from the track identity binding structure and perform speech transcription model inference and segment timestamp binding. The single-speaker audio track set will serve as the track-level reference basis when S400 performs adjacent segment splicing and conflict segment pruning. At the same time, the template library update instruction structure generated by S400 is backflowed to update the voiceprint template library structure after being written into the library, so that the cosine similarity threshold determination operation in subsequent rounds of this step continues to reference the updated template state and template voiceprint features.

[0114] In summary, the technical effects of this step are as follows: it completes the assembly and scheduling of interval tasks to track tasks, completes the separation reasoning of overlapping intervals and track numbering processing, forms a single-speaker audio track set and generates a track identity binding structure, providing track-level input for the subsequent S400 transcription and structured output link.

[0115] S400, based on the track identity binding structure, performs speech-to-text model inference, concatenates adjacent segments with the same speaker, and generates a template library update instruction structure.

[0116] The input source for this step is the track identity binding structure output by S300, which is associated with the track audio segment references and track timestamps in the single-speaker audio track set. The track identity binding structure refers to the data structure after binding the track number and speaker identifier, and at least includes the track number, speaker identifier, similarity score, unknown marker, track timestamp, internal device identifier index, and input channel identifier, pointing to the corresponding track audio segment reference. Specifically, after receiving the track identity binding structure, the system first establishes a track processing context based on the track number and track timestamp. The track processing context at least includes a session identifier, track number, track segment index table, segment status, and abnormal event record item index. The session identifier is synthesized from the internal device identifier index, input channel identifier, and timestamp segment. Further, when the track processing context detects a reversed track timestamp, missing segment reference, or inconsistency between the unknown marker and the similarity score, the system marks the corresponding record as a record to be reviewed and writes it into the abnormal event record item. Simultaneously, the track audio segment reference is still retained for subsequent speech-to-text model inference, preventing track interruption in this step.

[0117] After establishing the track processing context, the system extracts track audio segments from the track identity binding structure, performs speech-to-text model inference, and completes segment timestamp binding to obtain a set of speaker text segments. The speech-to-text model inference refers to the process of automatic speech recognition inference on the track audio segments (Automatic Speech Recognition, ASR). The system segments the track audio segments into segments to be transcribed according to track timestamps, performs amplitude normalization, silence boundary convergence, and segment length reduction on the segments to be transcribed, and then sends them to the ASR inference module to obtain text content and confidence markers. Segment timestamp binding refers to the process of binding the text content with the start and end timestamps of the segments to be transcribed. During binding, the system simultaneously writes the track number, speaker identifier, "unknown" marker, and similarity score to form segment-level records. Furthermore, when the ASR inference module returns empty text, abnormal characters, or a confidence marker below a preset threshold, the system marks the segment as a low-confidence segment and writes it into the abnormal event record, while retaining the original text content and confidence markers for subsequent splicing and trimming processing, maintaining the continuity of the processing chain. Furthermore, when a track audio segment has a track interruption marker or gap marker field, the system forcibly segments the segment to be transcribed at the gap boundary and writes the gap segmentation marker in the segment-level record to avoid the segment start and end timestamps drifting due to binding timestamps across gaps.

[0118] After obtaining the segment-level records, the system performs concatenation of adjacent segments with the same speaker and pruning of conflicting segments on the speaker's text segment set, generating a structured output data structure. Adjacent segments with the same speaker refer to segments under the same session identifier, with identical speaker identifiers and adjacent segment timestamp intervals less than a preset merging interval. The system performs concatenation processing on these segments, including text content linking, segment start and end timestamp expansion, confidence tag merging, and anomaly tag inheritance, and updates the index range in the track segment index table after concatenation. Conflicting segments refer to a set of segments under the same session identifier whose segment timestamp intervals overlap or intersect, and whose speaker identifiers or "unknown" tags differ. The system performs pruning processing on conflicting segments, including conflict interval location, boundary segmentation, and rewriting of retained segments. Specifically, the system first sorts the speaker's text fragment set by timestamp, detects fragment groups with overlapping timestamps and generates conflict intervals, then makes conflict handling decisions based on track timestamps, similarity scores, confidence markers, and interval type markers in the overlapping interval labeling structure. After completing boundary segmentation, it rewrites the start and end timestamps and text content indexes of the affected fragments. Furthermore, when a conflict interval crosses a gap segmentation marker or contains low-confidence fragments, the system marks the conflict interval as a high-risk conflict and writes it into the abnormal event record item, while retaining the fragment mapping relationship before and after pruning for the database writing stage record. The structured output data structure is the result structure for external storage in this step, and at least includes session identifier, speaker identifier, fragment start and end timestamps, text content, track number, unknown marker, similarity score, confidence marker, internal device identifier index, input channel identifier, and abnormal event record item index, and this structure maintains the timestamp monotonicity check after generation.

[0119] After generating the structured output data structure, the system performs a database write operation and generates a template library update instruction structure. The database write operation refers to the process of writing the structured output data structure to persistent storage. The system serializes the structured output data structure into a JavaScript Object Notation (JSON) record format and simultaneously writes it to both text file media and database media. When using a relational database (MySQL) for storage, the system establishes association keys for a session table, a fragment table, and an exception table, writes records according to the session identifier and fragment start and end timestamps, and indexes exception event records into the exception table. Furthermore, when database writes fail or primary key conflicts occur, the system performs idempotent retries and conflict resolution. Conflict resolution uses timestamp boundary recalculation and fragment sequence number reordering, and writes the number of retries, conflict keys, and resolution strategies into the exception event record. The template library update instruction structure refers to the update request structure oriented towards the voiceprint template library structure. During the database writing phase, the system scans the unknown marked segments in the structured output data structure, extracts the corresponding track audio segment references and track aggregated voiceprint feature references, and forms an update request record. The update request record includes at least the operation type, session identifier, track number, segment start and end timestamps, voiceprint feature references, data source index, and pending review mark. The operation type includes at least one of adding a template, updating a template, and freezing a template. Furthermore, when multiple unknown marked segments appear under the same session identifier and the similarity of track aggregated voiceprint features meets the preset aggregation conditions, the system merges the multiple unknown marked segments into a single update request record and writes the merged index range into the update request record to avoid duplicate generation of update requests. The template library update instruction structure is written into the update queue as the output field name template library update instruction structure, and is used by the voiceprint template library structure referenced in the voiceprint feature extraction and cosine similarity threshold determination operations in S300 to perform the update, thus forming a backflow connection across the main steps. At the same time, the speaker text fragment set and the structured output data structure are saved as the output field name speaker text fragment set and the output field name structured output data structure for subsequent retrieval calls.

[0120] In summary, the technical effects of this step are as follows: the speech transcription model inference of the track audio segments is completed and the segments are timestamped; adjacent segments of the same speaker are spliced ​​and conflicting segments are trimmed and written to the database; at the same time, the template library update instruction structure is generated to form a backflow update link.

[0121] Example 2: Figure 3 A structural block diagram of a real-time voiceprint recognition system for multiple overlapping speech according to an embodiment of the present invention is shown. Figure 3 As shown, the structure may include:

[0122] The multi-source device audio stream standardization module 01 receives the multi-source device audio stream acquisition configuration structure and the multi-source device audio stream set, performs device identifier mapping, input channel binding, sampling rate unification, channel unification, pulse code modulation encoding unification, and fixed-duration framing processing, and outputs the standard audio stream and streaming audio frame sequence structure to the speech effective frame set and overlapping interval labeling module. Specifically, the multi-source device audio stream standardization module receives the multi-source device audio stream acquisition configuration structure and reads the device identifier mapping item, input channel binding item, sampling rate unification item, channel unification item, pulse code modulation encoding unification item, and fixed-duration framing parameters therein. It then performs device identifier mapping for each channel in the multi-source device audio stream set and generates channel association relations consistent with the input channel binding. The system performs resampling on the target sampling rate indicated by the sampling rate unification item, performs channel merging or channel splitting on the target number of channels indicated by the channel unification item, and performs encoding conversion on the sample format indicated by the pulse code modulation encoding unification item, while preserving the continuity of timestamps and channel associations of the audio stream during the conversion process; in the fixed-duration framing stage, the standard audio stream is divided into continuous audio frames according to a fixed duration and written into the streaming audio frame sequence structure. Each audio frame in the streaming audio frame sequence structure carries a channel tag corresponding to the input channel and carries a timestamp; the multi-source device audio stream standardization module provides the standard audio stream and streaming audio frame sequence structure to the speech effective frame set and overlapping interval labeling module and triggers its extraction process of the audio frames to be processed.

[0123] The effective speech frame set and overlapping interval labeling module 02 is used to extract the audio frames to be processed from the streaming audio frame sequence structure, perform circular queue enqueue and dequeue, spectral subtraction and noise reduction, dual-threshold endpoint detection and sliding window traversal overlap detection model calling processing, and output the effective speech frame set and overlapping interval labeling structure to the single speaker audio track set generation module; specifically, the effective speech frame set and overlapping interval labeling module receives the streaming audio frame sequence structure output from the multi-source device audio stream standardization module, extracts the audio frames to be processed in timestamp order and writes them into a circular queue, the circular queue enqueue and dequeue schedule the audio frames to be processed in a predetermined order and maintain the correspondence between channel labels and timestamps; in the spectral subtraction and noise reduction stage, noise is statistically established based on the background segment of the audio frames to be processed. The system estimates and performs spectral subtraction to generate denoised audio frames to be processed. In the dual-threshold breakpoint detection stage, threshold discrimination is performed on the denoised audio frames to be processed, and the speech segment boundaries are output. The audio frames to be processed within the speech segment range are aggregated to generate a set of valid speech frames. In the sliding window traversal overlap detection model invocation and processing stage, continuous frame segments are extracted from the set of valid speech frames according to the sliding window parameters, and the overlap detection model is invoked to generate overlap judgment results. The overlap judgment results are aggregated according to timestamps to generate an overlap interval labeling structure. The set of valid speech frames and the overlap interval labeling module provide the set of valid speech frames and the overlap interval labeling structure to the single speaker audio track set generation module, and maintain the consistency of the timestamp index between the overlap interval labeling structure and the streaming audio frame sequence structure.

[0124] The single-speaker audio track set generation module 03 is used to receive the overlapping interval labeling structure and interval splitting rules, perform interval processing task assembly, separation model inference and track numbering processing, and output the single-speaker audio track set to the track identity binding structure generation module. Specifically, the single-speaker audio track set generation module receives the overlapping interval labeling structure output from the set of valid speech frames and the overlapping interval labeling module, and receives the interval splitting rules as the basis for interval splitting determination. First, it aligns the interval boundaries and timestamp indices of the overlapping interval labeling structure, maps the interval boundaries to the corresponding range of audio frames to be processed, and forms a reference relationship from interval to frame index. In the interval processing task assembly stage, each interval generates a task entity and writes it into the task scheduling channel. The task entity includes an interval type marker, an interval start and end timestamp, and a frame. The index range and input segment references, where the input segment references point to the corresponding audio segment in the standard audio stream or streaming audio frame sequence structure; during the separation model inference stage, tasks marked as overlapping intervals are inferred by the separation model and multiple candidate audio segments are output, while tasks marked as clean intervals are bypassed and a single audio segment is output; during the track numbering processing stage, candidate audio segments are assigned track numbers according to the order within the interval and track timestamps consistent with the start and end timestamps of the interval are generated, and audio segments with the same track number are sorted by timestamp and assembled into a single-speaker audio track set; the single-speaker audio track set generation module provides the single-speaker audio track set to the track identity binding structure generation module and retains the association between the track number and the interval type mark for subsequent conflict segment trimming.

[0125] The track identity binding structure generation module 04 is used to perform voiceprint feature extraction and cosine similarity threshold determination operations on the single-speaker audio track set, generate a track identity binding structure, and output the track identity binding structure to the structured output module. Specifically, the track identity binding structure generation module receives the single-speaker audio track set output by the single-speaker audio track set generation module, extracts track audio segments according to track number, and performs voiceprint feature extraction. Voiceprint feature extraction includes performing speech segment slicing, amplitude normalization, and segment length reduction on the track audio segments to generate track voiceprint features. For each track voiceprint feature... The cosine similarity threshold determination operation is performed, which calculates the similarity between the track voiceprint features and the preset voiceprint features and completes the threshold determination. The threshold determination output is bound to the track number. The track identity binding structure generation module associates and assembles the track number, threshold determination output, similarity score, track timestamp, internal device identifier index and input channel identifier with the track audio segment reference to generate the track identity binding structure. The track identity binding structure generation module provides the track identity binding structure to the structured output module and maintains the consistency between the track identity binding structure and the track number of the single speaker audio track set.

[0126] The structured output module 05 is used to extract track audio segments from the track identity binding structure, perform speech-to-text model inference, segment timestamp binding, concatenation of adjacent segments with the same speaker, and pruning of conflicting segments, construct a set of speaker text segments and a structured output data structure, and send the structured output data structure to the database writing module; specifically, the structured output module receives the track identity binding structure output from the track identity binding structure generation module, reads the track number and track audio segment reference from the track identity binding structure, and segments the track audio segments according to the track timestamp to form segments to be transcribed; in the speech-to-text model inference stage, the speech-to-text model is invoked to infer and output the text content of the segment to be transcribed, and the text content is generated synchronously. The system binds the timestamps of the segments to be transcribed to the corresponding timestamps of the segments to be transcribed. During the segment splicing stage adjacent to the speaker, the text content is spliced ​​according to the adjacency relationship between the speaker identifier and the segment timestamp, and the start and end timestamps of the spliced ​​segments are updated. During the conflict segment trimming stage, segments with overlapping timestamps are detected and boundary trimming is performed. After trimming, the segment timestamp binding records are rewritten synchronously while retaining the association with the track number. The structured output module assembles the speaker identifier, track number, segment timestamp binding records, and text content to form a speaker text segment set, and encapsulates the speaker text segment set into a structured output data structure. Then, the structured output data structure is sent to the database writing module and the database writing process is triggered.

[0127] The database writing module 06 is used to perform database writing and generate template library update instruction structures. Specifically, the database writing module receives the structured output data structure sent by the structured output module, writes it to the persistent storage medium according to the session dimension and the segment dimension, and maintains the consistency relationship between the writing order and the segment timestamp. During the writing process, consistency checks are performed on duplicate writing records, and conflicting records are overwritten or merged. In the template library update instruction structure generation stage, identity change information and track number information related to the track identity binding structure are extracted from the structured output data structure, assembled to generate the template library update instruction structure, and written to the update channel. The template library update instruction structure includes the association information of track number, speaker identifier, similarity score, and segment timestamp binding record. The database writing module provides the template library update instruction structure to the update channel called by the track identity binding structure generation module for reading. After reading the template library update instruction structure, the track identity binding structure generation module participates in the configuration update of the subsequent cosine similarity threshold determination operation, thereby forming a closed loop from the multi-source device audio stream standardization module to the database writing module.

Claims

1. A method for real-time voiceprint recognition of multiple overlapping speech, characterized in that, include: S100: Obtain the multi-source device audio stream acquisition configuration structure and multi-source device audio stream set, perform pulse code modulation encoding unification and fixed duration framing processing, and generate a streaming audio frame sequence structure; wherein, each audio frame is assigned a frame number and a timestamp; when packet loss occurs and the timestamp is discontinuous, a gap mark is inserted during the timestamp generation process and written into the abnormal event record item; S200. Based on the streaming audio frame sequence structure, extract the set of valid speech frames and perform sliding window traversal overlap detection model call processing to obtain the overlap interval labeling structure; wherein, when the set of valid speech frames contains a gap mark field, the gap position is used as the candidate point of the interval boundary during the interval generation stage, and the intervals on both sides of the gap are generated respectively and the gap segmentation mark is recorded in the overlap interval labeling structure. S300. Based on the overlapping interval labeling structure, perform interval processing task assembly, track numbering processing, and execute voiceprint feature extraction and cosine similarity threshold determination operations to generate a track identity binding structure. Specifically, when a timestamp gap mark exists or the interval boundary falls within the gap range, the interval is labeled as a gap-sensitive interval and written into the abnormal event record. Simultaneously, frame index correction is performed according to the interval boundary proximity mapping principle. When a gap mark field appears in the track audio segment sequence, the gap position is recorded as a track interruption mark and written into the abnormal event record. When a track audio segment has a track interruption mark, it is divided into multiple sub-segments according to the interruption boundary, and track voiceprint feature vectors are extracted separately. Aggregation is then performed at the track layer to obtain track aggregated voiceprint features. S400. Based on the track identity binding structure, perform speech transcription model inference and segment timestamp binding to form segment-level records; when the track audio segment has a track interruption mark or gap mark field, forcibly split the segment to be transcribed at the gap boundary and write the gap segmentation mark in the segment-level record.

2. The method according to claim 1, characterized in that, The process of pulse code modulation encoding unification and fixed-duration framing includes: The sampling bit width rules and byte order rules are read from the multi-source device audio stream acquisition configuration structure and bit width conversion and amplitude normalization are performed. Overflow clipping and silence segment preservation are completed during the bit width conversion process. The fixed-duration framing rules are read from the multi-source device audio stream acquisition configuration structure, and the standard audio stream is slidably segmented according to the frame length and frame shift to generate audio frames and assign frame numbers and timestamps. When the length of the standard audio stream is less than the frame length, the frame boundary padding strategy is used to perform padding or delay output.

3. The method according to claim 1, characterized in that, Before entering the set of valid voice frames for extraction, the streaming audio frame sequence structure also performs circular queue enqueue and dequeue processing. When timestamp reversal, frame sequence number jump, or inconsistency between internal device identifier index and input channel identifier is detected, abnormal frames are isolated and enqueued, and the isolation reason and associated frame index are written into the abnormal event record. When the queue level reaches the preset upper limit, overflow handling is performed. The overflow handling includes discarding the oldest audio frame, delaying the enqueue of newly arrived audio frames, or marking the newly arrived audio frames with gaps.

4. The method according to claim 1, characterized in that, The extraction of the effective speech frame set also includes spectral subtraction and noise reduction, and dual-threshold endpoint detection: The spectral subtraction and noise reduction includes a noise reduction process based on noise estimation and noise subtraction, maintaining a noise baseline cache which is obtained by aggregating recent silent candidate audio frames, and updating the noise baseline cache on a rolling basis according to a preset update cycle when the background noise drifts slowly. The dual-threshold endpoint detection involves calculating and smoothing frame-level energy statistics for each noise-reduced frequency frame, then comparing them with the start and end thresholds to determine the speech segment state. A continuous decision window and the shortest speech segment constraint are introduced to perform backoff processing for short-term jitter and impulse noise triggering.

5. The method according to claim 1, characterized in that, The sliding window traversal overlap detection model call process includes: selecting continuous valid frame segments by scrolling along the time axis according to the window length and window step size, extracting acoustic features and calling the overlap detection model to output the overlap probability sequence, and then generating the overlap interval and clean interval by threshold determination.

6. The method according to claim 1, characterized in that, The interval processing task assembly includes: The interval boundaries are verified for consistency and the interval-level description is transformed into an executable task entity. The task entity includes at least a task identifier, an interval type marker, an interval start and end timestamp, a frame index range, an input segment reference, a task priority, and a bypass marker. Perform a traffic diversion decision for each interval based on the interval diversion rules; Assigning values ​​to tasks, the assignment is based on factors including interval duration, average interval overlap probability, and task waiting time; When an overflow handling record appears in the circular queue, the priority of the associated interval tasks is downgraded or rearranged, and the reason for the rearrangement is written into the exception event record.

7. The method according to claim 1, characterized in that, The track numbering process includes: Perform separation model inference on the input segments of overlapping intervals to obtain a set of candidate track segments, and assign track numbers to each candidate single-speaker audio segment to construct an interval track number table; When the number of candidate track segments is inconsistent with the upper limit of the number of tracks configured in the interval diversion rule, the segments are pruned according to the energy sorting and the silent ratio filtering rule.

8. The method according to claim 1, characterized in that, The process of voiceprint feature extraction and cosine similarity threshold determination includes: The 3DSpeaker ERes2Net voiceprint feature extraction network is used to extract voiceprint feature vectors from track audio segments, and cosine similarity and threshold discrimination are performed between the vector vectors and template voiceprint features in the voiceprint template library. When a track audio segment has a track interruption marker, it is divided into multiple sub-segments according to the interruption boundary, and the track voiceprint feature vectors are extracted separately. The mean is then aggregated at the track layer to generate track aggregated voiceprint features.

9. The method according to claim 1, characterized in that, The S400 also includes: Segments with the same speaker ID under the same session ID and whose adjacent segment timestamp interval is less than the preset merging interval are spliced ​​together; Conflicting segments with overlapping or intersecting timestamp intervals are pruned. The pruning process includes locating the conflict intervals, dividing the boundaries, and rewriting the retained segments.

Images