An overlap speech separation method, system, device and medium based on characteristic space orthogonal projection

Abstract

The application discloses an overlapping speech separation method and system based on feature space orthogonal projection, a device and a medium, and relates to the technical field of speech recognition. The method comprises the following steps: real-time detection of a collected speech sequence is performed through a pre-trained segmentation model, speech active segments and non-active segments are divided, and multi-speaker overlapping speech segments are accurately located. For the overlapping segments, the identity of the main speaker is determined and the voiceprint features thereof are extracted by using context tracking information, and a known speaker feature subspace is constructed. Then, the mixed voiceprint features of the overlapping speech are extracted, the mixed voiceprint features are decomposed into a subspace parallel component and an orthogonal vertical component through matrix projection operation, so that residual features without the main speaker information are obtained, the cosine similarity with candidate speakers is calculated, the speaker with the highest similarity is selected as the secondary speaker, and the logical separation of the overlapping speech is realized. The known speaker features are eliminated through mathematical projection operation, and unknown speakers can be accurately identified in the residual space.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to an overlapping speech separation method, system, device and medium based on orthogonal projection of feature space. Background Technology

[0002] Speaker separation technology aims to identify and distinguish speech segments from different speakers in mixed speech, and is a key step in the front-end processing of speech recognition. In real-world applications (such as meetings and interviews), the phenomenon of multiple speaker overlap is common.

[0003] Existing mainstream solutions mainly fall into two categories: 1. Waveform-level speech separation: This method uses models such as Deep Clustering, TasNet, or MossFormer to restore mixed waveforms to clean multi-channel waveforms. While theoretically feasible, this method has extremely high computational complexity (usually requiring massive GPU resources) and is difficult to deploy with low latency on mobile devices or real-time servers.

[0004] 2. Clustering / Embedding methods: such as the Diarization model, typically assume that speech segments are non-overlapping and perform clustering by extracting voiceprint features (such as x-vector, ECAPA-TDNN). However, when overlap occurs, the feature vectors of the mixed speech often exhibit a "winner-takes-all" behavior in the embedding space or are located in the middle of two cluster centers, causing traditional clustering algorithms to fail.

[0005] Currently, there is a lack of efficient algorithms that can handle overlapping speech without requiring huge computational power. Especially in scenarios where the identities of some speakers are known (such as registered users), existing technologies cannot effectively utilize prior information for feature decoupling. Summary of the Invention

[0006] The purpose of this invention is to provide a method, system, device and medium for overlapping speech separation based on orthogonal projection of feature space, in order to solve or improve at least one of the above-mentioned technical problems.

[0007] To achieve the above objectives, the present invention provides the following solution: An overlapping speech separation method based on orthogonal projection of feature space includes: Voice sequences from smart logistics are collected, and a pre-trained segmentation model is used to perform real-time flow detection on the input voice sequences, dividing them into active and inactive segments, and locating overlapping speech segments from multiple speakers; the pre-trained segmentation model adopts a hybrid architecture of 4-layer 1D convolution and 2-layer BiLSTM. Whenever the overlapping speech segments of multiple speakers occur, the identity of the main speaker is determined by using context tracking information, and the voiceprint feature vector of the main speaker is retrieved to construct a known speaker feature subspace; the context tracking information selects the speech of 1 second before the overlapping segment as the preceding context information; Extract the mixed voiceprint features of the overlapping speech segments of the multiple speakers, use the QR decomposition algorithm to calculate the orthogonal basis matrix of the known speaker feature subspace, and decompose the mixed voiceprint features into subspace parallel components and orthogonal perpendicular components through matrix projection operation to obtain the residual features after removing the main speaker information; Calculate the cosine similarity between the residual features and the voiceprint features of the candidate speaker, select the candidate with the highest similarity as the secondary speaker, and complete the logical separation of overlapping speech.

[0008] Optionally, the process of collecting speech sequences in smart logistics involves using a pre-trained segmentation model to perform real-time flow detection on the input speech, dividing it into active and inactive segments, and locating overlapping speech fragments from multiple speakers. Specifically, this includes: Construct training data; the training data includes Mel frequency cepstral coefficient features of speech and corresponding multidimensional Softmax probability vectors; wherein each dimension corresponds to the activity probability of a potential speaker; The initial segmentation model is loaded, and the initial segmentation model is pre-trained with the goal of minimizing the loss between the Mel frequency cepstral coefficient features of the speech and the multidimensional Softmax probability vector, so as to obtain a segmentation model that meets the set training requirements. Voice sequences from smart logistics are collected and input into the segmentation model to divide voice into active and inactive segments. When the probability of two or more dimensions in the model output vector exceeds a preset threshold, it is determined to be an overlapping voice segment with multiple speakers.

[0009] Optionally, whenever the overlapping speech segments of multiple speakers occur, the identity of the main speaker is determined using context tracking information, and the main speaker's voiceprint feature vector is retrieved to construct a known speaker feature subspace, specifically including: Based on streaming time series, the speaker state of each multi-speaker overlapping speech segment is used as the preceding context information in the first second of the speech segment. Based on the aforementioned contextual information, the speaker's activity duration percentage is statistically analyzed. The speaker with the highest activity duration percentage is identified as the main speaker. The speaker's corresponding voiceprint feature vector is retrieved from the designated feature library Gallery, and a feature subspace for known speakers is constructed.

[0010] Optionally, the step of extracting the mixed voiceprint features of the overlapping speech segments from multiple speakers, calculating the orthogonal basis matrix of the known speaker feature subspace using the QR decomposition algorithm, and decomposing the mixed voiceprint features into parallel components and orthogonal perpendicular components of the subspace through matrix projection operations to obtain residual features after removing the main speaker information, specifically includes: The overlapping speech segments from multiple speakers are input into the voiceprint feature extraction model to obtain a 512-dimensional voiceprint embedding vector of the mixed speech, and then L2 norm normalization is performed to obtain the mixed voiceprint features; the voiceprint feature extraction model adopts the ResNet34 architecture. The known speaker feature subspace in the hybrid voiceprint features is subjected to QR decomposition to obtain the orthogonal basis matrix; Based on the orthogonal basis matrix and matrix projection operation, the residual features after removing the speaker information are determined.

[0011] Optionally, the calculation of the cosine similarity between the residual features and the voiceprint features of the candidate speaker, and the selection of the candidate with the highest similarity as the secondary speaker, to complete the logical separation of overlapping speech, specifically includes: Exclude confirmed main speakers from the feature library Gallery, determine candidate speakers, and perform projection operation on the voiceprint features of the candidate speakers to remove components in the known subspace, thereby obtaining the orthogonal residual features of the candidate speakers. The cosine similarity is calculated between the residual features of the main speaker (after removing the main speaker information) and the orthogonal residual features of the candidate speakers. The candidate with the highest similarity score is selected as the secondary speaker in the current overlapping segment, and the final multi-speaker separation result is output in combination with the main speaker.

[0012] The present invention also provides an overlapping speech separation system based on orthogonal projection of feature space, comprising: The overlapping speech segmentation module is used to acquire speech sequences in smart logistics. It uses a pre-trained segmentation model to perform real-time flow detection on the input speech sequence, divides the speech into active and inactive segments, and locates overlapping speech segments from multiple speakers. The pre-trained segmentation model adopts a hybrid architecture of 4-layer 1D convolution and 2-layer BiLSTM. The main speaker identification module is used to determine the identity of the main speaker whenever the overlapping speech segments of the multiple speakers occur, and to retrieve the main speaker's voiceprint feature vector to construct a known speaker feature subspace; the context tracking information selects the speech of the previous second of the overlapping segment as the preceding context information. The hybrid feature extraction and orthogonal projection processing module is used to extract the hybrid voiceprint features of the overlapping speech segments of the multiple speakers, calculate the orthogonal basis matrix of the known speaker feature subspace using the QR decomposition algorithm, and decompose the hybrid voiceprint features into parallel components and orthogonal perpendicular components of the subspace through matrix projection operation to obtain the residual features after removing the main speaker information. The secondary speaker recognition module is used to calculate the cosine similarity between the residual features and the candidate speaker voiceprint features, select the candidate with the highest similarity as the secondary speaker, and complete the logical separation of overlapping speech.

[0013] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to cause the electronic device to perform the overlapping speech separation method based on orthogonal projection of feature space as described above.

[0014] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the overlapping speech separation method based on orthogonal projection of feature space as described above.

[0015] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This invention discloses a method, system, device, and medium for overlapping speech separation based on orthogonal projection of feature space. The method includes acquiring speech sequences from smart logistics, performing real-time flow detection on the input speech sequences using a pre-trained segmentation model to divide the speech into active and inactive segments, and locating overlapping speech segments from multiple speakers. The pre-trained segmentation model adopts a hybrid architecture of 4-layer 1D convolution and 2-layer BiLSTM. Whenever the overlapping speech segments from multiple speakers occur, the identity of the main speaker is determined using context tracking information, and the main speaker's voiceprint feature vector is retrieved to construct a known speaker feature subspace. The context tracking information selects the speech one second before the overlapping segment as the preceding context information. The mixed voiceprint features of the overlapping speech segments from multiple speakers are extracted, and the orthogonal basis matrix of the known speaker feature subspace is calculated using a QR decomposition algorithm. Through matrix projection operations, the mixed voiceprint features are decomposed into parallel components and orthogonal perpendicular components of the subspace to obtain residual features after removing the main speaker information. The cosine similarity between the residual features and the voiceprint features of candidate speakers is calculated, and the candidate with the highest similarity is selected as the secondary speaker, thus completing the logical separation of overlapping speech. This invention utilizes the principles of high-dimensional space geometry to eliminate known speaker features through mathematical projection operations without waveform reconstruction, thereby accurately identifying unknown speakers in the residual space and achieving a perfect balance between high precision and low computational power. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the overall architecture and feature projection in this embodiment; Figure 2 This is a flowchart of the operational logic in this embodiment. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The purpose of this invention is to provide a method, system, device and medium for overlapping speech separation based on orthogonal projection of feature space, in order to solve or improve at least one of the above-mentioned technical problems.

[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] As a first aspect, such as Figures 1-2 As shown, this invention provides an overlapping speech separation method based on orthogonal projection of feature space, comprising: S1. Collect speech sequences from smart logistics, and use a pre-trained segmentation model to perform real-time flow detection on the input speech sequence, divide the speech into active and inactive segments, and locate overlapping speech segments from multiple speakers; the pre-trained segmentation model adopts a hybrid architecture of a deep neural network with 4 layers of 1D convolution and 2 layers of BiLSTM, including convolutional recurrent layers and fully connected layers.

[0022] As one specific implementation method, this step specifically includes: Construct training data; the training data includes Mel-frequency cepstral coefficient features of speech and corresponding multidimensional Softmax probability vectors; wherein each dimension corresponds to the activity probability of a potential speaker; load an initial segmentation model, and pre-train the initial segmentation model with the goal of minimizing the loss between the Mel-frequency cepstral coefficient features of the speech and the multidimensional Softmax probability vectors, to obtain a segmentation model that meets the set training requirements; collect speech sequences in smart logistics, input them into the segmentation model to divide speech into active and inactive segments, and when the probability of two or more dimensions in the model output vector exceeds a preset threshold, it is determined to be a multi-speaker overlapping speech segment.

[0023] S2. Whenever the overlapping speech segments of multiple speakers occur, the identity of the main speaker is determined by using context tracking information, and the voiceprint feature vector of the main speaker is retrieved to construct a known speaker feature subspace; the context tracking information selects the speech of the previous second of the overlapping segment as the preceding context information.

[0024] As one specific implementation method, this step specifically includes: Based on the time series of streaming processing, the speaker state of the previous second of each of the multi-speaker overlapping speech segments is used as the preceding context information; based on the preceding context information, the activity duration ratio of each speaker is counted, and the speaker with the highest activity duration ratio is taken as the main speaker. The voiceprint feature vector corresponding to the main speaker is retrieved from the set feature library Gallery, and a known speaker feature subspace is constructed.

[0025] S3. Extract the mixed voiceprint features of the overlapping speech segments of the multiple speakers, calculate the orthogonal basis matrix of the known speaker feature subspace using the QR decomposition algorithm, and decompose the mixed voiceprint features into parallel components and orthogonal perpendicular components of the subspace through matrix projection operation to obtain the residual features after removing the main speaker information.

[0026] As one specific implementation method, this step specifically includes: The overlapping speech segments from multiple speakers are input into the voiceprint feature extraction model to obtain a 512-dimensional voiceprint embedding vector of the mixed speech, and L2 norm normalization is performed to obtain the mixed voiceprint features. The voiceprint feature extraction model adopts the ResNet34 architecture. QR decomposition is performed on the known speaker feature subspace in the mixed voiceprint features to obtain the orthogonal basis matrix. Based on the orthogonal basis matrix and matrix projection operation, the residual features after removing the main speaker information are determined.

[0027] S4. Calculate the cosine similarity between the residual features and the voiceprint features of the candidate speaker, select the candidate with the highest similarity as the secondary speaker, and complete the logical separation of overlapping speech.

[0028] As one specific implementation method, this step specifically includes: The confirmed main speaker is excluded from the feature library Gallery to determine candidate speakers. The voiceprint features of the candidate speakers are also subjected to projection operation to remove the components in the known subspace, resulting in orthogonal residual features of the candidate speakers. The cosine similarity of the residual features after removing the main speaker information and the orthogonal residual features of the candidate speakers is calculated. The candidate with the highest similarity score is selected as the secondary speaker in the current overlapping segment, and the final multi-speaker separation result is output in combination with the main speaker.

[0029] Based on the above technical solution, the following embodiment is provided.

[0030] S1. System initialization and parameter configuration.

[0031] S1-1, Building the basic model library: Deploy a speaker segmentation model that uses a hybrid architecture of 4 layers of 1D convolution and 2 layers of BiLSTM. The input is 80-dimensional Mel spectrum features, the frame length is 25ms, the frame shift is 10ms, and the output is a multi-dimensional Softmax probability distribution with an upper limit of 8 dimensions (supporting up to 8 concurrent speakers).

[0032] A voiceprint feature extraction model is deployed. This model adopts the ResNet34 architecture, takes 40-dimensional MFCC features as input, and outputs a 512-dimensional L2-normalized voiceprint embedding vector.

[0033] System parameters are set as follows: overlap threshold Th_overlap=0.65, feature similarity threshold Th_sim=0.7, micro-slice window length T_window=200ms, micro-slice step size T_step=50ms.

[0034] S1-2. Establish a speaker feature database: The system collects 5-10 seconds of clean speech samples from known speakers through a pre-registration process; generates a 512-dimensional feature vector for each registered speaker using a voiceprint feature extraction model; assigns a unique identifier (ID) to each feature vector and stores it in the feature library Gallery; for unregistered speakers, the system dynamically creates temporary IDs and updates the feature library during operation.

[0035] S1-3, Speech Preprocessing: The original WAV format input speech is resampled to a uniform 16kHz sampling rate; a sliding window mechanism is used with a window length of 2 seconds and a step size of 300ms to segment the speech stream; 80-dimensional Mel-spectral features are extracted from the speech within each window, with a frame length of 25ms and a frame shift of 10ms, which are used as input to the segmentation model.

[0036] S1-4, Overlapping Speech Detection: The preprocessed features are input into the segmentation model to obtain the speaker activity probability distribution for each frame; the output probabilities are post-processed by using median filtering (window size = 11 frames) to smooth the probability curve; the overlap determination rule is: when the probability of two or more speakers in any consecutive 10 frames is greater than Th_overlap = 0.65, it is determined to be an overlapping speech segment; the start and end timestamps [t_start, t_end] of the overlapping segments are recorded, and the number of main speakers N_main (usually 1) is marked.

[0037] S1-5. Contextual analysis to determine the main speaker: Extract the 1-second speech [t_start-1.0s, t_start] before the overlapping segment as the preceding context; use a segmentation model to analyze the active speakers in the preceding context and count the percentage of activity time for each speaker; select the speaker with the highest percentage of activity time as the main speaker S_main; retrieve the corresponding voiceprint feature vector E_main from the feature library Gallery.

[0038] S2, Hybrid feature extraction and orthogonal projection processing.

[0039] S2-1, Feature extraction of overlapping segments: Extract the original audio of the overlapping speech segments [t_start, t_end]; use 40-dimensional MFCC features as input and feed them into the voiceprint feature extraction model; obtain the 512-dimensional voiceprint embedding vector E_mix of the mixed speech and perform L2 norm normalization on it.

[0040] S2-2, Constructing a subspace of known speaker features: The main speaker features E_main are constructed into a feature matrix K∈R^(512×1) (single main speaker case); for multiple main speakers (N_main>1), the feature vectors of multiple main speakers are concatenated horizontally to form K∈R^(512×N_main); the feature matrix K is decomposed into QR: K = Q·R, where Q∈R^(512×N_main) is the orthonormal basis matrix and R is an upper triangular matrix; Verify the orthogonality of Q: Calculate Q^T·Q, ensuring the result is close to the identity matrix (error < 1e). -6 ).

[0041] S2-3 Perform orthogonal projection to obtain residual features: Calculate the projection matrix P = Q·Q^T; calculate the projection components of the mixed features on the known subspace: E_parallel = P·E_mix; calculate the orthogonal residual features: E_residual = E_mix - E_parallel; perform L2 normalization on E_residual to make its magnitude 1.

[0042] S3. Secondary speaker identification and result optimization.

[0043] S3-1, Orthogonalization of candidate features: Exclude confirmed main speakers from the feature library Gallery to obtain a set of candidate speakers C = {c_1,c_2, ..., c_m}; • For each candidate c_i's feature vector E_cand_i, calculate its projection onto the known subspace: E_cand_parallel = P·E_cand_i; • Calculate the orthogonal residual features of the candidates: E_cand_orth_i = E_cand_i - E_cand_parallel; • Perform L2 normalization on E_cand_orth_i.

[0044] S3-2, Similarity Calculation and Preliminary Judgment: Calculate the cosine similarity between the residual features and the orthogonal features of each candidate: Score_i = E_residual · E_cand_orth_i Set a similarity threshold Th_sim=0.7 to filter candidates whose Score_i>Th_sim; • If multiple candidates meet the condition, select the one with the largest Score_i as the initial second speaker S_secondary; • If no candidate meets the condition, create a new speaker ID and update the feature library Gallery.

[0045] S3-3, Microslice Voting Optimization: Divide the overlapping segment [t_start, t_end] into N micro-windows, each with a length T_window=200ms and a step size T_step=50ms; Execute processing steps S3-1 to S4-2 independently for each micro-window to obtain N secondary speaker determination results {r_1, r_2, ..., r_N}; Count the frequency of each speaker ID and select the ID with the highest frequency as the final secondary speaker S_secondary_final; If the highest frequency does not exceed 60% of the total number of windows, it is determined as an "unknown speaker" and the voiceprint registration process is triggered.

[0046] S4. System Output and Post-processing.

[0047] S4-1. Generate speaker separation results: The overlapping segments [t_start, t_end] are marked as dual-speaker states: [S_main, S_secondary_final]; the non-overlapping segments are directly marked as single-speaker states according to the segmentation model output; and a time-aligned speaker label sequence is generated in the format [(t_0, t_1, speaker_id), (t_1, t_2, speaker_id), ...].

[0048] S4-2. Post-processing of results: Perform speaker coherence checks: if the same speaker is segmented into multiple short segments (<300ms), merge them; perform minimum interval filtering: if the interval between two segments of the same speaker is <200ms, merge them into a single continuous segment; generate the final speaker separation result, output in JSON or RTTM standard format.

[0049] S4-3, Dynamic updates to the feature library: For newly identified speakers, the system automatically collects their non-overlapping speech segments; when the accumulated clean speech reaches 3 seconds, its voiceprint feature vector is recalculated; and the feature library Gallery is updated to provide more accurate reference features for subsequent overlap processing.

[0050] Based on the above technical solution, the following embodiment two is provided.

[0051] In the context of actual smart logistics operations, efficient voice interaction recognition is key to achieving automated scheduling. The following is a description of the specific implementation process within a busy logistics sorting center: Scenario Setting: In the dispatch room of a large logistics sorting center, the system needs to process in real time a mixture of voice commands from headsets, walkie-talkies, and ambient microphones worn on-site. Operator "Old Zhang" (already registered) is directing the automated sorting line, while novice "Xiao Wang" (already registered) is simultaneously inquiring about the parcel anomaly handling process via walkie-talkie; their voices overlap on the timeline.

[0052] Step 1: Real-time speech stream segmentation and overlap detection (corresponding to S1) After the system starts up, initialization and parameter configuration are completed first (S1-1). All known operators (including Lao Zhang, Xiao Wang, etc.) have gone through the pre-registration process, and their 5-10 second clean speech samples are converted into 512-dimensional voiceprint feature vectors and stored in the feature library Gallery (S1-2).

[0053] Speech stream acquisition and preprocessing (S1-3): The microphone array in the dispatch room acquires a 16kHz speech stream in real time. The system uses a 2-second sliding window (300ms step) to segment the speech stream. The audio within each window is converted into 80-dimensional Mel-spectral features, with a frame length of 25ms and a frame shift of 10ms, to prepare input data for subsequent deep learning models.

[0054] Real-time determination of overlapping speech (S1-4): The preprocessed features are fed into a pre-trained segmentation model. This model consists of 4 layers of 1D convolutions and 2 layers of BiLSTM, and outputs a multi-dimensional Softmax probability distribution (with the upper limit of dimension set to 8).

[0055] At a certain moment, the system detected that for more than 10 consecutive frames, the probability vector output by the model had two dimensions (corresponding to "Old Zhang" and "Little Wang") whose probability values ​​simultaneously exceeded the preset threshold of 0.65.

[0056] The system immediately determined that a multi-speaker overlapping speech segment had occurred and accurately recorded the start and end timestamps of the segment [t_start, t_end].

[0057] Step 2: Context tracking and speaker confirmation (corresponding to S2) Extracting contextual information (S2-1): The system automatically extracts the audio from the first second of the overlapping segment [t_start-1.0s, t_start] as the preceding contextual information. The segmentation model's analysis of this audio segment shows that within this 1 second, "Old Zhang's" speaking activity accounts for as much as 85% of the total time, far exceeding that of other speakers.

[0058] Identifying the main speaker and constructing a subspace (S2-2): Based on contextual information, the system determines that "Lao Zhang" is the main speaker. Subsequently, the system retrieves the voiceprint feature vector E_main corresponding to "Lao Zhang" from the feature library Gallery. Since only one main speaker has been identified so far, the system constructs E_main into a single-column feature matrix K, and calculates the orthonormal basis matrix Q of this feature subspace using the QR decomposition algorithm.

[0059] Step 3: Hybrid feature extraction and speaker information removal (corresponding to S3) Extracting hybrid voiceprint features (S3-1): The system extracts the hybrid audio of overlapping segments [t_start, t_end], inputs it into the voiceprint feature extraction model based on the ResNet34 architecture, and obtains a 512-dimensional hybrid voiceprint embedding vector E_mix, which is then normalized to the L2 norm.

[0060] Orthogonal projection to remove speaker information (S3-2): This is the core technology of the entire process. The system uses the orthogonal basis matrix Q obtained in the second step to calculate the projection matrix P = Q·Q^T. Through matrix projection operations, the mixed voiceprint feature E_mix is ​​decomposed into two parts: Parallel component E_parallel: represents the contribution of "Lao Zhang's" voiceprint features to the mixed speech.

[0061] The orthogonal vertical component E_residual, also known as the residual feature, removes the voiceprint information of "Lao Zhang" and theoretically only contains the acoustic features of "Xiao Wang" or other unknown speakers. The system performs L2 normalization on E_residual again for subsequent secondary speaker identification.

[0062] Step 4: Second speaker identification and logical separation (corresponding to S4) Candidate Feature Preprocessing (S4-1): The system excludes "Lao Zhang" from the feature library Gallery and lists "Xiao Wang" and several other operators who may be present as candidates. For the voiceprint features of each candidate, the system also performs the same orthogonal projection operation, removing components parallel to the "Lao Zhang" subspace, to obtain the orthogonal residual feature E_cand_orth_i of each candidate.

[0063] Cosine similarity calculation and microslice voting (S4-2, S4-3): The system calculates the cosine similarity between the residual feature E_residual and the orthogonal residual features of all candidates. The results show that "Xiao Wang" has the highest feature score.

[0064] To improve robustness, the system performed micro-slice voting optimization on the 200ms overlapping segment. This overlapping segment was divided into multiple micro-windows with 50ms increments, and the above recognition process was repeated for each micro-window. In the judgment results of all micro-windows, the frequency of "Xiao Wang" far exceeded the 60% threshold.

[0065] Final output and post-processing (S4-4, S4-5): Based on the combined similarity calculation and micro-slice voting results, the system finally confirms that the secondary speaker of the overlapping speech segment is "Xiao Wang". The system marks this time stamp as the dual-speaker state of [Lao Zhang, Xiao Wang] and updates the final speaker label sequence.

[0066] Through this series of steps, the system successfully separated the previously mixed and difficult-to-identify voice commands of "Old Zhang" and "Little Wang" into a logical sequence, providing an accurate and clear voice data foundation for subsequent command understanding and automated scheduling, and ensuring the high reliability of the smart logistics system in complex acoustic environments.

[0067] As a second aspect, the present invention also provides an overlapping speech separation system based on orthogonal projection of feature space, comprising: The overlapping speech segmentation module is used to acquire speech sequences in smart logistics. It uses a pre-trained segmentation model to perform real-time flow detection on the input speech sequence, divides the speech into active and inactive segments, and locates overlapping speech segments from multiple speakers. The pre-trained segmentation model adopts a hybrid architecture of 4-layer 1D convolution and 2-layer BiLSTM. The main speaker identification module is used to determine the identity of the main speaker whenever the overlapping speech segments of the multiple speakers occur, and to retrieve the main speaker's voiceprint feature vector to construct a known speaker feature subspace; the context tracking information selects the speech of the previous second of the overlapping segment as the preceding context information. The hybrid feature extraction and orthogonal projection processing module is used to extract the hybrid voiceprint features of the overlapping speech segments of the multiple speakers, calculate the orthogonal basis matrix of the known speaker feature subspace using the QR decomposition algorithm, and decompose the hybrid voiceprint features into parallel components and orthogonal perpendicular components of the subspace through matrix projection operation to obtain the residual features after removing the main speaker information. The secondary speaker recognition module is used to calculate the cosine similarity between the residual features and the candidate speaker voiceprint features, select the candidate with the highest similarity as the secondary speaker, and complete the logical separation of overlapping speech.

[0068] As a third aspect, the present invention also provides an electronic device, including a memory and a processor, the memory for storing a computer program, the processor for running the computer program to cause the electronic device to perform the overlapping speech separation method based on orthogonal projection of feature space as described above.

[0069] As a fourth aspect, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the overlapping speech separation method based on orthogonal projection of feature space as described above.

[0070] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0071] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for separating overlapping speech based on orthogonal projection of feature space, characterized in that, include: Voice sequences from smart logistics are collected, and a pre-trained segmentation model is used to perform real-time flow detection on the input voice sequences, dividing them into active and inactive segments, and locating overlapping speech segments from multiple speakers; the pre-trained segmentation model adopts a hybrid architecture of 4-layer 1D convolution and 2-layer BiLSTM. Whenever the overlapping speech segments of multiple speakers occur, the identity of the main speaker is determined by using context tracking information, and the voiceprint feature vector of the main speaker is retrieved to construct a known speaker feature subspace; the context tracking information selects the speech of 1 second before the overlapping segment as the preceding context information; Extract the mixed voiceprint features of the overlapping speech segments of the multiple speakers, use the QR decomposition algorithm to calculate the orthogonal basis matrix of the known speaker feature subspace, and decompose the mixed voiceprint features into subspace parallel components and orthogonal perpendicular components through matrix projection operation to obtain the residual features after removing the main speaker information; Calculate the cosine similarity between the residual features and the voiceprint features of the candidate speaker, select the candidate with the highest similarity as the secondary speaker, and complete the logical separation of overlapping speech.

2. The overlapping speech separation method based on orthogonal projection of feature space according to claim 1, characterized in that, The process of collecting speech sequences in smart logistics involves using a pre-trained segmentation model to perform real-time flow detection on the input speech, dividing it into active and inactive segments, and locating overlapping speech fragments from multiple speakers. Specifically, this includes: Construct training data; the training data includes Mel frequency cepstral coefficient features of speech and corresponding multidimensional Softmax probability vectors; wherein each dimension corresponds to the activity probability of a potential speaker; The initial segmentation model is loaded, and the initial segmentation model is pre-trained with the goal of minimizing the loss between the Mel frequency cepstral coefficient features of the speech and the multidimensional Softmax probability vector, so as to obtain a segmentation model that meets the set training requirements. Voice sequences from smart logistics are collected and input into the segmentation model to divide voice into active and inactive segments. When the probability of two or more dimensions in the model output vector exceeds a preset threshold, it is determined to be an overlapping voice segment with multiple speakers.

3. The overlapping speech separation method based on orthogonal projection of feature space according to claim 1, characterized in that, Whenever overlapping speech segments from multiple speakers occur, the identity of the main speaker is determined using context tracking information, and the main speaker's voiceprint feature vector is retrieved to construct a known speaker feature subspace, specifically including: Based on streaming time series, the speaker state of each multi-speaker overlapping speech segment is used as the preceding context information in the first second of the speech segment. Based on the aforementioned contextual information, the speaker's activity duration percentage is statistically analyzed. The speaker with the highest activity duration percentage is identified as the main speaker. The speaker's corresponding voiceprint feature vector is retrieved from the designated feature library Gallery, and a feature subspace for known speakers is constructed.

4. The overlapping speech separation method based on orthogonal projection of feature space according to claim 1, characterized in that, The process involves extracting the mixed voiceprint features from the overlapping speech segments of multiple speakers, calculating the orthogonal basis matrix of the known speaker feature subspace using the QR decomposition algorithm, and then decomposing the mixed voiceprint features into parallel and orthogonal perpendicular components of the subspace through matrix projection operations to obtain residual features after removing the main speaker information. Specifically, this includes: The overlapping speech segments from multiple speakers are input into the voiceprint feature extraction model to obtain a 512-dimensional voiceprint embedding vector of the mixed speech, and then L2 norm normalization is performed to obtain the mixed voiceprint features; the voiceprint feature extraction model adopts the ResNet34 architecture. The known speaker feature subspace in the hybrid voiceprint features is subjected to QR decomposition to obtain the orthogonal basis matrix; Based on the orthogonal basis matrix and matrix projection operation, the residual features after removing the speaker information are determined.

5. The overlapping speech separation method based on orthogonal projection of feature space according to claim 1, characterized in that, The calculation of the cosine similarity between the residual features and the voiceprint features of the candidate speaker, and the selection of the candidate with the highest similarity as the secondary speaker, completes the logical separation of overlapping speech, specifically including: Exclude confirmed main speakers from the feature library Gallery, determine candidate speakers, and perform projection operation on the voiceprint features of the candidate speakers to remove components in the known subspace, thereby obtaining the orthogonal residual features of the candidate speakers. The cosine similarity is calculated between the residual features of the main speaker (after removing the main speaker information) and the orthogonal residual features of the candidate speakers. The candidate with the highest similarity score is selected as the secondary speaker in the current overlapping segment, and the final multi-speaker separation result is output in combination with the main speaker.

6. An overlapping speech separation system based on orthogonal projection of feature space, characterized in that, include: The overlapping speech segmentation module is used to acquire speech sequences in smart logistics. It uses a pre-trained segmentation model to perform real-time flow detection on the input speech sequence, divides the speech into active and inactive segments, and locates overlapping speech segments from multiple speakers. The pre-trained segmentation model adopts a hybrid architecture of 4-layer 1D convolution and 2-layer BiLSTM. The main speaker identification module is used to determine the identity of the main speaker whenever the overlapping speech segments of the multiple speakers occur, and to retrieve the main speaker's voiceprint feature vector to construct a known speaker feature subspace; the context tracking information selects the speech of the previous second of the overlapping segment as the preceding context information. The hybrid feature extraction and orthogonal projection processing module is used to extract the hybrid voiceprint features of the overlapping speech segments of the multiple speakers, calculate the orthogonal basis matrix of the known speaker feature subspace using the QR decomposition algorithm, and decompose the hybrid voiceprint features into parallel components and orthogonal perpendicular components of the subspace through matrix projection operation to obtain the residual features after removing the main speaker information. The secondary speaker recognition module is used to calculate the cosine similarity between the residual features and the candidate speaker voiceprint features, select the candidate with the highest similarity as the secondary speaker, and complete the logical separation of overlapping speech.

7. An electronic device, characterized in that, The device includes a memory and a processor, the memory being used to store a computer program, and the processor running the computer program to cause the electronic device to perform the overlapping speech separation method based on orthogonal projection of feature space according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the overlapping speech separation method based on orthogonal projection of feature space as described in any one of claims 1-5.

Images