Speech extraction method and device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of artificial intelligence, and provides a voice extraction method and device, electronic equipment and storage medium, the method comprising: obtaining a mixed voice segment at a current time and identity representation information of a target speaker; obtaining a historical voice segment of the target speaker at at least one historical time; fusing mixed voice features extracted from the mixed voice segment, historical context features extracted from the historical voice segment, and the identity representation information to obtain target fusion features; and extracting a target voice segment of the target speaker at the current time from the mixed voice segment according to the target fusion features. The present application introduces the historical voice segment containing rich phonemes, prosody and other sound states as a dynamic context reference, breaking the limitations of traditional stateless models, not only giving the model a memory ability for voice content, significantly improving the continuity of the output, thereby effectively avoiding the generation of stuttering, jumps and artifacts at the block boundary.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a speech extraction method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of smart home devices, remote conferencing, and other technologies, extracting the voice of a specific person from a complex mixed speech environment has become crucial, leading to the emergence of Target Speaker Extraction (TSE) technology.

[0003] Current target speaker extraction methods typically segment long mixed speech into fixed-length segments, convert the current mixed speech segment into high-dimensional features using an encoder, and then fuse these features using pre-extracted static embedding vectors of the target speaker into a separation network to directly estimate the target speech within that segment. However, when processing continuous long speech, this method is prone to auditory artifacts such as phase discontinuities and energy abrupt changes at the boundaries of speech blocks, resulting in stuttering or distortion in the final output speech. Furthermore, when the target speaker is briefly silent or completely masked by strong noise, the model is prone to losing the target in subsequent time steps and is difficult to reconnect, leading to the easy erroneous extraction of the interfering speaker's voice in low signal-to-noise ratio environments, resulting in poor overall robustness. Summary of the Invention

[0004] This invention provides a speech extraction method, apparatus, electronic device, and storage medium to address the deficiencies in the prior art.

[0005] This invention provides a speech extraction method, comprising the following steps: Obtain the mixed speech segment at the current moment and the identity representation information of the target speaker; Obtain at least one historical speech segment of the target speaker at a historical moment; The mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information are fused together to obtain the target fused features; Based on the target fusion features, the target speaker's target speech segment at the current moment is extracted from the mixed speech segment; the historical speech segment and the target speech segment at the current moment have a contextual relationship.

[0006] According to a speech extraction method provided by the present invention, the steps for obtaining the mixed speech segment sample and the historical clean speech segment sample include: Acquire a continuous sample mixed speech stream and a continuous clean speech stream of the sample speaker corresponding to the sample mixed speech stream; A sliding window with the same time length and the same overlap ratio is used to synchronously segment the sample mixed speech stream and the continuous clean speech stream; The segmented mixed speech corresponding to the current moment is used as the mixed speech segment sample, and the segmented clean speech corresponding to the at least one historical moment is used as the historical clean speech segment sample.

[0007] According to a speech extraction method provided by the present invention, the step of acquiring historical speech segments of the target speaker at least at a historical moment includes: If the current time is the starting time, the initial voice segment will be used as the historical voice segment; If the current time is not the starting time, the target speech segment that has been extracted before the current time and corresponds to the at least one historical time is obtained as the historical speech segment.

[0008] According to a speech extraction method provided by the present invention, the step of acquiring a target speech segment that has been extracted before the current time and corresponds to the at least one historical time, as the historical speech segment, includes: When processing the continuously input mixed speech stream up to the current moment, the target speech segment corresponding to the at least one historical moment is retrieved from the context buffer space as the historical speech segment; the context buffer space stores the target speech segments of the target speaker extracted sequentially before the current moment during the processing of the mixed speech stream, in chronological order.

[0009] According to a speech extraction method provided by the present invention, the step of extracting the historical context features includes: Perform multi-level feature extraction operations along the time dimension of the historical speech segment to capture the speech content and changes in the speech state contained in the historical speech segment; The change information is encoded into a context feature vector in the target dimension to obtain the historical context features.

[0010] According to a speech extraction method provided by the present invention, the step of fusing the mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information to obtain target fused features includes: The identity representation information is mapped to a preset target dimension to obtain identity projection features; Dynamic control weights are generated based on the identity projection features and the historical context features; Based on the dynamic control weights, each channel of the mixed speech features is filtered and modulated to generate the target fusion feature, which is jointly determined by the mixed speech features, the historical context features, and the identity representation information.

[0011] According to a speech extraction method provided by the present invention, the method involves fusing the mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information to obtain target fusion features; and extracting the target speech segment of the target speaker at the current moment from the mixed speech segment based on the target fusion features, including: The mixed speech segment and the historical speech segment are input into the encoding network of the speech extraction model, and the encoding network extracts the mixed speech features from the mixed speech segment and the historical context features from the historical speech segment; The mixed speech features, the historical context features, and the identity representation information are input into the fusion and separation network of the speech extraction model. The fusion and separation network then fuses the mixed speech features, the historical context features, and the identity representation information to obtain the target fusion features. The target fusion features are input into the decoding network of the speech extraction model, and the decoding network reconstructs the signal based on the target fusion features to output the target speech segment.

[0012] According to a speech extraction method provided by the present invention, the training steps of the speech extraction model include: Obtain a mixed speech segment sample at the current moment, a historical clean speech segment sample of at least one historical moment of the sample speaker, and an identity representation information sample of the sample speaker; The mixed speech segment sample, the historical clean speech segment sample, and the identity representation information sample are input into the initial speech extraction model to obtain the predicted target speech segment corresponding to the current moment. The loss value is calculated based on the difference between the predicted target speech segment and the sample label, and the parameters of the initial speech extraction model are updated based on the loss value to obtain the speech extraction model. The sample label is the clean speech segment of the sample speaker corresponding to the current moment.

[0013] The present invention also provides a speech extraction device, comprising the following modules: The first acquisition module is used to acquire the mixed speech segment at the current moment and the identity representation information of the target speaker; The second acquisition module is used to acquire at least one historical speech segment of the target speaker at a historical moment; The fusion processing module is used to fuse the mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information to obtain the target fusion feature; The speech extraction module is used to extract the target speaker's target speech segment at the current moment from the mixed speech segment based on the target fusion features; the historical speech segment and the target speech segment at the current moment have a contextual relationship.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech extraction methods.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech extraction method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described speech extraction methods.

[0017] The speech extraction method, apparatus, electronic device, and storage medium provided by this invention, by introducing historical speech segments containing rich phonemes, prosody, and other vocal states as dynamic context references, break through the limitations of traditional stateless models. This not only endows the model with the ability to remember speech content and significantly improves the continuity of output, but also effectively avoids stuttering, jumps, and artifacts at the boundaries of processing blocks. At the same time, this embodiment can also accurately remove interference with similar timbres by utilizing the consistency of the preceding and following content, avoiding the defect of target loss under extreme conditions such as low signal-to-noise ratio or brief silence, and greatly improving the robustness and accuracy of the extraction system. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the speech extraction method provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the processing flow of the speech extraction model provided by the present invention.

[0021] Figure 3This is a schematic diagram of the speech extraction device provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0024] Current target speaker extraction techniques, especially deep learning-based methods, typically employ a stateless or weakly stateful processing mode. Specifically, when processing long speech segments, the system usually uses a block-based approach, dividing the long speech into fixed-length segments and then performing the extraction process independently for each segment. When processing the current speech block, the encoder receives the time-domain waveform or short-time Fourier transform spectrum of the mixed speech and converts it into a high-dimensional feature representation. Simultaneously, the conditional module fuses the pre-extracted static embedding vector of the target speaker into the high-dimensional features, enabling subsequent networks to recognize the target to be extracted. Subsequently, the subject separation network processes the fused features to estimate the features or mask of the target speech. Finally, the decoder converts the estimated features or mask back into a time-domain waveform.

[0025] While this processing model has achieved some success, it suffers from inherent drawbacks in practical applications due to the lack of memory between blocks: First, it lacks contextual information. Static speaker embeddings only provide timbre information ("who is this person"), completely ignoring the continuity of speech content ("what this person said a second ago"). When processing speech blocks at the current moment, the model completely lacks dynamic, content-related context. Second, it handles discontinuities. Due to block-based processing and the lack of content memory, auditory artifacts such as phase discontinuities and energy abrupt changes are easily generated at the boundaries of speech blocks, leading to stuttering or distortion in the final output speech. Third, it has poor robustness. When the target speaker is briefly silent or completely masked by strong noise or other people's speech, the memoryless model struggles to continue in the next moment. It must rely entirely on the mixed speech and static embedding of the current block to re-identify the target, which is prone to failure or incorrect extraction of interfering speakers in low signal-to-noise ratio environments.

[0026] To address this issue, the present invention provides a speech extraction method. This method aims to acquire the mixed speech segment at the current moment and the target speaker's identity information, while simultaneously introducing and acquiring at least one historical speech segment of the target speaker from a previous time. Then, it fuses the mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity information to obtain target fusion features. Finally, based on the target fusion features, the target speaker's speech segment at the current moment is extracted from the mixed speech segment. This approach utilizes the target speaker's own historical speech as a dynamic contextual reference, overcoming the deficiency of content memory in related technologies. It not only leverages the continuity of speech content to achieve smooth transitions at block boundaries to eliminate artifacts and stuttering, but also maintains stable target tracking under extreme conditions such as strong noise or brief silence, significantly improving the system's robustness and extraction accuracy.

[0027] In this invention, all actions involving the acquisition of signal information or data are carried out in accordance with the relevant data protection laws and policies of the country where the device is located, and with the authorization granted by the owner of the device.

[0028] Figure 1 This is a flowchart illustrating the speech extraction method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0029] Step 110: Obtain the mixed speech segment at the current moment and the identity representation information of the target speaker.

[0030] Here, a mixed speech segment can be understood as a slice of speech signal collected in a complex acoustic environment that contains at least two sound sources. The mixed speech segment can take the form of a time-domain waveform or a short-time Fourier transform spectrum. For example, in a complex mixed speech environment, a mixed audio slice received by smart home or remote conferencing devices, containing the target speaker's voice, interfering speaker voices, and environmental noise, is a mixed speech segment.

[0031] Identity representation information is used to identify the target speaker to be extracted. It primarily provides the system with the speaker's vocal characteristics, indicating "who this person is." As an optional implementation, this identity representation information can be obtained by extracting a static embedding vector from a clean reference speech segment of the target speaker using a pre-trained speaker recognition model.

[0032] Step 120: Obtain the historical speech segment of the target speaker at at least one historical moment.

[0033] Specifically, considering that speech signals are highly continuous, the preceding phoneme, word, and prosodic pattern has a strong predictive power for the phonation state at the next moment. If independent speech block extraction is performed based solely on mixed speech segments and static identity representation information, then when processing the speech segment at the current moment, there is no knowledge of what the target speaker said at previous moments, resulting in a lack of dynamic contextual information. This processing method, lacking content memory, is prone to producing auditory artifacts such as phase discontinuities and energy abrupt changes at the boundaries of speech blocks when processing continuous long speech, causing the final output speech to be stuttering or distorted.

[0034] Therefore, to compensate for the lack of dynamic context information, this embodiment, in addition to acquiring identity representation information, also introduces historical speech segments of the target speaker at historical moments. These historical speech segments contain the speech content emitted by the target speaker at a historical moment, such as crucial predictive cues like phonemes, intonation, and energy just spoken by the target speaker. A historical moment refers to a node on the timeline earlier than the current moment. Here, at least one historical moment can refer to the previous processing moment immediately adjacent to the current moment, a historical node separated by a set time span before the current moment, or a set of multiple consecutive historical processing moments before the current moment. Regardless of whether a single or multiple historical moments are used, their corresponding historical speech segments can provide coherent dynamic content support for the speech extraction at the current moment.

[0035] In this case, the target speech segment that has been predicted and output at a historical moment can be used as the historical speech segment at the current moment, or the real and pure speech of the target speaker that is obtained in advance and corresponds to the historical moment can be used as the historical speech segment.

[0036] Step 130: The mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information are fused to obtain the target fused features.

[0037] Specifically, mixed speech features are high-dimensional feature representations that reflect the noise at the current moment; historical context features are feature representations that reflect dynamic contextual information such as the speech content, phonemes, prosody, and energy that the target speaker has just spoken.

[0038] As an optional embodiment, the mixed speech segments can be processed by a master feature coding network to extract mixed speech features; at the same time, historical speech segments can be processed by a lightweight coding network to extract historical context features; and identity representation information can be mapped to an appropriate feature dimension to obtain mapped identity dimension features.

[0039] Furthermore, after obtaining the mixed speech features, historical context features, and identity representation information, they are fused together. Fusion processing refers to the comprehensive interaction of feature information from different dimensions, so that the subsequent processing network not only knows the target timbre to be extracted, but also knows the content of the target speaker's previous speech.

[0040] As an optional implementation, the fusion process can integrate information through direct concatenation. For example, the mixed speech features, historical context features, and mapped identity representation information can be directly concatenated along the feature dimension to obtain the target fused features. This process can be represented as follows: ; in, This indicates a feature concatenation operation. Indicates target fusion features, Indicates mixed speech features, Indicates historical context features, It represents identity information.

[0041] As an alternative implementation, to achieve deeper feature interaction, the fusion process can also employ a feature linear modulation mechanism. For example, dynamic gating weights can be generated using historical context features and mapped identity representation information, and then channel-level filtering and modulation can be performed on the mixed speech features to obtain the target fused features.

[0042] Step 140: Based on the target fusion features, extract the target speaker's target speech segment at the current moment from the mixed speech segment; there is a contextual relationship between the historical speech segment and the target speech segment at the current moment.

[0043] Specifically, the target speech segment refers to the clean speech prediction result that is finally separated from the mixed signal and belongs to the target speaker.

[0044] Because the target fusion features deeply integrate the current moment's mixed acoustic observations, the target speaker's static timbre guidance, and the crucial dynamic historical content memory, they can provide a multi-dimensional strong constraint for the feature separation network to accurately estimate the specific vocalization state of the target speech at the current moment.

[0045] Specifically, human speech signals exhibit a high degree of continuity in the temporal dimension. The preceding phoneme, word, and prosodic pattern have a strong predictive effect on the acoustic state at the next moment. Simply relying on static identity representation information can only solve the problem of matching the timbre of the extracted object, while the historical context features introduced into the target fusion feature endow it with the dynamic perception ability of the target object's previous vocal content. On the one hand, under extreme conditions of low signal-to-noise ratio, such as when the target speaker is briefly silent or completely submerged by strong ambient noise, the system can utilize the historical contextual memory in the target fusion features and, based on the coherence and inertia of the speech content, accurately predict and smoothly continue the speaker's likely ongoing vocalizations. This effectively overcomes the shortcomings of traditional memoryless models that are prone to losing the target. On the other hand, when there are interference sources in the mixed speech segment that are very similar to the target speaker's timbre, it is difficult to effectively distinguish them using only identity representation information. However, since the speech content expressed by the interference source and the target speaker are fundamentally different, the system can utilize the coherence difference between the historical content and the current observation features in the target fusion features in the contextual dimension to more accurately extract interference components with similar timbre but mismatched content. Therefore, based on the above-mentioned multi-dimensional feature synergy and content consistency constraints, the extraction system can stably and accurately extract the target speaker's current speech segment from complex mixed speech segments.

[0046] As an optional embodiment, the features or mask of the target speech can be estimated by the subject separation network based on the target fusion features, and the estimated features or mask can be sent to the decoder. The decoding network converts it back to the time domain waveform, thereby reconstructing and extracting the target speech segment at the current moment.

[0047] Furthermore, considering that existing memoryless models are prone to losing the target when the target speaker is briefly silent or completely masked by strong noise, relying solely on mixed speech segments and static identity representation information, this leads to delays and errors when re-capturing the target. This embodiment preferably employs the aforementioned autoregressive mechanism based on historical speech segments, extracting target fusion features that include historical contextual characteristics. Even if the target signal is weak in the current mixed speech segment, it can infer that the target speaker is likely continuing their previous vocalization state, thus maintaining more stable target tracking.

[0048] The speech extraction method provided in this embodiment, by introducing historical speech segments containing rich phonemes, prosody, and other vocal states as dynamic context references, breaks through the limitations of traditional stateless models. This not only endows the model with the ability to remember speech content and significantly improves the continuity of output, but also effectively avoids stuttering, abrupt changes, and artifacts at the boundaries of processing blocks. At the same time, this embodiment can also accurately remove interfering elements with similar timbres by utilizing the consistency of preceding and following content, avoiding the defect of target loss under extreme conditions such as low signal-to-noise ratio or brief silence, and greatly improving the robustness and accuracy of the extraction system.

[0049] Considering that in the inference phase of actual deployment and use, it is impossible to obtain the target speaker's true, clean speech tag data, in order to build dynamic contextual references during streaming or long speech processing, the model needs to rely on its own output features to form a memory loop, thereby achieving continuous and smooth speech extraction. Based on this, this embodiment obtains at least one historical speech segment of the target speaker at a historical moment, including: If the current time is the starting time, the initial speech segment will be used as the historical speech segment; If the current time is not the starting time, the target speech segment that has been extracted before the current time and corresponds to at least one historical time is obtained as the historical speech segment.

[0050] When the system first starts and processes the first speech block, there are no previous extraction results to recall, creating a logical gap in historical memory acquisition. By forcibly assigning an initial state, the data gap in the cold start phase can be effectively filled, allowing the model to smoothly begin the extraction and calculation of the first frame.

[0051] The start time refers to the time point at which the first processing segment is extracted when processing a long speech or streaming speech. The initial speech segment refers to the pre-set placeholder data representation used to replace the input of historical speech segments when historical information is lacking.

[0052] As an optional implementation, if the current time is the starting time, since there is no historical record of that time, a zero vector or a special start marker pre-learned by the network can be generated as the initial speech segment. For example, the zero vector or special start marker can be used as the historical speech segment of the starting time and input into the model network to ensure the integrity of all input branches in the model structure at the first time.

[0053] After extracting the first speech segment and establishing the initial state by using the initial speech segment, considering that the system has accumulated the processing results of previous moments as the speech stream continues to be input, in order to achieve dynamic and realistic and coherent transmission of context content, in the case that the current moment is not the starting moment, the target speech segment that has been extracted before the current moment and corresponds to at least one historical moment is obtained as the historical speech segment.

[0054] Furthermore, human speech production possesses strong continuity, and the sound content predicted by the model in previous moments contains crucial predictive cues such as the target speaker's phonemes, words, and prosodic patterns. By feeding the system's output back to the input through an autoregressive mechanism, the stateless limitation of block processing is overcome, significantly improving the continuity and naturalness of the output speech, as well as enhancing the system's robustness under extreme acoustic conditions.

[0055] Here, "non-starting time" refers to any processing time node immediately following the first processing segment. The target speech segment extracted before the current time and corresponding to at least one historical time refers to the clean speech output data of the target speaker that the model has calculated and reconstructed in past operation cycles based on the mixed speech segments and corresponding historical references.

[0056] As an optional implementation, during the autoregressive loop inference process, for the second processing time, the model uses the predicted output of the previous step (i.e., the first time) as the current input history reference. For any non-starting time, iterative processing is adopted, that is, the model continuously consumes new mixed speech segments and uses its own previous time prediction output as a reference for historical speech segments, repeating the process to achieve streaming processing.

[0057] Considering that in actual streaming speech processing or long speech iterative prediction processes, the model needs to continuously receive mixed audio and output clean speech in real time. To support the autoregressive loop calls at the physical device and system architecture levels, an effective data flow and temporary storage mechanism is needed to ensure that the historical data of its own prediction output can be securely saved and quickly and accurately extracted when needed. Based on this, this embodiment obtains target speech segments that have been extracted before the current time and correspond to at least one historical time, as historical speech segments, including: When processing the continuously input mixed speech stream up to the current moment, the target speech segment corresponding to at least one historical moment is retrieved from the context buffer space as a historical speech segment; the context buffer space stores the target speech segments of the target speaker extracted sequentially before the current moment during the processing of the mixed speech stream, in chronological order.

[0058] During uninterrupted streaming processing, if the speech data generated by the previous computing node is not physically stored in a dedicated storage area, it will be lost with memory refreshes or the input of a new round of features, causing the next computing node to be unable to obtain the necessary dynamic context reference. Based on this, this embodiment constructs a dedicated cache space and performs queued storage and retrieval according to the timeline, achieving the effect of providing a stable, continuous, and orderly historical data supply source for the autoregressive loop.

[0059] In this context, continuously input mixed speech streams refer to long segments of real-time audio data, including the target speaker's voice, background noise, and other interfering sounds, that the system continuously receives in actual deployment scenarios. The context buffer space refers to a dedicated historical context buffer or memory unit within the target speaker extraction system used to temporarily store predicted output data. This context buffer space, as an objectively existing system-level storage state, stores, in chronological order, the target speech segments of the target speaker extracted sequentially up to the current moment during the processing of the mixed speech stream; that is, a series of clean speech results generated by the system through predictions at various past time points. Retrieving from the context buffer space means that, at the current processing progress point, the system executes a data read command on this storage unit to obtain the required historical speech segment.

[0060] As an optional embodiment, the target speaker extraction system internally establishes and maintains a first-in-first-out (FIFO) data queue as a context buffer space. During block-by-block iterative processing of the continuously input mixed speech stream, after each extraction calculation, the system writes the extracted target speech segment of the current moment as predicted output data into this historical context buffer area for storage. As the timeline progresses, when processing of the continuously input mixed speech stream reaches the current moment, the system's internal reading mechanism intervenes, accurately retrieving the predicted output result from the previous moment stored in the context buffer space, or retrieving a set of predicted output results from several past moments. This set is then used as the target speech segment corresponding to at least one historical moment and directly provided to the system's internal autoregressive encoder for context feature extraction, thereby completing the fusion and separation task for the current moment.

[0061] This embodiment constructs a context buffer space and stores historical extraction results in chronological order. This allows for the retrieval of target speech segments corresponding to at least one historical moment from the context buffer space as historical speech segments when processing a continuously input mixed speech stream up to the current moment. By utilizing a dedicated buffer area to solidify the data read / write flow and temporary storage mechanism, the historical speech prediction results are securely retained and efficiently retrieved. This avoids the risk of autoregressive link breakage due to underlying data loss during streaming processing, further enhancing the stability of the target speaker extraction system in real-world deployment scenarios such as long dialogues.

[0062] Considering that historical speech segments are inherently time-domain waveforms or spectral sequences containing rich temporal signals, directly inputting them into subsequent fusion and separation networks would lead to excessively high computational complexity and make it difficult to highlight the core semantic and acoustic cues relevant to the current target extraction task. To efficiently transform past sound signals into condensed feature memories that can be directly utilized by the separation network, the historical context feature extraction steps in this embodiment include: Perform multi-level feature extraction operations along the time dimension of historical speech segments to capture the speech content and changes in the speech state contained in the historical speech segments; The change information is encoded into a context feature vector in the target dimension to obtain historical context features.

[0063] Specifically, "along the time dimension of historical speech segments" refers to performing operations along the chronological development axis of historical audio signals. Multi-level feature extraction refers to a sequential processing method that utilizes multiple stacked network layers in a deep neural network to progressively abstract and expand the receptive field of the input signal. Speech content refers to the specific linguistic units just uttered by the target speaker, such as phonemes or words. Information on changes in vocalization state refers to the physical and acoustic changes in the target speaker's vocalization process, such as prosodic variations, energy levels, and the continuous trajectory of fundamental frequency changes—dynamic information.

[0064] As an optional embodiment, a lightweight autoregressive encoder can be added to the main architecture of the target speaker extraction system. This autoregressive encoder employs a temporal convolutional network structure, performing multi-level feature extraction operations along the time dimension of historical speech segments by stacking multiple layers of dilated convolutional layers with different dilation coefficients. Since dilated convolution can exponentially expand the receptive field without increasing the number of model parameters, it can extremely efficiently process and capture the changes in speech production content and vocalization state, such as phonemes, prosody, and energy, contained in historical speech segments.

[0065] After performing multi-level feature extraction operations along the time dimension of historical speech segments to capture the speech content and changes in the speech state contained in the historical speech segments, considering that these changes are usually very complex in terms of data structure and channel dimension, and may not be consistent with the dimension of the current mixed features extracted by the main encoder, subsequent mathematical multiplication and addition operations and feature fusion cannot be directly performed. Therefore, the changes are encoded into context feature vectors in the target dimension to obtain historical context features.

[0066] The change information refers to the acoustic and semantic features captured in the previous step, such as phonemes, prosody, and energy. The target dimension refers to the pre-defined number of network channels or the size of the feature matrix to meet the unified input format requirements of subsequent fusion processing modules. The context feature vector refers to the high-density, fixed-size data array formed after dimensionality reduction, convergence, or linear mapping. The historical context features refer to the high-dimensional feature matrix that represents the historical dynamic acoustic state of the target speaker in the final output.

[0067] As an optional embodiment, after extracting the complex change information mentioned above, the autoregressive encoder encodes the change information into a context feature vector in the target dimension through a fully connected layer or a one-dimensional convolutional layer as the output mapping layer, thereby obtaining historical context features.

[0068] This embodiment employs multi-level dilated convolution and other feature extraction methods and performs target dimension encoding, enabling the system to efficiently extract and utilize dynamic features such as phonemes and prosody just spoken by the target speaker. This avoids the waste of computational resources caused by directly processing the original long speech and the dimension mismatch problem during multi-path feature fusion, greatly improving the efficiency of dynamic context information utilization and the overall performance of the target speaker extraction system.

[0069] Considering that when fusing features from multiple sources with different modalities or dimensions, simply using matrix concatenation often fails to enable the network model to deeply understand the target speaker's acoustic timbre characteristics and the correlation of continuous speech content in the context, it is difficult to accurately extract the target speaker's feature information from complex acoustic aliasing features. To achieve efficient three-way information fusion, allowing static identity and dynamic content to actively intervene in the extraction process of mixed features, this embodiment fuses the mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and identity representation information to obtain the target fused features, including: The identity representation information is mapped to a preset target dimension to obtain the identity projection feature; Dynamic control weights are generated based on identity projection features and historical context features; Based on dynamic control weights, filtering and feature modulation are performed on each channel of the mixed speech features to generate target fusion features jointly determined by the mixed speech features, historical context features, and identity representation information.

[0070] Pre-extracted identity representation information typically comes from external, independent speaker recognition networks. Its data dimensions are fixed and often do not match the high-dimensional acoustic feature dimensions processed by the internal network of the target speaker extraction system, making direct subsequent matrix multiplication or channel concatenation impossible. This embodiment achieves compatibility between various feature dimensions and facilitates direct computation by the subsequent network through dimensional alignment and transformation.

[0071] In this embodiment, the preset target dimension refers to the standard number of channels or the hidden layer feature dimension set by the subject separation network within the speech extraction model when processing acoustic features. Identity projection features refer to the feature data representation of identity representation information after being transformed by linear or nonlinear network layers to the aforementioned standard dimension.

[0072] As an optional embodiment, an embedded projection layer can be set within the model to project the E-values ​​representing the identity features of the target speaker. target The input is fed into the embedding projection layer, and the linear mapping module projects it onto the appropriate hidden layer dimension, thereby obtaining the dimension-matched identity projection feature, which can be denoted as H. embed .

[0073] After obtaining the identity projection features, considering that the separation network needs to decide which components to retain and which components to suppress from the current mixed sound features based on external conditions, the guidance information must be transformed into adjustment coefficients that can directly affect the backbone acoustic features. Then, dynamic control weights are generated based on the identity projection features and historical context features.

[0074] This embodiment, by comprehensively utilizing identity projection features representing timbre and historical context features representing continuous speech semantic content, can construct an extremely rich conditional guidance feature containing temporal logic. After being converted into weights, this achieves the effect of providing precise channel scaling ratios and offset values ​​for subsequent feature modulation processes.

[0075] Here, dynamic control weights refer to the network parameter matrix generated in real time as the input historical speech segments and target identity information change. It is mainly used to perform channel-by-channel calculation intervention on the main feature map.

[0076] As an optional implementation, a fully connected layer structure can be used to perform a non-linear fusion mapping between the identity projection features representing "who to look for" and the historical context features representing "what was just said," generating two sets of dynamic control weight parameters, which serve as the scaling control weights and offset control weights required for subsequent processes. These two sets of parameters can be calculated using specific gating functions.

[0077] After generating dynamic control weights, considering that the mixed speech features contain information from all sound sources as well as background noise, the generated weight guidance information is used as a digital filter to remove interference and enhance the target. Then, based on the dynamic control weights, each channel of the mixed speech features is filtered and modulated to generate target fusion features determined by the mixed speech features, historical context features, and identity representation information.

[0078] By performing channel-level affine transformations on the mixed speech features, it is equivalent to deeply injecting conditional control information into the acoustic feature extraction and processing flow, achieving the purification effect of accurately activating the relevant frequency bands and acoustic attributes of the target speaker and forcibly suppressing irrelevant interference noise.

[0079] Here, each channel of the mixed speech features refers to a numerical slice of the high-dimensional acoustic features extracted by the main encoder at different depth feature dimensions. Filtering and feature modulation refer to performing element-wise multiplication and addition operations on the mixed speech features using the dynamically generated parameter matrix. Target fusion features refer to the comprehensive network feature representation that thoroughly integrates the three inputs: current acoustic observation, historical context memory, and target static timbre.

[0080] As an optional embodiment, the target speaker extraction system utilizes a feature linear modulation mechanism to perform the aforementioned fusion operation. Specifically, the separation network performs element-wise multiplication of the mixed speech features with the generated scaling control weights to perform a filtering operation, followed by the addition of offset control weights to perform a modulation operation. The principle formula for this filtering and feature modulation processing is as follows: ; in, This indicates the target fusion features. This represents the high-dimensional mixed speech features extracted from the mixed speech segment at the current moment. The symbol represents element-level dot product operations. This represents the scaling gate function, which outputs the aforementioned scaling control weights. This represents the offset gating function, which outputs the aforementioned offset control weights. This represents the historical context features extracted from historical speech segments. This represents the identity projection features obtained after dimensionality reduction through mapping. The target fusion features, ultimately determined by the combined effect of the three signals, are generated through the mathematical formulas described above.

[0081] This embodiment obtains identity projection features by mapping identity representation information to a preset target dimension. Dynamic control weights are generated based on these identity projection features and historical context features. These dynamic control weights are then used to filter and modulate the features of each channel of the mixed speech features, achieving an efficient and targeted three-way information deep integration mechanism. Because the weights generated by jointly using static identity projection features and dynamic historical context features directly intervene and adjust the mixed speech features, the subsequent separation network fully understands the target timbre to be extracted and the coherent context of the previous sentence. This avoids the shortcomings of traditional simple concatenation operations that result in insufficient control over conditional information, significantly enhancing the extraction model's ability to distinguish and separate specific speakers in multi-signal interleaving scenarios.

[0082] Based on the above embodiments, the mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information are fused to obtain the target fusion feature; based on the target fusion feature, the target speech segment of the target speaker at the current moment is extracted from the mixed speech segment, including: The mixed speech segment and the historical speech segment are input into the encoding network of the speech extraction model, and the encoding network extracts mixed speech features from the mixed speech segment and historical context features from the historical speech segment. The mixed speech features, historical context features, and identity representation information are input into the fusion and separation network of the speech extraction model. The fusion and separation network then fuses the mixed speech features, historical context features, and identity representation information to obtain the target fused features. The target fusion features are input into the decoding network of the speech extraction model, and the decoding network reconstructs the signal based on the target fusion features to output the target speech segment.

[0083] Here, the speech extraction model refers to the deep neural network architecture used to perform the entire task of target speaker speech separation and extraction. The encoding network is the front-end feature extraction component of the speech extraction model, which can include parallel processing branches specifically designed for different input sources, to independently encode the mixed signal acquired at the current moment and the buffered historical signal, respectively.

[0084] As an optional implementation, the speech extraction model preferably uses a temporal convolutional neural network architecture as its foundation. Specifically, the encoding network can be divided into two independent processing modules: a master encoder and an autoregressive encoder. The master encoder receives and processes the mixed speech segment at the current moment, extracting mixed speech features through convolutional operations. The autoregressive encoder, as a newly added lightweight encoding module, is specifically used to receive and process historical speech segments corresponding to at least one historical moment. The autoregressive encoder can employ a temporal convolutional network structure or a recurrent neural network structure, efficiently and layer-by-layer expanding the receptive field through mechanisms such as dilated convolution, thereby fully extracting the phoneme, prosody, and energy evolution patterns contained in long historical speech segments, ultimately encoding and outputting them as historical context features.

[0085] After extracting mixed speech features and historical context features using the encoding network of the speech extraction model, considering that the two types of acoustic features and the target speaker's identity representation information belong to different feature spaces and are physically isolated from each other, without deep integration and conditional constraints, the subsequent network structure will not be able to accurately identify the specific target to be extracted, nor will it know what the target's likely vocal state is. To enable the model to accurately locate the target object and smoothly continue the previous vocal content in a complex, noisy context, the fusion separation network of the speech extraction model is used to fuse the mixed speech features, historical context features, and identity representation information to obtain the target fused features. This step completely breaks down the barriers between the independent information sources, constructing a three-dimensional feature interaction mechanism where static timbre provides identity guidance and dynamic historical content provides state correction.

[0086] Here, the fusion separation network is the core computational hub of the entire speech extraction model. It not only has the ability to align and splice together multimodal or multi-source features, but also contains multiple deep computational network layers for processing the fused features, so as to effectively separate the target speech signal from the interference signal.

[0087] As an optional embodiment, when using a fusion-separation network for feature processing, the fusion-separation network can include an embedded projection layer component. The embedded projection layer first maps the extracted identity representation information to a target dimension space that matches the aforementioned acoustic features. Subsequently, in the main separation processing flow of the fusion-separation network, for example, within each layer or processing block of the fusion-separation network, the historical context features output by the encoding network and the mapped identity representation information are used as conditional features and injected into the processing channels for the mixed speech features. The fusion-separation network can merge the three features along the channel dimension through direct concatenation, or it can employ a feature modulation layer based on a gating mechanism, using the mixed speech features as the backbone input, and utilizing historical context features and identity representation information to jointly generate dynamic control weights for weighted scaling and offsetting of each channel of the backbone input features, thereby calculating and outputting the target fusion feature containing the pure feature information or mask information of the target speaker.

[0088] After obtaining the target fusion features through deep interaction and processing using the fusion separation network of the speech extraction model, considering that the target fusion features at this point still belong to an abstract high-dimensional digital matrix space representation, which cannot be directly perceived by the human auditory system, nor can it be directly output as the final playable audio, in order to restore the enhanced features in the latent space back to the human-understandable physical temporal acoustic space, the decoding network of the speech extraction model is used to reconstruct the signal based on the target fusion features, outputting the target speech segment. Through this step, the clean high-dimensional features, which have been freed from interference noise and other speaker voices, are inversely transformed to generate the final coherent, natural, and high-quality target speaker speech waveform signal.

[0089] Here, the decoding network refers to the back-end signal reconstruction component of the speech extraction model. Its function is exactly the opposite of the front-end encoding network. It usually contains transposed convolutional layers or corresponding inverse transform operator structures to map high-dimensional latent features back to the original speech signal dimension.

[0090] As an optional embodiment, the decoding network receives the target fusion feature output from the fusion separation network. If the target fusion feature is represented as an estimated mask matrix for the target speech, the system first performs an element-wise multiplication filtering operation on the estimated mask matrix and the corresponding output feature of the original mixed speech segment in the main encoder to obtain the clean speech feature after mask filtering. Then, the clean speech feature is input into the decoding network, which reconstructs and converts it back into a time-domain waveform signal, and finally generates and outputs the target speech segment at the current time.

[0091] This embodiment utilizes the encoding network of a speech extraction model to extract mixed speech features and historical context features separately. A fusion separation network then performs deep fusion processing on the mixed speech features, historical context features, and identity representation information to obtain the target fusion features. Finally, a decoding network reconstructs the signal based on the target fusion features to output the target speech segment. This achieves a complete, dedicated deep learning processing architecture with an autoregressive memory path. Because the network design specifically incorporates an autoregressive encoding branch and a three-way feature fusion mechanism, the model can efficiently utilize the dynamic speech content just spoken by the target speaker in the underlying computational logic of the extraction architecture. This effectively avoids the contextual gaps caused by the inability to receive historical information in traditional dual-input models, greatly improving the temporal smoothness and separation accuracy when processing long-term continuous speech features.

[0092] in, Figure 2 This is a schematic diagram of the processing flow of the speech extraction model provided by the present invention, as shown below. Figure 2 As shown, the specific processing flow of this speech extraction model when processing speech data at the current moment is as follows: First, the speech extraction model receives three independent inputs: the mixed speech segment at the current moment, the identity representation information of the target speaker to be extracted, and the historical speech segment of the target speaker at least one historical moment. During the model training phase, the historical speech segment directly uses the real, clean speech labels corresponding to the historical moment as input to execute the teacher-enforced strategy, thereby avoiding error accumulation in the early stages of training. During the inference phase of actual model deployment, due to the lack of real, clean speech, the historical speech segment uses the target speech segment extracted by the model before the historical moment as historical context input, which is fed back to the model input through an autoregressive loop to construct a streaming dynamic contextual memory.

[0093] Then, the encoding network of the speech extraction model is used to perform dimensionality reduction and feature extraction on the received signals. Specifically, the main encoder in the encoding network processes the mixed speech segments to extract high-dimensional mixed speech features; at the same time, a newly added lightweight autoregressive encoder in the encoding network processes historical speech segments to efficiently extract the vocal state change information such as phonemes, prosody, and energy contained in the historical speech segments, thus obtaining historical context features; in addition, the model also maps identity representation information to a preset appropriate dimension by embedding a projection layer to obtain the corresponding identity projection features.

[0094] Next, the fusion-separation network of the speech extraction model is used to deeply fuse the mixed speech features, historical context features, and mapped identity representation information. The fusion-separation network, as the main network of the model, can specifically employ a dual-path recurrent neural network or a temporal convolutional separation network structure. Through feature concatenation or gating mechanisms, the three sets of features are interactively injected, allowing the fusion-separation network to fully understand the target object to be extracted and the speech content it has just spoken. This enables it to estimate the mask or target features of the target speech, resulting in the target fusion features determined by the three sets of information.

[0095] Finally, the decoding network of the speech extraction model is used to reconstruct the target signal based on the target fusion features. The decoding network can perform a multiplication and filtering operation on the target fusion features estimated by the fusion separation network and the mixed speech features output by the main encoder. Then, the filtered features are converted back to time domain waveforms through decoding operations, and finally the target speech segment of the target speaker at the current time is generated and output to achieve high-quality, continuous and artifact-free speech extraction.

[0096] Considering that when constructing a speech extraction system based on an autoregressive context mechanism, if the model's own predictions are relied upon as historical context during the initial training phase, any deviation in the prediction from the previous moment will be passed as historical input to the next moment, causing errors to accumulate over time. To enable the model to correctly utilize historical context during training and effectively avoid error accumulation in the early stages of training, the training steps of the speech extraction model in this embodiment include: Obtain the mixed speech segment sample at the current moment, the historical clean speech segment sample of at least one historical moment of the sample speaker, and the identity representation information sample of the sample speaker; The mixed speech segment sample, the historical clean speech segment sample, and the identity representation information sample are input into the initial speech extraction model to obtain the predicted target speech segment corresponding to the current moment; The loss value is calculated based on the difference between the predicted target speech segment and the sample label, and the parameters of the initial speech extraction model are updated based on the loss value to obtain the speech extraction model. The sample label is the clean speech segment of the sample speaker corresponding to the current time.

[0097] Here, the mixed speech segment sample at the current moment can be understood as an audio segment sample containing multiple sound sources used to simulate a real complex acoustic environment when constructing the training dataset. The historical clean speech segment sample of at least one historical moment of the sample speaker can be understood as a clean speech segment sample located before the current moment, directly provided to the model as a real label. The identity representation information sample of the sample speaker can be understood as a high-dimensional vector sample pre-extracted from the sample speaker's reference audio to represent its static acoustic characteristics such as timbre.

[0098] As an optional embodiment, the above sample data can be obtained by segmenting the original audio dataset. Prepare a relatively long sample mixed speech, clean speech of the sample speaker, and identity representation information samples. Divide the long sample mixed speech and clean speech into multiple segments using the same overlap window, for example, using a 50% overlap ratio, thereby obtaining multiple mixed speech segment samples that are strictly aligned in the time series and corresponding multiple clean speech segments. When processing the current moment in the time series, directly obtain the current mixed segment in the sequence as the mixed speech segment sample for the current moment, and obtain the previous clean speech segment in the sequence whose time sequence precedes the current moment as a historical clean speech segment sample for at least one historical moment.

[0099] After obtaining the mixed speech segment sample at the current moment, the historical clean speech segment sample of at least one historical moment of the sample speaker, and the identity representation information sample of the sample speaker, considering that the network architecture in the initial state needs to fuse multi-source acoustic and content features through the forward propagation mechanism to generate the speech prediction result at the current moment, this embodiment inputs the mixed speech segment sample, the historical clean speech segment sample, and the identity representation information sample into the initial speech extraction model to obtain the predicted target speech segment corresponding to the current moment.

[0100] Here, the initial speech extraction model refers to the target speaker extraction neural network whose network parameters have not yet been fully optimized and converged, and which is in the process of training preparation or training. The predicted target speech segment refers to the estimated speech segment for the sample speaker at the current moment, output by the initial speech extraction model based on the current multi-channel input information, after internal operations such as encoding, fusion, and decoding.

[0101] As an optional implementation, the initial speech extraction model simultaneously receives the above three inputs, and the specific principle of its internal information fusion and output of the predicted target speech segment can be expressed as the formula: ; in, Indicates the current time The corresponding predicted target speech segment, Representing historical moments Corresponding historical clean speech samples, This represents a mixed speech segment sample. This represents a sample of identity representation information. Furthermore, for... At that time, since there was no data for that historical moment, A sequence of all zero features or a pre-set starting marker feature can be used to replace the input to the initial speech extraction model.

[0102] After obtaining the predicted target speech segment corresponding to the current time, in order to quantify the prediction error of the initial speech extraction model and guide the optimization direction of the node weights within the network to improve the extraction accuracy, this embodiment calculates the loss value based on the difference between the predicted target speech segment and the sample label, and updates the parameters of the initial speech extraction model based on the loss value to obtain the speech extraction model, wherein the sample label is the clean speech segment of the sample speaker corresponding to the current time.

[0103] Here, the sample label refers to a genuine speech segment from the original dataset that corresponds strictly to the current time point and is free from any noise or interference; it serves as the standard answer for supervised learning. The loss value is a mathematical measure of the difference between the predicted target speech segment and the sample label in terms of signal quality, waveform similarity, etc., calculated using a specific evaluation function. The speech extraction model refers to the final network model that, after multiple rounds of parameter iteration and updates, achieves convergence of the loss value and possesses the ability to extract the target speaker's speech in complex real-world environments.

[0104] As an optional implementation, when calculating the loss value, the loss function is only calculated on the speech block at the current time. The scale-invariant signal-to-noise ratio loss function can be used to calculate the difference, and the specific calculation formula is as follows: ; in, Indicates the current time The corresponding loss value, Indicates the current time Corresponding sample labels This represents the scale-invariant signal-to-noise ratio (SNR) function. The loss value at the current time step is equal to the negative value of the SNR function. The loss value at the current time step represents the error metric in which the network processes the current speech segment. The SNR function value is calculated as the ratio of signal distortion between the predicted target speech segment and the sample label as input parameters after eliminating the effect of amplitude scaling. Using the backpropagation algorithm, the loss value at the current time step is used to calculate gradients and update weights for the parameters of the main encoder, autoregressive encoder, and segregating network modules within the initial speech extraction model. As the entire training dataset is continuously traversed and the parameters are iterated, a speech extraction model with satisfactory performance is finally obtained.

[0105] Considering that during sample-based supervised training of speech extraction models, if the model relies on its own inaccurate predictions as historical context in the early training stages, errors are easily accumulated over time. To enable the model to quickly learn to use the correct historical context for feature inference, this embodiment constructs a time-aligned, absolutely time-aligned current mixed input and historically clean reference during the data preparation stage. Based on this, the steps for obtaining mixed speech segment samples and historically clean speech segment samples in this embodiment include: Obtain a continuous mixed speech stream of samples and a continuous clean speech stream of the sample speaker corresponding to the mixed speech stream of samples; A sliding window with the same time length and the same overlap ratio is used to simultaneously segment the mixed speech stream and the continuous clean speech stream. The segmented mixed speech corresponding to the current moment is used as the mixed speech segment sample, and the segmented clean speech corresponding to at least one historical moment is used as the historical clean speech segment sample.

[0106] Here, a continuous sample mixed speech stream refers to a long segment of audio data that is pre-recorded or synthesized in a training scenario, is uncut, and contains background noise and other interfering human voices. A continuous clean speech stream refers to a long segment of clean audio data that is recorded synchronously with the above-mentioned sample mixed speech stream and contains only the voice of the speaker of the sample to be extracted.

[0107] As an optional embodiment, clean speech data of the target speaker and corresponding environmental mixing data can be extracted by reading pairs of audio files in a pre-constructed speech separation training dataset, thereby obtaining a continuous sample mixed speech stream and a continuous clean speech stream of the sample speaker corresponding to the sample mixed speech stream.

[0108] After acquiring continuous mixed speech streams and corresponding continuous clean speech streams from the speakers, a rigid segmentation of long speech segments is considered. This is because directly and forcefully slicing long speech segments would result in the loss of edge information at the slits and abrupt splicing. Furthermore, if the segmentation scales of the mixed and clean audio are inconsistent, it would cause spatiotemporal distortion during model training, making it impossible for historical features to accurately correspond to the previous state of the current feature. Therefore, a sliding window with the same time length and the same overlap ratio is used to simultaneously segment the mixed speech streams and the continuous clean speech streams.

[0109] Synchronous segmentation using sliding windows with the same parameters ensures that the input question (sample mixed speech stream) and the standard answer (continuous pure speech stream) are perfectly aligned on the time axis without any discrepancies. At the same time, the overlap mechanism ensures that there are no abrupt breaks in the high-level semantics and acoustic content between speech segments, thus achieving the effects of eliminating data temporal misalignment during model training and suppressing boundary auditory artifacts.

[0110] Here, sliding windows with the same time length and the same overlap ratio indicate that the two data truncation scales used to segment mixed speech and clean speech are not only completely identical in the time span of a single truncation, but also have completely identical crossover ratios of the step size of each forward slide relative to the total window length. Synchronous segmentation processing means applying the above sliding windows simultaneously to truncate and segment the two speech streams at the same time starting point.

[0111] As an optional embodiment, the time span of the sliding window can be set to two seconds, and the overlap ratio can be set to 50%. Using this sliding window, sliding reading can be performed at the same time starting point of the continuous sample mixed speech stream and the continuous clean speech stream, thereby completing the synchronous segmentation processing of the sample mixed speech stream and the continuous clean speech stream, and thus dividing the continuous long speech into multiple blocks of data with fixed durations.

[0112] After synchronously segmenting the mixed speech stream and the continuous clean speech stream using a sliding window with the same time length and the same overlap ratio, considering that the model needs to clearly distinguish the complex observation signal to be processed at the moment of the current iteration and the historical reference signal as an auxiliary guide in a single iteration calculation, the mixed speech segment corresponding to the current moment after segmentation is taken as the mixed speech segment sample, and the clean speech segment corresponding to at least one historical moment after segmentation is taken as the historical clean speech segment sample.

[0113] By physically allocating slices at different times as data for different input branches, the absolutely correct answer from the previous time step can be directly forced into the model as historical memory, realizing a teacher-forced mechanism during the training phase. This avoids the propagation and accumulation of errors caused by the model's inaccurate predictions in the early stages, thereby accelerating model convergence and improving the accuracy of context utilization.

[0114] Here, the mixed speech segment corresponding to the current moment is the target mixed object that the model needs to separate and denoise in this computing node, and the clean speech segment corresponding to at least one historical moment is the real vocal state cue required by the model in this computing node.

[0115] As an optional embodiment, after synchronous segmentation processing, the sample mixed speech stream is segmented into set M, and the continuous clean speech stream is segmented into set S. label Where M includes M1 to M2. N Multiple time-ordered chunks, S label Includes S label1 To S labelNMultiple time-ordered segments are generated. In the parameters above, M represents the feature representation set of the mixed speech, S represents the speech signal, label represents the pure label of the speech signal, N represents the total number of speech blocks obtained after segmentation with the same overlap ratio, and 1 to N represent the time sequence number. When training progresses to the data block corresponding to time t, where t is the current time number greater than 1, the M corresponding to the current time t is... t As a sample of mixed speech segments; simultaneously, the clean data corresponding to the previous time step t-1 is obtained, and S label（t-1） As samples of clean historical speech segments corresponding to at least one historical moment, they are jointly input into the subsequent model network process.

[0116] This embodiment achieves temporal alignment of data by synchronously segmenting two types of speech streams using sliding windows with the same duration and overlap ratio. By ensuring a precise temporal correspondence between the mixed speech and historical clean speech, the authenticity and effectiveness of contextual cues are guaranteed, avoiding learning biases caused by spatiotemporal distortions in the model.

[0117] The speech extraction device provided by the present invention will be described below. The speech extraction device described below can be referred to in correspondence with the speech extraction method described above.

[0118] Based on the above embodiments, Figure 3 This is a schematic diagram of the speech extraction device provided by the present invention, as shown below. Figure 3 As shown, the device includes: The first acquisition module 310 is used to acquire the mixed speech segment at the current moment and the identity representation information of the target speaker; The second acquisition module 320 is used to acquire at least one historical speech segment of the target speaker at a historical moment. The fusion processing module 330 is used to fuse the mixed speech features extracted from the mixed speech segments, the historical context features extracted from the historical speech segments, and the identity representation information to obtain the target fusion features. The speech extraction module 340 is used to extract the target speaker's speech segment at the current moment from the mixed speech segment based on the target fusion features; there is a contextual relationship between the historical speech segment and the target speech segment at the current moment.

[0119] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communications interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute the speech extraction method.

[0120] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech extraction methods provided by the above methods.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the speech extraction methods provided by the methods described above.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech extraction method, characterized in that, include: Obtain the mixed speech segment at the current moment and the identity representation information of the target speaker; Obtain at least one historical speech segment of the target speaker at a historical moment; The mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information are fused together to obtain the target fused features; Based on the target fusion features, extract the target speaker's target speech segment at the current moment from the mixed speech segment; The historical speech segment and the target speech segment at the current moment have a contextual relationship.

2. The speech extraction method according to claim 1, characterized in that, The acquisition of at least one historical speech segment of the target speaker at a historical moment includes: If the current time is the starting time, the initial voice segment will be used as the historical voice segment; If the current time is not the starting time, the target speech segment that has been extracted before the current time and corresponds to the at least one historical time is obtained as the historical speech segment.

3. The speech extraction method according to claim 2, characterized in that, The step of acquiring the target speech segment that has been extracted before the current time and corresponds to the at least one historical time, as the historical speech segment, includes: When processing the continuously input mixed speech stream up to the current moment, the target speech segment corresponding to the at least one historical moment is retrieved from the context buffer space as the historical speech segment; the context buffer space stores the target speech segments of the target speaker extracted sequentially before the current moment during the processing of the mixed speech stream, in chronological order.

4. The speech extraction method according to any one of claims 1 to 3, characterized in that, The steps for extracting the historical context features include: Perform multi-level feature extraction operations along the time dimension of the historical speech segment to capture the speech content and changes in the speech state contained in the historical speech segment; The change information is encoded into a context feature vector in the target dimension to obtain the historical context features.

5. The speech extraction method according to any one of claims 1 to 3, characterized in that, The process of fusing the mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information to obtain the target fused features includes: The identity representation information is mapped to a preset target dimension to obtain identity projection features; Dynamic control weights are generated based on the identity projection features and the historical context features; Based on the dynamic control weights, each channel of the mixed speech features is filtered and modulated to generate the target fusion feature, which is jointly determined by the mixed speech features, the historical context features, and the identity representation information.

6. The speech extraction method according to any one of claims 1 to 3, characterized in that, The mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information are fused together to obtain the target fused features; Based on the target fusion features, extract the target speaker's target speech segment at the current time from the mixed speech segment, including: The mixed speech segment and the historical speech segment are input into the encoding network of the speech extraction model, and the encoding network extracts the mixed speech features from the mixed speech segment and the historical context features from the historical speech segment; The mixed speech features, the historical context features, and the identity representation information are input into the fusion and separation network of the speech extraction model. The fusion and separation network then fuses the mixed speech features, the historical context features, and the identity representation information to obtain the target fusion features. The target fusion features are input into the decoding network of the speech extraction model, and the decoding network reconstructs the signal based on the target fusion features to output the target speech segment.

7. The speech extraction method according to claim 6, characterized in that, The training steps of the speech extraction model include: Obtain a mixed speech segment sample at the current moment, a historical clean speech segment sample of at least one historical moment of the sample speaker, and an identity representation information sample of the sample speaker; The mixed speech segment sample, the historical clean speech segment sample, and the identity representation information sample are input into the initial speech extraction model to obtain the predicted target speech segment corresponding to the current moment. The loss value is calculated based on the difference between the predicted target speech segment and the sample label, and the parameters of the initial speech extraction model are updated based on the loss value to obtain the speech extraction model. The sample label is the clean speech segment of the sample speaker corresponding to the current moment.

8. The speech extraction method according to claim 7, characterized in that, The steps for obtaining the mixed speech segment samples and the historical clean speech segment samples include: Acquire a continuous sample mixed speech stream and a continuous clean speech stream of the sample speaker corresponding to the sample mixed speech stream; A sliding window with the same time length and the same overlap ratio is used to synchronously segment the sample mixed speech stream and the continuous clean speech stream; The segmented mixed speech corresponding to the current moment is used as the mixed speech segment sample, and the segmented clean speech corresponding to the at least one historical moment is used as the historical clean speech segment sample.

9. A speech extraction device, characterized in that, include: The first acquisition module is used to acquire the mixed speech segment at the current moment and the identity representation information of the target speaker; The second acquisition module is used to acquire at least one historical speech segment of the target speaker at a historical moment; The fusion processing module is used to fuse the mixed speech features extracted from the mixed speech segment, the historical context features extracted from the historical speech segment, and the identity representation information to obtain the target fusion feature; The speech extraction module is used to extract the target speaker's target speech segment at the current moment from the mixed speech segment based on the target fusion features; the historical speech segment and the target speech segment at the current moment have a contextual relationship.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech extraction method as described in any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech extraction method as described in any one of claims 1 to 8.

Images