Speech reconstruction method and speech reconstruction system

Abstract

The embodiment of the specification relates to the technical field of speech signal processing, and includes a speech reconstruction method and a speech reconstruction system, wherein the speech reconstruction method comprises the following steps: obtaining mixed speech containing a target object and other sound sources, and reference speech of the target object; performing frequency band division on mixed spectrum features of the mixed speech and reference spectrum features of the reference speech to obtain mixed band features and reference band features, and extracting a target object condition for representing an acoustic identity of the target object from the reference band features; for any time frame of the mixed speech, determining recursive parameters of a state space model in the time frame based on the mixed band features and the target object condition, and recursively determining output features of the time frame according to the recursive parameters; and performing signal reconstruction on the mixed spectrum features based on the output features of each time frame to obtain target speech of the target object. The extraction accuracy and reconstruction quality of the target speech are improved, and the accuracy of speech recognition is higher.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of speech signal processing technology, and in particular to a speech reconstruction method and a speech reconstruction system. Background Technology

[0002] With the rapid development of intelligent voice interaction and online audio and video communication, the technology of extracting the voice of a target object from mixed speech containing multiple sound sources based on the target object's identity has been widely used in scenarios such as online meetings, remote calls, voice assistants, and content production, and has put forward higher requirements for real-time performance and extraction accuracy.

[0003] Currently, to address the need for extracting such target speech, the usual approach is to first obtain the reference speech of the target object, extract conditional information from it to characterize the acoustic identity of the target object, then fuse the conditional information with the acoustic features of the mixed speech, and feed it into a temporal modeling network to obtain the speech of the target object. The conditional information is usually fused with the acoustic features in the form of concatenation, element-wise multiplication, additive bias, or feature modulation. The temporal modeling network is usually implemented using recurrent networks, temporal convolutional networks, or self-attention networks.

[0004] However, the aforementioned technical solutions are prone to retaining interference components in the extraction results when there are similar timbres, overlapping speech, or inconsistent channel conditions, making it difficult to meet the requirements for the accuracy of target speech extraction and reconstruction quality. Furthermore, to maintain target discrimination capability, such solutions often require significant computational and storage overhead, making it difficult to balance real-time processing with low-cost deployment. Therefore, there is an urgent need for a speech reconstruction method with higher accuracy and better suitability for real-time processing. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a speech reconstruction method. One or more embodiments of this specification also relate to a speech reconstruction system, a computing device, a computer-readable storage medium, and a computer program product, to solve the aforementioned technical problems.

[0006] According to a first aspect of the embodiments of this specification, a speech reconstruction method is provided, comprising: Acquire mixed speech containing the target object and other sound sources, as well as the reference speech of the target object; Frequency bands are divided into mixed frequency band features and reference frequency band features of mixed speech and reference speech, respectively, and target object conditions for characterizing the acoustic identity of the target object are extracted from the reference frequency band features. For any time frame of mixed speech, based on the mixed frequency band features and the target object conditions, the recursive parameters of the state space model for that time frame are determined, and the output features of that time frame are recursively determined according to the recursive parameters. Based on the output features of each time frame, the mixed spectral features are used to reconstruct the signal and obtain the target speech of the target object.

[0007] According to a second aspect of the embodiments of this specification, a speech reconstruction system is provided, comprising: The acquisition module is used to acquire mixed speech containing the target object and other sound sources, as well as the reference speech of the target object; The condition determination module is used to divide the frequency bands of the mixed spectral features of the mixed speech and the reference spectral features of the reference speech, obtain the mixed frequency band features and the reference frequency band features, and extract the target object conditions used to characterize the acoustic identity of the target object from the reference frequency band features. The recursive determination module is used to determine the recursive parameters of the state space model for any time frame of mixed speech based on the mixed frequency band features and the target object conditions, and to recursively determine the output features of the time frame according to the recursive parameters. The reconstruction module is used to reconstruct the signal from the mixed spectral features based on the output features of each time frame, so as to obtain the target speech of the target object.

[0008] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described speech reconstruction method.

[0009] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the above-described speech reconstruction method.

[0010] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including computer-executable instructions that, when executed by a processor, implement the steps of the above-described speech reconstruction method.

[0011] One embodiment of this specification implements a speech reconstruction method, comprising: acquiring mixed speech containing a target object and other sound sources, and reference speech of the target object; dividing the mixed spectral features of the mixed speech and the reference spectral features of the reference speech into frequency bands to obtain mixed frequency band features and reference frequency band features, and extracting target object conditions from the reference frequency band features to characterize the acoustic identity of the target object; for any time frame of the mixed speech, determining the recursive parameters of the state space model for that time frame based on the mixed frequency band features and the target object conditions, and recursively determining the output features of that time frame according to the recursive parameters; and reconstructing the mixed spectral features based on the output features of each time frame to obtain the target speech of the target object.

[0012] By acquiring mixed speech containing the target object and other sound sources, as well as the target object's reference speech, a unified input basis is provided for subsequent processing. By dividing the mixed spectral features and the reference spectral features into frequency bands, the two features are aligned within the same frequency band structure, providing a basis for extracting conditional information and performing frequency band-by-frequency processing. Target object conditions characterizing the acoustic identity of the target object are extracted from the reference frequency band features, preserving the target object's identity information at the frequency band level. For any given time frame, the recursive parameters of the state space at that time frame are determined based on the mixed frequency band features and the target object conditions, and the recursion is performed according to these parameters. By determining the output features of a given time frame, the target object condition participates in the frame-by-frame recursive process, thereby continuously strengthening components related to the target object and suppressing interference components, improving the extraction accuracy and reconstruction quality of the target speech. The target speech of the target object is obtained by reconstructing the mixed spectral features based on the output features of each time frame. The entire processing is performed frame-by-frame and does not depend on future frames. The maintained state scale is fixed, making it easy to adapt to application scenarios requiring real-time processing and low-cost deployment. Based on the reconstructed target speech, it can provide subsequent speech recognition with less interference and higher quality speech input, thus improving the accuracy of speech recognition. Attached Figure Description

[0013] Figure 1 This is a flowchart of a speech reconstruction method provided in one embodiment of this specification; Figure 2a This is a schematic diagram of the framework of a speech reconstruction method provided in one embodiment of this specification; Figure 2b This is a schematic diagram illustrating the processing procedure of a state-space model provided in one embodiment of this specification; Figure 2c This is a schematic diagram of a non-uniform frequency band division process provided in one embodiment of this specification; Figure 2d This is a schematic diagram illustrating the processing procedure of a dual-path block according to one embodiment of this specification; Figure 2e This is a schematic diagram of a target object condition extraction process provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the structure of a speech reconstruction system provided in one embodiment of this specification; Figure 4 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are obtained through open-source datasets or public datasets that comply with their license agreements, or are obtained with full authorization from the relevant parties. Moreover, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0019] Attention mechanism is a computational method that aggregates information in a weighted manner based on relevance. It uses the degree of matching between the query and the key to determine the weight, and then sums the values ​​in a weighted manner, so that information that is more relevant to the query is given a greater weight.

[0020] A phoneme is the smallest unit of speech that can distinguish the meaning of words. The speech system of a language consists of a finite number of phonemes, and continuous speech is formed by the sequential connection of different phonemes in time.

[0021] A multilayer perceptron (MLP) is a classic feedforward artificial neural network consisting of an input layer, one or more hidden layers, and an output layer connected in sequence. Each layer contains several neurons and transmits information through a fully connected manner. With the help of non-linear activation functions, it can approximate any continuous function and is a fundamental model component in deep learning.

[0022] Signal-to-noise ratio (SNR) is a measure of the ratio of signal power to noise power, usually expressed in dB. A higher value indicates that the noise component accounts for a smaller proportion of the signal.

[0023] Overlap-Add (OLA) is a method for reassembling segmented time-domain signals into a complete signal. It preserves overlapping areas between adjacent segments and adds the overlapping parts point by point, so that the signals at the splicing point can be smoothly connected.

[0024] The Short-Time Fourier Transform (STFT) is a time-frequency analysis method that obtains local feature information of the signal in both time and frequency dimensions by windowing the signal into segments and performing Fourier transform on each segment. This overcomes the shortcoming of the traditional Fourier transform, which cannot reflect the frequency change over time.

[0025] The Inverse Short-Time Fourier Transform (iSTFT) is the inverse time-frequency transform corresponding to the Short-Time Fourier Transform. It is used to restore the complex spectrum of each time frame obtained by the Short-Time Fourier Transform to the time-domain signal.

[0026] A window function is a function used in frame processing to weight the signal of each frame. Its value is higher at the center of the frame and gradually decreases to near zero at both ends of the frame to reduce spectral leakage caused by signal abrupt changes at the frame boundaries.

[0027] A filter bank is a collection of filters whose center frequencies and bandwidths are distributed according to a set pattern and whose adjacent frequency bands are interconnected. The input signal is processed by each filter and outputs a set of sub-band components.

[0028] An activation function is a non-linear function that acts on the output of neurons in a neural network. It is used to introduce non-linear transformation capabilities into the network, enabling multi-layer structures to fit the non-linear relationship between input and output.

[0029] The sigmoid linear unit (SiLU) is a self-gated nonlinear activation function whose value is determined by the product of the input itself and the sigmoid function value of that input. It retains a small non-zero response even when the input is negative.

[0030] Zero-order hold (ZOH) is a method for converting a continuous-time system into a discrete-time system. It maintains the input values ​​at the sampling time within a sampling interval and discretizes the continuous state equation accordingly to obtain a discrete state matrix and a discrete input matrix related to the sampling interval.

[0031] Room Impulse Response (RIR) is a time-domain function that describes the acoustic transmission characteristics between a sound source and a receiver. It represents the response received at the receiver when an impulse is applied at the sound source, and includes direct sound, early reflections, and late reverberation components.

[0032] Scale-Invariant Signal-to-Noise Ratio (SI-SNR) is a metric used to evaluate the similarity between an estimated signal and a target signal. Before calculation, the estimated signal is scaled to eliminate the influence of amplitude differences. A higher value indicates that the estimated signal is closer to the target signal.

[0033] The Adaptive Moment Estimation Optimizer (Adam) is a deep learning parameter optimization algorithm that combines momentum method with adaptive learning rate strategy. It dynamically adjusts the learning rate for each parameter by maintaining the first-order moment estimate and second-order moment estimate of the gradient respectively. It features fast convergence speed, low hyperparameter sensitivity and moderate memory usage.

[0034] A graphics processing unit (GPU) is an electronic processor designed specifically for large-scale parallel computing. Initially used for graphics rendering and image acceleration, it has been widely applied in general-purpose parallel computing fields such as deep learning model training and inference due to its large number of computing cores and high memory bandwidth.

[0035] Short-Time Objective Intelligibility (STOI) is an objective metric used to evaluate speech intelligibility. It is obtained by comparing the time-frequency representation correlation between the processed speech and the clean reference speech over a short period of time. The closer the value is to 1, the higher the intelligibility.

[0036] Speaker similarity is an indicator used to evaluate how close two audio segments are in terms of speaker identity. It is usually calculated by the similarity between the speaker representations of the two audio segments. The higher the value, the more likely the two audio segments are to come from the same speaker.

[0037] Referenceless speech quality scoring is an objective evaluation method that scores speech quality without requiring a clean reference speech. The score is estimated from the acoustic features of the speech signal itself, and the higher the score, the better the speech quality.

[0038] With the rapid development of intelligent voice interaction and online audio and video communication, the technology of extracting the voice of a target object from mixed speech containing multiple sound sources based on the target object's identity has been widely used in scenarios such as online meetings, remote calls, voice assistants, and content production, and has put forward higher requirements for real-time performance and extraction accuracy.

[0039] Currently, to address the need for extracting such target speech, the usual approach is to first obtain the reference speech of the target object, extract conditional information from it to characterize the acoustic identity of the target object, then fuse the conditional information with the acoustic features of the mixed speech, and feed it into a temporal modeling network to obtain the speech of the target object. The conditional information is usually fused with the acoustic features in the form of concatenation, element-wise multiplication, additive bias, or feature modulation. The temporal modeling network is usually implemented using recurrent networks, temporal convolutional networks, or self-attention networks.

[0040] However, the above-mentioned technical solutions are prone to leaving interference components in the extraction results when the timbre is similar, the speech overlaps, and the channel conditions are inconsistent. The extraction accuracy and reconstruction quality of the target speech are difficult to meet the requirements. At the same time, in order to maintain the target discrimination ability, such solutions often require a large amount of computing power and storage overhead, making it difficult to balance real-time processing and low-cost deployment.

[0041] In view of this, this specification provides a speech reconstruction method, and also relates to a speech reconstruction system, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0042] Specifically, see Figure 1 , Figure 1 A flowchart of a speech reconstruction method according to an embodiment of this specification is shown, such as... Figure 1 As shown, the method specifically includes the following steps.

[0043] Step 102: Obtain the mixed speech containing the target object and other sound sources, as well as the reference speech of the target object.

[0044] The speech reconstruction method provided in one or more embodiments of this specification can be applied to various scenarios that require extracting the speech of a specified object from mixed speech from multiple sources. Specifically, it can include speech enhancement of a specified speaker in online meetings and remote calls, extraction of the target user's speech in voice assistants and intelligent customer service, separation of a specified speaker in live broadcasts and interview recordings, and target speech separation as a pre-processing step for speech recognition and speaker verification. The specific application can be a speech processing task deployed in the cloud, on an edge server, or on a low-power edge device. This task can receive mixed speech frame by frame and continuously output the speech of the target object based on the reference speech of the target object.

[0045] The target object is the sound source object targeted in this speech reconstruction, that is, the object whose speech needs to be preserved and recovered from the mixed speech. Specifically, the target object can be a designated speaker in an online meeting, a target user served by a voice assistant, or a designated speaker in an interview or recording scenario. Its identity can be selected by the user through an interactive interface or specified by the system according to content requirements. The acoustic identity of the target object is provided by the reference speech and is represented in the form of target object conditions in subsequent processing. Furthermore, the embodiments of this specification do not rely on an additional pre-trained speaker recognition network to extract the fixed-dimensional embedding of the target object. The frequency band coding of the reference speech and the subsequent speech reconstruction can be jointly trained end-to-end, so that the target object conditions are adapted to the speech reconstruction task itself.

[0046] It should be noted that the target object is not limited to human speakers, but can also be other sound source objects with identifiable acoustic features, such as specific musical instrument sound sources, specific device sound sources, or specific animal sound sources. In such cases, the reference speech is the reference audio segment of the sound source object, and the target object condition characterizes the acoustic identity of the sound source object. The frequency band division, condition extraction, and frame-by-frame recursion process of the embodiments in this specification are also applicable.

[0047] Other sound sources refer to all sound source components in mixed speech other than the target object. Specifically, other sound sources can include other speakers speaking simultaneously with the target object, background noise in the environment, and late-stage reverberation. Their spectral characteristics can overlap with the target object's spectrum in both time and frequency. For example, when two speakers have similar timbres and overlapping vocalization times, their harmonic structures are highly similar in the low-frequency region. Furthermore, the presence of other sound sources causes the mixed speech to exhibit a multi-component superposition in the spectrum. Speech reconstruction needs to preserve the target object components while suppressing other sound source components. Therefore, target object conditions need to continuously participate in the selection of acoustic features during frame-by-frame processing.

[0048] Mixed speech is an observed signal, including the speech of the target subject and other sound source components, acquired by audio acquisition devices in a real acoustic environment. Specifically, mixed speech can originate from the microphone array of an online conferencing terminal, the pickup channel of a call device, the audio track of a recording device, or a content production system. It can be a single-channel or multi-channel signal, with a sampling rate of, for example, 16kHz. At a 16kHz sampling rate, the effective frequency range of the audio signal is 0 to 8kHz. Mixed speech can arrive continuously frame by frame in streaming scenarios, for example, by framing with a window length of 512 points and a frame shift of 256 points. Furthermore, in streaming processing scenarios, the various time frames of mixed speech enter the processing flow sequentially according to time order. The processing does not access time frames that have not yet arrived. The algorithm latency is, for example, one analysis window length, i.e., 32ms, thus adapting to latency-sensitive applications such as online conferencing and real-time calls.

[0049] Reference speech is a pre-obtained speech segment of the target object, used to provide acoustic identity information of the target object. Specifically, the reference speech can be a clean speech segment actively recorded by the target object during the registration process, with a length of, for example, 3 seconds, or it can be extracted by the system from the target object's historical speech. The language content of the reference speech can differ from the mixed speech to be processed, so that the target object's conditions do not depend on specific language content, thereby completing the speech reconstruction of the target object under text-independent conditions. Furthermore, the information provided by the reference speech includes timbre information at the overall identity level and articulation and spectrogram information at the local level, which are used for the stable localization and detail recovery of the target object, respectively. The reference speech only participates in the determination of the target object's conditions before processing begins and does not participate in the calculation repeatedly in subsequent frame-by-frame processing, thereby avoiding the processing overhead from increasing with the duration of the reference speech.

[0050] In practical applications, obtaining mixed speech containing the target object and other sound sources, as well as the reference speech of the target object, can be achieved through various methods.

[0051] One alternative method is to have an audio acquisition device capture the mixed speech in real time and feed it into the processing flow frame by frame. Specifically, online conferencing terminals or communication devices can acquire observed signals from the acoustic environment through microphones, obtain the mixed speech after analog-to-digital conversion and pre-emphasis processing, and then divide it into frames according to preset window lengths and frame shifts. Each time frame is then fed into subsequent processing sequentially via a circular buffer. The reference speech can be recorded and buffered by the target subject before the meeting begins, or a clean speech segment can be extracted from the target subject's previous speech by the system as the reference speech. Through real-time acquisition and frame-by-frame feeding, the arrival rhythm of the mixed speech is aligned with the processing rhythm.

[0052] Another alternative method is for the processing device to read the recorded audio file from a storage medium or network, and then read the mixed speech frame by frame in chronological order. Specifically, the processing device can obtain the mixed speech data from the recording file, the audio track of the content production system, or the audio service on the network side, divide it into multiple time frames according to a preset window length and frame shift, and read the reference speech from a specified segment of the same audio file or from a separate reference speech file. During the reading process, the processing device can maintain reading points so that reading can resume from those points after an interruption. By reading the recorded audio, the sources of the mixed speech and the reference speech can be independent of each other.

[0053] Another alternative approach is to separately acquire the mixed speech and reference speech from multiple audio streams, depending on the application scenario. Specifically, in online conferencing scenarios, the mixed speech can be taken from the conference mixing channel, while the reference speech can be taken from the recording of the target speaker speaking alone. In interview and content creation scenarios, the mixed speech can be taken from the on-site audio pickup channel, while the reference speech can be taken from previously recorded audio material of the target speaker. By acquiring the two audio streams separately, it is possible to avoid the inclusion of other sound source components in the reference speech.

[0054] Furthermore, preprocessing of the mixed speech can be performed during the acquisition stage. For example, when the mixed speech is a multi-channel signal, time delay alignment and channel merging can be performed on the multi-channel signal to obtain a single-channel mixed speech; when the signal amplitude difference is large, amplitude normalization can be performed on the mixed speech and the reference speech respectively to bring them into the same dynamic range.

[0055] For example, let's further illustrate this with the example of speech enhancement for a designated speaker in an online meeting. Specifically, the mixed speech can be the entire audio segment of the meeting captured by the microphone of the meeting terminal, which includes the target speaker, other participants, and the noise from the meeting room's air conditioning and room reverberation; the reference speech can be a 3-second clean speech segment recorded by the target speaker before the meeting starts, whose language content is different from the statements to be processed in the meeting; each time frame arrives sequentially and enters the processing flow, and the state scale used in the processing is determined by the model configuration and does not increase with the duration of the meeting.

[0056] In this step, by acquiring mixed speech containing the target object and other sound sources, as well as the reference speech of the target object, a unified input basis is provided for subsequent frequency band division, condition extraction, and frame-by-frame recursion. This allows the two speech streams to be organized into a signal form that can be processed frame by frame within the same processing framework. Since the mixed speech carries observation information containing other sound source components, while the reference speech carries the acoustic identity information of the target object, the two speech streams correspond to the acoustic feature side and the condition side, respectively, in subsequent processing. This allows the determination of the target object and the modeling of acoustic content to be carried out collaboratively in the same process. By organizing the mixed speech by time frames, the processing can be executed immediately after each time frame arrives, providing a prerequisite for continuous processing in streaming scenarios.

[0057] Step 104: Divide the mixed spectral features of the mixed speech and the reference spectral features of the reference speech into frequency bands to obtain mixed frequency band features and reference frequency band features, and extract target object conditions from the reference frequency band features to characterize the acoustic identity of the target object.

[0058] Hybrid spectral features are complex representations of mixed speech in the time-frequency domain, carrying acoustic information of the mixed speech in both time and frequency dimensions. Specifically, hybrid spectral features can be obtained from the mixed speech through time-frequency transformation. They contain real and imaginary parts and have multiple frequency points on the frequency axis. For example, under a sampling rate of 16kHz and a window length of 512 points, the number of non-repeating frequency points can be 257, with a frequency point interval of 31.25Hz. Each time frame of the hybrid spectral feature corresponds one-to-one with each analysis window of the mixed speech and can serve as a reference for subsequent signal reconstruction. That is, after obtaining the target spectrum, the masking operation related to the target spectrum is performed on the hybrid spectral features. Furthermore, the hybrid spectral features retain amplitude and phase information, allowing amplitude and phase to be corrected simultaneously during the reconstruction stage, thereby improving the reconstruction quality of the target speech.

[0059] Reference spectral features are complex representations of reference speech in the time-frequency domain, used to carry the acoustic identity information of the target object. Specifically, reference spectral features can be obtained from the reference speech through the same time-frequency transformation as the mixed speech, and its frequency axis structure is consistent with the mixed spectral features, containing the same number of frequency points and frequency intervals. The time length of the reference spectral features is determined by the duration of the reference speech; for example, 3 seconds of reference speech corresponds to several time frames under a frame shift of 256 points. Furthermore, reference spectral features can provide two types of information: one is the timbre information collectively reflected in the entire reference speech, and the other is the articulation and spectral information reflected in local time segments. Therefore, its processing accuracy directly affects the distinguishability of the target object under conditions of similar timbre.

[0060] Hybrid frequency band features are feature representations obtained by dividing hybrid spectral features into frequency bands, using frequency bands as the unit. Specifically, hybrid frequency band features can have multiple frequency bands in the frequency band dimension, with each frequency band corresponding to a continuous frequency point in the hybrid spectral features. For example, 257 frequency points can be divided into 36 non-uniform frequency bands. Hybrid frequency band features can maintain a time-frame-by-time, frequency-band-by-frequency structure, meaning that in each time frame, there exists a feature component that corresponds one-to-one with each frequency band, serving as input for subsequent frame-by-frame recursion and inter-band modeling. Furthermore, since hybrid frequency band features are organized by frequency band, frame-by-frame recursion can be performed separately for each frequency band and its hidden state can be maintained, thereby reducing the sequence length while preserving the frequency band-level modeling granularity.

[0061] Reference frequency band features are feature representations obtained by dividing reference spectral features into the same frequency bands, using frequency bands as the unit. Specifically, reference frequency band features can use the same number of frequency bands and frequency band boundaries as mixed frequency band features, ensuring that the information provided by the reference speech and the acoustic features of the mixed speech correspond to each other on the same frequency band. Reference frequency band features can also use normalization and projection parameters shared with the mixed spectral feature parameters to reduce the number of parameters and preserve the frequency band-wise time-frequency structure of the reference speech. This allows them to be used to extract invariant conditions for the entire segment and to support retrieval based on the current time frame. Furthermore, the consistency in frequency band structure between reference frequency band features and mixed frequency band features allows the target object conditions extracted from the reference frequency band features to be applied to the determination of recursive parameters on the same frequency band.

[0062] Target object conditions are conditional information used to characterize the acoustic identity of a target object. They originate from reference frequency band features and influence the determination of recursive parameters in subsequent frame-by-frame recursion. Specifically, target object conditions can be in vector form, i.e., a fixed-length representation that remains unchanged throughout the entire processing; or in sequence form, i.e., a representation that provides local acoustic information as the current time frame changes; their dimension can be the same as the channel dimension of the frequency band features, or it can be an independent preset dimension. Furthermore, target object conditions do not replace the mixed frequency band features. The mixed frequency band features still retain their frame-by-frame, frequency band structure as the backbone input, and the target object conditions participate as conditional signals in the determination of recursive parameters, thus ensuring that the selection of the target object occurs within the recursive process rather than only during the fusion stage of input or output features.

[0063] In practical applications, frequency band division of mixed spectral features and reference spectral features to obtain mixed frequency band features and reference frequency band features can be achieved through various methods.

[0064] One possible approach is to divide the frequency bands non-uniformly according to the information density of different frequency regions. Specifically, the amount of information carried in each frequency region can be statistically analyzed, using narrower bands in frequency regions with higher information density and wider bands in those with lower information density. For example, the low-frequency region carries the fundamental frequency and low-order harmonics, so a narrower band can be used to preserve them; the mid-frequency region carries formants and intelligibility information, so a medium-width band can be used; and the high-frequency region mainly consists of noise and consonant components, so it can be merged into a wider band to reduce the length of the band sequence. Through non-uniform partitioning, the computational load can be allocated according to the information density of the frequency regions.

[0065] Another alternative approach is to employ a learnable partitioning method for the number and boundaries of frequency bands. Specifically, the frequency axis can be divided into multiple initial frequency bands, and learnable boundary and merging parameters can be set for each band. During training, the boundary positions and merging methods of each band are updated based on the speech reconstruction target, ensuring that the resulting frequency band structure adapts to the acoustic characteristics of the training data. Through this learnable partitioning method, the frequency band structure can be determined data-driven.

[0066] Another alternative approach is to adaptively determine the frequency band division based on the acoustic scenario. Specifically, a preset frequency band division scheme can be dynamically selected based on the frequency distribution characteristics of the current signal, such as the proportion of low-frequency energy and the intensity of noise distribution in the high-frequency region. Alternatively, the frequency band boundaries can be merged and subdivided to further compress the number of high-frequency bands in scenarios with strong noise or limited high-frequency information. Through adaptive division, the frequency band structure can maintain consistent processing granularity and computational overhead under different acoustic environments.

[0067] Furthermore, the above methods can be combined. For example, a basic partitioning scheme can be determined first based on information density, and then the high-frequency bands can be merged or subdivided according to the acoustic scenario during operation. Regardless of the partitioning method used, the mixed spectral features and the reference spectral features use the same set of band partitioning results to ensure that their band structures correspond to each other.

[0068] In practical applications, extracting target object conditions from reference frequency band features to characterize the acoustic identity of the target object can also be achieved through various methods.

[0069] One alternative approach is to aggregate the reference frequency band features along both the frequency band and time dimensions, and then project the aggregation result. Specifically, the values ​​of the reference frequency band features in each frequency band and time frame can be aggregated to obtain a fixed-length aggregated representation. This representation is then mapped to a preset dimension using linear projection or a multilayer perceptron (MLP) to obtain the target object conditions that remain unchanged throughout the entire processing. By aggregating along two dimensions, the target object conditions can stably characterize the overall timbre of the target object.

[0070] Another alternative approach is to preserve the band-by-band time-frequency structure of the reference band features and then aggregate them after transformation to obtain the target object conditions. Specifically, the reference band features can be segmented along the time dimension, and each segment can be aggregated separately before fusion. Alternatively, the time-frequency structure of the reference band features can be encoded using a convolutional structure and an attention mechanism before aggregation, so that the target object conditions can preserve the overall timbre information while also taking into account the statistical characteristics of local pronunciation.

[0071] Another alternative approach is to perform weighted aggregation based on the reliability of each component of the reference frequency band features. Specifically, weighting coefficients can be determined based on the energy intensity, signal-to-noise ratio (SNR) estimation, or confidence level related to the target object for each frequency band or time frame. Higher values ​​are assigned greater weights, thereby suppressing the influence of potential silence and noise segments in the reference speech on the target object conditions. Through weighted aggregation, the extraction process of target object conditions becomes more adaptable to quality fluctuations in the reference speech.

[0072] Furthermore, when there are multiple segments of reference speech, conditions can be extracted from each segment separately, and then the conditions from each segment can be merged. For example, after obtaining the condition representations from multiple segments of reference speech, they can be averaged or weighted by reliability to obtain more stable target object conditions when the quality of multiple segments of reference speech is inconsistent.

[0073] For example, taking a sampling rate of 16kHz, a window length of 512 points, and a frame shift of 256 points as an example, the mixed speech, after time-frequency transformation, yields a mixed spectral feature with 257 non-repetitive frequency points. This feature can be divided into 36 non-uniform frequency bands. The width of each frequency band, in terms of frequency points, is as follows: 8 bands with a width of 2, 8 bands with a width of 4, 6 bands with a width of 8, 13 bands with a width of 11, and 1 band with a width of 18. The total number of frequency bands is 36, and the total number of frequency points is 257. With a frequency point interval of 31.25Hz, this division corresponds to the preservation of the fundamental frequency and low-order harmonics from 0 to 0.5kHz, the modeling of low-order formants and timbre contours from 0.5 to 1.5kHz, the coverage of the main intelligibility information from 1.5 to 3kHz, and the modeling of the frequency organization of high-frequency consonants and noise components from 3 to 8kHz.

[0074] For the k-th frequency band, the real and imaginary parts of the mixed spectral features can be concatenated along the channel dimension. After normalization and linear projection, the frequency band features of dimension D can be obtained, and their expression can be:

[0075] in, Indicates the first The frequency band in the first Complex spectrum over time frames and These represent taking the real part and taking the imaginary part, respectively. Indicates channel-dimensional splicing. Indicates the first Normalization operation for each frequency band Indicates the first A linear projection of each frequency band, with an input dimension of... , The number of frequency points in this frequency band, with the output dimension being... , Indicates the first The frequency band in the first Frequency band features across time frames. Each frequency band can use its own independent normalization and projection parameters to accommodate the differences in dynamic range between different frequency bands; the reference speech can use the same or parameter-shared frequency band coding structure to obtain reference frequency band features.

[0076] In an alternative embodiment, the number of frequency bands and the frequency band boundaries can be set to fixed, learnable, or adaptively determined according to the acoustic scenario, and the same set of frequency band division results and frequency band coding parameters can be shared between the reference speech and the mixed speech. This ensures that the number of parameters used to generate recursive parameters remains unchanged when the frequency band scheme is replaced, thereby facilitating the scaling of the model according to the computing power, bandwidth, and latency requirements of the end-side.

[0077] In this step, by dividing the hybrid spectral features and the reference spectral features into frequency bands, the two features correspond to each other under the same frequency band structure, providing a basis for conditional information to act on the hybrid frequency band features at the frequency band level. By organizing the frequency axis from the frequency point level to the frequency band level, the sequence length of the spectrum modeling is compressed, and the number of units to be processed in frame-by-frame recursion is reduced accordingly, thereby reducing the amount of computation and cache usage. By performing non-uniform division according to information density, the amount of computation can be distributed according to the information density of each frequency region, reducing high-frequency redundant computation while preserving the fundamental frequency, harmonics and timbre clues in the low-frequency region.

[0078] Step 106: For any time frame of the mixed speech, based on the mixed frequency band features and the target object conditions, determine the recursive parameters of the state space model for that time frame, and recursively determine the output features of that time frame according to the recursive parameters.

[0079] A time frame is a processing unit defined on the timeline of mixed speech, and it is also the basic temporal granularity for frame-by-frame recursion. Specifically, a time frame can be obtained by dividing the mixed speech into frames according to a preset window length and frame shift, such as a window length of 512 points, a frame shift of 256 points, and an overlap rate of 50% between adjacent time frames. Each time frame corresponds to the acoustic content of the mixed speech within an analysis window. In a streaming scenario, each time frame arrives sequentially in chronological order, and the processing procedure performs a recursion independently for each time frame. Therefore, the processing result is independent of the specific position of the time frame in the speech. Furthermore, since the recursion process only uses the state determined by the current time frame and previous time frames, the processing procedure does not access time frames that have not yet arrived, thus supporting continuous processing while maintaining causality.

[0080] A state-space model is a model that models sequence dependencies through implicit state recursion. Specifically, a state-space model can include states that carry historical information, parameters that control information writing to states, parameters that control information reading from states, and parameters that control the time scale and forgetting level of states. Its state dimension is determined by the model configuration and does not increase with the length of the processed sequence. The state-space model performs one state update and one readout at each time frame, thereby modeling long-term dependencies in a frame-by-frame recursive manner. Furthermore, the state update rule of the state-space model is determined by the recursive parameters; therefore, whether the recursive parameters are determined based on the target object conditions directly determines whether the state update process has selectivity at the target object level. In the embodiments of this specification, the recursive parameters are jointly determined based on mixed frequency band characteristics and target object conditions, so that the selection of the target object occurs within the state update process.

[0081] Recursive parameters are used in the recursive process of the state-space model to determine how information from the current time frame is written into and read from the state, as well as the effective memory length of the state on the time axis. Specifically, recursive parameters can include parameters related to information writing, parameters related to information reading, and parameters related to the state's time scale. These parameters can be determined for each time frame or for each frequency band within each time frame. Their values ​​are jointly determined by the mixed frequency band characteristics and target object conditions of the current time frame. Therefore, in time frames and frequency bands where the target object component is dominant, the state can retain more information consistent with the target object; in time frames and frequency bands where the interference component is dominant, the effective memory length of the state can be correspondingly shortened. Furthermore, recursive parameters and states are distinct: recursive parameters determine how the state is updated, while the state carries historical information from processed time frames; together, they determine the output characteristics of the current time frame.

[0082] The output features are the feature representations read from the state-space model in the current time frame, used to characterize the acoustic content of the target object in that time frame. Specifically, the output features can be organized by frequency band, meaning that there are output features corresponding one-to-one with each frequency band in each time frame; their dimension can be the same as the channel dimension of the frequency band features, or it can be another corresponding dimension; the output features are read from the updated state and can be determined by combining the acoustic features of the current time frame, thus preserving the historical information carried by the state while taking into account the real-time information of the current time frame. Furthermore, the output features are generated frame by frame in chronological order, and the output features of each time frame together constitute a feature sequence for signal reconstruction, providing a foundation for subsequent organization of masks by frequency band and reconstruction of mixed spectral features.

[0083] In practical applications, for any given time frame, the recursive parameters of the state-space model for that time frame can be determined using various methods based on the hybrid frequency band characteristics and the conditions of the target object.

[0084] One alternative approach is to use the mixed frequency band features of the current time frame and the target object conditions as common inputs, mapping them to obtain the recursive parameters for that time frame. Specifically, the inputs can be dimensionally aligned and combined into a set of input representations. These input representations then simultaneously determine the values ​​of various parameters related to information writing, information reading, and the state time scale, ensuring that the parameters are mutually matched. By jointly determining each set of parameters, the values ​​of the recursive parameters can simultaneously reflect the acoustic content of the current time frame and the acoustic identity of the target object.

[0085] Another alternative approach is to first determine the basic parameters based on the mixed frequency band characteristics of the current time frame, and then modulate these basic parameters using the target object conditions. Specifically, a set of basic parameters can be generated according to the acoustic content of the current time frame, and then scaled, biased, or gated using modulation coefficients generated by the target object conditions to obtain the recursive parameters for that time frame. By determining the basic parameters first and then performing conditional modulation, the scope of the conditional signal can be limited to the parameter level.

[0086] Another alternative approach is to determine the recursive parameters for each frequency band within each time frame. Specifically, for the k-th frequency band in the t-th time frame, a set of recursive parameters can be determined based on the mixed frequency band characteristics of that frequency band and the target object conditions corresponding to that time frame, so that different frequency bands have their own parameter values ​​in the same time frame. By determining the recursive parameters band by band, the state update rhythm of each frequency band can be independent of each other.

[0087] Furthermore, recursive parameters can be determined in real-time according to the arrival order of time frames during streaming processing. For example, after each time frame arrives, the recursive parameters for that time frame are determined first, and then used in the state update of the current time frame, without waiting for subsequent time frames to arrive, thus keeping the determination process of recursive parameters consistent with the streaming processing rhythm.

[0088] In practical applications, the output characteristics of a time frame can be determined recursively according to the recursive parameters, which can also be achieved through various methods.

[0089] One alternative approach is to first update the state corresponding to the time frame based on recursive parameters, and then read the output features from the updated state. Specifically, the recursive process can write the acoustic information of the current time frame into the state according to the parameters related to information writing, determine the degree of retention of historical information in the current time frame according to the parameters related to the state time scale, and read the feature components related to the target object from the updated state according to the parameters related to information reading, using the read results as the output features of that time frame. By updating the state before reading, historical information can be continuously accumulated in the state and selectively read out according to the target object.

[0090] Another alternative approach is to control the time scale of the state based on recursive parameters, and then perform state updates and feature readouts accordingly. Specifically, in time frames and frequency bands where the target object component is dominant, the effective memory length of the state can be extended, allowing more historical information consistent with the target object to be retained; in time frames and frequency bands where the interference component is dominant, the effective memory length can be shortened, allowing interference information already in the state to be forgotten more quickly. By controlling the time scale with recursive parameters, the influence of interference components on the state can be limited to a finite time range.

[0091] Another alternative approach is to maintain the state and perform recursion separately for each frequency band. Specifically, for each frequency band within the same time frame, its own state can be maintained. Each frequency band independently completes state updates and readouts based on its own recursive parameters, and then the readout results of each frequency band are combined to form the output features of that time frame. By maintaining the state independently for each frequency band, the states of each frequency band do not interfere with each other.

[0092] In addition, the output features of each time frame can be cached in chronological order. For example, after obtaining the output features of each time frame, the processing device can write them into the cache according to the correspondence between frequency bands and time frames, so that they can be used by the signal reconstruction stage according to the frequency band boundaries. The cache content only includes the output features and status, and there is no need to save the input data of historical time frames.

[0093] For example, the processing procedure in a streaming scenario will be further explained. Specifically, after each time frame of the mixed speech arrives, the mixed frequency band features of that time frame are obtained as described above, and the recursive parameters of that time frame are determined in combination with the determined target object conditions. Then, the state corresponding to that time frame is updated and read out according to the recursive parameters to obtain the output features of that time frame. The above process is repeated after each time frame arrives. The state used when processing the t-th time frame is only the state obtained by recursion from the previous time frames. The processing does not need to save the input data of historical time frames. The scale of the maintained state is determined by the model configuration and does not increase with the duration of the processed speech.

[0094] In this step, the recursive parameters of the state-space model for any given time frame are determined based on the hybrid frequency band characteristics and the target object conditions. This allows the target object conditions to directly participate in the determination of the recursive parameters, thereby enabling the selection of the target object to be integrated into the state update process. By enhancing the writing and reading of information related to the target object in time frames and frequency bands where the target object component is dominant, and shortening the effective memory length of the state in time frames and frequency bands where the interference component is dominant, the contamination of the state by the interference component can be limited to a finite time range, thereby reducing the accumulation of the interference component along the time axis. By recursively determining the output features according to the recursive parameters, the historical information carried by the state can be read out selectively according to the target object, providing a feature sequence of suppressed interference for subsequent signal reconstruction.

[0095] Step 108: Reconstruct the signal from the mixed spectral features based on the output features of each time frame to obtain the target speech of the target object.

[0096] The target speech is the output of this speech reconstruction, that is, the speech signal obtained after preserving the speech components belonging to the target object and suppressing other sound source components in the mixed speech. Specifically, the target speech can be a time-domain waveform signal with a sampling rate consistent with the mixed speech, such as 16kHz; the duration of the target speech corresponds to the duration of the mixed speech, and in streaming processing scenarios, it can be output sequentially according to time frames to form a continuous speech stream; the target speech can be used for speaker voice playback and transcription in online meetings, target user voice interaction in voice assistants, generation of specified speaker audio tracks in content creation, and can also be provided as a preprocessing result for downstream tasks such as speech recognition and speaker verification. Furthermore, the acquisition of the target speech depends on the correspondence between the output features and the mixed spectral features of each time frame. The output features provide the acoustic content related to the target object, and the mixed spectral features provide the amplitude and phase references required for reconstruction. The two are combined under the same frequency band structure, so that the reconstruction result retains the target object components while having a spectral organization consistent with the original observation.

[0097] In practical applications, the target speech of the target object can be obtained by reconstructing the mixed spectral features based on the output features of each time frame through various methods.

[0098] One alternative approach is to reconstruct the target speech in the frequency domain. Specifically, based on the correspondence between time frames and frequency bands, the output features of each time frame are converted into a mask that acts on the mixed spectral features. This mask is then combined point-by-point with the mixed spectral features in the frequency domain to obtain the target spectrum of the target object. An inverse time-frequency transform is then performed on the target spectrum, and the two signals are overlapped and added (OLA) according to the overlap relationship of the time frames to obtain the target speech in the time domain. Through frequency domain reconstruction, the reconstruction process can directly utilize the amplitude and phase information carried by the mixed spectral features.

[0099] Another alternative approach is to reconstruct the target speech based on frequency band organization. Specifically, the output features corresponding to each frequency band can be organized into frequency band-level reconstruction results according to the frequency band boundaries. Then, the frequency band-level reconstruction results of each frequency band are spliced ​​together in frequency order to obtain a reconstructed spectrum covering the entire frequency range. Subsequently, the target speech is obtained by inverse time-frequency transformation and overlapping. By organizing the reconstruction by frequency band, each frequency band can use its own projection parameters and reconstruction method.

[0100] Another alternative method is to combine target activity information to complete the reconstruction and obtain the target speech. Specifically, during the reconstruction process, it can be determined whether the target object is speaking in the current time frame based on target activity information that is from the same source as the output features. The output can be attenuated or zeroed in time frames where the target object is not speaking, thereby suppressing residual output during periods when the non-target object is speaking. By combining target activity information, the reconstruction result can maintain low residual energy in silent segments and periods where only other sound sources are speaking.

[0101] In addition, the target speech can be obtained in a streaming manner. For example, the processing device can output the corresponding speech segment after reconstructing several time frames, so that the target speech is continuously output according to the arrival rhythm of the time frames, without having to wait for the entire mixed speech to be processed.

[0102] For example, let's take the voice enhancement of a designated speaker in an online meeting as an example for further explanation. Specifically, the microphone of the meeting terminal continuously collects mixed speech, and each time frame enters the processing flow in sequence to obtain output features. After organizing the output features of each time frame by frequency band and combining them with the mixed spectrum features, the target spectrum corresponding to that time frame can be obtained. Performing inverse time-frequency transformation and overlapping addition on the target spectrum of multiple consecutive time frames can obtain the speaker's continuous speech, whose sampling rate is consistent with the mixed speech, for example, 16kHz. The output target speech can be directly sent to the meeting transcription module for further processing.

[0103] In an optional embodiment, the speech reconstruction method provided in this specification can also be deployed on a cloud server, an edge server, or a low-power edge device. When deployed on the edge, since the processing is performed frame-by-frame and the scale of the maintained state is fixed, the memory required for reconstruction does not increase with call duration, making it suitable for continuous execution on devices that run online for extended periods. When deployed in the cloud or at the edge, the same processing flow can be executed in parallel for multiple mixed speech streams, and can simultaneously serve multiple target objects through batch dimension or target dimension expansion.

[0104] In this step, the mixed spectral features are reconstructed based on the output features of each time frame. This reorganizes the feature sequences selectively read out according to the target object into a spectral form with the same frequency band structure as the original observation. As a result, the reconstruction result retains the acoustic content of the target object while maintaining the frequency organization consistent with the observation. By using the mixed spectral features as the reconstruction benchmark, the reconstruction process can utilize both amplitude and phase information, thereby improving the continuity of the target speech timbre and the listening quality. By completing the reconstruction sequentially according to the time frames, the target speech can be continuously output as the processing progresses, adapting to scenarios requiring streaming output, such as online conferences and real-time calls.

[0105] In the embodiments of this specification, by acquiring mixed speech containing the target object and other sound sources, as well as reference speech of the target object, a unified input basis is provided for subsequent processing; by dividing the mixed spectral features and reference spectral features into frequency bands respectively, the two features are aligned under the same frequency band structure, providing a basis for the extraction of conditional information and frequency band-by-frequency processing; by extracting target object conditions used to characterize the acoustic identity of the target object from the reference frequency band features, the identity information of the target object is preserved in a frequency band-level form; by determining the recursive parameters of the state space for any given time frame based on the mixed frequency band features and target object conditions, and following the recursive... The parameters are recursively determined to identify the output features of the time frame, allowing the target object condition to participate in the frame-by-frame recursive process. This continuously strengthens the components related to the target object and suppresses interference components, improving the extraction accuracy and reconstruction quality of the target speech. By reconstructing the mixed spectral features based on the output features of each time frame, the target speech of the target object is obtained. The entire processing is performed frame by frame and does not depend on future frames. The maintained state scale is fixed, making it easy to adapt to application scenarios requiring real-time processing and low-cost deployment. Based on the reconstructed target speech, it can provide subsequent speech recognition with less interference and higher quality speech input, thereby improving the accuracy of speech recognition.

[0106] In an optional embodiment of this specification, after acquiring the mixed speech including the target object and other sound sources, and the reference speech of the target object, the method further includes: Time-frequency transformations are performed on the mixed speech and the reference speech of the target object, respectively, to obtain the mixed spectral features and the reference spectral features.

[0107] Time-frequency transformation (TF-F) is a processing method that converts time-domain speech signals into frequency-domain representations. It is used to convert time-varying speech waveforms into complex spectra that simultaneously contain time and frequency information. Specifically, TF-F can perform frame-by-frame transformation on the speech signal after windowing, obtaining a complex spectrum that varies with time frames and frequency indices. Each time frame corresponds to the acoustic content of the speech signal within an analysis window, and each frequency index corresponds to a frequency point. Taking a sampling rate of 16kHz, a window length of 512 points (32ms), and a frame shift of 256 points (16ms) as an example, the overlap rate between adjacent time frames can be 50%, resulting in 257 non-repeating frequency points with a frequency interval of 31.25Hz and an effective frequency range of 0 to 8kHz. It can also output both real and imaginary parts, preserving amplitude and phase information in both the mixed spectral features and the reference spectral features.

[0108] Furthermore, the time-frequency transformation and the length of the analysis window determine the time granularity of the processing and the algorithm delay. Without accessing future time frames, the algorithm delay can be the length of an analysis window, such as 32ms. The mixed speech and the reference speech use the same time-frequency transformation parameters, making their frequency axis structures completely consistent, which provides a prerequisite for subsequent frequency band division and condition extraction under the same frequency band structure.

[0109] In practical applications, time-frequency transformations are performed on the mixed speech and the reference speech of the target object to obtain the mixed spectral features and the reference spectral features, respectively. This can be achieved through various methods.

[0110] One alternative approach is to perform a Short-Time Fourier Transform (STFT) on each of the two speech streams. Specifically, the mixed speech and the reference speech can be windowed and framed according to a preset window length and frame shift. The window function can be a Hanning window or a Hamming window. A Discrete Fourier Transform is performed on each time frame to obtain the complex spectrum of that time frame. With a sampling rate of 16kHz and a window length of 512 points, each time frame yields 257 non-repeating frequency points, each corresponding to a frequency interval of 31.25Hz. A frame shift of 256 points ensures that the waveforms of adjacent time frames overlap by 50%. Through the STFT, the two speech streams can obtain complex representations containing amplitude and phase on the same frequency axis.

[0111] Another alternative approach is to organize and store the transformation results separately for their real and imaginary parts. Specifically, after obtaining the complex spectrum, the real and imaginary parts can be saved as independent channels, allowing subsequent frequency band coding to directly concatenate the real and imaginary parts at the channel dimension. For the reference speech, the real and imaginary parts can also be cached sequentially by time frame for reuse when extracting target object conditions. By organizing the real and imaginary parts separately, the channel layout of the complex information remains consistent in subsequent processing.

[0112] Another alternative approach is to incrementally perform time-frequency transformation according to the arrival order of time frames. Specifically, in a streaming scenario, the processing device can maintain a buffer with a length equal to the analysis window. When a new sample point arrives, the previously arrived sample points are removed and the newly arrived sample points are written in, thus forming the analysis window for the current time frame. Performing a transformation on this analysis window yields the spectrum of the current time frame. Since there is overlap between adjacent time frames, only a portion of the sample points need to be updated each time, thereby reducing redundant calculations. By incrementally performing the time-frequency transformation, the transformation process is aligned with the streaming processing rhythm.

[0113] Furthermore, the transformed spectrum can be amplitude normalized before output. For example, normalization can be performed separately according to the overall amplitude levels of the reference speech and the mixed speech, so that the two spectra are within the same dynamic range, thereby avoiding the impact of amplitude differences on the stability of subsequent frequency band coding and condition extraction.

[0114] For example, a conference audio sampled at 16kHz is used as an example for further explanation. Specifically, the mixed speech is framed with a window length of 512 points and a frame shift of 256 points. Each time frame is subjected to a short-time Fourier transform to obtain 257 non-repeating frequency points, with a frequency range of 0 to 8kHz and a frequency point interval of 31.25Hz. The reference speech is transformed using the same window length, frame shift, and frequency axis parameters, and the spectra of the two speech streams are aligned point by point on the frequency axis.

[0115] In an alternative embodiment, the time-frequency transform can be replaced with a learnable filter bank, a subband analysis filter bank, or a waveform encoder, so that the method of obtaining spectral features is not limited to short-time Fourier transform. Since the determination of recursive parameters and the state update process in the embodiments of this specification do not depend on the specific transform form, the processing flow of frequency band division, condition extraction, and frame-by-frame recursion can remain unchanged after replacing the front-end transform method.

[0116] In the embodiments of this specification, by performing time-frequency transformations on the mixed speech and the reference speech respectively, the two speech streams obtain complex representations containing amplitude and phase under the same frequency axis structure, providing a unified input basis for subsequent frequency band division and condition extraction under the same frequency band structure; by using the same transformation parameters, the acoustic identity information provided by the reference speech corresponds point-by-point with the mixed frequency band features of the mixed speech on the frequency axis, enabling the target object conditions to act on the mixed frequency band features at the frequency band level; by performing the transformation on a time frame basis and not accessing future time frames, the algorithm latency is limited to within an analysis window length, making the processing adaptable to scenarios with streaming processing requirements such as online conferencing and real-time calls.

[0117] In one optional embodiment of this specification, frequency band division is performed on the mixed spectral features of the mixed speech and the reference spectral features of the reference speech to obtain mixed frequency band features and reference frequency band features, including: Based on the information density of the mixed spectral features in different frequency regions, the mixed spectral features of mixed speech are divided into multiple frequency bands to obtain mixed frequency band features; The reference spectral features of the reference speech are divided into multiple frequency bands to obtain the reference frequency band features; Extract target object conditions from the reference frequency band features to characterize the acoustic identity of the target object, including: Pooling is performed on the reference frequency band features along the frequency band dimension and the time dimension to obtain the conditions of the target object.

[0118] A frequency region is a continuous range of frequencies in the spectrum, divided in descending order of frequency, used to distinguish frequency components carrying different acoustic information. Specifically, frequency regions can be divided into low-frequency, mid-frequency, and high-frequency regions based on the information distribution characteristics of the speech spectrum. For example, under conditions of a 16kHz sampling rate, 257 frequency points, and a frequency point interval of 31.25Hz, the low-frequency region can be 0 to 0.5kHz, the mid-frequency region can be 0.5 to 1.5kHz and 1.5 to 3kHz, and the high-frequency region can be 3 to 8kHz. Furthermore, the speech information carried by different frequency regions differs in type and concentration. The low-frequency region carries the fundamental frequency and low-order harmonics, closely related to the timbre and identity cues of the target object; the mid-frequency region carries low-order formants and main intelligibility information; and the high-frequency region contains high-frequency consonants, sibilants, and noise components. Since bandwidth is allocated in units of frequency regions, the division of frequency regions directly determines the fineness of the frequency band structure in each frequency segment.

[0119] Information density is the amount of information carried per unit frequency range that contributes to the expression of speech content and the identification of the target object. It is used to measure the importance of each frequency region in the spectrum in modeling. Specifically, information density can be reflected by the correlation of adjacent frequency values, the richness of harmonic structure, the number of formants, and the contribution of the frequency range to intelligibility and identification. For example, in the low-frequency region, the fundamental frequency and multiple harmonics make the values ​​of adjacent frequency points highly correlated, and there are more identity clues that can be extracted per unit frequency range, resulting in a correspondingly high information density. In the high-frequency region, the energy distribution is relatively flat and dominated by consonants and noise components, resulting in a correspondingly low information density. Furthermore, information density is the basis for allocating bandwidth. Frequency regions with high information density can use narrower bandwidths to preserve details, while frequency regions with low information density can be merged into wider bandwidths to shorten the bandwidth sequence. Information density can be determined by statistical quantities such as bandwidth energy and harmonicity, or it can be preset based on the known distribution characteristics of the speech spectrum.

[0120] The frequency band dimension is the dimension in the frequency band feature that corresponds to the frequency band index, used to organize the feature components of each frequency band in the same time frame. Specifically, the frequency band feature has multiple components in the frequency band dimension, and the number of components is equal to the number of frequency bands. For example, after dividing 257 frequency points into 36 non-uniform frequency bands, both the mixed frequency band feature and the reference frequency band feature have 36 components in the frequency band dimension. Each component corresponds to the feature representation of a continuous frequency point in the mixed spectrum feature or the reference spectrum feature. Furthermore, the frequency band dimension and the time dimension together constitute the two-dimensional structure of the frequency band feature. The frequency band dimension reflects the harmonic, formant, and spectral envelope relationships between different frequency components at the same time. Pooling along the frequency band dimension means eliminating the differences in frequency band positions, merging the information carried by each frequency band into a representation that is independent of the frequency band index.

[0121] The temporal dimension is the dimension in the frequency band feature that corresponds to the time frame index, used to organize the feature components across each time frame. Specifically, the frequency band feature has multiple components in the temporal dimension, the number of which equals the number of time frames already defined. For example, under a frame shift of 256 points (16ms), each time frame corresponds to one component of the frequency band feature in the temporal dimension. During streaming processing, the components in the temporal dimension are obtained sequentially according to their chronological order, and the processing does not access time frames that have not yet arrived. Furthermore, the temporal dimension carries the temporal changes in speech, including the start and end of pronunciation, transitions between phonemes, and fluctuations in speech style; pooling along the temporal dimension means eliminating differences in temporal position.

[0122] Pooling is an operation that aggregates tensors along a specified dimension to compress or eliminate differences in the components along that dimension. It is used to transform component-wise varying feature representations into coarser-grained representations. Specifically, pooling can include forms such as mean pooling, peak pooling, weighted pooling, and attention pooling. The dimension in which it operates is set by the operation. For example, performing mean pooling simultaneously along both the frequency band and time dimensions can aggregate frame-by-frame, frequency band-by-frequency two-dimensional feature representations into a fixed-length representation that does not change with either the frequency band index or the time frame index. Furthermore, pooling does not change the acoustic meaning carried by the features, but only changes the granularity of the feature organization. On the reference speech side, pooling is used to eliminate differences between specific frequency band positions and specific time positions, so that the acoustic identity information embodied in the reference speech is aggregated into a stable conditional representation.

[0123] In practical applications, based on the information density of the mixed spectral features in different frequency regions, the mixed spectral features of mixed speech can be divided into multiple frequency bands to obtain mixed frequency band features, which can be achieved through various methods.

[0124] One alternative approach is to allocate bandwidth according to the order of information density from high to low, using narrower bandwidths in frequency regions with higher information density and wider bandwidths in frequency regions with lower information density.

[0125] Specifically, the 257 frequency points can be divided into 36 non-uniform frequency bands. The width of each band, in units of frequency points, is as follows: 8 bands with a width of 2, 8 bands with a width of 4, 6 bands with a width of 8, 13 bands with a width of 11, and 1 band with a width of 18. The number of frequency bands is 8 + 8 + 6 + 13 + 1, totaling 36. The number of frequency points is 16 + 32 + 48 + 143 + 18, totaling 257. With a frequency point interval of 31.25Hz, the above bandwidth distribution corresponds to the frequency organization of preserving the fundamental frequency and low-order harmonics from 0 to 0.5kHz, modeling low-order formants and timbre contours from 0.5 to 1.5kHz, covering the main intelligibility information from 1.5 to 3kHz, and modeling high-frequency consonants, fricatives, and noise components from 3 to 8kHz. The three dividing frequencies can fall on the cumulative 16th, 48th, and 96th frequency points.

[0126] Another alternative approach is to employ a learnable partitioning method for the number and boundaries of frequency bands. Specifically, the frequency axis can be divided into multiple initial frequency bands, and learnable width parameters, boundary parameters, or merging parameters can be set for each band. During training, the boundary positions and merging methods of the frequency bands can be updated according to the optimization goal of speech reconstruction. For example, 257 frequency points can be evenly divided into multiple candidate frequency bands, and adjacent candidate frequency bands can be merged through a learnable merging operation, so that the frequency band structure formed after partitioning adapts to the acoustic characteristics in the training data.

[0127] Another alternative approach is to adaptively determine the frequency band division according to the acoustic scenario. Specifically, the frequency band boundaries can be determined based on the frequency distribution characteristics of the mixed speech, such as the proportion of low-frequency energy, the noise distribution intensity in the high-frequency region, or the frequency distribution range of speech activity. One of the preset division schemes can be selected, and the determined frequency band boundaries can be merged and subdivided. For example, in scenarios with strong noise or little high-frequency information, the frequency bands in the high-frequency region can be merged, while in scenarios with abundant low-frequency information or where the interference source and the target object have similar timbre, the frequency bands in the low-frequency region can be subdivided to retain more identity clues, so that the processing granularity and computational overhead are adapted to the acoustic environment.

[0128] In addition, a fixed number of frequency bands and frequency band boundaries can be used, that is, the width of each frequency band is determined at once according to the pre-set dividing frequency point, and no adjustment is made during the processing; or the above methods can be combined, for example, the basic frequency band division scheme is determined first based on the information density, and some frequency bands are merged or subdivided according to the acoustic scene during the operation.

[0129] After obtaining the mixed frequency band features, the reference spectral features of the reference speech can be divided according to the same frequency band structure to obtain reference frequency band features that correspond to the mixed frequency band features in terms of frequency band. In practical applications, dividing the reference spectral features of the reference speech into multiple frequency bands to obtain reference frequency band features can also be achieved through various methods.

[0130] One alternative approach is to use the same number of frequency bands, frequency band boundaries, and frequency band arrangement order obtained from dividing the mixed frequency band features when defining the reference spectral features. Specifically, the same set of frequency band division results can be directly applied to the reference spectral features, making the number of components in the frequency band dimension of the reference frequency band features consistent with that of the mixed frequency band features, and ensuring that the frequency point range covered by each frequency band is the same segment by segment. This allows the target object conditions to participate in subsequent processing at the same frequency band position as the mixed frequency band features.

[0131] Another alternative approach is to share a frequency band coding structure between the two speech streams while using the same set of frequency band division results. Specifically, the reference speech can adopt the same or shared normalization and projection parameters as the mixed speech. That is, the normalization operation and linear projection on each frequency band are performed using the same set of parameters for both speech streams. For example, for the k-th frequency band, both speech streams can concatenate the real and imaginary parts of the complex spectrum in the channel dimension, and obtain frequency band features of a preset dimension through normalization and linear projection. The preset dimension is, for example, 128. Through parameter sharing, the frequency band coding on the reference speech side does not require additional parameters.

[0132] Another alternative approach is to preprocess the reference spectral features in the same way as the mixed spectral features before partitioning, and then complete the partitioning according to the same set of frequency band boundaries. Specifically, the real and imaginary parts of the reference spectral features can be normalized frame-by-frame or dynamically adjusted to make the value distribution of the reference spectral features at each frequency point similar to that of the mixed spectral features. Then, the reference spectral features are divided into multiple frequency bands according to the same set of frequency band boundaries, ensuring that the numerical conditions of the two frequency band features remain consistent during the partitioning process.

[0133] Furthermore, the reference spectral features can be divided all at once before processing begins, and the reference frequency band features can be cached in time frame order. For example, the length of the reference speech can be 3 seconds, which corresponds to multiple time frames under the condition of a frame shift of 256 points, or 16ms. The reference frequency band features can be divided and cached frame by frame for use in pooling and conditional extraction, so that the reference spectral features do not need to be accessed repeatedly during frame-by-frame processing.

[0134] Since the reference frequency band features retain a complete time-frequency structure in both the frequency band and time dimensions, pooling these features along both dimensions yields target object conditions used to characterize the acoustic identity of the target object. In practical applications, pooling the reference frequency band features along both dimensions to obtain target object conditions can also be achieved through various methods.

[0135] One alternative approach is to perform mean pooling in both the frequency band and time dimensions. Specifically, the values ​​of the reference frequency band features can be averaged across each frequency band and time frame, ensuring that the pooling result has the same value across all frequency bands and time frames, thus eliminating the differences between frequency band and time positions. Mean pooling assigns the same weight to each time frame and each frequency band, without introducing pooling parameters that need to be trained, allowing the target object conditions to stably reflect the average acoustic characteristics of the reference speech over the entire range.

[0136] Another alternative approach is to weight the components during pooling according to their contribution to the acoustic identity of the target object. Specifically, weighting coefficients can be determined based on the energy intensity, signal-to-noise ratio estimation, or identity discrimination confidence of each frequency band or time frame. Components with higher values ​​or better reliability are given larger weights, while silent segments, noisy segments, and time frames occupied by other sound sources are given smaller weights. The weighting coefficients are applied to the reference frequency band features band by band and time frame before pooling, making the extraction process of the target object conditions more adaptable to possible quality fluctuations in the reference speech.

[0137] Another alternative approach is to perform pooling in stages, that is, pooling along one dimension first, and then pooling along another dimension. Specifically, the reference frequency band features can be segmented and pooled along the time dimension first, dividing the reference speech into multiple time periods and obtaining the representation of each time period, and then pooling the representation of each time period along the frequency band dimension; alternatively, the information in each frequency band can be merged into a time-frame representation along the frequency band dimension first, and then pooling the time-frame representation along the time dimension, so that the pooling order and segmentation method can be set according to the length and quality distribution of the reference speech.

[0138] In addition, the reference frequency band features can be normalized before pooling, for example, by normalizing according to the dynamic range of each frequency band, so that the values ​​of different frequency bands are within a comparable range; when the reference speech is in multiple segments, the frequency band features of each segment of the reference speech can be pooled separately to obtain the target object conditions of each segment, and the target object conditions of each segment can be averaged or combined by reliability weighting.

[0139] For example, a 16kHz sampling rate and a reference speech length of 3s are used for further explanation. Specifically, after time-frequency transformation, the reference speech yields a reference spectral feature with 257 non-repeating frequency points, which is then divided into 36 non-uniform frequency bands. The reference frequency band feature has 36 components in the frequency band dimension. By simultaneously performing pooling on the reference frequency band feature in both the frequency band and time dimensions, a fixed-length representation that does not change with either the frequency band index or the time frame index can be obtained, for example, a 128-dimensional representation.

[0140] In an alternative embodiment, pooling can also occur only during the target object condition extraction process on the reference speech side. The mixed frequency band features of the mixed speech do not participate in pooling and maintain the frame-by-frame, frequency band structure for subsequent processing. The target object conditions obtained by pooling do not replace the mixed frequency band features, but are used as condition signals to participate in the determination of subsequent recursive parameters.

[0141] In the embodiments of this specification, the mixed spectral features are divided into multiple frequency bands according to the information density of the mixed spectral features in different frequency regions, so that the bandwidth matches the amount of information carried by each frequency region. This shortens the frequency band sequence while preserving the low-frequency harmonics and timbre cues that contribute to the identification of the target object. By dividing the reference spectral features according to the same frequency band structure as the mixed spectral features, the acoustic identity information of the target object is aligned with the mixed frequency band features of the mixed speech at the frequency band level. By pooling the reference frequency band features along the frequency band dimension and the time dimension, the acoustic identity information embodied in the entire reference speech is aggregated into target object conditions that do not change with the frequency band and time frame, so that the condition representation is decoupled from the duration and language content of the reference speech.

[0142] In an optional embodiment of this specification, after dividing the mixed spectral features of the mixed speech into multiple frequency bands based on the information density of the mixed spectral features in different frequency regions and obtaining the mixed frequency band features, the method further includes: The mixed frequency band features are input into multiple cascaded dual-path blocks, wherein each dual-path block includes a spectral path and a temporal path; The spectral path modeling results are obtained by modeling the spectral path along the frequency band axis within this time frame; The spectral path modeling results are used as input to the time path. Causal modeling is performed along the time axis in each frequency band, and the hidden state of the state space model is maintained according to the frequency band. The output features of the dual-path block are determined based on the updated hidden state. The output features of each dual-path block are used as the input of the next dual-path block in sequence, and the output features of the last dual-path block are used as the output features.

[0143] A dual-path block is a processing unit sequentially connected in the backbone network, used to model frame-by-frame and frequency-band features. Each dual-path block includes a spectral path and a temporal path. Specifically, a dual-path block first receives the mixed frequency-band features of the mixed speech branch. The spectral path then models the frequency-band sequence within that time frame, obtaining a spectral path modeling result incorporating inter-band relationships. This spectral path modeling result is then sent to the temporal path, where causal modeling is performed along the time axis within each frequency band, and hidden states are maintained by frequency band. For example, the backbone can be composed of 36 structurally identical dual-path blocks connected in series, with a feature dimension of, for example, 128. Furthermore, the spectral path and temporal path within a dual-path block are sequential; the output feature of the dual-path block is the output of the temporal path within that block. The dual-path block processes backbone features that retain the frame-by-frame and frequency-band structure, with target object conditions serving as conditional signals in determining the recursive parameters within the block.

[0144] A spectral path is a processing path within a dual-path block that models each frequency band along the frequency band axis within the same time frame. Specifically, the spectral path takes the mixed frequency band characteristics of that time frame or the frequency band-by-frequency characteristics output from the preceding dual-path block as input, processes each frequency band component of that time frame once along the frequency band axis, and models the harmonic relationships, formant positions, and spectral envelope shapes between different frequency bands. Its output is the spectral path modeling result. Furthermore, the frequency axis is not constrained by streaming causality, and the spectral path can adopt a bidirectional structure along two scanning directions on the frequency band axis, without maintaining cross-frame states, and the frequency band axis modeling of each time frame is independent of each other.

[0145] The time path is the processing path in a dual-path block that performs causal modeling along the time axis within each frequency band. Specifically, the time path takes the spectral path modeling results as input, performs recursion sequentially within each frequency band according to time frames, and maintains the hidden states of the state-space model by frequency band. For example, with 36 frequency bands, a dual-path block maintains 36 sets of independent hidden states; the output of the time path of this block is the output feature of the dual-path block. Furthermore, the time path only uses information from the current time frame and previous time frames. The hidden states of each frequency band are independent and can be processed in parallel. The relationships between different frequency bands within a frame are handled by the spectral path.

[0146] The frequency band axis is the direction corresponding to the order of frequency bands in a frequency band feature, used to organize the feature components of each frequency band within the same time frame. Specifically, a frequency band feature has the same number of components as the number of frequency bands on the frequency band axis. For example, after dividing the 257 frequency points of the mixed spectral feature into 36 non-uniform frequency bands, the mixed frequency band feature has 36 components on the frequency band axis, arranged in order from low frequency to high frequency. Modeling along the frequency band axis involves modeling the relationships between the components of each frequency band within the same time frame, and its processing range does not span across time frames. Furthermore, processing along the frequency band axis is not subject to streaming causality constraints and is used to supplement the harmonic, formant, and spectral envelope relationships between different frequency bands within the same frame.

[0147] The spectral path modeling result is the feature representation output after modeling the spectral path along the frequency band axis within the same time frame, and is used as the input to the time path. Specifically, the spectral path modeling result has the same number of components on the frequency band axis as the number of frequency bands. For example, in the case of 36 frequency bands, it has 36 frequency band components. Its dimension can be the same as the channel dimension of the frequency band feature, and it is obtained within the same time frame, carrying the interrelationship between the frequency bands in that time frame. Furthermore, the spectral path modeling result corresponds to the input features of that time frame component by component on the frequency band axis, without changing the frame-by-frame, frequency band structure of the features; since the spectral path does not maintain cross-frame state, the historical information in the time dimension is carried by the latent state of the time path.

[0148] Causal modeling is a modeling approach that uses only information from the current and previous time frames when modeling the current time frame, excluding time frames that have not yet arrived. Specifically, causal modeling proceeds sequentially along the time axis, with the modeling result for each time frame determined by the current input and the updated latent states. For example, the time path maintains cross-frame states within each frequency band, and the latent state used to process the t-th time frame is recursively obtained only from that frame and several previous time frames. Furthermore, because causal modeling does not access time frames that have not yet arrived, the processing can provide results frame by frame as the speech arrives.

[0149] The hidden state is an intermediate state in the state-space model used to carry historical information of processed time frames, and it is maintained separately for each frequency band in the time path. Specifically, the hidden state has a dimension determined by the model configuration and does not increase with the duration of processed speech. After each time frame arrives, the hidden state is updated according to the recursive parameters corresponding to that time frame and frequency band, and the updated hidden state determines the output characteristics of the dual-path block. For example, when the number of frequency bands is 36, the 36 sets of hidden states in a dual-path block are updated with the input of their respective frequency bands. Furthermore, the hidden states are not exchanged between frequency bands, and the writing, reading, and effective memory length of historical information in the hidden state are controlled by the recursive parameters.

[0150] In practical applications, the mixed frequency band characteristics can be input into multiple cascaded dual-path blocks, which can be achieved through various methods.

[0151] One alternative approach is to feed the mixed frequency band features into concatenated dual-path blocks using a time-frame-by-time, frequency-band-by-frequency organization method. Specifically, for any given time frame, the frequency band components of the mixed frequency band features for that time frame can be organized into a set of input representations according to frequency band order and fed into the first dual-path block. This dual-path block is first modeled on the frequency band axis by the spectral path and then causally modeled along the time axis by the temporal path. The resulting output features serve as the input to the second dual-path block, and each dual-path block executes the same processing flow sequentially according to the concatenated order. Through this time-frame-by-time, frequency-band-by-frequency organization method, the frequency band structure of the mixed speech branches is preserved during the concatenation process.

[0152] Another alternative approach is to align the inputs of each dual-path block on a frequency band basis. Specifically, the number of frequency points covered by each frequency band of the hybrid spectral feature varies. Each frequency band can be first mapped to a frequency band feature of the same dimension via frequency band coding, such as a frequency band feature with dimension 128, ensuring that the dimensions of each component of the hybrid frequency band feature are consistent along the frequency band axis. During the mapping process from frequency points to frequency band features, each frequency band can use its own independent normalization and projection parameters to accommodate differences in the dynamic range of different frequency bands. By aligning the inputs by frequency band, the input organization can be adjusted accordingly when the number of frequency bands and the frequency band boundaries change.

[0153] Another alternative method is to feed the mixed frequency band features frame by frame into the concatenated dual-path blocks according to the arrival order of the time frames. Specifically, after each time frame arrives, the mixed frequency band features of that time frame are first fed into the first dual-path block, and its output features are then fed into the second dual-path block. This process continues through each dual-path block to obtain the output features of that time frame. The progression of the same time frame between the dual-path blocks is independent of the progression of the time frame on the time axis. Each dual-path block maintains its own hidden state in its time path and writes it back to overwrite the content of the previous time frame. By feeding the features frame by frame according to the arrival order, the processing does not require saving the input data of historical time frames.

[0154] Furthermore, the number and feature dimensions of each cascaded dual-path block can be configured. For example, the backbone can use 36 structurally identical dual-path blocks, and the feature dimension can be 128. When the computing power, bandwidth, and latency requirements on the edge side are different, the number of dual-path blocks can be reduced or increased, and the feature dimension can be increased or decreased.

[0155] After inputting mixed frequency band features into multiple cascaded dual-path blocks, the spectral path within any dual-path block can first model the input features along the frequency band axis within the same time frame. In practical applications, obtaining the spectral path modeling result by modeling the spectral path along the frequency band axis within this time frame can also be achieved through various methods.

[0156] One alternative approach is to use a bidirectional state-space module to model along both scanning directions of the frequency band axis. Specifically, within the same time frame, a state recursion can be performed once from low to high frequency according to the frequency band arrangement, and another state recursion can be performed in the order from high to low frequency. The representations obtained from the two directions are then fused, for example, by addition or splicing, and projected as the spectral path modeling result. Using a bidirectional state-space module, relationships can also be established between frequency bands that are far apart within the same time frame; both scans on the frequency band axis are completed within the same time frame, without involving access to time frames that have not yet arrived.

[0157] Another alternative approach is to use a bidirectional cyclic module for modeling along the frequency band axis. Specifically, two sets of cyclic processing units can be set up on the frequency band axis, traversing each frequency band component of the time frame in forward and reverse order respectively, and fusing the representations obtained in the two directions into the spectral path modeling result. The values ​​of the two sets of cyclic processing units remain unchanged between time frames and do not carry the state between time frames. Through the bidirectional cyclic module, the relationship on the frequency band axis can be established step by step according to the traversal order, and intermediate representations are not retained across time frames.

[0158] Another alternative approach is to use convolutional modules to model along the frequency band axis. Specifically, the frequency band components of the time frame can be organized into a one-dimensional sequence according to their frequency band arrangement, and a convolutional kernel sliding along the frequency band axis can be applied, so that the output of each frequency band is jointly determined by the components of that frequency band and its adjacent frequency bands. The value of the convolutional kernel can be preset or determined based on the optimization goal of speech reconstruction, and its length can be determined during training, so that the local relationships between frequency bands can be modeled with a small amount of computation.

[0159] Furthermore, it allows spectral paths to be modeled independently within the same time frame and output as modeling results, without maintaining cross-frame states. For example, spectral path modeling for each time frame is completed independently within that time frame, and processing the spectral path for the t-th time frame does not require referencing the processing results of previous or subsequent time frames. Since the frequency axis is not subject to streaming causality constraints, access to time frames that have not yet been reached is not introduced when the spectral path adopts a bidirectional or convolutional structure.

[0160] Since the spectral path modeling results incorporate the relationships between frequency bands within the same time frame, these results can continue to be used as input for the time path, performing causal modeling along the time axis within each frequency band. In practical applications, the spectral path modeling results can be used as input for the time path, performing causal modeling along the time axis within each frequency band, maintaining the hidden states of the state-space model by frequency band, and determining the output features of the dual-path block based on the updated hidden states. This can also be achieved through various methods.

[0161] One alternative method is to use the components of the frequency band in the spectral path modeling result as the acoustic input of that frequency band in the current time frame, update the hidden state according to the recursive parameters corresponding to that time frame and frequency band, and read out the updated hidden state. Specifically, for the k-th frequency band in the t-th time frame, the components of that frequency band in the spectral path modeling result can be used as the current acoustic input, and a state update and a readout can be completed according to the recursive parameters determined in the aforementioned embodiment. The readout result is used as the output feature of the dual-path block in that time frame and frequency band. By completing the time-dimensional modeling centered on the update and readout of the hidden state, the size of the hidden state remains unchanged.

[0162] Another alternative approach is to maintain independent hidden states for each frequency band and process each band in parallel. Specifically, a dual-path block can maintain the same number of hidden states as the number of frequency bands. For example, with 36 frequency bands, 36 sets of hidden states can be maintained. The hidden states of each frequency band are updated only by the input of its own frequency band, and they do not exchange hidden states with each other. In a forward processing step, the recursion of each frequency band can be expanded in parallel along the frequency band dimension, and the hidden states of each frequency band can be organized into the same set of tensors along the frequency band dimension. By maintaining hidden states independently along the frequency band, the relationships between frequency bands are handled by the spectral path, thereby separating the responsibilities of frequency band axis modeling from those of time axis modeling.

[0163] Another alternative approach is to advance causal modeling sequentially along the time frame axis. Specifically, upon arrival of each time frame, the spectral path first completes frequency band modeling on that time frame, followed by a latent state update and readout within each frequency band via the time path. The latent state used to process the t-th time frame is obtained recursively from that frame and several previous time frames. By advancing sequentially along the time frame axis, the output features of each time frame are obtained in chronological order, collectively forming a feature sequence for signal reconstruction.

[0164] Furthermore, the output of the time path within the dual-path block can be directly used as the output feature of the dual-path block. For example, the output feature can maintain a frame-by-frame, frequency-band-by-frequency structure, that is, there are output feature components that correspond one-to-one with each frequency band in the t-th time frame, and its dimension is, for example, 128; the output features of each dual-path block can adopt the same organization method, so that the output feature of one dual-path block can be used as the input of the next dual-path block without additional transformation.

[0165] Furthermore, after processing the spectral and temporal paths within a dual-path block, the output features can be passed sequentially between the dual-path blocks in a cascaded order. In practical applications, the output features of each dual-path block can be used as the input to the next dual-path block, and the output features of the final dual-path block can be used as the output features. This can be achieved through various methods.

[0166] One alternative approach is to pass the output features block by block in a cascaded order. Specifically, the first dual-path block receives the mixed frequency band features and outputs its own output features, which serve as the input for the second dual-path block. Each dual-path block is passed sequentially in the same organized manner until the final dual-path block. Within each dual-path block, frequency band modeling of the spectral path is performed first, followed by causal modeling of the temporal path, so that the features are enhanced step by step along the network depth.

[0167] Another alternative approach is to use the output features of the terminal dual-path block as the band-wise hidden features of the backbone output. Specifically, the output of the dual-path block at the end of the cascaded structure is the output of the backbone network in its time path. This output is maintained in a frame-by-frame, band-by-band form, meaning that each time frame has hidden feature components that correspond one-to-one with each frequency band. The number of frequency bands is, for example, 36, and the dimension is, for example, 128. These can be organized according to the frequency band boundaries for subsequent use in generating masks by frequency band.

[0168] Another alternative approach is to allow the same time frame to flow sequentially between each dual-path block. Specifically, after each time frame arrives, a forward processing operation is performed sequentially from the first dual-path block to the last dual-path block to obtain the output features of that time frame. Each dual-path block maintains its own hidden state in its time path, so the advancement of the same time frame between dual-path blocks and the recursion between different time frames are interconnected, and there is no need to save additional intermediate features between dual-path blocks during the processing.

[0169] Furthermore, each dual-path block can adopt the same input and output organization method. For example, each dual-path block can take frame-by-frame, frequency-by-band features as input and take features of the same dimension as output, so the number of cascaded dual-path blocks can be increased or decreased according to the computing power, bandwidth and latency requirements of the end side.

[0170] For example, a sampling rate of 16kHz, 36 frequency bands, and 257 frequency points are used for further explanation. Specifically, the mixed frequency band features have 36 components on the frequency band axis and maintain a time-frame structure. Each frequency band component is mapped to a frequency band feature with a dimension of 128 through frequency band coding and is fed into a backbone composed of 36 identical dual-path blocks connected in series. In any dual-path block, the spectral path completes a modeling along the frequency band axis on the 36 frequency band components within the same time frame, obtaining a spectral path modeling result that incorporates the relationship between frequency bands. The spectral path modeling result is then fed into the temporal path, which completes a causal recursion along the time axis within each frequency band. The 36 frequency bands maintain their respective hidden states and can be processed in parallel. The output feature of the 36th dual-path block is the frame-by-frame, frequency band hidden feature output by the backbone, with 36 frequency band components and a dimension of 128, which is used for subsequent generation of masks by frequency band and reconstruction of the speech of the target object.

[0171] In an optional embodiment, the spectral path can be modeled bidirectionally along the frequency band axis, and the temporal path can adopt a causal state-space recursive structure. Specifically, the bidirectional structure of the spectral path refers only to the two scanning directions along the frequency axis, that is, modeling is performed in both the forward and reverse directions according to the frequency band arrangement order within the same time frame, and the results of the two directions are fused. It does not involve accessing time frames that have not yet been reached, and therefore does not affect the causality of the processing. The temporal path is recursively processed along the time axis in the order of time frames, and cross-frame states are maintained according to frequency bands. In addition, the number of dual-path blocks and the feature dimensions can be scaled according to the computing power, bandwidth, and latency requirements of the edge.

[0172] In the embodiments of this specification, by inputting mixed frequency band features into multiple cascaded dual-path blocks, the frame-by-frame and frequency band mixed frequency band features are sequentially enhanced in processing units with identical structures. By modeling along the frequency band axis by the spectral path within the same time frame within the dual-path block and using the spectral path modeling result as the input of the temporal path, the harmonics, formants, and spectral envelope relationships between different frequency bands within the same frame are first incorporated into the features, and then causal modeling is performed along the time axis, so that the inter-band relationships and temporal dependencies are respectively undertaken by two paths and interconnected. Since the spectral path does not maintain cross-frame states and the temporal path maintains hidden states by frequency band, the hidden states of each frequency band are independent and can be processed in parallel, and the scale of the maintained hidden states is determined by the model configuration. By using the output features of each dual-path block as the input of the next dual-path block and using the output features of the last dual-path block as the output features, the frequency band hidden features are refined step by step along the network depth, providing a feature basis for subsequent generation of masks by frequency band and reconstruction of the speech of the target object.

[0173] In one optional embodiment of this specification, pooling is performed on the reference frequency band features along the frequency band dimension and the time dimension to obtain the target object conditions, including: Pool the reference frequency band features along the frequency band dimension and the time dimension, and project the pooling results to obtain the global object conditions; Projecting the features of the reference frequency band yields keys and values, forming a frame-level reference memory; Based on the mixed frequency band characteristics of the time frame, the characteristics of the current frame are determined, and the frame-level reference memory is retrieved using the current frame characteristics as the query to obtain the frame-level retrieval conditions. Use global object conditions and / or frame-level search conditions as target object conditions.

[0174] The global object condition is a conditional representation obtained by pooling the reference frequency band features along the frequency band and time dimensions and projecting the pooling results. It is used to characterize the overall acoustic identity of the target object and remains unchanged throughout the entire processing. Specifically, pooling can aggregate the values ​​of the reference frequency band features across each frequency band and time frame. Projection can employ a two-layer perceptron, with the hidden layer dimension being the same as the channel dimension D of the frequency band features. The output dimension is, for example, 128, and the activation function is, for example, a sigmoid linear unit (SiLU) activation function. The global object condition is calculated all at once before processing begins. Furthermore, the global object condition eliminates the differences between frequency band and time positions, thus not carrying over localized phonological and spectral details that change frame by frame. Its role is to provide a stable and low-cost basis for target localization during processing and to participate as a conditional signal in determining subsequent recursive parameters.

[0175] A key is a feature representation obtained by projecting the reference frequency band features. It is used to perform matching operations with a query to determine the part of the reference speech that is relevant to the current processing time. Specifically, keys can be organized according to the time frames of the reference speech. Each memory entry corresponds to a set of key components. For example, if the length of the reference speech is 3 seconds, the number of keys obtained by frame-by-frame projection corresponds to the number of time frames of the reference speech. The key is obtained by linear projection of the reference frequency band features, and its dimension can be the same as the channel dimension D of the frequency band features. Furthermore, keys and values ​​appear in pairs and together constitute a frame-level reference memory.

[0176] Values ​​are content representations that appear in pairs with keys and are used to be retrieved during the retrieval process. They carry the acoustic content of the reference speech in the memory entry corresponding to the key. Specifically, values ​​are also obtained by linear projection of reference frequency band features, and they correspond one-to-one with keys in the same set of memory entries according to time frames. Their dimension can be the same as the channel dimension D of the frequency band features, or it can be an independent preset dimension. When using a memory with a limited length, the number of value entries is determined by a preset constant, such as 128. Furthermore, during the retrieval process, the relevance between the key and the query determines the weight of each memory entry. After the weights are applied to the values, a weighted sum is obtained to obtain the retrieval result.

[0177] Frame-level reference memory is a serialized conditional representation consisting of keys and values ​​obtained by projecting reference frequency band features. It is used to read local information related to the target object from the reference speech at the current processing time. Specifically, frame-level reference memory preserves the time-frequency structure of the reference speech. Each entry contains a set of keys and a set of values. The number of entries can correspond to the number of time frames of the reference speech, or it can be set to a preset constant with a limited length. For example, when the length of the reference speech is 3 seconds and the number of entries is 128, the resident memory of the keys and values ​​is approximately 0.06 MB. Furthermore, compared with global object conditions, frame-level reference memory does not eliminate differences in temporal position and can provide phoneme-level pronunciation and local spectroscopic details. The keys and values ​​can be pre-calculated and cached during the initialization phase. During frame-by-frame processing, it is only necessary to access the cached memory using the current frame features as the query.

[0178] The current frame feature is the acoustic hidden representation obtained by the mixed speech branch at the current processing moment, used as a frame-level reference memory for query access. Specifically, the current frame feature originates from the mixed speech, which undergoes time-frequency transformation, frequency band division, and frequency band coding, and is modeled in the preceding dual-path block to obtain the current frame feature at the current time frame. Furthermore, the current frame feature can be organized separately by frequency band, or it can be organized after aggregation between frequency bands; each deep dual-path block can use its independently learned query projection to map the current frame feature to a query.

[0179] A query is a feature representation used in the retrieval process to match keys in the frame-level reference memory, determining which local information should be retrieved from the reference speech. Specifically, the query can be obtained by linearly projecting the features of the current frame, with dimensions consistent with the keys, so that the degree of matching between the query and each memory entry can be determined through a relevance measure in the form of an inner product; the parameters of the linear projection can be learned independently in each deep dual-path block. Furthermore, the query changes with the current time frame, so the content retrieved from the same frame-level reference memory differs across different time frames.

[0180] Frame-level retrieval criteria are conditional representations obtained by retrieving frame-level reference memories using the features of the current frame as the query. They are used to supplement the local pronunciation and spectral information of the target object during processing. Specifically, frame-level retrieval criteria can be obtained by time frame, or by time frame and frequency band separately. The retrieval is based on the correlation between the query and the key, determining the weight of each memory entry, and then weighted summing the values. Its dimension can be, for example, the same as the global object criteria. Furthermore, frame-level retrieval criteria do not eliminate differences in temporal position; their frame-by-frame computational cost is determined by the number of memory entries. They, along with global object criteria, are suitable for providing conditions at different processing positions.

[0181] In practical applications, the global object conditions can be obtained by pooling the reference frequency band features along the frequency band dimension and the time dimension, and then projecting the pooling results.

[0182] One alternative approach is to perform mean pooling on both the frequency band and time dimensions, and then project the pooling results onto a preset dimension using a two-layer perceptron. Specifically, the values ​​of the reference frequency band features across each frequency band and time frame can be averaged to obtain a fixed-length pooling result. This pooling result is then mapped onto a preset dimension using a two-layer perceptron, with the output dimension being, for example, 128. Through mean pooling and two-layer perceptron projection, global object conditions can characterize the overall timbre of the target object without introducing an additional discriminant network.

[0183] Another alternative approach is to weight the pooling process according to the contribution of each component to the acoustic identity of the target object, and then project the weighted pooling result. Specifically, weighting coefficients can be determined based on the energy intensity, signal-to-noise ratio estimation, or identity discrimination confidence of each frequency band or time frame, assigning larger weights to components with better reliability. The weighted pooling result is then mapped to a preset dimension via linear projection or a projection module with normalization. Through weighted pooling and projection, the global object condition is more adaptable to potential quality fluctuations in the reference speech.

[0184] Another alternative approach is to perform pooling in stages and employ different projection structures. Specifically, one can first perform segmented pooling of the reference frequency band features along the time dimension, and then pool the representations of each time segment along the frequency band dimension; alternatively, one can first merge the information in each frequency band into a time-frame representation along the frequency band dimension, and then pool the time-frame representations along the time dimension. The pooling results obtained in both sequences are mapped to a preset dimension via linear projection, a multilayer perceptron, or a projection module with normalization, respectively. Through the optional configuration of staged pooling and projection structures, the computational complexity and representation granularity of the global object conditions can be determined based on the length and quality distribution of the reference speech.

[0185] In addition, pooling and projection can be completed all at once before processing begins, and the resulting global object conditions can be cached, so that the frame-by-frame processing stage only reads the cached global object conditions and does not need to repeatedly access the reference frequency band features.

[0186] In addition to being used to generate global object conditions, the frame-by-frame time-frequency structure preserved by the reference frequency band features can also be used to form frame-level reference memories. In practical applications, projecting the reference frequency band features to obtain keys and values ​​to form frame-level reference memories can also be achieved through various methods.

[0187] One alternative approach is to perform two sets of linear projections on the reference frequency band features to obtain keys and values. Specifically, two sets of linear projections with mutually exclusive parameters can be used to map the reference frequency band features to keys and values, with the keys and values ​​corresponding one-to-one on the same set of memory entries according to time frames. The dimension of each entry can, for example, be the same as the channel dimension D of the frequency band features. The parameters of the two sets of projections can be determined once before processing begins, and the calculation and caching of keys and values ​​can be completed during the initialization phase.

[0188] Another alternative approach is to decouple the number of memory entries from the duration of the reference speech by employing a limited-length method. Specifically, the reference frequency band features can be segmented and pooled along the time dimension, merging multiple adjacent time frames into a single memory entry, making the number of entries equal to a preset constant. Alternatively, clustering can be performed on the projection results of each time frame, merging time frames with similar acoustic content into the same memory entry, and filtering each time frame according to similarity and energy intensity to retain a preset number of memory entries. For reference speech exceeding a preset duration, it can be truncated to the preset duration before projection. With limited-length memory, for example, when the reference speech is 3 seconds long and the number of entries is 128, the computational cost of frame-by-frame retrieval and the resident key-value memory are both determined by this preset constant, independent of the actual duration of the reference speech.

[0189] Another alternative approach is to share projection parameters during memory formation. Specifically, keys and values ​​can be obtained by combining the same set of base projections with two sets of output mappings, or multiple deep dual-path blocks can share the same set of key-value projections, meaning each deep dual-path block accesses the same cached set of keys and values, using its own independent projection parameters only on the query side; when the reference speech is multi-segmented, the reference frequency band features of each segment can also generate keys and values ​​separately through the same set of projections.

[0190] Furthermore, the reference frequency band features can be segmented along the time dimension, and keys and values ​​can be generated for each segment. For example, the entire reference speech can be divided into multiple time periods, and keys and values ​​can be projected onto each time period. The results of each segment can then be concatenated into a frame-level reference memory. Alternatively, frame-level reference memories can be generated for each segment, and their weight in the retrieval can be determined according to the reliability of each segment.

[0191] Because frame-level reference memory preserves the time-frequency structure of the reference speech, during frame-by-frame processing, reference information related to the current time frame can be read from it according to the acoustic representation at the current processing moment. In practical applications, based on the mixed frequency band characteristics of the time frame, the features of the current frame are determined, and the frame-level reference memory is retrieved using the current frame features as the query to obtain frame-level retrieval conditions. This can also be achieved through various methods.

[0192] One alternative approach is to determine the current frame features using the acoustic hidden representation of the mixed speech branch at the current processing moment, and then perform attention retrieval using these current frame features as the query. Specifically, the current frame features can be mapped to the query via linear projection, the correlation between the query and each set of keys in the frame-level reference memory can be calculated, and the correlations can be normalized into weights before a weighted sum of the values ​​is obtained to arrive at the frame-level retrieval criteria. Through attention retrieval, reference speech segments with high relevance to the current time frame are retrieved.

[0193] Another alternative approach is to organize the current frame features by frequency band and perform retrieval separately for each band. Specifically, for the k-th frequency band in the t-th time frame, the acoustic hidden representation of that frequency band at the current processing moment can be used as the current frame feature. The query obtained from the current frame feature is used to access the frame-level reference memory to obtain the frame-level retrieval conditions for that frequency band. If the acoustic content is similar across frequency bands, the current frame features can be aggregated across frequency bands first, and then a retrieval can be performed using the aggregated current frame features as the query. By organizing the retrieval by frequency band or by time frame, the frame-level retrieval conditions correspond to the granularity of the frequency band division.

[0194] Another alternative approach is to allow each deep dual-path block to use its own independently learned query projection. Specifically, since the semantic levels corresponding to acoustic hidden representations of different depths are different, independent query projection parameters can be configured in each deep dual-path block. This allows each deep dual-path block to determine the query and perform the retrieval based on its own semantic level, while the keys and values ​​are pre-calculated and cached during the initialization phase for use by all deep dual-path blocks.

[0195] Furthermore, linear projection and normalization can be performed on the weighted summation result obtained after retrieval. For example, the read result obtained by weighted summation can be mapped to a preset dimension through a linear projection, and normalization processing can be applied after projection to make the value distribution of frame-level retrieval conditions close to the value distribution of global object conditions, so that the two can be used together in a selective or combined manner.

[0196] After obtaining the global object conditions and frame-level retrieval conditions, they can be used separately or together as target object conditions, depending on the processing location, to participate in the determination of subsequent recursive parameters. In practical applications, using global object conditions and / or frame-level retrieval conditions as target object conditions can also be achieved through various methods.

[0197] One alternative approach is to choose between two conditions based on the processing location. Specifically, a global object condition can be used as the target object condition in some processing locations, while a frame-level retrieval condition can be used as the target object condition in others. The choice can be based on the order of the processing locations in network depth, or on the computing power budget, interference intensity, or target existence confidence at the current processing moment. By choosing one condition, the granularity of the condition representation is matched to the needs of each processing location.

[0198] Another alternative approach is to concatenate the global object condition with the frame-level retrieval condition to obtain the target object condition. Specifically, the global object condition and the frame-level retrieval condition can be concatenated along the channel dimension, and the concatenated result can be mapped back to a preset dimension via linear projection to obtain the target object condition. For example, if both dimensions are 128, the concatenated result has 256 components, which can be mapped back to 128 dimensions via linear projection. Through this concatenation and combination, the target object condition simultaneously includes the stable identity information embodied in the entire reference speech segment and the local pronunciation and spectral information embodied in the current time frame.

[0199] Another alternative approach is to weight and fuse the global object criteria and the frame-level retrieval criteria. Specifically, the two can be weighted and summed according to preset weight coefficients. These weight coefficients can be pre-set based on the processing location or determined based on the target presence confidence or interference intensity in the current time frame. Through weighting and fusion, the proportion of each criterion in the target object criteria can be adjusted according to the processing location and the content of the current time frame.

[0200] Furthermore, gating fusion can be applied to global object conditions and frame-level retrieval conditions. For example, a gating value can be determined based on the features of the current frame, mapped to a range of 0 to 1, and then applied to both global object conditions and frame-level retrieval conditions. The weighted results of the two approaches are then summed to obtain the target object condition. The gating value can be determined frame by frame, allowing the fusion ratio between the two conditions to change as the processing progresses.

[0201] For example, a sampling rate of 16kHz, a reference speech length of 3s, 36 frequency bands, and 128 memory entries are used for further explanation. Specifically, pooling is performed on the reference frequency band features in both the frequency band and time dimensions. The pooling result is projected through two layers of perceptrons, with an output dimension of 128, resulting in global object conditions that are determined once before processing begins and remain unchanged throughout the entire processing. Simultaneously, two sets of linear projections are performed on the reference frequency band features to obtain keys and values, and the number of memory entries is set to 128 according to a limited length, forming a frame-level reference memory. The keys and values ​​are pre-calculated and cached during the initialization phase. During frame-by-frame processing, the acoustic hidden representation of the current processing moment of the mixed speech branch is used to determine the current frame features, and the frame-level reference memory is retrieved using the current frame features as the query to obtain frame-level retrieval conditions. The global object conditions and frame-level retrieval conditions are selected or combined using methods such as splicing, weighted sum fusion, or gated fusion, and used as target object conditions to participate in the determination of subsequent recursive parameters.

[0202] In an optional embodiment, global object conditions and frame-level retrieval conditions can be distinguished from each other in function while also complementing each other. Specifically, global object conditions do not carry local details that change frame by frame; their function is to provide stable and low-cost target localization criteria during processing. Frame-level retrieval conditions change with the current frame; their function is to supplement the timbre, articulation, and spectrogram details of the target object at processing locations where detail recovery is required. Since the functions and costs of the two conditions are different, the selection and combination of conditions can be determined according to the processing location, interference intensity, or computational budget.

[0203] In the embodiments of this specification, by projecting the pooling results, the acoustic identity information of the reference speech over the entire segment is aggregated into global object conditions that are determined once before processing begins and remain unchanged throughout the entire processing; by projecting the reference frequency band features to obtain keys and values ​​and forming frame-level reference memory, the frame-by-frame time-frequency structure of the reference speech is preserved, so that phoneme-level pronunciation and local spectrogram information can be read according to the current time frame, complementing the global object conditions in terms of granularity; by performing retrieval using the current frame features as the query, the obtained conditions change with the current acoustic content, so that the stability of target localization and the fineness of detail recovery are mutually coordinated.

[0204] In one optional embodiment of this specification, the hidden states of the state-space model are maintained separately in multiple cascaded two-path blocks according to frequency bands; Use global object criteria and / or frame-level retrieval criteria as target object criteria, including: In the first preset number of dual-path blocks, the global object condition is used as the target object condition; In the remaining dual-path blocks, frame-level retrieval conditions are used as target object conditions.

[0205] For details regarding the technical aspects of dual-path blocks, please refer to the description of dual-path blocks in the foregoing embodiments; these details will not be repeated here.

[0206] The preset quantity is a predetermined value used to define the condition allocation method for dual-path blocks. It represents the number of dual-path blocks in a series that consecutively use global object conditions as target object conditions from the starting end. Specifically, the preset quantity and the total number of dual-path blocks together determine the condition allocation method. The number of remaining dual-path blocks is equal to the difference between the total number of dual-path blocks and the preset quantity. For example, if the backbone consists of 36 series-connected dual-path blocks, the preset quantity can be 30, meaning the first 30 dual-path blocks use global object conditions, and the remaining 6 dual-path blocks use frame-level retrieval conditions. Furthermore, the value of the preset quantity can be configured or dynamically adjusted based on model size, edge computing power budget, reference speech length, interference intensity, and scene latency requirements, thereby reducing the computational overhead introduced by frame-by-frame retrieval.

[0207] In practical applications, using global object conditions as target object conditions in a pre-set number of dual-path blocks can be achieved through various methods.

[0208] One alternative approach is to divide the processing positions according to the order of the dual-path blocks in the cascaded structure, so that a predetermined number of dual-path blocks at the beginning use global object conditions. Specifically, a subset of dual-path blocks at the beginning are responsible for target localization and coarse-grained sound source differentiation, requiring stable and low-overhead identity conditions. Therefore, a predetermined number of dual-path blocks can be selected consecutively starting from the first dual-path block, so that these dual-path blocks all use global object conditions as target object conditions when determining recursive parameters. By dividing the processing positions sequentially, the allocation of condition granularity corresponds to the processing responsibilities undertaken by each dual-path block.

[0209] Another alternative approach is to directly read pre-calculated and cached global object conditions within each dual-path block. Specifically, the global object conditions are calculated once before processing begins and are permanently stored in the processing device. A predetermined number of dual-path blocks each read the same global object conditions during time-frame-by-band processing and send the reading results to the condition projection and recursive parameter determination process of that block. By reading the cached conditions, each dual-path block does not need to repeatedly access the reference band features, nor does it need to re-perform pooling and projection.

[0210] Another alternative approach is to configure a conditional projection for each dual-path block that is compatible with its acoustic input, allowing the global object conditions to directly participate in the determination of recursive parameters within each dual-path block. Specifically, the global object conditions have eliminated positional differences in the frequency band and time dimensions, and therefore do not carry components related to the frequency band index or time frame index. Each dual-path block can map its global object conditions to a conditional embedding space consistent with the channel dimensions of its acoustic input through its configured conditional projection, and then concatenate or add it to the acoustic input of that block along the channel dimension.

[0211] In addition, the preset number can be determined according to the actual needs of the processing location. For example, the preset number can be set to a value less than the total number of dual-path blocks, so that a portion of the dual-path blocks at the beginning of the serial structure use global object conditions, while the remaining dual-path blocks use frame-level retrieval conditions; if the total number of dual-path blocks changes, the preset number can be re-determined in a way that adapts to the total number of dual-path blocks.

[0212] Meanwhile, after a predetermined number of dual-path blocks at the beginning have completed the processing determined by global object conditions, the remaining dual-path blocks can obtain target object conditions at a finer granularity. In practical applications, using frame-level retrieval conditions as target object conditions in the remaining dual-path blocks can also be achieved through various methods.

[0213] One alternative approach is to divide the processing positions according to the order of the dual-path blocks in the cascaded structure, so that each dual-path block after a preset number uses frame-level retrieval conditions as the target object condition. Specifically, the remaining dual-path blocks at the end of the cascaded structure are closer to the output end along the network depth. Their processing responsibility is to suppress residual interference and restore the timbre details of the target object. Therefore, these dual-path blocks can use frame-level retrieval conditions as the target object condition during frame-by-frame processing. That is, each time frame performs a retrieval of the frame-level reference memory with the current frame features as the query, so that the supplemented local articulation and spectrogram information directly affects the processing position closer to the output end.

[0214] Another alternative approach is to match the query obtained by linear projection of the current frame features with the cached keys, and then sum the values ​​using weighted methods to obtain the frame-level retrieval criteria. Specifically, the keys and values ​​in the frame-level reference memory are pre-calculated and cached during the initialization phase. When processing frame by frame, only the query needs to be determined by the current frame features, without reprocessing the reference speech. During the retrieval process, the weights are determined by the correlation between the query and each memory entry. After the weights are applied to the values, they are summed using weighted methods to obtain the frame-level retrieval criteria for that time frame, which are then fed into the conditional projection and recursive parameter determination process of the dual-path block.

[0215] Another alternative approach is to configure independent query projections for each subsequent dual-path block, allowing each block to determine the query and perform the retrieval according to its own semantic level. Specifically, the semantic levels corresponding to acoustic hidden representations of different depths differ, so the remaining dual-path blocks can use their independently learned query projection parameters to map the current frame features to the query, while the keys and values ​​are pre-calculated and cached during the initialization phase for all dual-path blocks to access.

[0216] In addition, multi-path retrieval can be used on the terminal dual-path blocks in the remaining dual-path blocks. For example, multiple retrieval heads can be used to perform retrieval on the frame-level reference memory in the terminal dual-path block. Each retrieval head uses its own independent query projection to obtain the query and reads the content from the same set of keys and values. The reading results of each retrieval head are concatenated and mapped to a preset dimension through a linear projection.

[0217] For example, a sampling rate of 16kHz, 36 frequency bands, 128 feature dimensions, and 36 dual-path blocks are used for further explanation. Specifically, the preset number is 30, meaning that in the 36 cascaded dual-path blocks, the first 30 dual-path blocks use global object conditions, and the remaining 6 dual-path blocks use frame-level retrieval conditions. The first 30 dual-path blocks each read the global object conditions that are calculated and cached once before processing begins, and participate in the determination of the recursive parameters of the block per time frame and per frequency band after their respective conditional projection mapping. The remaining 6 dual-path blocks each use the current frame features as the query and perform a 4-head retrieval on the frame-level reference memory. The keys and values ​​are pre-calculated and cached during the initialization phase and shared among these 6 dual-path blocks. The preset number and the number of retrieval heads can be adjusted according to the model size, the end-side computing power budget, the reference speech length, and the interference intensity.

[0218] In an alternative embodiment, the condition granularity can also be dynamically scheduled, rather than being limited to a fixed allocation in the order of dual-path blocks. Specifically, switching can be based on target presence confidence, which can be obtained from the cosine similarity between the target-related representation of the current frame and the global object condition, or from the output of lightweight binary classification or the proportion of mask energy, and dual-threshold hysteresis can be set to avoid frequent switching between the two conditions; switching can also be based on interference intensity, which can be determined by spectral flatness, estimated signal-to-noise ratio, or residual energy; switching can also be based on frequency band, enabling frame-level retrieval only in low-frequency bands with dense speaker cues; switching can also be based on computational budget, or soft weighting can be applied between the two conditions through learnable gating.

[0219] In this embodiment, by having a preset number of dual-path blocks at the beginning of the serial structure use global object conditions and the remaining dual-path blocks use frame-level retrieval conditions, the granularity of the conditions corresponds to the processing responsibilities undertaken by each processing position. The starting dual-path blocks complete target localization and coarse-grained sound source differentiation with stable and low-overhead identity conditions, while the dual-path blocks near the output end complete residual interference suppression and detail recovery with local information retrieved according to the current frame content. By having each dual-path block at the beginning read the global object conditions calculated and cached once before processing begins, this part of the processing position does not generate frame-by-frame overhead related to the reference speech length. The keys and values ​​accessed by the remaining dual-path blocks for retrieval are also pre-calculated and cached. Since the preset number can be configured or dynamically adjusted according to the model size, end-side computing power budget, reference speech length, interference intensity, and scene latency requirements, the distribution of condition granularity in the network depth can be determined according to deployment conditions and processing requirements.

[0220] In one optional embodiment of this specification, the number of memory entries for the frame-level reference memory is a preset constant, and the frame-level reference memory is obtained through at least one of the following methods: The reference frequency band features are segmented and pooled along the time dimension, and multiple adjacent time frames are merged into a memory entry. Clustering is performed on the projection results for each time frame; After filtering each time frame according to similarity or energy intensity, a preset number of memory entries are retained. Reference audio that exceeds the preset duration is truncated before projection.

[0221] The number of memory entries is the number of entries in the frame-level reference memory obtained by arranging keys and values ​​in chronological order. It is used to determine the number of relevance calculations during frame-by-frame retrieval and the resident storage scale of keys and values. Specifically, the number of memory entries can correspond to the number of time frames of the reference speech, or it can be directly given by a preset constant. For example, when the length of the reference speech is 3 seconds, the number of memory entries can be determined by a preset constant as 128. In this case, each entry corresponds to the merged result of several adjacent time frames in the reference speech. Keys and values ​​correspond one-to-one in chronological order on the same group of entries, and the dimension of each entry is the same as the channel dimension of the frequency band feature. Furthermore, once the number of memory entries is determined, the number of entries accessed during frame-by-frame retrieval and the resident memory of keys and values ​​no longer change with the actual duration of the reference speech. Therefore, the reference speech can be taken from speech segments of different durations without changing the processing structure.

[0222] The preset constant is a pre-defined value used to determine the number of memory entries, and its value does not change with the actual duration of the reference speech. Specifically, the preset constant can be determined according to the computational cost budget for frame-by-frame retrieval, the resident memory budget for keys and values, or the typical duration of the reference speech, for example, a value of 128. During frame-by-frame processing, the preset constant remains unchanged, and the number of entries accessed by the same frame-level reference memory in different time frames is consistent. Furthermore, the preset constant can be preset as a model configuration along with the number of frequency bands and the number of dual-path blocks, or it can be selected from candidate values ​​based on the duration and quality of the reference speech before processing begins, thus decoupling the retrieval overhead from the duration of the reference speech.

[0223] Limited-length memory is a memory organization method that determines the number of entries in frame-level reference memory by a preset constant, independent of the duration of the reference speech. Specifically, limited-length memory operates during the formation stage of frame-level reference memory. It maps reference speech of arbitrary duration to a preset constant number of entries by performing segmented pooling of reference frequency band features along the time dimension, clustering of the projection results of each time frame, filtering of each time frame, or truncating the reference speech. Furthermore, limited-length memory decouples the duration of the reference speech from the overhead of frame-by-frame retrieval; the length of the reference speech no longer determines the resident size of keys and values.

[0224] In practical applications, setting the number of memory entries in the frame-level reference memory to a preset constant and processing the reference frequency band characteristics as the key and value of that number of memory entries can be achieved through various methods.

[0225] One alternative approach is to perform segmented pooling of the reference frequency band features along the time dimension. Specifically, the time dimension of the reference frequency band features can be divided into a corresponding number of segments according to the number of memory entries. Each segment can be determined by a preset window length or by equal division of time frames. The values ​​of each time frame within a segment are aggregated by mean or weighted aggregation to obtain a memory entry corresponding to that segment. The key and value of the entry are obtained by projecting each segment separately. As the duration of the reference speech changes, the segmentation method is adjusted accordingly, while the number of memory entries remains determined by a preset constant. Through segmented pooling, the acoustic content repeatedly carried in adjacent time frames is merged, and the number of keys and values ​​is controlled while preserving the overall speech information.

[0226] Another alternative approach is to perform clustering on the time-frame projection results. Specifically, the reference frequency band features of each time frame can be projected to obtain frame-by-frame representations. Clustering is then performed based on the similarity between these frame-by-frame representations, merging time frames with similar acoustic content into the same memory entry. The center representation of each cluster serves as the key and value of that entry. The number of clusters is determined by the number of memory entries, and the correspondence between frame-by-frame representations and cluster centers can be determined and cached once before processing begins. Through clustering, time frames with similar articulation states are grouped into the same entry, allowing a limited number of entries to cover recurring articulation content in the reference speech.

[0227] Another alternative method is to filter time frames based on similarity or energy intensity. Specifically, time frames can be sorted according to their energy intensity or the correlation between each time frame and the features of the current frame. A predetermined number of memory entries are retained based on this sorting, while the remaining time frames are discarded or merged into adjacent entries. The resulting selections are still arranged in chronological order. Through this filtering process, time frames with low relevance to the target object are no longer counted as entries, concentrating the limited number of entries on effective pronunciation segments.

[0228] In addition, reference audio that exceeds a preset duration can be truncated before projection. For example, if the duration of the reference audio exceeds the preset duration, only a segment of continuous audio can be retained for projection, while the rest can be discarded, ensuring that the number of projected entries does not exceed the number of stored entries.

[0229] In the embodiments of this specification, the number of memory entries in the frame-level reference memory is determined by a preset constant. The number of entries accessed during frame-by-frame retrieval and the resident key-value memory are both determined by this preset constant, regardless of the actual duration of the reference speech. By using methods such as segmented pooling, clustering, filtering, or truncation, the reference frequency band features are processed into keys and values ​​of the number of entries. The effective pronunciation information in the reference speech is preserved in a limited number of entries, thereby making the processing compatible with reference speech of different durations and maintaining a stable retrieval overhead in frame-by-frame processing.

[0230] In one optional embodiment of this specification, for any time frame of the mixed speech, based on the mixed frequency band characteristics and the target object conditions, the recursive parameters of the state-space model for that time frame are determined, including: Project the conditions of the target object to obtain conditional projection features; The conditional projection features are fused with the mixed frequency band features of the time frame to obtain the fused features; Based on the fusion features, the recursive parameters for this time frame are generated through a parameter generation network.

[0231] Conditional projection features are feature representations obtained by projecting the conditions of a target object onto a conditional space that can be fused with the acoustic input. Specifically, conditional projection performs three functions: dimension alignment, nonlinear transformation, and conditional compression. For example, the dimension of the target object conditions can be 128, and the channel dimension of the acoustic input entering the time path can be 256. Conditional projection can map the conditions from 128 dimensions to 256 dimensions through a single-layer linear mapping and apply a sigmoid linear unit activation function, making the conditional projection features consistent with the acoustic input in the channel dimension. The implementation of conditional projection can include linear layers, multilayer perceptrons, or projection modules with normalization. Furthermore, the conditional projection features and the target object conditions are semantically related. Their role is not to re-identify the identity of the target object, but to enable global object conditions or frame-level retrieval conditions to participate in the generation of recursive parameters in a form that matches the acoustic input.

[0232] The fused feature is a feature representation obtained by fusing the conditional projection feature with the mixed frequency band feature of the current time frame, and is used as input to the parameter generation network. Specifically, fusion mainly uses channel-dimensional concatenation. For example, when the channel dimension of the acoustic input entering the time path is 256 and the channel dimension of the conditional projection feature is 256, the resulting fused feature has 512 components in the channel dimension. Fusion can also employ element-wise addition, gating, or low-rank modulation. Furthermore, the fused feature simultaneously carries the acoustic content of the current time frame and the acoustic identity information of the target object. Therefore, the recursive parameters determined by the fused feature change with both the current acoustic content and the target object conditions. Fusion only changes the combination form of the conditions and acoustic input, without altering the frame-by-frame and frequency band structure of the mixed frequency band feature. It should be noted that the acoustic input entering the time path is a derived feature obtained by spectral path modeling and / or local convolution processing of the mixed frequency band features of the time frame, which carries the acoustic content of the same time frame; the recursive parameters determined based on the mixed frequency band features in the embodiments of this specification include both the case of directly using the mixed frequency band features as input and the case of using the derived features as input.

[0233] The parameter generation network (PGN) is used to determine the recursive parameters of a given time frame based on fused features. Its input is a fused representation of the acoustic input and conditional projection features, and its output is the recursive parameters for that time frame. Specifically, the PGN can be implemented as a linear layer, a two-layer perceptron, or a low-rank supernetwork, preferably as a single-layer linear mapping. For example, a fused feature with an input dimension of 512 corresponds to an output dimension of 40, where the first 16 components constitute the information writing parameters, the next 16 components constitute the information reading parameters, and the last 8 components are a low-rank representation of the time-scale parameters. Furthermore, the output of the PGN can be determined per dual-path block, per time frame, and per frequency band. Each frequency band within the same dual-path block shares the same PGN, enabling the state update rule itself to have target object selectivity.

[0234] In practical applications, projecting conditions onto a target object to obtain conditional projection features can be achieved through various methods.

[0235] One alternative approach is to project the target object conditions using a single-layer linear projection with nonlinear activation. Specifically, the target object conditions can be linearly mapped to the same dimension as the acoustic input channel, for example, from 128 dimensions to 256 dimensions. After mapping, a sigmoid linear unit activation function is applied, allowing the projection to simultaneously achieve dimension alignment and condition compression. The number of parameters is determined by the condition dimension and the acoustic input channel dimension. By combining single-layer linear projection with nonlinear activation, conditional projection aligns the target object conditions with the acoustic input in the channel dimension while maintaining low computational cost.

[0236] Another alternative approach is to project the conditions of the target object using a multilayer perceptron. Specifically, the conditions of the target object can be mapped through two layers of perceptron. The hidden layer dimension can be the same as the channel dimension of the frequency band feature, for example, 128, and the output dimension can be the same as the acoustic input channel dimension, for example, 256. Nonlinear activation can be applied between the hidden layer and the output layer to ensure a stable mapping effect when the condition projection has a high condition dimension or a large range of values ​​included in the condition.

[0237] Another alternative approach is to project the target object conditions using a projection module with normalization. Specifically, the target object conditions can be normalized before linear mapping to ensure that the values ​​of the conditions are within a comparable dynamic range, or normalization can be applied to the projection result after linear mapping to make the distribution of the value of the conditional projection features approximate the distribution of the value of the acoustic input. Normalization can be performed along the feature dimension, making the conditional projection more adaptable to the fluctuations in the value of the target object conditions under different reference speech qualities.

[0238] In addition, the output dimension of conditional projection can be made consistent with the acoustic input channel dimension, and the same set of conditional projection parameters can be shared between frequency bands within the same dual-path block, or shared only on some projection weights; when the target object condition is composed of a combination of global object condition and frame-level retrieval condition, conditional projection can be applied to the combined condition, or it can be applied to the two separately and then fused.

[0239] After obtaining the conditional projection features, these features can be fused with the mixed frequency band features of the time frame. In practical applications, fusing the conditional projection features with the mixed frequency band features of the time frame to obtain the fused features can also be achieved through various methods.

[0240] One alternative approach is to concatenate the conditional projection features with the acoustic input entering the time path from the mixed frequency band features of the time frame along the channel dimension. Specifically, the two features can be concatenated end-to-end along the channel dimension. For example, when both the acoustic input and the conditional projection features have a channel dimension of 256, the resulting fused feature has 512 components along the channel dimension. Each component retains its own value before and after concatenation, and the correspondence between the acoustic input in the frequency band and the time frame does not change due to concatenation. This allows the parameter generation network to read the acoustic content and conditional information from different segments of the concatenated result.

[0241] Another alternative approach is to add the conditional projection features element-wise to the acoustic input of that time frame. Specifically, the channel dimension of the conditional projection features can be made consistent with the channel dimension of the acoustic input, and the two can be added element-wise along the channel dimension to obtain a fused feature with the same channel dimension as the acoustic input of that time frame. Since the additive fusion does not introduce an expansion along the channel dimension, the input dimension of the parameter generation network is reduced accordingly, and the number of parameters required can be reduced accordingly, making it suitable for deployments with tight computing budgets.

[0242] Another alternative method is to perform gated fusion of the two features. Specifically, the gate value can be determined based on the mixed frequency band features or conditional projection features of the time frame. After mapping the gate value to a range of 0 to 1, it is applied to the two features respectively, and the weighted results of the two features are added together to obtain the fused feature. The gate value can be determined frame by frame or frequency band by frequency band, so that the proportion of the two features in the fused feature can be adjusted according to the current acoustic content and target object conditions.

[0243] In addition, low-rank modulation can be applied to the acoustic input, or weighted summation can be combined with gating fusion. For example, channel-by-channel scaling and bias can be applied to the acoustic input according to a preset low-rank basis, or two features can be spliced ​​along the channel dimension and gating weights can be applied to the splicing result, so that the fused features have both the distinguishability of the components retained by splicing and the adjustable proportion brought by gating.

[0244] After obtaining the fused features, these features can be fed into a parameter generation network to determine the recursive parameters for that time frame. In practical applications, the recursive parameters for that time frame can be generated by the parameter generation network based on the fused features, or this can be achieved through various methods.

[0245] One alternative approach is to use a single-layer linear mapping as the parameter generation network. Specifically, the fused features can be directly output as the recursive parameters for that time frame through a single linear mapping. The output components correspond sequentially to the low-rank representations of the information writing parameters, information reading parameters, and time scale parameters. For example, when the channel dimension of the fused features is 512, the output dimension can be 40. The branch of the time scale parameters is then subjected to a single linear projection and a positive activation function to obtain the channel-wise and always positive time scale parameters. This allows the parameter generation network to maintain a low number of parameters and computational cost while generating recursive parameters frame by frame and frequency band by frequency band.

[0246] Another alternative approach is to use a two-layer perceptron as the parameter generation network. Specifically, a hidden layer can be placed between the input and output with non-linear activation applied. The dimension of the hidden layer can be the same as or half the channel dimension of the fused features, thereby establishing a non-linear mapping relationship between the fused features and the recursive parameters, so that the effect of the target object conditions on the recursive parameters is not limited to linear scaling and bias.

[0247] Another alternative approach is to use a low-rank supernetwork as the parameter generation network. Specifically, a set of low-rank factors can be generated from the fusion features, and these low-rank factors can be reconstructed into the values ​​of the recursive parameters according to the low-rank decomposition relationship. For example, two sets of low-dimensional representations can be determined first from the fusion features, and the information reconstructed from the product of the two sets of low-dimensional representations can be written into the parameters and read out the parameters. When the output dimension is large, the low-rank supernetwork can express the recursive parameters with fewer parameters, thus decoupling the number of parameters in the parameter generation network from the scale of the recursive parameters, making it easier to control the number of parameters according to the state dimension and the number of frequency bands.

[0248] Furthermore, the output of the parameter generation network can be determined per dual-path block, per time frame, and per frequency band, and each frequency band within the same dual-path block can share the same set of parameter generation networks. On the branches of the time-scale parameters, a learnable bias offset can be superimposed after linear projection to ensure that the initial time-scale parameters fall within the expected range.

[0249] For example, let's take a case where the output dimension and acoustic input channel dimension of the conditional projection are both 256, the state dimension is 16, and the low-rank representation dimension of the time-scale parameter is 8. Specifically, the target object condition is mapped to a 256-dimensional conditional projection feature by a single-layer linear projection with a sigmoid linear unit activation function. The acoustic input entering the time path of this time frame and the conditional projection feature are concatenated by the channel dimension to form a 512-dimensional fusion feature. The fusion feature is fed into a parameter generation network composed of a single-layer linear mapping, with an output dimension of 40, where the first 16 components are information writing parameters, the next 16 components are information reading parameters, and the last 8 components are the low-rank representation of the time-scale parameter. The branch of the time-scale parameter is then subjected to linear projection from 8 to 256 and a positive activation function to obtain a channel-wise and always positive time-scale parameter.

[0250]

[0251] in, Write parameters to the information. For reading parameters of information, For time scale parameters, This is the acoustic input for entering the time path in the current time frame. For the current condition characteristics, For conditional projection, Generate a network for the parameters. This indicates channel-dimensional splicing. The target object conditions and the current acoustic input thus jointly participate in the generation of recursive parameters, giving the state update rule target object selectivity. In this section, the information writing parameters are the input matrix (described later), the information reading parameters are the output matrix (described later), and the time scale parameter, after projection and a positive activation function, yields the discretized step size (described later).

[0252] In an optional embodiment, the target object conditions can further participate in the generation or modulation of other parameters in the state-space model, in addition to the recursive parameters. Specifically, the target object conditions can be used to generate or modulate the state matrix, jump parameters, gating parameters, local convolution kernels, normalization parameters, or output projections; taking the state matrix as an example, its values ​​can take a negative exponential form to ensure the stability of state transitions, and its expression can be:

[0253] in, The state matrix, The parameters of the linear mapping, For conditional projection, For the current condition characteristics, The exponent is a natural constant, and the negative sign before the exponent ensures that the state matrix always takes a negative value. This form maintains stability in the state transitions along the time axis, with the effects of the target object conditions on the state itself and on the recursive parameters working in tandem. It should be noted that the state matrix in the previous embodiment was a fixed, learnable parameter that does not change with the input, while the state matrix in this embodiment varies with the target object conditions; either can be used.

[0254] In the embodiments of this specification, conditional projection features are obtained by projecting the conditions of the target object, so that the conditions of the target object and the acoustic input entering the time path are aligned in terms of channel dimension and value distribution, and the conditional information participates in subsequent processing in a form consistent with the acoustic input; by fusing the conditional projection features with the mixed frequency band features of the time frame, fusion features are obtained, and the acoustic content of the current time frame and the acoustic identity information of the target object are organized into the same set of input representations before the input parameters are generated, so that the determined recursive parameters simultaneously reflect the acoustic content and the conditions of the target object; by generating the recursive parameters of the time frame based on the fusion features by the parameter generation network, the information writing, information reading and time scale of the state space model are all determined by the conditions, and the selection of the target object plays a role within the state update process.

[0255] In one optional embodiment of this specification, the recursive parameters include the input matrix, the output matrix, and the discretization step size; The output characteristics of this time frame are determined recursively according to the recursive parameters, including: Based on the discretization step size, the input matrix and the state matrix of the state space model are discretized respectively to obtain the discretized input matrix and the discretized state matrix; The hidden state is recursively updated based on the previous time step value of the hidden state in the discretized input matrix, discretized state matrix, and state space model. The updated hidden state is read out based on the output matrix, and the output features are determined by combining the mixed frequency band features of the time frame.

[0256] The input matrix is ​​a parameter in the state-space model used to project the current acoustic input into the hidden state space. Its value determines how the information of the current time frame is written into the hidden state. Specifically, the input matrix can be configured by channel, and its value is generated frame by frame by the parameter generation network based on the fusion features of the acoustic input entering the time path of the current time frame and the target object conditions. For example, if the number of channels in the internal dimension is 256 and the state dimension is 16, the input matrix generated for this time frame can be organized into 256 groups by channel, with 16 components in each group. Furthermore, the input matrix is ​​used to control the degree to which current information consistent with the target object is written into the hidden state, making the writing process itself selective for the target object; all frequency bands within the same depth share the same set of conditional projection and parameter generation networks.

[0257] The output matrix is ​​a parameter in the state-space model used to read feature components from the hidden state. Its value determines which information is read from the hidden state and the relative weight of each component in the readout result. Specifically, the output matrix can be configured by channel, and its value is also generated frame-by-frame by the parameter generation network based on the fused features. For example, with a state dimension of 16 and 256 channels, the output matrix generated for that time frame can be organized into 256 groups of 16 components each, and the readout result is the feature representation of that time frame and frequency band in the internal dimension. Furthermore, the input matrix is ​​used for writing to the hidden state, and the output matrix is ​​used for reading out the hidden state; both are determined by the current acoustic input and the target object conditions.

[0258] The discretization step size is a parameter used to convert parameters from continuous-time form to a frame-by-frame recursive form. Its value determines the update magnitude and effective memory length of the state on the time axis. Specifically, the discretization step size can be determined channel-by-channel and is always positive. For example, the parameter generation network first outputs a low-rank representation of the time scale with a dimension of 8, then performs a linear projection mapping to 256 channels and applies a positive activation function, and offsets the initial value with a learnable bias to ensure it falls within the expected range. Furthermore, the discretization step size, in conjunction with the state matrix and input matrix, jointly determines the degree of retention and writing ratio of the hidden state between adjacent time frames, allowing the effective memory length of the state to vary with the current acoustic content and target object conditions.

[0259] The state matrix is ​​a parameter describing the variation of each component of the hidden state along the time axis. Its value remains fixed during processing and does not change with time frames or frequency bands. Specifically, the state matrix can be configured as a set of parameters that do not change with the input, per channel. For example, with a state dimension of 16 and an internal dimension of 256 channels, the state matrix can be organized into 256 groups, each with 16 components. Its value remains consistent across all time frames and frequency bands of the dual-path block, and a negative exponential form can be used to ensure the stability of state transitions. Furthermore, the state matrix constrains the transition rules of the states themselves, while the input matrix, output matrix, and discretized step size generated frame by frame control the writing, reading, and updating of information.

[0260] The discretized input matrix is ​​a parameter obtained by discretizing the input matrix according to a discretization step size. It is used together with the discretized state matrix for the recursive update of the hidden state. Specifically, discretization can be performed in a zero-order hold (ZOH) manner. For example, the discretization step size of the current time frame is multiplied by the channel-by-channel values ​​of the input matrix to obtain the discretized input matrix for that time frame and frequency band, and its organization is consistent with that of the input matrix. Furthermore, the discretized input matrix determines the proportion of acoustic input in the current time frame written into the hidden state, and its values ​​vary per dual-path block, per time frame, and per frequency band.

[0261] The discretized state matrix is ​​a parameter obtained by discretizing the state matrix according to a discretization step size. It is used to determine the degree to which the value of the hidden state at the previous time step is retained in the current time frame. Specifically, discretization can be performed in an exponential form, for example, by exponentiating the product of the discretization step size and the channel-by-channel value of the state matrix, resulting in a retention coefficient with values ​​between zero and one. This coefficient is determined by channel; the larger the discretization step size, the smaller the retention coefficient. For different frequency bands within the same time frame, the discretization step size is determined by the acoustic input and conditional characteristics of each frequency band, thus the discretized state matrices of each frequency band can be different. Furthermore, the discretized state matrix determines the time range within which the historical information carried by the hidden state can continue on the time axis.

[0262] Readout is the process of extracting feature components from the updated latent state and forming the output features for the current time frame. Specifically, readout multiplies the output matrix with the updated latent state in each channel and state dimension to obtain the feature components of that time frame and frequency band in the internal dimensions. For example, with a state dimension of 16 and 256 channels, the readout result still has 256 channel components. Furthermore, the readout result is combined with the mixed frequency band features of that time frame, so that the output features simultaneously carry the historical information accumulated by the latent state and the real-time acoustic content of the current time frame.

[0263] In practical applications, the input matrix and the state matrix of the state-space model are discretized based on the discretization step size to obtain the discretized input matrix and the discretized state matrix, which can be achieved through various methods.

[0264] One alternative approach is to treat the continuously varying input within a time frame as constant using the zero-order preservation method, and then rearrange the analytical solution of the continuous state equation under this constant input into a frame-by-frame recursive form. Specifically, the discretization step size can be multiplied by the channel-by-channel values ​​of the state matrix, and the exponent can be taken to obtain the channel-by-channel retained coefficients as the discretized state matrix. The discretization step size, the state matrix, and the input matrix are then combined according to the same set of relationships to obtain the discretized input matrix for that time frame. Through the zero-order preservation method, the correspondence between the parameters in the continuous form and the frame-by-frame recursive form is maintained; the discretization process only changes the values ​​and combinations of the parameters.

[0265] Another alternative approach is to calculate the product of the discretization step size and the state matrix channel by channel using an exponential function, and then use the resulting value as the value of the discretized state matrix. Specifically, the input to the exponential function can be the product of the discretization step size and the channel-by-channel value of the state matrix; its output is between zero and one, decreasing as the product increases. The state matrix can be calculated using a negative exponential form, preventing the value of the discretized state matrix from diverging with the number of processed frames. By calculating the value using the exponential function, the historical information of each channel can decay according to its respective time scale.

[0266] Another alternative approach is to combine the discretization step size, state matrix, and input matrix according to a channel-by-channel multiplication-addition relationship, allowing discretization to be performed separately at the channel granularity. Specifically, the product of the discretization step size and the state matrix values ​​can be determined first for each channel. Then, this product is combined with the channel-by-channel values ​​of the input matrix according to the same channel correspondence to obtain the discretized input matrix. Since each frequency band has its own acoustic input and conditional characteristics in the same time frame, the discretization step size of each frequency band can be different, thus the discretized input matrix and discretized state matrix of each frequency band are determined separately. By combining them by channel, the state update rhythms of different frequency bands within the same time frame can be independent of each other.

[0267] Furthermore, the size of each matrix can be kept consistent with that before discretization after discretization, so that discretization only changes the values ​​without changing the shape of the state update; the discretization step size can be determined channel by channel and always be positive, or it can be taken according to the shared part of the channel within the same dual-path block, so as to reduce the amount of computation per frame.

[0268] In practical applications, the hidden state can be recursively updated based on the previous time step value of the discretized input matrix, the discretized state matrix, and the hidden state of the state-space model. This can also be achieved through various methods.

[0269] One alternative approach is to perform an update based on a combination of retention and writing. Specifically, the discretized state matrix can be multiplied by the previous time step value of the hidden state to obtain the retained portion of the historical information from the previous time step in the current time frame. The discretized input matrix can then be multiplied by the acoustic input of the current time frame to obtain the portion of the hidden state written into the current time frame. The two portions are then added together to obtain the updated hidden state. For example, the updated hidden state can be represented as:

[0270] in, This is the updated hidden state. The value is the value taken in the previous time step of the hidden state. For the discretized state matrix, To discretize the input matrix, This is the acoustic input for entering the time path in the current time frame. Through a combination of retention and writing, historical and current information are merged in the same hidden state according to their respective proportions.

[0271] Another alternative approach is to maintain and update the hidden state separately for each frequency band. Specifically, each frequency band within the same time frame holds its own hidden state, and each band independently updates itself based on its own discretized state matrix, discretized input matrix, and the value from the previous time step. The hidden states of different frequency bands do not exchange values ​​and can be expanded in parallel along the frequency band dimension. By maintaining and updating the hidden state separately for each frequency band, the relationships between frequency bands are handled by the spectral path.

[0272] Another alternative approach is to update sequentially according to the arrival order of time frames, writing the new value back to the original location after each update. Specifically, during streaming processing, after each time frame arrives, an update is performed based on the discretized state matrix, discretized input matrix, and hidden state of that frequency band. The updated hidden state is written back to the original storage location, overwriting the value of the previous time frame. By writing the new value back to overwrite, the processing does not need to save the hidden state sequence of historical time frames.

[0273] Furthermore, the updates of the hidden state can be made independent of time frames that have not yet arrived within the same time frame; in time frames and frequency bands where the target object component is dominant, the value of the discretized state matrix can be increased accordingly, so that more historical information consistent with the target object can be preserved.

[0274] In practical applications, the updated hidden state can be read out based on the output matrix, and the output features can be determined by combining the mixed frequency band features of the time frame. This can also be achieved through various methods.

[0275] One alternative approach is to read out the updated hidden state using the output matrix and supplement the readout result with the mixed frequency band features of that time frame as skip terms. Specifically, the readout component can be obtained by multiplying the output matrix with the updated hidden state, and the acoustic inputs entering the time path of that time frame according to the same set of channel correspondences can be superimposed on the readout component as skip terms to obtain the output features of that time frame; for example, the output features can be expressed as:

[0276] in, For output features, For the output matrix, This is the updated hidden state. For jump parameters, This is the acoustic input for entering the time path in the current time frame. By superimposing the readout components with the jump terms, the output features contain both historical information accumulated from the latent states and the instantaneous acoustic content of the current time frame.

[0277] Another alternative approach is to perform readouts band-by-band and then weight and superimpose the mixed-band features of the time frame channel-by-channel. Specifically, each band in the same time frame performs readouts on its respective hidden state using its own output matrix, obtaining the readout components for each band. Subsequently, each band applies channel-by-channel scaling and biasing to the corresponding band components in the mixed-band features of the time frame, and the scaling and biasing results are superimposed on the readout components. By performing readouts and weighted superposition band-by-band, the output features maintain a frame-by-frame, band-by-band structure.

[0278] Another alternative approach is to make the hop parameters, like the state matrix, channel-wise learnable parameters that do not change with time frames or frequency bands. Specifically, the hop parameters can be configured per channel, with their values ​​remaining consistent across all time frames and frequency bands of the dual-path block, and shared by all frequency bands within the same depth. Since the hop parameters do not change with time frames, no additional parameter generation overhead is incurred during frame-by-frame processing. By keeping the hop parameters fixed, the portion of the output features from the acoustic input of the current time frame has a stable transmission path.

[0279] In addition, the readout components and skip terms can be combined according to the channel-by-channel weighting relationship, so that the relative contribution of the two parts in the output features can be adjusted according to the acoustic content of the current time frame; the readout process is executed once in each time frame, and the results can be organized according to the correspondence between frequency bands and time frames.

[0280] For example, a discretization step size of 16 channels with a constant positive value, a state dimension of 16, an internal dimension of 256 channels, and a time-scale low-rank representation dimension of 8 are used for further explanation. Specifically, the parameter generation network outputs recursive parameters based on the fusion representation of the acoustic input and conditional features of the time frame and frequency band. Both the input and output matrices are organized into 256 groups, with 16 components in each group. The discretization step size is obtained by linear projection from 8 to 256 and a positive activation function to obtain channel-wise values. The discretized state matrix is ​​obtained by exponentiation of the product of the discretization step size and the channel-wise values ​​of the state matrix. The discretized input matrix is ​​obtained by multiplying the discretization step size and the channel-wise values ​​of the input matrix. The hidden state is updated once based on the values ​​of both and the previous time step. The updated hidden state is read out by the output matrix. The readout result is then superimposed with the mixed frequency band features of the time frame by channel weights to obtain the output features of the time frame.

[0281] In an optional embodiment, the initial value of the hidden state and the switching between sessions can also be processed. Specifically, at the beginning of each streaming session, the hidden state can be set to zero, or it can be generated by the target object conditions. For example, the global object conditions can be linearly mapped and reshaped according to the state dimension and internal dimension to serve as the initial hidden state, so that the processing carries the prior information of the target object at the beginning stage. In the case of session switching or long silence, the hidden state can be decayed by a coefficient ratio instead of being directly cleared to zero, so that the prior identity of the target object can be preserved, while the expired interference information can be released.

[0282] In an optional embodiment, the scale of the persistent state during frame-by-frame recursion can also be quantified. Specifically, the number of channels in the internal dimension can be twice that of the feature dimension; for example, when the feature dimension is 128, the number of channels in the internal dimension is 256. The number of hidden state elements in a single layer and single frequency band can be the product of the internal dimension and the state dimension; for example, the product of 256 and 16 is 4096. Each frequency band can write the acoustic input of the current time frame into a circular convolution buffer and retrieve the local convolution result as the acoustic input for recursion. The number of elements in the circular convolution buffer can be the product of the internal dimension and the length of the local causal convolution kernel minus one; for example, when the kernel length is 4, the product of 256 and 3 is 768. The total of the two is 4864. Since the spectral path does not maintain cross-frame states, the persistent state of the entire network comes from the time paths of each dual-path block. For example, when the number of dual-path blocks is 36 and the number of frequency bands is 36, the product of 36 and 36 and 4864 is approximately 6.30 × 10⁻⁶. 6 Each element is approximately 12.0 MB in 16-bit floating-point representation; this value is uniquely determined by the network structure and does not increase with the duration of the input speech.

[0283] In an alternative embodiment, training data can also be constructed and training configurations set as follows.

[0284] Specifically, a dual-speaker hybrid can be constructed using corpora of Chinese and English speakers. The clean speech is uniformly resampled to, for example, 16kHz. Reverberation is simulated using room impulse response (RIR) and superimposed with real environmental noise. The reverberation time is, for example, 0.2 to 1.0 s, and the signal-to-noise ratio is, for example, [missing information]. 6 to 3 dB, the relative energy difference between the two speakers is, for example, The data is divided into 7:2:1 segments, with a speaker-microphone distance of 5 to 5 dB and a speaker-microphone distance of 0.66 to 2.0 m. The reference speech is a clean 3-second segment whose content differs from the sentence to be processed. The training objective can be composed of both scale-invariant signal-to-noise ratio (SI-SNR) loss and multi-resolution time-frequency domain loss, with the multi-resolution time-frequency domain loss using transformation sizes of 512, 1024, and 2048, for example. The training process can employ an adaptive moment estimation optimizer (Adam), with an initial learning rate of, for example, 1 × 10⁻⁶. -4 If the validation loss does not decrease for 5 consecutive rounds, the learning rate can be halved, the batch size can be 16, and 8 graphics processing units (GPUs) can be used to train for 200 rounds, with a total number of model parameters of approximately 15.89M.

[0285] In an optional embodiment, the effects of target object conditions in recursive parameter generation and frame-by-frame recursion by frequency band can be illustrated through comparative experiments. Specifically, in the comparison of conditionalization methods, compared with the method of fusing conditions into frequency band features by element-wise multiplication and the method of retrieving reference speech information by using current frame features as the query reference layer by layer, the scale-invariant signal-to-noise ratio improvement is, for example, increased from 9.23dB and 11.25dB to 13.63dB, and the short-time objective intelligence (STOI) is, for example, increased from 0.83 and 0.88 to 0.89. In the comparison of temporal modeling methods, compared with the method of maintaining hidden states by frequency band and recursively advancing along the time axis, the scale-invariant signal-to-noise ratio improvement of the method adopted in the embodiments of this specification is, for example, increased by 5.37dB relative to the former and 3.11dB relative to the latter.

[0286] In publicly available real-time target speech extraction evaluations, the scheme employed in the embodiments of this specification, when the number of parameters is reduced from approximately 25.05M and 27.24M to approximately 15.89M, and the multiply-accumulate operation per second is reduced from approximately 38.91G / s and 38.57G / s to approximately 30.01G / s, the target extraction error rate is reduced from 0.809 and 0.805 to 0.713, the target activity detection accuracy is improved from 0.821 and 0.836 to 0.849, the speaker similarity is improved from 0.422 and 0.456 to 0.524, and the no-reference speech quality score is improved from 1.714 and 1.670 to 2.151. The above comparisons demonstrate that by involving the target object conditions in the generation of recursive parameters and maintaining the hidden state by frequency band in frame-by-frame recursion, the extraction quality and speaker similarity of the target speech can be improved without increasing the number of parameters.

[0287] In the embodiments of this specification, the input matrix and state matrix are discretized based on the discretization step size to obtain the discretized input matrix and discretized state matrix. The continuous parameters are converted into a frame-by-frame recursive form. The writing ratio and retention degree of the state on the time axis are determined by the values ​​of the input matrix and state matrix, respectively, and the update amplitude is uniformly controlled by the discretization step size. The hidden state is updated recursively based on the discretized input matrix, the discretized state matrix, and the previous time value of the hidden state. Historical information is retained and supplemented in the updated hidden state according to the proportion corresponding to the current acoustic content. The size of the hidden state is determined by the model configuration and does not increase with the duration of the processed speech. The updated hidden state is read out based on the output matrix and the mixed frequency band features of the time frame are combined to determine the output features. The historical information accumulated by the hidden state and the real-time acoustic content of the current time frame are reflected in the same output, providing a stable source of features for subsequent generation of masks by frequency band and reconstruction of the speech of the target object.

[0288] In one optional embodiment of this specification, signal reconstruction is performed on the mixed spectral features based on the output features of each time frame to obtain the target speech of the target object, including: Gated projection is performed on the output features of each time frame to obtain a complex mask; The complex mask is concatenated along the frequency band boundaries to obtain the complete mask, and the complete mask is multiplied by the mixed spectral features to obtain the target spectrum. The target speech of the target object is obtained by performing an inverse time-frequency transform and overlapping on the target spectrum.

[0289] Gated projection is a feature transformation performed on the output features of each time frame to generate a complex mask. The mapping process involves element-wise weighted modulation of candidate values ​​by a gating variable. Specifically, gated projection can employ a two-layer perceptron with a gating structure. The two perceptrons generate gating variables and candidate values ​​respectively. The gating variable is then mapped to a range of 0 to 1 via an activation function and multiplied element-wise with the candidate value. The complex mask can be estimated separately for each frequency band; for example, the complex mask M_k for the k-th frequency band is obtained. Furthermore, the output dimension of gated projection corresponds to the number of frequency points covered by the frequency band on the frequency axis. Its output includes both real and imaginary components, thus determining both the amplitude and phase correction values ​​simultaneously. Gated projection is performed frame-by-frame.

[0290] A complex mask is a mask representation obtained by gated projection, containing both real and imaginary parts, used to correct the complex values ​​of mixed spectral features at each frequency point. Specifically, complex masks can be organized by frequency band, with each frequency band of each time frame corresponding to the number of real and imaginary components of the frequency points covered by that band. The real and imaginary parts can be organized by channel dimension, corresponding in layout to the real and imaginary parts of the mixed spectral features. Furthermore, compared to real masks, complex masks can correct both amplitude and phase simultaneously; when multiplied by mixed spectral features, they are equivalent to scaling the amplitude and rotating the phase.

[0291] A frequency band boundary is the dividing position between adjacent frequency bands on the frequency axis, used to determine the range of continuous frequency points covered by each frequency band. Specifically, the frequency band boundary can be determined by the frequency band division result. For example, when 257 frequency points are divided into 36 non-uniform frequency bands, each frequency band covers 2, 4, 8, 11, and 18 frequency points on the frequency axis respectively, forming three dividing positions at the 16th, 48th, and 96th frequency points. Furthermore, the frequency band boundary provides a positional basis for the splicing of complex masks, allowing the complex masks of each frequency band to be written into the corresponding frequency point intervals according to their order on the frequency axis; the frequency band boundary is consistent with the frequency band division result.

[0292] A complete mask is a mask representation that covers the entire frequency range, obtained by concatenating the complex masks of each frequency band along the band boundaries. Specifically, the frequency axis structure of the complete mask is consistent with the mixed spectral characteristics, and its number and spacing of frequency points are the same as those of the mixed spectral characteristics. For example, under the condition of 257 non-repeating frequency points and a frequency point spacing of 31.25Hz, the complete mask also contains real and imaginary components at 257 frequency points. Furthermore, the complete mask and the mixed spectral characteristics correspond one-to-one at frequency points, allowing mask operations to be performed frequency-by-frequency on the same frequency axis; concatenation only changes the arrangement order of the complex masks on the frequency axis.

[0293] The target spectrum is a complex spectrum obtained by performing a complex multiplication of the complete mask and the mixed spectral features. It is used to characterize the spectral components belonging to the target object in the mixed speech. Specifically, the target spectrum contains real and imaginary parts. The real part is determined by subtracting the product of the imaginary part of the mixed spectral features and the imaginary part of the complete mask from the product of the real part of the mixed spectral features and the real part of the complete mask. The imaginary part is determined by adding the product of the real part of the mixed spectral features and the imaginary part of the complete mask to the product of the imaginary part of the mixed spectral features and the real part of the complete mask. Furthermore, the amplitude and phase of the target spectrum are both masked and corrected. Their values ​​reflect the intensity and phase relationship of the components belonging to the target object in the current time frame and frequency band, providing a frequency domain basis for subsequent reconstruction into a time-domain waveform.

[0294] Inverse time-frequency transform (IFT) is a process that converts the target spectrum in the frequency domain back into a time-domain waveform. It is used to restore the masked spectrum into a speech signal suitable for playback and transmission. Specifically, IFT can correspond to the aforementioned time-frequency transform. For example, when the time-frequency transform uses short-time Fourier transform (SFT), the IFT can use inverse short-time Fourier transform (iSTFT). Its window length and frame shift are the same as the forward transform; for example, a window length of 512 points (32ms) and a frame shift of 256 points (16ms) can be performed frame-by-frame. Furthermore, IFT utilizes both the amplitude and phase of the target spectrum, allowing the phase correction introduced by the complete mask to be reflected in the time-domain waveform.

[0295] Overlapping and summing is a processing method that splices together time-domain segments from each time frame obtained by inverse time-frequency transform according to the overlap relationship between adjacent time frames. It is used to combine the segments reconstructed frame by frame into a continuous time-domain waveform. Specifically, when the frame shift is less than the window length, the time-domain segments of adjacent time frames overlap on the time axis. For example, when the window length is 512 points and the frame shift is 256 points, the overlap rate between adjacent time frames is 50%. Overlapping and summing can accumulate the values ​​of the time-domain segments of each time frame within the overlapping region and apply compensation corresponding to the windowing to the values ​​of each time frame. Furthermore, overlapping and summing allows the results reconstructed frame by frame to be spliced ​​into a continuous speech stream.

[0296] In practical applications, gating projection of the output features of each time frame to obtain a complex mask can be achieved through various methods.

[0297] One alternative approach is to use a two-layer perceptron with a gating structure for gated projection. Specifically, the output features of the k-th frequency band in the time frame can be mapped through a two-layer perceptron to obtain two outputs: a gated value and candidate values. The gated value is then mapped to a value between 0 and 1 using an activation function and multiplied element-wise with the candidate values ​​to obtain a complex mask for the frequency band. The complex mask can be determined according to the following gating relationship:

[0298] in, For the first Complex masks for each frequency band, For gating quantity, For candidate values, An activation function that maps values ​​to the range 0 to 1. For element-wise multiplication, The frequency band is numbered. Through a gating structure, candidate values ​​are filtered by a gating quantity before being written into the complex mask, thus constraining the range of values ​​during the estimation process.

[0299] Another alternative approach is to estimate the complex mask separately for each frequency band and share some projection parameters. Specifically, each frequency band can use its own independent output projection to accommodate the differences in dynamic range between different frequency bands, while sharing the hidden layer parameters of the two-layer perceptron to control the number of parameters; for frequency bands with different widths, the output dimension of the candidate values ​​corresponds to the number of frequency points covered by that frequency band, so that the estimation granularity of the complex mask is consistent with the granularity of the frequency band division.

[0300] Another alternative approach is to use different branching structures to generate the gating variable. Specifically, the gating variable can be obtained from a single-layer linear mapping combined with normalization, or from a gating branch in a two-layer perceptron that is independent of the candidate value branch. The hidden layer dimension of the gating branch can be smaller than that of the candidate value branch. Furthermore, a bias term can be superimposed on the gating variable to make the initial value of the complex mask approximate the identity mapping. Through these branching structures, the computational complexity and expressive power of the gating projection can be adjusted according to the deployment conditions.

[0301] In addition, the output features can be normalized by frequency band before gating projection, for example, by normalizing along the channel dimension, so that the values ​​of each frequency band are in a consistent dynamic range, thereby reducing the mask estimation deviation caused by the difference in bandwidth.

[0302] In practical applications, complex masks can be spliced ​​together according to frequency band boundaries to obtain a complete mask, which can also be achieved through various methods.

[0303] One alternative method is to sequentially write the complex masks of each frequency band into the frequency point intervals corresponding to the complete mask, according to the order of the frequency bands on the frequency axis. Specifically, the range of continuous frequency points covered by each frequency band can be determined according to the frequency band division result. For example, when 257 frequency points are divided into 36 non-uniform frequency bands, each frequency band covers 2, 4, 8, 11, and 18 frequency points respectively. The mask components of each frequency band are written into the corresponding frequency point positions to obtain a complete mask that is continuously arranged on the frequency axis, and its frequency axis structure is completely consistent with the hybrid spectrum characteristics.

[0304] Another alternative method is to splice the masks according to the channel organization of the mask components. Specifically, the complex masks for each frequency band can be organized into two components, one for the real part and one for the imaginary part. During splicing, the real components are first spliced ​​into a complete real mask according to the frequency band boundaries, and the imaginary components are then spliced ​​into a complete imaginary mask. The complete mask is then output according to the same channel organization, thus avoiding cross-recombination of the real and imaginary parts during the splicing process, and ensuring that the complete mask corresponds to the channel layout of the mixed spectral characteristics.

[0305] Another alternative method is to perform boundary alignment on the complex masks of each frequency band before splicing. Specifically, an index can be generated based on the position of the endpoint frequencies of each frequency band on the full frequency axis, and the masks of each frequency band can be written into the full mask according to the index. If the frequency band boundaries are not aligned with the center of the frequency points, the masks can be resampled according to the center frequency of each frequency band before being written. Through boundary alignment, the splicing result will not have frequency misalignment due to changes in bandwidth.

[0306] In addition, the values ​​of the complete mask can be limited after splicing, so that the values ​​of the complete mask are within a preset range.

[0307] In practical applications, the target spectrum can be obtained by performing complex multiplication on the complete mask and the mixed spectral features. This can be achieved through various methods.

[0308] One alternative method is to calculate the real and imaginary parts of the target spectrum separately according to the complex multiplication rules. Specifically, the real part of the target spectrum is obtained by subtracting the product of the imaginary part of the mixed spectral features and the imaginary part of the complete mask from the product of the real part of the mixed spectral features and the real part of the complete mask; the imaginary part of the target spectrum is obtained by adding the product of the real part of the mixed spectral features and the imaginary part of the complete mask to the product of the imaginary part of the mixed spectral features and the real part of the complete mask. The formula can be:

[0309]

[0310] in, It has mixed spectral characteristics. For the complete mask, For the target spectrum, and These represent the real and imaginary parts, respectively. By calculating the real and imaginary parts according to the rules of complex multiplication, the amplitude and phase corrections carried by the complex mask can simultaneously apply to the mixed spectral characteristics.

[0311] Another alternative approach is to perform complex multiplication in terms of amplitude and phase. Specifically, the mixed spectral features and the complete mask can be represented as combinations of amplitude and phase, respectively. Multiplying their amplitudes and adding their phases, then converting the result back to real and imaginary parts, yields the target spectrum. Using amplitude and phase representation, the effect of the complex mask on the mixed spectral features can be directly interpreted as amplitude scaling and phase rotation.

[0312] Another alternative approach is to perform complex multiplication on a band-by-band basis. Specifically, the mixed spectral features and the complete mask can be divided into the same frequency bands according to the band boundaries. Within each frequency band, a point-by-point multiplication can be performed, and the results of each band can be concatenated according to the band boundaries to form a target spectrum covering the entire frequency range. This ensures that the organization of complex multiplication is consistent with the estimation method of complex mask.

[0313] In addition, scaling factors can be applied to one or both of the mixed spectral features and the complete mask before complex multiplication to keep the energy of the target spectrum within the expected range.

[0314] In practical applications, the target speech of the target object can be obtained by performing inverse time-frequency transformation and overlapping addition on the target spectrum. This can also be achieved through various methods.

[0315] An alternative method is to perform an inverse time-frequency transformation on the target spectrum frame by frame, and then overlap and add the transformation results of adjacent time frames. For specific implementation details, please refer to the aforementioned embodiments, which will not be repeated here.

[0316] Another alternative approach is to incrementally perform inverse time-frequency transformation and overlap addition according to the arrival order of time frames. Specifically, an output buffer covering several time frames can be maintained in the streaming process. After the target spectrum of each time frame completes the inverse transformation, its time-domain segment is accumulated into the overlapping area of ​​the output buffer. The accumulated portion is output as a segment of the target speech, while the unaccumulated portion is retained for accumulation in subsequent time frames, so that the target speech can be continuously output according to the arrival rhythm of the time frames.

[0317] Another alternative approach is to perform frequency-axis restoration of the target spectrum before the inverse time-frequency transform. Specifically, the target spectrum can be restored to a frequency point arrangement consistent with the characteristics of the mixed spectrum according to the frequency band boundaries, and the non-repeating frequency points can be conjugately symmetrically expanded before performing the inverse transform. For implementations using learnable filter banks or subband analysis filter banks, the inverse transform can be performed according to the synthesis filter bank corresponding to the forward transform, so that the input of the inverse time-frequency transform and the output of the forward transform maintain a frequency-axis correspondence.

[0318] In addition, the target speech can be normalized as a whole after overlapping and adding, or the splicing results of adjacent time frames in the overlapping area can be smoothed to keep the target speech continuous on the time axis.

[0319] For example, a configuration with a sampling rate of 16kHz, a window length of 512 points (32ms), a frame shift of 256 points, 257 non-repetitive frequency points, and 36 frequency bands is used for further explanation. Specifically, the output features of each time frame are projected by frequency bands through a two-layer perceptron and a gated projection network to obtain complex masks of 36 frequency bands. These masks are then concatenated sequentially along the frequency band boundaries to obtain a complete mask covering 257 frequency points. This mask is multiplied by the mixed spectral features frequency-by-frequency according to the complex multiplication rule to obtain the target spectrum. An inverse short-time Fourier transform is performed on the target spectrum, and the two spectra are overlapped and added according to a 50% overlap rate between adjacent time frames to obtain the target speech.

[0320] In an optional embodiment, a target activity detection module can also be set up based on the mask estimation. The target activity detection module shares the backbone representation with the mask estimation and outputs the target object activity probability in parallel to represent the target object activity status in the current time frame. When the target object activity probability is lower than a preset threshold, the output of the current time frame can be attenuated or set to zero, thereby suppressing the false triggering output during non-target object sounding periods.

[0321] In the embodiments of this specification, a complex mask is obtained by gating the output features of each time frame. The value of the mask is determined by the gating relationship of the output features themselves, thereby suppressing components inconsistent with the target object while estimating the spectral correction amount. By splicing the complex masks of each frequency band according to the frequency band boundaries into a complete mask covering the entire frequency range, the organization of the mask is consistent with the frequency structure of the mixed spectral features. By applying the complete mask to the mixed spectral features with complex multiplication, the real and imaginary parts of the target spectrum are corrected simultaneously. Therefore, compared with the method of correcting only the amplitude, the target speech is improved in both timbre and phase continuity.

[0322] In an optional embodiment of this specification, when there are multiple target objects, the method further includes: For any target object, the target object conditions are determined based on the reference frequency band characteristics of the reference speech of the target object. For any given time frame, based on the mixed frequency band characteristics of that time frame and the target object conditions corresponding to that target object, the recursive parameters of that target object in that time frame are determined, and the output characteristics of that target object are recursively determined according to the recursive parameters of that target object in that time frame. Signal reconstruction is performed based on the output features of each target object, and the target speech of multiple target objects is output in parallel.

[0323] Multi-target parallel output is a method of simultaneously performing condition determination, frame-by-frame recursion, and signal reconstruction for multiple target objects within a single processing flow of mixed speech, and outputting multiple target speech streams. Specifically, the multiple target objects addressed by multi-target parallel output each have independent reference speech. Each target object is processed separately in the condition determination and frame-by-frame recursion stages, while the mixed speech carrying acoustic content remains the same input. Therefore, the multiple outputs correspond to each other in acoustic content and are distinguished in terms of target object attribution.

[0324] Furthermore, multi-objective parallel output can include three coordinated processes: multi-condition determination, multi-path recursion, and multi-path reconstruction. The parameters of the backbone network are shared among the target objects, while recursion is performed in parallel on batch dimensions or newly added target dimensions. The overhead related to the number of target objects in the processing flow is concentrated on the recursion side; that is, the state size is proportional to the number of target objects, while the number of parameters does not increase with the number of target objects. Therefore, when adding a new target object, it is only necessary to determine the target object conditions and allocate independent hidden states and caches for it; the backbone network does not need to be reconfigured or copied.

[0325] In practical applications, for any target object, the target object conditions corresponding to the target object can be determined based on the reference frequency band characteristics of the reference speech of the target object, which can be achieved through various methods.

[0326] One alternative approach is to pool and project the reference frequency band features of each target object separately, and form a frame-level reference memory for each target object. Specifically, for any target object, the reference frequency band features of the target object's reference speech can be pooled along the frequency band dimension and the time dimension, and the pooling result can be projected to obtain the global object condition corresponding to the target object; at the same time, the reference frequency band features of the target object can be projected to obtain keys and values, forming the frame-level reference memory corresponding to the target object, which is used to retrieve the local articulation and spectral information of the target object according to the current time frame, so that multiple target objects can each obtain a complete conditional representation.

[0327] Another alternative approach is to have all target objects share the same set of reference coding parameters, using their respective reference speech as input to obtain the target object conditions. Specifically, the reference speech of each target object can be processed using the same frequency band division results and the same normalization and projection parameters. The coding parameters are shared among all target objects, so increasing the number of reference speech entries does not cause an increase in the number of parameters. The conditions of each target object are calculated separately using the same reference coding structure, maintaining a consistent dimension, which facilitates unified organization according to the target dimension in subsequent processing.

[0328] Another alternative approach is to group target objects and determine the conditions for each target object in parallel on the batch dimension or a newly added target dimension. Specifically, the reference frequency band features of each target object can be organized into a batch input on the target dimension, and the target object conditions for each target object can be obtained in a single forward computation. For the case where the same target object has multiple reference speech segments, the condition representations of each reference speech segment can be extracted separately and then combined according to reliability weights to obtain the condition representation of the target object.

[0329] In addition, the target object conditions and frame-level reference memory of each target object can be determined and cached once before processing begins. In subsequent frame-by-frame processing, only the corresponding conditions and memory are read according to the target object, without repeatedly accessing the reference speech, thereby decoupling the frame-by-frame overhead from the duration of the reference speech.

[0330] After each target object obtains its own conditional representation, the recursive process for each target object within the same time frame needs to be completed separately with the participation of its respective conditions. In practical applications, for any given time frame, the recursive parameters of the target object in that time frame can be determined based on the mixed frequency band characteristics of that time frame and the target object conditions corresponding to that target object. This can also be achieved through various methods.

[0331] One alternative approach is to share the hybrid frequency band features among the target objects, while each target object participates in the generation of recursive parameters based on its own conditions. Specifically, the hybrid frequency band features obtained after frequency band division of the same mixed speech stream are the same for all target objects. The target object conditions corresponding to this target object can be projected into conditional projection features, and these conditional projection features are fused with the hybrid frequency band features of the current time frame. The parameter generation network then generates the recursive parameters of the target object for that time frame based on the fused features, so that multiple target objects obtain different recursive parameters in the same time frame.

[0332] Another alternative approach is to first determine a set of basic parameters based on the mixed frequency band characteristics, and then modulate these basic parameters according to the conditions of each target object. Specifically, basic parameters related to information writing, information reading, and state time scales can be generated according to the mixed frequency band characteristics of the time frame. For any target object, modulation coefficients are generated based on the target object conditions corresponding to that target object, and the basic parameters are scaled, biased, or gated to obtain the recursive parameters of that target object in that time frame. Calculations related to acoustic content are shared among all target objects, while calculations related to the identity of the target object are performed separately for each target object.

[0333] Another alternative method is to combine the conditions of each target object with the mixed frequency band features of the time frame on the target dimension to obtain the recursive parameters of each target object in that time frame in one step. Specifically, the mixed frequency band features of the time frame can be copied on the target dimension and combined with the conditions of each target object to form a batch input according to the target dimension. The parameter generation network then generates the recursive parameters of each target object in that time frame in batches. The recursive parameters can also be generated separately according to the frequency band, that is, a set of recursive parameters can be generated for any target object and any frequency band in any time frame.

[0334] Furthermore, the condition granularity can be selected separately for each target object. For example, for any target object, the global object condition of the target object can be used to participate in the generation of recursive parameters in shallow dual-path blocks, while the frame-level retrieval condition of the target object can be used to participate in the generation of recursive parameters in deep dual-path blocks. This allows the condition granularity of each target object to be determined separately with the network depth, and the condition selection between each target object does not affect each other.

[0335] Since the recursive parameters for each target object in this time frame have been determined separately, the recursive process for each target object can be executed independently according to its respective recursive parameters. In practical applications, determining the output characteristics of a target object recursively according to its recursive parameters in this time frame can also be achieved through various methods.

[0336] One alternative method is to update the hidden state corresponding to the target object according to the recursive parameters of the target object in that time frame, and then read the output features of the target object in that time frame from the updated hidden state. Specifically, the acoustic information of the current time frame can be written into the hidden state corresponding to the target object according to the recursive parameters of the target object in that time frame, and the degree of retention of historical information in that time frame can be determined according to parameters related to the state time scale, thereby reading the feature components related to the target object as the output features of the target object in that time frame.

[0337] Another alternative approach is to maintain a separate local convolutional buffer for each target object and perform the recursion in parallel along the target dimension. Specifically, each target object maintains its own local convolutional buffer and hidden state within its own frequency band. The recursion of each target object in the same time frame can be executed in parallel along the target dimension. The state update of each target object only uses its own target object conditions and recursion parameters, and they do not exchange hidden states with each other, thus ensuring that the recursion process of each target object does not interfere with each other.

[0338] Another alternative approach is to share the readout mapping among the target objects, while each target object provides the readout content from its own hidden state. Specifically, when each target object reads feature components from the updated hidden state, the output matrix and skip connection parameters used for the readout can be shared among the target objects. However, the hidden states of each target object carry different content due to different conditions, thus allowing for distinct output features even with a shared readout mapping. The output features of each target object can be cached and organized separately for each target object, enabling subsequent reconstruction by target object.

[0339] Furthermore, the size of the hidden state and the cache can be determined according to the number of target objects. For example, the hidden state and local convolutional cache of each target object can be maintained separately, and their size is determined by the network structure and the number of target objects, and does not increase with the duration of processed speech; at the beginning of the streaming session corresponding to each target object, its hidden state can be set to zero, or it can be generated by the target object conditions of that target object, so that the streaming processing has the identity prior of the target object at the beginning stage.

[0340] After the output features of each target object are determined, the reconstruction process of each target object can be completed separately on the same mixed spectral features. In practical applications, signal reconstruction can be performed separately based on the output features of each target object, and the target speech of multiple target objects can be output in parallel, which can also be achieved through various methods.

[0341] An alternative approach is to perform mask estimation and signal recovery separately for each target object. For specific implementation details, please refer to the aforementioned embodiments, which will not be repeated here.

[0342] Another alternative approach is to organize multi-path reconstruction computations in batches along the target dimension. Specifically, the complete masks of each target object can be organized as batch inputs along the target dimension, and complex multiplications can be performed in batches with the same mixed spectral features to obtain multiple target spectra at once. Inverse time-frequency transform and overlap addition can then be performed on each target object to obtain multiple target speech, allowing the reconstruction process in multi-target scenarios to share the same reconstruction structure as in single-target scenarios.

[0343] Another alternative approach is to allow all target objects to share the same reconstruction structure, while outputting the reconstruction results of each path to a separate output channel corresponding to that target object. Specifically, the projection parameters used for mask estimation and signal recovery can be shared among the target objects. Each target object obtains its own complete mask and target spectrum on the same time frame, and each target speech is written into the output channel corresponding to that target object and output in parallel on a time frame basis. When multiple target speech paths need to be sent to different downstream processing channels, each output can be organized and labeled according to the target object, so that when the number of target objects increases, only the output channels need to be expanded.

[0344] In addition, output control can be set separately for each target object. For example, the output of a target object can be attenuated or set to zero during the time frame when the target object is not speaking, according to the target activity information of each target object; or the output gain or output channel can be set separately for each target speech according to the usage needs, so that multiple target speech are output separately according to their corresponding target objects.

[0345] For example, the simultaneous enhancement of the voices of two designated speakers in an online meeting is used as an example for further explanation. Specifically, the mixed speech is transformed by time-frequency to obtain mixed spectral features with 257 frequency points, which can be divided into 36 non-uniform frequency bands; the 3s reference speech recorded by each of the two speakers is divided and encoded by the same frequency band to obtain two reference frequency band features, and two target object conditions and two frame-level reference memories are determined respectively; after each time frame arrives, the two recursive parameters are determined by the mixed frequency band features and their respective target object conditions in that time frame, and the two hidden states and local convolutional buffers are updated respectively to obtain two output features, which are then output in parallel after mask estimation and signal reconstruction. The state scale is twice that of the single-target configuration, while the number of parameters is the same as that of the single-target configuration.

[0346] In an optional embodiment, frame-level retrieval can also be performed separately in deep dual-path blocks in a multi-target scenario. Specifically, for any target object, each deep dual-path block can use its independently learned query projection to determine the current frame features based on the mixed frequency band features of that time frame, and use the current frame features as the frame-level reference memory for querying the target object to obtain the frame-level retrieval conditions for that target object; the key and value of the target object can be pre-calculated and cached during the initialization phase, and multiple deep dual-path blocks can share the same set of key-value projections, thereby ensuring that the resident overhead structure in a multi-target scenario remains consistent with that in a single-target scenario.

[0347] In the embodiments of this specification, by determining the target object conditions corresponding to each target object when there are multiple target objects, each target object obtains an independent identity condition and condition representation, thereby locating multiple target objects simultaneously in the same processing flow; by sharing the parameters of the backbone network among the target objects, while maintaining the hidden state, local convolutional buffer, and mask output separately for each target object, multi-path recursion can be carried out in parallel on batch dimensions or newly added target dimensions, making the number of target objects a scalable dimension in the processing flow; by determining the recursive parameters of each target object in any given time frame based on the conditions of each target object and recursively determining their respective output features, the extraction process of each target object is separated from each other in the state update stage.

[0348] In one embodiment of this specification, a practical application description of the speech reconstruction method is also provided. Specifically, the following description is in conjunction with the appendix. Figure 2a To be continued Figure 2e Taking the application of the speech reconstruction method provided in this manual in an online meeting scenario as an example, this document further explains the speech reconstruction method. Figure 2a A schematic diagram of the framework of a speech reconstruction method provided in one embodiment of this specification is shown. Figure 2b This diagram illustrates a processing procedure for a state-space model provided in one embodiment of this specification. Figure 2c This specification illustrates a schematic diagram of a non-uniform frequency band division process according to an embodiment of the present specification. Figure 2d This diagram illustrates a processing procedure for a dual-path block according to an embodiment of this specification. Figure 2e This diagram illustrates a process for extracting target object conditions according to an embodiment of this specification.

[0349] like Figure 2a As shown, the speech reconstruction method provided in this specification can be implemented by two parts: a reference speech branch and a hybrid speech branch. The two branches share the same frequency band division result and the same set of frequency band coding structures. Specifically, in the reference speech branch, the reference speech undergoes time-frequency transformation, frequency band division, and frequency band coding processing in sequence to obtain reference frequency band features. Then, target object conditions are extracted to obtain target object conditions that characterize the acoustic identity of the target object. In the hybrid speech branch, the hybrid speech undergoes time-frequency transformation, frequency band division, and frequency band coding processing in sequence to obtain hybrid frequency band features. The hybrid frequency band features are fed into a backbone composed of 36 identical dual-path blocks connected in series. The output features of the terminal dual-path blocks are fed into mask estimation to obtain a complete mask covering all frequency points. The target object conditions obtained from the reference speech branch are fed into each dual-path block in the hybrid speech branch to participate in the determination of recursive parameters for each time frame. The mixed spectral features obtained by time-frequency transformation of the mixed speech are directly fed into complex multiplication and multiplied by the complete mask at each frequency point to obtain the target spectrum. The target spectrum is then subjected to inverse time-frequency transformation and overlap and added to obtain the target speech of the target object. Figure 2a The overall connection relationship between the modules is given, and the internal processing procedures for frequency band division and frequency band coding, dual path blocks, and target object condition extraction are described in detail below. Figure 2c , Figure 2d and Figure 2e .

[0350] Figure 2b It shows Figure 2d The processing procedure of the state-space model used in the mid-time path. For example... Figure 2bAs shown, the acoustic input of the time frame and the frequency band entering the time path, and the target object conditions are input separately. The target object conditions are first conditionally projected and then concatenated with the acoustic input along the channel dimension to obtain fused features. The fused features are used by a parameter generation network to generate recursive parameters for the time frame. The recursive parameters include an input matrix, an output matrix, and a discretization step size. The discretization step size and the state matrix, which remain fixed during processing, are discretized in a zero-order preserved manner to obtain a discretized input matrix and a discretized state matrix. The discretized input matrix and the discretized state matrix are combined with the previous time step value of the hidden state to recursively update the hidden state to obtain the updated hidden state. The output matrix reads out the updated hidden state, and the acoustic input of the time frame participates in the supplementation of the readout result as a skip term, thereby obtaining the output features of the time frame. Figure 2b The processing procedure shown is in Figure 2d It is executed frame by frame in the time path, and the parameters are determined by the target object conditions and the current acoustic input.

[0351] Figure 2c It shows Figure 2a The process of intermediate frequency band division and band coding. For example... Figure 2c As shown, the hybrid spectral features or reference spectral features have 257 frequency points. These 257 frequency points are divided into 36 non-uniform frequency bands. The widths of each frequency band, in units of frequency points, are: 8 bands with a width of 2, 8 bands with a width of 4, 6 bands with a width of 8, 13 bands with a width of 11, and 1 band with a width of 18. Each frequency band is normalized and linearly projected, mapping it to frequency band features of the same dimension, so that the resulting hybrid frequency band features or reference frequency band features have 36 components in the frequency band dimension and 128 components in the channel dimension. The hybrid speech and the reference speech use the same frequency band division result and the same set of frequency band coding structures, thus the two features are aligned with each other under the same frequency band structure. Figure 2a Extraction of target object conditions and Figure 2d The frequency band-wise processing of each dual-path block in the middle provides the representation basis.

[0352] Figure 2d It shows Figure 2a The internal processing of any two-path block. For example... Figure 2dAs shown, the dual-path block receives mixed frequency band features or frequency band-wise features output from the preceding dual-path block. The block includes two processing paths: a spectral path and a temporal path. The spectral path models along the frequency band axis within the same time frame without maintaining cross-frame states, yielding the spectral path modeling result. This spectral path modeling result serves as input to the temporal path, which performs causal modeling along the time axis within each frequency band and maintains the hidden states of the state space model according to the frequency band. Using only information from the current and previous time frames, it obtains the output features of the dual-path block. The target object conditions are fed into the temporal path to participate in the determination of recursive parameters within that path; the internal processing is described in [reference needed]. Figure 2b Each cascaded dual-path block transmits its output features in the same way; that is, the output features of each dual-path block are sequentially used as the input to the next dual-path block, and the output feature of the final dual-path block is... Figure 2a The output features of the mask estimation are fed into the input.

[0353] Figure 2e It shows Figure 2a The process of extracting conditions from target objects. For example... Figure 2e As shown, the reference frequency band features are fed into the global object condition branch and the frame-level reference memory branch, respectively. In the global object condition branch, the reference frequency band features are globally pooled along the frequency band and time dimensions. The pooling result is projected through a multilayer perceptron to obtain the global object condition. The global object condition is determined once before processing begins and remains unchanged throughout the entire processing. In the frame-level reference memory branch, the reference frequency band features are linearly projected to obtain keys and values, forming a frame-level reference memory that preserves the time-frequency structure of the reference speech. The mixed frequency band features determine the current frame features in the current time frame. The current frame features are used as queries to perform retrieval on the frame-level reference memory. The weights are determined by the correlation between the query and the key, and the values ​​are weighted and summed to obtain the frame-level retrieval conditions. The frame-level retrieval conditions change with the current time frame and are used to supplement local pronunciation and spectral details. The global object condition and the frame-level retrieval conditions are used together as the target object conditions, and the condition granularity is allocated according to the order of the dual-path blocks in the network depth. That is, the global object condition is taken from the first preset number of dual-path blocks, and the frame-level retrieval conditions are taken from the remaining dual-path blocks.

[0354] In the embodiments of this specification, by sharing the same frequency band division result and the same set of frequency band coding structure with the reference speech branch and the mixed speech branch, the features of the two paths are aligned with each other under the same frequency band structure, providing a unified representation basis for the extraction of target object conditions and frequency band-by-frequency band processing; by having the spectral path handle the modeling of the relationship between frequency bands within the same time frame and the temporal path handle the modeling of temporal dependence within the dual-path block, and maintaining the hidden state by frequency band, the relationship between frequency bands and the temporal dependence are handled by the two paths respectively, and the recursion of each frequency band is independent and can be expanded in parallel; by generating recursive parameters together with the target object conditions and the current acoustic input, the information writing, information reading and update amplitude of the state space model all change with the target object, and the selection of the target object falls within the state update process; by sequentially passing the output features between each dual-path block, the frame-by-frame and frequency band features are refined level by level along the network depth; by allocating global object conditions and frame-level retrieval conditions according to the processing position, the stable identity information embodied by the entire reference speech and the local pronunciation and spectral details embodied by the current time frame are respectively supplied to the required processing positions. The above processing is performed frame by frame without accessing time frames that have not yet arrived. The scale of the maintained state is determined by the model configuration, thereby improving the accuracy of target speech extraction and reconstruction quality, while facilitating application scenarios that require real-time processing and low-cost deployment.

[0355] Corresponding to the above method embodiments, this specification also provides embodiments of a speech reconstruction system. Figure 3 A schematic diagram of the structure of a speech reconstruction system provided in one embodiment of this specification is shown. Figure 3 As shown, the speech reconstruction system includes: The acquisition module 302 is used to acquire mixed speech including the target object and other sound sources, as well as the reference speech of the target object; The condition determination module 304 is used to divide the frequency bands of the mixed spectral features of the mixed speech and the reference spectral features of the reference speech, obtain the mixed frequency band features and the reference frequency band features, and extract the target object conditions used to characterize the acoustic identity of the target object from the reference frequency band features. The recursive determination module 306 is used to determine the recursive parameters of the state space model for any time frame of mixed speech based on the mixed frequency band features and the target object conditions, and to recursively determine the output features of the time frame according to the recursive parameters. The reconstruction module 308 is used to reconstruct the mixed spectral features based on the output features of each time frame to obtain the target speech of the target object.

[0356] The acquisition module is responsible for receiving input speech in the speech reconstruction system. It obtains mixed speech including the target object and other sound sources, as well as reference speech of the target object. Specifically, the acquisition module may include an audio access interface and a preprocessing unit. The audio access interface establishes a connection with the audio acquisition device, the pickup channel of the conference terminal, the recording device, or the audio service on the network side and continuously receives incoming speech data. The preprocessing unit performs analog-to-digital conversion, amplitude normalization, and framing processing on the received speech data. For example, it divides continuous speech into multiple time frames according to a window length of 512 points and a frame shift of 256 points. Furthermore, the acquisition module acquires the reference speech once before processing begins and caches the data required for its processing. During streaming processing, it continuously receives time frames of mixed speech, thereby providing a unified input source for the condition determination module and the recursive determination module, and decoupling the overhead of subsequent processing from the duration of the reference speech.

[0357] The condition determination module is responsible for frequency band division and condition extraction in the speech reconstruction system. It performs frequency band division on the mixed spectral features and reference spectral features, obtaining mixed frequency band features and reference frequency band features, and extracts target object conditions from the reference frequency band features to characterize the acoustic identity of the target object. Specifically, the condition determination module may include a time-frequency transformation unit, a frequency band division unit, and a condition extraction unit. The time-frequency transformation unit performs time-frequency transformation on the mixed speech and the reference speech respectively. The frequency band division unit divides the two spectral features into multiple frequency bands according to the same set of frequency band division results, for example, dividing 257 frequency points into 36 non-uniform frequency bands, and performs channel-dimensional splicing, normalization, and linear projection on each frequency band. The condition extraction unit aggregates and projects the reference frequency band features along the frequency band dimension and the time dimension. Furthermore, the condition determination module distinguishes the processing of the reference speech branch and the mixed speech branch, and completes the determination of the target object conditions before processing begins and outputs them to the recursive determination module, so that subsequent frame-by-frame processing does not repeatedly access the reference speech.

[0358] The recursive determination module is responsible for frame-by-frame recursion in the speech reconstruction system. For any time frame of mixed speech, it determines the recursive parameters of the state-space model for that time frame based on mixed frequency band features and target object conditions, and then recursively determines the output features of that time frame according to the recursive parameters. Specifically, the recursive determination module can include a parameter determination unit and a state update unit. The parameter determination unit takes the mixed frequency band features and target object conditions of the current time frame as input to determine the recursive parameters used to control information writing, information reading, and the state time scale for that time frame. The state update unit updates the state corresponding to that time frame according to the recursive parameters and reads the output features of that time frame from the updated state. The state of each frequency band is maintained separately. Furthermore, the recursive determination module performs a recursion immediately after each time frame arrives. The target object conditions participate in the determination of the recursive parameters, ensuring that the selection of the target object is completed within the state update process. The maintained state scale is determined by the model configuration.

[0359] The reconstruction module is responsible for outputting the target speech in the speech reconstruction system. It reconstructs the signal from the mixed spectral features based on the output features of each time frame to obtain the target speech of the target object. Specifically, the reconstruction module may include a mask estimation unit, a frequency band splicing unit, and a signal recovery unit. The mask estimation unit converts the output features into a mask that acts on the mixed spectral features according to the correspondence between time frames and frequency bands. The frequency band splicing unit splices the masks of each frequency band into a mask covering the entire frequency range according to the frequency band boundaries. The signal recovery unit combines the spliced ​​mask with the mixed spectral features point-by-point in the frequency domain to obtain the target spectrum, and performs inverse time-frequency transformation and overlap addition on the target spectrum to obtain the target speech in the time domain. Furthermore, the reconstruction module uses the amplitude and phase carried by the mixed spectral features as the reconstruction reference and outputs the reconstruction results sequentially according to the time frames, so that the target speech is continuously output as the processing progresses.

[0360] It should be noted that the meanings of the concepts involved in the above modules, such as mixed speech, reference speech, mixed spectral features, reference spectral features, mixed frequency band features, reference frequency band features, target object conditions, recursive parameters, output features, and target speech, can be found in the relevant descriptions in the aforementioned method embodiments, and will not be repeated here.

[0361] The speech reconstruction system provided in the embodiments of this specification integrates mixed speech and reference speech into the system through an acquisition module, providing standardized and corresponding input sources for the condition determination module and the recursive determination module. The condition determination module forms mixed frequency band features and reference frequency band features under the same frequency band structure and extracts the target object conditions, so that the acoustic identity information of the target object participates in subsequent processing in a frequency band-level form. The recursive determination module determines the recursive parameters based on the mixed frequency band features and the target object conditions at each time frame and completes the state update and readout, so that the selection of the target object is implemented within the state update process.

[0362] The above is an illustrative scheme of a speech reconstruction system according to this embodiment. It should be noted that the technical solution of this speech reconstruction system and the technical solution of the speech reconstruction method described above belong to the same concept. For details not described in detail in the technical solution of the speech reconstruction system, please refer to the description of the technical solution of the speech reconstruction method described above.

[0363] Figure 4 A structural block diagram of a computing device 400 according to one embodiment of this specification is shown. The components of the computing device 400 include, but are not limited to, a memory 410 and a processor 420. The processor 420 is connected to the memory 410 via a bus 430, and a database 450 is used to store data.

[0364] The computing device 400 also includes an access device 440, which enables the computing device 400 to communicate via one or more networks 460. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 440 may include one or more of any type of wired or wireless network interface (e.g., a network interface controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Worldwide Interoperability for Microwave Access (WiMAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0365] In one embodiment of this specification, the aforementioned components of the computing device 400 and Figure 4 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 4 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0366] The computing device 400 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 400 can also be a mobile or stationary server.

[0367] The processor 420 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described speech reconstruction method.

[0368] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the above-described speech reconstruction method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-described speech reconstruction method.

[0369] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described speech reconstruction method.

[0370] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above-described speech reconstruction method belong to the same concept, and all details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the above-described speech reconstruction method.

[0371] An embodiment of this specification also provides a computer program product, including computer-executable instructions that, when executed by a processor, implement the steps of the above-described speech reconstruction method.

[0372] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above-described speech reconstruction method belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the above-described speech reconstruction method.

[0373] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0374] Computer-executable instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable storage media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0375] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0376] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0377] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A speech reconstruction method, comprising: Acquire mixed speech containing the target object and other sound sources, as well as reference speech of the target object; The mixed spectral features of the mixed speech and the reference spectral features of the reference speech are divided into frequency bands to obtain mixed frequency band features and reference frequency band features, and target object conditions for characterizing the acoustic identity of the target object are extracted from the reference frequency band features. For any time frame of the mixed speech, based on the mixed frequency band features and the target object conditions, the recursive parameters of the state space model for that time frame are determined, and the output features of that time frame are recursively determined according to the recursive parameters. Based on the output features of each time frame, the mixed spectral features are reconstructed to obtain the target speech of the target object.

2. The method according to claim 1, further comprising, after acquiring the mixed speech including the target object and other sound sources, and the reference speech of the target object: Time-frequency transformations are performed on the mixed speech and the reference speech of the target object to obtain the mixed spectral features and the reference spectral features respectively.

3. The method according to claim 1, wherein dividing the mixed spectral features of the mixed speech and the reference spectral features of the reference speech into frequency bands to obtain mixed frequency band features and reference frequency band features includes: Based on the information density of the hybrid spectral features in different frequency regions, the hybrid spectral features of the hybrid speech are divided into multiple frequency bands to obtain the hybrid frequency band features; The reference spectral features of the reference speech are divided into the multiple frequency bands to obtain the reference frequency band features; The step of extracting target object conditions for characterizing the acoustic identity of the target object from the reference frequency band features includes: pooling the reference frequency band features along the frequency band dimension and the time dimension to obtain the target object conditions.

4. The method according to claim 3, after dividing the mixed spectral features of the mixed speech into multiple frequency bands based on the information density of the mixed spectral features in different frequency regions and obtaining the mixed frequency band features, further includes: The mixed frequency band features are input into multiple cascaded dual-path blocks, wherein any dual-path block includes a spectral path and a temporal path; The spectral path modeling result is obtained by modeling along the frequency band axis within the time frame using the spectral path. The spectral path modeling result is used as the input of the time path. Causal modeling is performed along the time axis in each frequency band, and the hidden state of the state space model is maintained according to the frequency band. The output features of the dual-path block are determined based on the updated hidden state. The output features of each two-path block are used as the input of the next two-path block in sequence, and the output features of the last two-path block are used as the output features.

5. The method according to claim 3, wherein pooling the reference frequency band features along the frequency band dimension and the time dimension to obtain the target object conditions includes: The reference frequency band features are pooled along the frequency band dimension and the time dimension, and the pooling results are projected to obtain the global object conditions; Projecting the reference frequency band features yields keys and values, forming a frame-level reference memory; Based on the mixed frequency band characteristics of the time frame, the current frame characteristics are determined, and the frame-level reference memory is retrieved using the current frame characteristics as a query to obtain frame-level retrieval conditions; The global object conditions and / or the frame-level retrieval conditions are used as the target object conditions.

6. The method according to claim 5, wherein the hidden states of the state-space model are maintained separately in multiple cascaded dual-path blocks according to frequency bands; The step of using the global object conditions and / or the frame-level retrieval conditions as the target object conditions includes: In the first preset number of dual-path blocks, the global object condition is used as the target object condition; In the remaining dual-path blocks, the frame-level retrieval conditions are used as the target object conditions.

7. The method according to claim 5, wherein, The number of memory entries in the frame-level reference memory is a preset constant; The frame-level reference memory is obtained through at least one of the following methods: The reference frequency band features are segmented and pooled along the time dimension, and multiple adjacent time frames are merged into a memory entry; Clustering is performed on the projection results for each time frame; After filtering each time frame according to similarity or energy intensity, a preset number of memory entries are retained. Reference audio that exceeds the preset duration is truncated before projection.

8. The method according to any one of claims 1 to 7, wherein determining the recursive parameters of the state-space model for any time frame of the mixed speech, based on the mixed frequency band features and the target object conditions, comprises: The conditions of the target object are projected to obtain conditional projection features; The conditional projection features are fused with the mixed frequency band features of the time frame to obtain fused features; Based on the fusion features, the recursive parameters for this time frame are generated through a parameter generation network.

9. The method according to claim 8, wherein the recursive parameters include an input matrix, an output matrix, and a discretization step size; The step of recursively determining the output features of the time frame according to the recursive parameters includes: Based on the discretization step size, the input matrix and the state matrix of the state space model are discretized respectively to obtain the discretized input matrix and the discretized state matrix; Based on the discretized input matrix, the discretized state matrix, and the previous time value of the hidden state of the state space model, the hidden state is recursively updated. The updated hidden state is read out based on the output matrix, and the output feature is determined by combining the mixed frequency band features of the time frame.

10. The method according to claim 1, wherein the step of reconstructing the mixed spectral features based on the output features of each time frame to obtain the target speech of the target object comprises: Gated projection is performed on the output features of each time frame to obtain a complex mask; The complex mask is concatenated along the frequency band boundaries to obtain a complete mask, and the complete mask is multiplied by the mixed spectral features to obtain the target spectrum. The target speech of the target object is obtained by performing inverse time-frequency transform and overlap addition on the target spectrum.

11. The method according to claim 1, wherein when there are multiple target objects, the method further comprises: For any target object, the target object conditions corresponding to the target object are determined based on the reference frequency band characteristics of the reference speech of the target object; For any given time frame, based on the hybrid frequency band characteristics of that time frame and the target object conditions corresponding to the target object, the recursive parameters of the target object in that time frame are determined, and the output characteristics of the target object are recursively determined according to the recursive parameters of the target object in that time frame. Signal reconstruction is performed based on the output features of each target object, and the target speech of multiple target objects is output in parallel.

12. A speech reconstruction system, comprising: The acquisition module is used to acquire mixed speech containing the target object and other sound sources, as well as reference speech of the target object; The condition determination module is used to divide the frequency bands of the mixed spectral features of the mixed speech and the reference spectral features of the reference speech to obtain mixed frequency band features and reference frequency band features, and extract the target object conditions for characterizing the acoustic identity of the target object from the reference frequency band features; The recursive determination module is used to determine the recursive parameters of the state space model for any time frame of the mixed speech based on the mixed frequency band features and the target object conditions, and to recursively determine the output features of the time frame according to the recursive parameters. The reconstruction module is used to reconstruct the mixed spectral features based on the output features of each time frame to obtain the target speech of the target object.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising computer-executable instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images