Weakly supervised target speaker extraction method and system based on pseudo-label signal generation

Abstract

A weakly supervised target speaker extraction method and system based on pseudo-label signal generation are disclosed, including obtaining a to-be-processed far-field multi-channel mixed audio and a corresponding target speaker timestamp; the amplitude spectrum of the far-field multi-channel mixed audio is spliced along the channel dimension and mapped into speech coding hidden features; according to the target speaker timestamp, a speech segment in which the target speaker is active is cut out from a reference channel of the far-field multi-channel mixed audio; a speaker embedding vector of the speech segment is calculated and dimensionally expanded to obtain target speaker embedding features; the speech coding hidden features and the target speaker embedding features are fused and input into a target speaker extraction model for feature extraction to obtain target speaker speech features; the target speaker speech features are mapped into an amplitude spectrum mask, the mask is applied to the amplitude spectrum of the reference channel, and inverse transformation is performed in combination with the phase of the reference channel to obtain clean target speaker speech.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to a weakly supervised target speaker extraction method and system based on pseudo-label signal generation. Background Technology

[0002] Target Speaker Extraction (TSE) aims to extract the clean speech signal of a target speaker from mixed audio containing noise, reverberation, and overlapping speech from multiple speakers, using given auxiliary cues (such as the target speaker's registered speech, lip movements, and location information). This technology has broad application prospects in devices such as smart speakers, headphones, hearing aids, and conference systems, and can significantly improve the intelligibility and clarity of target speech.

[0003] Current TSE methods primarily rely on supervised learning frameworks, utilizing simulated synthetic datasets (such as WSJ0-mix, WHAM!, WHAMR!, etc.) for model training. However, simulated data typically only considers simple speaker overlap patterns or idealized acoustic environments, differing significantly from complex real-world scenarios (such as intricate overlap patterns and variable acoustic environments). This mismatch between the "simulation domain" and the "real domain" leads to poor generalization performance of models trained on simulated data in real-world scenarios. Although some research has attempted to improve generalization by simulating more diverse overlap scenarios, it still cannot completely compensate for the differences in data distribution.

[0004] On the other hand, traditional TSE models typically assume that the target speaker's enrollment speech can be pre-acquired as an auxiliary cue. However, in many practical applications, such as impromptu multi-person meetings or sudden conversations, the system often cannot pre-record or acquire the target speaker's clean enrollment speech, which limits the application scope of traditional TSE models. To address this issue, existing technologies have developed schemes that utilize timestamps generated by Speaker Diarization (SD) systems as guiding information (such as ADEnet). However, such methods have significant limitations: firstly, timestamps as guiding information are coarse-grained and fail to provide fine-grained voiceprint features; secondly, most existing unregistered speech extraction methods only use single-channel audio as input, failing to fully utilize the spatial information provided by the multi-microphone arrays typically equipped in conference recording equipment, thus limiting the model's extraction performance in complex acoustic environments.

[0005] In summary, how to improve the generalization ability of the model by utilizing real-time data, and how to achieve high-precision target speaker extraction under the premise of missing registered speech and making full use of multi-channel spatial information, are the technical problems that urgently need to be solved. Summary of the Invention

[0006] To address the aforementioned technical problems in the existing technology, this invention proposes a weakly supervised target speaker extraction method and system based on pseudo-label signal generation, thereby solving the above-mentioned technical problems.

[0007] According to a first aspect of the present invention, a weakly supervised target speaker extraction method based on pseudo-label signal generation is proposed, comprising:

[0008] S1: Obtain the far-field multi-channel mixed audio to be processed and the corresponding target speaker timestamp; concatenate the amplitude spectrum of the far-field multi-channel mixed audio along the channel dimension and map it as speech coding latent features;

[0009] S2: Based on the target speaker's timestamp, segment the active speech segments of the target speaker from the reference channel of the far-field multi-channel mixed audio; calculate the speaker embedding vector of the speech segment and expand its dimension to obtain the target speaker embedding features;

[0010] S3: Fuse the latent features of speech coding with the embedded features of the target speaker, and input them into the target speaker extraction model for feature extraction to obtain the speech features of the target speaker;

[0011] S4: Map the target speaker's speech features to an amplitude spectrum mask, apply the mask to the amplitude spectrum of the reference channel, and perform an inverse transform in combination with the phase of the reference channel to obtain the clean speech of the target speaker.

[0012] In some specific embodiments, the target speaker extraction model is obtained through weakly supervised training based on pseudo-label signals generated from the transcript dataset. The training process includes:

[0013] Obtain a real-time dataset containing synchronously acquired far-field mixed audio, near-field recordings of the target speaker, and timestamps;

[0014] Based on the timestamp, the active part of the non-target speaker in the near-field recording is zeroed out. The time delay points between the zeroed near-field recording and the far-field mixed audio are calculated and time delay compensation is performed to obtain the time-domain aligned signal.

[0015] Multi-frame linear filtering is applied to the time-domain aligned signal to align its amplitude with the far-field mixed audio, thereby generating a pseudo-tag signal.

[0016] The far-field mixed audio is input into the model to be trained, and the pseudo-label signal is used as the supervision target. The loss function is calculated and the model parameters are updated.

[0017] In some specific embodiments, the number of delay points is calculated using a generalized cross-correlation-phase transform weighted algorithm. The formula is: ,in, and These represent the recordings of the lecture. and far-field mixed audio Discrete-time Fourier transform, Represents a set of integers, for Make an offset to align in time. ,get .

[0018] In some specific embodiments, the process of performing multi-frame linear filtering on the time-domain aligned signal includes: constructing an optimization problem: ,in, express The short-time Fourier transform coefficients, These represent the time frame index and the frequency index, respectively. Indicates the frequency to be estimated The filter coefficient vector at that point, L time-frequency units were stacked. express STFT coefficient, Set to order 2 Representing L-dimensional complex space, Indicates Hermite transpose. It is a weighted term, and its calculation formula is: ,in, Returns the maximum value in all time-frequency cells of a given power spectrum. express The short-time Fourier transform spectrum, Returns the larger of the two values, applied to the calculated MFLF filter. This yields the final high-quality pseudo-label signal: ,in This indicates the inverse short-time Fourier transform operation.

[0019] In some specific embodiments, the loss function is the amplitude spectrum composite loss. The specific calculation formula is as follows: Among them, mean square error loss Target perception cosine similarity loss T represents the total number of time frames, and F represents the total number of frequency points. and These represent the model outputs respectively. and pseudo-tag signals STFT coefficient, This is an indicator function used to determine whether the target signal exists. It takes a value of 1 when the pseudo-tag signal is a non-silent segment, and a value of 0 otherwise. As a regulating factor, it is used to adjust The percentage of the loss value.

[0020] In some specific embodiments, the target speaker extraction model includes a speech encoder, a speaker encoder, a feature fusion layer, a speaker extractor, and a speech decoder; the speaker extractor includes stacked crossband building modules and narrowband building modules; the crossband building module includes two frequency convolutional modules and a full-band linear module for modeling the contextual dependencies of the intra-frame spectrum; the narrowband building module includes a multi-head self-attention module and a temporal convolutional feedforward module for global modeling along the time axis within the narrowband.

[0021] In some specific embodiments, in S1, splicing the amplitude spectrum of the far-field multi-channel mixed audio along the channel dimension specifically includes: Short-time Fourier transform spectrum of each channel The input features are obtained by concatenation. , This represents a splicing operation along the channel axis, and the channel dimensions are concatenated using a one-dimensional convolutional layer. Mapped to encoding dimension Obtain speech coding latent features In S3, the specific process of feature fusion is as follows: embedding the target speaker with expanded dimensions into features. With speech coding latent features Splicing is performed on the frequency axis. The dimension of the speaker embedding vector is used to obtain the fused features. And the frequency dimension is compressed back through a one-dimensional convolutional layer. The features input to the target speaker extraction model are obtained. , This represents a one-dimensional convolution operation.

[0022] In some specific embodiments, in S4, mapping the target speaker's speech features to an amplitude spectrum mask specifically includes: mapping the channel dimension of the target speaker's speech features to 1 through a linear layer, and limiting the output value to positive through a ReLU activation function to obtain the amplitude spectrum mask. , The model represents the predicted speech features of the target speaker; the inverse transformation combined with the phase of the reference channel specifically includes: masking the amplitude spectrum. Amplitude spectrum with reference channel The enhanced amplitude spectrum is obtained by performing the Hadamard product operation. This will enhance the phase term of the amplitude spectrum relative to the reference channel. Coupled, the clean speech prediction value of the target speaker is obtained through inverse short-time Fourier transform. .

[0023] According to a second aspect of the invention, a computer-readable storage medium is provided on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0024] According to a third aspect of the present invention, a weakly supervised target speaker extraction system based on pseudo-label signal generation is proposed, comprising:

[0025] The feature encoding module is configured to acquire the far-field multi-channel mixed audio to be processed and the corresponding target speaker timestamp; the amplitude spectrum of the far-field multi-channel mixed audio is concatenated along the channel dimension and mapped to speech coding latent features;

[0026] The speaker encoding module is configured to segment active speech segments of the target speaker from the reference channel of the far-field multi-channel mixed audio based on the target speaker's timestamp; calculate the speaker embedding vector of the speech segment and expand its dimension to obtain the target speaker embedding features;

[0027] The speaker extraction module is configured to fuse the speech coding latent features with the target speaker embedded features, and input them into the target speaker extraction model for feature extraction to obtain the target speaker's speech features;

[0028] The speech reconstruction module is configured to map the speech features of the target speaker to an amplitude spectrum mask, apply the mask to the amplitude spectrum of the reference channel, and perform an inverse transform in combination with the phase of the reference channel to obtain the clean speech of the target speaker.

[0029] The weakly supervised target speaker extraction method and system based on pseudo-label signal generation of the present invention have the following beneficial technical effects:

[0030] This invention proposes a weakly supervised training method based on pseudo-label signal generation, effectively solving the problem of poor generalization performance in real-world scenarios caused by the over-reliance on simulation data in existing technologies. Specifically, this method fully utilizes high-quality near-field audio recordings acquired synchronously from real-world data, performs time delay compensation using the Generalized Cross-Correlation-Phase Transform Weighted (GCC-PHAT) algorithm, and combines multi-frame linear filtering (MFLF) to achieve amplitude alignment, thereby generating high-quality pseudo-label signals that are highly consistent with far-field mixed audio in the time-frequency domain. These pseudo-labels generated based on physical acoustic associations, compared to simple model prediction or data augmentation, can more realistically reflect the characteristics of complex acoustic environments, overcoming the domain mismatch problem between simulation and real data. Furthermore, by introducing the amplitude spectrum composite (CMS) loss function and using cosine similarity as a relaxation constraint, the model's tolerance to potential noise in the pseudo-labels is further enhanced, significantly improving the robustness and convergence stability of the model during weakly supervised training.

[0031] To address the pain point of real-world applications such as multi-person conferences where it's impossible to obtain the registered speech of the target speaker in advance, this invention constructs a multi-channel target speaker extraction model (EM-TSE) that does not require registered speech, achieving source extraction in "cold start" scenarios. This model no longer relies on pre-recorded clean speech as auxiliary cues; instead, it creatively utilizes timestamp information generated by a speaker log (SD) system to directly segment active segments of the target speaker from the mixed audio reference channel to extract voiceprint embedding features. This innovative design breaks the strong dependence of traditional target speaker extraction techniques on registered speech, enabling the technology to be directly deployed in open real-world scenarios such as impromptu meetings and spontaneous dialogues, greatly expanding the application boundaries and practical value of the technology.

[0032] At the model architecture level, this invention overcomes the limitations of existing methods that only utilize single-channel information. Through a specially designed network structure, it fully mines and utilizes the spatial information of a multi-microphone array. The system concatenates the amplitude spectra of multi-channel audio at the input end, and in the feature extraction stage, it uses stacked crossband building blocks (CBs) and narrowband building blocks (NBs) to perform deep modeling of frequency domain contextual dependencies and time-domain information within narrowbands, respectively. This design not only captures rich spectral features but also effectively utilizes the spatial differences between channels, significantly compensating for the performance shortcomings of single-channel models in complex reverberation and noisy environments. Experimental results show that, combined with the aforementioned weakly supervised training strategy and multi-channel model architecture, this invention outperforms existing models trained on simulation data in key metrics such as speech recognition error rate (CER) and speech quality score (DNSMOS), achieving high-precision target speech extraction in real-world scenarios. Attached Figure Description

[0033] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Other features, objects, and advantages of this application will become more apparent from reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0034] Figure 1 This is a flowchart of a weakly supervised target speaker extraction method based on pseudo-label signal generation, according to an embodiment of this application.

[0035] Figure 2 This is a flowchart of a weakly supervised training process based on pseudo-label signal generation, which is a specific embodiment of this application.

[0036] Figure 3This is a structural diagram of the EM-TSE model of a specific embodiment of this application;

[0037] Figure 4 This is a framework diagram of a weakly supervised target speaker extraction system based on pseudo-label signal generation, according to one embodiment of this application.

[0038] Figure 5 This is a schematic diagram of the structure of a computer system used to implement the electronic device of the present application. Detailed Implementation

[0039] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0040] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0041] Figure 1 A flowchart illustrating a weakly supervised target speaker extraction method based on pseudo-label signal generation according to an embodiment of this application is shown. Figure 1 As shown, the method includes the following steps:

[0042] S1: Obtain the far-field multi-channel mixed audio to be processed and the corresponding target speaker timestamp; concatenate the amplitude spectrum of the far-field multi-channel mixed audio along the channel dimension and map it as a speech coding latent feature.

[0043] S2: Based on the target speaker's timestamp, segment the active speech segments of the target speaker from the reference channel of the far-field multi-channel mixed audio; calculate the speaker embedding vector of the speech segment and expand its dimension to obtain the target speaker embedding features.

[0044] S3: The latent features of speech coding are fused with the embedded features of the target speaker and input into the target speaker extraction model for feature extraction to obtain the speech features of the target speaker.

[0045] S4: Map the target speaker's speech features to an amplitude spectrum mask, apply the mask to the amplitude spectrum of the reference channel, and perform an inverse transform in combination with the phase of the reference channel to obtain the clean speech of the target speaker.

[0046] In a specific embodiment, the target speaker extraction model is obtained through weakly supervised training based on pseudo-label signals generated from the transcript dataset. Figure 2 A flowchart illustrating a specific embodiment of this application of weakly supervised training based on pseudo-label signal generation is shown, as follows: Figure 2 As shown, the training process includes:

[0047] Obtain a real-time dataset containing synchronously acquired far-field mixed audio, near-field recordings of the target speaker, and timestamps;

[0048] Based on the timestamp, the active part of the non-target speaker in the near-field recording is zeroed out. The time delay points between the zeroed near-field recording and the far-field multi-channel mixed audio are calculated and time delay compensation is performed to obtain the time-domain aligned signal.

[0049] Multi-frame linear filtering is applied to the time-domain aligned signal to align its amplitude with the far-field mixed audio, thereby generating a pseudo-tag signal.

[0050] The far-field mixed audio is input into the model to be trained, and the pseudo-label signal is used as the supervision target. The loss function is calculated and the model parameters are updated.

[0051] In a specific embodiment, the number of delay points is calculated using a generalized cross-correlation-phase transform weighted algorithm. The formula is: ,in, and These represent the recordings of the lecture. and far-field mixed audio Discrete-time Fourier transform, Used to represent the complex exponential form of a signal, reflecting the signal's rotational or oscillatory characteristics in the frequency domain. Represents far-field mixed audio signals The complex conjugate frequency domain representation of . Indicates the modulo value. Represents a set of integers, for Make an offset to align in time. ,get .

[0052] In a specific embodiment, the process of performing multi-frame linear filtering on the time-domain aligned signal includes: by... Perform multi-frame linear filtering (MFLF) to achieve the same Amplitude alignment between them, where the MFLF filter The following optimal problem was solved to obtain the following result: ,in, It is the optimization objective to be solved; it is the frequency. The filter response under the given conditions, express The short-time Fourier transform coefficients, These represent the time frame index and the frequency index, respectively. Indicates the frequency to be estimated The filter coefficient vector at that point, L time-frequency units were stacked. express STFT coefficient, Set to order 2 Representing L-dimensional complex space, Indicates Hermite transpose. It is a weighted term, and its calculation formula is: ,in, Returns the maximum value in all time-frequency cells of a given power spectrum. The time-frequency domain power spectrum of the far-field mixed audio signal. express The short-time Fourier transform spectrum, This represents the power value of the current time-frequency unit. Returns the larger of the two values, applied to the calculated MFLF filter. This yields the final high-quality pseudo-label signal: ,in This indicates the inverse short-time Fourier transform operation.

[0053] In a specific embodiment, the loss function is the amplitude spectrum composite loss. The specific calculation formula is as follows: Among them, mean square error loss Target-Aware Cosine Similarity Loss (TA-Cos Loss) T represents the total number of time frames, and F represents the total number of frequency points. and These represent the model outputs respectively. and pseudo-tag signals STFT coefficient, This is an indicator function used to determine whether the target signal exists. It takes a value of 1 when the pseudo-tag signal is a non-silent segment, and a value of 0 otherwise. As a regulating factor, it is used to adjust The proportion of the loss value. During model training, long speech segments are usually randomly divided into short segments as input. When the segmented segment happens to correspond to the silent interval of the target speaker, the pseudo-label signal is all zero, and the traditional cosine similarity loss no longer has a clear physical meaning. Therefore, this application introduces the aforementioned indicator function to enable cosine similarity constraints only when the target speaker is not silent. Unlike mean squared error loss, cosine similarity loss does not require the model's prediction to be completely consistent with the pseudo-label signal, making it a more lenient constraint. Therefore, introducing cosine similarity loss can improve the robustness of model training, especially when the pseudo-label signal is not accurate enough.

[0054] Figure 3 A structural diagram of the EM-TSE model of a specific embodiment of this application is shown, as follows: Figure 3 As shown, this model aims to solve the problem of not being able to obtain the registered voice of the target speaker in advance in real-world meeting scenarios, and to make full use of the spatial information of a multi-microphone array. Its specific processing flow consists of the following five main parts:

[0055] 1. Speaker Encoding Branch (the branch at the top of the diagram): This branch is responsible for solving the cold start problem of "unregistered speech" and extracting voiceprint cues from the target speaker. Specifically, it includes:

[0056] Input: Get the timestamp of the target speaker (generated by the speaker log system SD) and the mixed audio of the reference channel.

[0057] Speech segmentation: Based on timestamps, segment the speech that the target speaker is active in from a reference channel (e.g., channel 1).

[0058] Speaker embedding extraction: Using the segmented speech fragments, the embedding vector (e.g., i-vector) of the target speaker is calculated through a pre-trained speakerprint model, denoted as... .

[0059] Dimension expansion layer: for use with speech coding features The embedding vectors are merged by copying them. Expand the dimensions to fit the timeline. and channel axis Top alignment yields expanded target speaker embedding features. , This represents the dimension of the speaker embedding vector.

[0060] 2. Speech coding branch (bottom branch of the diagram): This branch is responsible for processing far-field multi-channel mixed audio and extracting acoustic features containing spatial information. Specifically, it includes:

[0061] Input: Get Multi-channel far-field mixed audio signal.

[0062] STFT Transform: First, perform a Short Time Fourier Transform (STFT) on the multi-channel audio to convert the time-domain signal into a frequency-domain signal.

[0063] Amplitude spectrum concatenation: Extract the amplitude spectra of all channels and concatenate them along the channel dimensions to form the input features. The specific formula is as follows: ,in, This indicates a splicing operation along the channel axis. For time frames, This represents the number of frequency points.

[0064] One-dimensional convolutional layer: The concatenated features are passed through a one-dimensional convolutional layer (1D-Conv) with weight sharing between subbands, mapping the channel dimension to the encoding dimension. To obtain the latent features of speech coding This step is similar to the independent sub-band operation in beamforming, which can improve modeling accuracy.

[0065] 3. Feature Fusion Layer (Middle of the Diagram): This module is responsible for combining acoustic features with voiceprint cues to guide subsequent separation. Specifically, it includes:

[0066] Concatenation and fusion: Unlike common additive fusion, this case uses a concatenation method along the frequency axis. This encodes latent features of the speech. Embedded features of the target speaker By piecing them together, we obtain This method preserves as much of the original information as possible and avoids feature aliasing.

[0067] One-dimensional convolutional layer: Features after concatenation Then, through another one-dimensional convolutional layer, the frequency dimension is changed from... Compress back , obtain fusion features As input to the extractor, This represents a one-dimensional convolution operation.

[0068] 4. Speaker Extractor (the core network in the middle of the diagram), is the core part of the model, using a stacked SpatialNet structure to extract the target speech. Specifically, it includes:

[0069] Structure stacking: includes A repeated extraction layer.

[0070] Cross-band Block (CB): As shown in the green module in the figure, it is used to model the contextual dependencies of the intra-frame spectrum. It contains a frequency convolution module and a full-band linear module, which can capture the harmonic structure of speech and cross-frequency spatial information.

[0071] Narrow-band Block (NB): As shown in the green module in the figure, it is used for global modeling along the time axis within a narrow band. It includes a multi-head self-attention (MHSA) module and a temporal convolutional feedforward module to capture narrow-band spatial information such as steering vectors and inter-channel level differences, as well as temporal stationarity characteristics.

[0072] Output: After The alternating processing of the CB and NB modules outputs refined speech features of the target speaker. .

[0073] 5. Speech Decoding Branch (right side of the diagram): This branch is responsible for restoring the extracted features to a time-domain speech waveform. Specifically, it includes:

[0074] Linear layers and ReLU layers: features First, a linear layer maps the channel dimension to 1, followed by a ReLU activation function. ReLU restricts the output value to positive values, as the amplitude spectrum cannot be negative, thus generating an amplitude spectrum mask. .

[0075] Masking function: to mask Amplitude spectrum with reference channel Performing the Hadamard product (element-wise multiplication) operation yields the enhanced amplitude spectrum. .

[0076] Phase coupling with reference channel: directly utilizing the original phase of the reference channel. Compared with the enhanced amplitude spectrum coupling.

[0077] iSTFT Transform: Finally, an inverse short-time Fourier transform (iSTFT) is performed to reconstruct the clean speech of the target speaker in the time domain. .

[0078] Figure 4 This invention illustrates a framework diagram of a weakly supervised target speaker extraction system based on pseudo-label signal generation, according to one embodiment of this application. Figure 4As shown, the system includes a feature encoding module 401, a speaker encoding module 402, a speaker extraction module 403, and a speech reconstruction module 404. The feature encoding module 401 is configured to acquire the far-field multi-channel mixed audio to be processed and the corresponding target speaker timestamp; it concatenates the amplitude spectrum of the far-field multi-channel mixed audio along the channel dimension to map it as a speech encoding latent feature; the speaker encoding module 402 is configured to segment the active speech segments of the target speaker from the reference channel of the far-field multi-channel mixed audio based on the target speaker timestamp; it calculates the speaker embedding vector of the speech segment and expands its dimension to obtain the target speaker embedding feature; the speaker extraction module 403 is configured to fuse the speech encoding latent feature with the target speaker embedding feature and input it into the target speaker extraction model for feature extraction to obtain the target speaker speech feature; the speech reconstruction module 404 is configured to map the target speaker speech feature as an amplitude spectrum mask, apply the mask to the amplitude spectrum of the reference channel, and perform an inverse transform combined with the phase of the reference channel to obtain the clean speech of the target speaker.

[0079] The weakly supervised training method based on pseudo-label signal generation proposed in this invention can effectively generate high-quality pseudo-label signals for recorded data. Using these signals for weakly supervised training of the TSE model on recorded data significantly improves the generalization ability of the TSE model in real-world scenarios. Specifically, after model enhancement, the Character Error Rate (CER) and DNSMOS scores of the audio significantly surpass those of previous models trained on simulation data. Furthermore, the CMS loss proposed in this invention effectively improves the robustness and stability of the model during training. On the other hand, the multi-channel target speaker extraction model proposed in this invention, which does not require registered speech, can extract clean speech signals of the target speaker from mixed audio using timestamp information even when the target speaker's registered speech is missing. Compared to previous TSE methods that rely on registered speech, this invention successfully extends the TSE method to real-world application scenarios such as multi-person conferences and significantly improves model performance by effectively utilizing the spatial information provided by multi-channel audio. Finally, by combining the two methods proposed in this invention, not only can the target speaker's speech be extracted in real-world application scenarios such as multi-person conferences where registered voices are missing, but the generalization performance of the model in real-world scenarios can also be significantly improved through the weakly supervised training method based on pseudo-label signal generation, allowing the TSE method to be truly implemented in real-world scenarios.

[0080] The following is for reference. Figure 5 It shows a schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0081] like Figure 5 As shown, the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 502 or programs loaded from storage section 508 into random access memory (RAM) 503. RAM 503 also stores various programs and data required for the operation of system 500. CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0082] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a liquid crystal display (LCD) and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card and a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.

[0083] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0084] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0086] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0087] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: acquire the far-field multi-channel mixed audio to be processed and the corresponding target speaker timestamp; concatenate the amplitude spectrum of the far-field multi-channel mixed audio along the channel dimension to map it as a speech coding latent feature; segment speech segments active only by the target speaker from the reference channel of the far-field multi-channel mixed audio according to the target speaker timestamp; calculate the embedding vector of the speech segment and perform dimensional expansion to obtain the target speaker embedding feature; fuse the speech coding latent feature with the target speaker embedding feature and input it into the target speaker extraction model for feature extraction to obtain the target speaker speech feature; map the target speaker speech feature as an amplitude spectrum mask, apply the mask to the amplitude spectrum of the reference channel, and perform an inverse transform combined with the phase of the reference channel to obtain the clean speech of the target speaker.

[0088] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A weakly supervised target speaker extraction method based on pseudo-label signal generation, characterized in that, The method comprises the following steps: S1: obtaining a to-be-processed far-field multi-channel mixed audio and a corresponding target speaker timestamp; amplitude spectra of the far-field multi-channel mixed audio are spliced along a channel dimension and mapped into speech coding hidden features; S2: according to the target speaker timestamp, a speech segment in which the target speaker is active is cut out from a reference channel of the far-field multi-channel mixed audio; a speaker embedding vector of the speech segment is calculated and dimensionally expanded to obtain a target speaker embedding feature; S3: the speech coding hidden features and the target speaker embedding feature are fused and input into a target speaker extraction model for feature extraction to obtain a target speaker speech feature; S4: the target speaker speech feature is mapped into an amplitude spectrum mask, the mask is applied to the amplitude spectrum of the reference channel, and inverse transformation is performed in combination with the phase of the reference channel to obtain clean speech of the target speaker; The target speaker extraction model is obtained through weak supervision training based on a pseudo-label signal generated from a live recording data set, and the training process comprises the following steps: a live recording data set containing synchronously collected far-field mixed audio, target speaker close-talking recording and timestamps is obtained; non-target speaker active parts in the close-talking recording are set to zero according to the timestamps, and the number of time delay points between the close-talking recording after being set to zero and the far-field mixed audio is calculated and time delay compensation is performed by using a generalized cross-correlation-phase transform weighting algorithm to obtain a time domain alignment signal; multi-frame linear filtering is performed on the time domain alignment signal to align the amplitude value thereof with the far-field mixed audio, and the pseudo-label signal is generated; the far-field mixed audio is input into a to-be-trained model, the pseudo-label signal is used as a supervision target, a loss function is calculated, and model parameters are updated; The loss function is a magnitude spectrum compound loss The specific calculation formula is: Wherein, the mean square error loss The target perceptual cosine similarity loss T represents the total number of time frames, F represents the total number of frequency points, And Respectively represent the STFT coefficients of the model output And the pseudo-label signal Indicate function, used to judge whether the target signal exists or not, when the pseudo-label signal is a non-silence segment, the value is 1, otherwise the value is 0, As an adjustment factor, used to adjust The proportion in the loss value.​ 2. The weakly supervised target speaker extraction method based on pseudo label signal generation of claim 1, wherein, Computing the number of time delay points using generalized cross-correlation - phase transform weighting algorithm The formula is: where, and denote the discrete-time Fourier transform of the near talker recording and the far-field mixed audio , respectively, denote the complex exponential form used to represent signals, reflecting the rotational or oscillatory nature of signals in the frequency domain, denote the complex conjugate frequency domain representation of the far-field mixed audio signal , and denote a set of integers, offset by to align in time , resulting in . 3.The weakly supervised target speaker extraction method based on pseudo-label signal generation of claim 1, wherein, The process of performing multi-frame linear filtering on the time-domain aligned signal includes: constructing an optimization problem: MFLF filter ,in, express The short-time Fourier transform coefficients, These represent the time frame index and the frequency index, respectively. Indicates the frequency to be estimated The filter coefficient vector at that point, L time-frequency units were stacked. express STFT coefficient, Set to order 2 Representing L-dimensional complex space, Indicates Hermite transpose. It is a weighted term, and its calculation formula is: ,in, Returns the maximum value in all time-frequency cells of a given power spectrum. express The short-time Fourier transform spectrum, Returns the larger of the two values, applied to the calculated MFLF filter. This yields the final high-quality pseudo-label signal: ,in This indicates the inverse short-time Fourier transform operation.

4. The weakly supervised target speaker extraction method based on pseudo label signal generation according to any one of claims 1-3, characterized in that, The target speaker extraction model comprises a speech encoder, a speaker encoder, a feature fusion layer, a speaker extractor and a speech decoder; the speaker extractor comprises stacked cross-band modeling modules and narrow-band modeling modules; the cross-band modeling modules comprise two frequency convolution modules and one full-band linear module, which are used for modeling the context dependence of the frequency spectrum within a frame; the narrow-band modeling modules comprise a multi-head self-attention module and a time convolution feedforward module, which are used for global modeling along the time axis within a narrow band.

5. The weakly supervised target speaker extraction method based on pseudo label signal generation of claim 1, wherein, In step S1, splicing the amplitude spectrum of the far-field multi-channel mixed audio along the channel dimension specifically includes: Short-time Fourier transform amplitude spectrum of each channel The input features are obtained by concatenation. , This represents a splicing operation along the channel axis, and the channel dimensions are concatenated using a one-dimensional convolutional layer. Mapped to encoding dimension To obtain the latent features of speech coding In step S3, the specific process of feature fusion is as follows: embedding the target speaker's features after dimensional expansion. With the speech coding latent features Splicing is performed on the frequency axis. The dimension of the speaker embedding vector is used to obtain the fused features. And the frequency dimension is compressed back through a one-dimensional convolutional layer. The features input to the target speaker extraction model are obtained. , This represents a one-dimensional convolution operation.

6. The weakly supervised target speaker extraction method based on pseudo label signal generation of claim 5, wherein, The mapping of the target speaker voice feature to the amplitude spectrum mask in the S4 specifically includes: mapping the channel dimension of the target speaker voice feature to 1 through a linear layer, and limiting the output value to be positive through a ReLU activation function to obtain the amplitude spectrum mask , The target speaker voice feature predicted by the model; the inverse transformation of the phase of the combined reference channel specifically includes: performing Hadamard product operation on the amplitude spectrum mask and the amplitude spectrum of the reference channel to obtain an enhanced amplitude spectrum , and coupling the enhanced amplitude spectrum with the phase item of the reference channel to obtain a target speaker clean voice prediction value through inverse short-time Fourier transform .

7. A computer readable storage medium having stored thereon one or more computer programs. The one or more computer programs are executed by a computer processor to implement the method of any one of claims 1-6.

8. A weakly supervised target speaker extraction system based on pseudo label signal generation, characterized by, The method comprises the following steps: a feature encoding module configured to obtain a to-be-processed far-field multi-channel mixed audio and a corresponding target speaker timestamp; and splice amplitude spectra of the far-field multi-channel mixed audio along a channel dimension and map the amplitude spectra into speech coding hidden features; a speaker encoding module configured to cut out a speech segment in which a target speaker is active from a reference channel of the far-field multi-channel mixed audio according to the target speaker timestamp; calculate a speaker embedding vector of the speech segment; and dimensionally expand the speaker embedding vector to obtain a target speaker embedding feature; a speaker extraction module configured to fuse the speech coding hidden features and the target speaker embedding feature, input the features into a target speaker extraction model for feature extraction, and obtain a target speaker speech feature; and a speech decoding module configured to map the target speaker speech feature into an amplitude spectrum mask, apply the mask to the amplitude spectrum of the reference channel, perform inverse transformation in combination with the phase of the reference channel, and obtain clean speech of the target speaker. The voice reconstruction module is configured to map the target speaker voice feature to an amplitude spectrum mask, apply the mask to an amplitude spectrum of the reference channel, and combine a phase of the reference channel to obtain inverse transformation to obtain clean voice of the target speaker; The target speaker extraction model is obtained through weak supervision training based on a pseudo-label signal generated by a live recording data set, and the training process includes: obtaining a live recording data set containing synchronously collected far-field mixed audio, target speaker close-talking recording and time stamps; According to the time stamp, the non-target speaker active part in the close-talking recording is set to zero, the number of time delay points between the zeroed close-talking recording and the far-field mixed audio is calculated using a generalized cross-correlation-phase transform weighting algorithm, and time delay compensation is performed to obtain a time domain alignment signal; The time domain alignment signal is subjected to multi-frame linear filtering to align its amplitude with the far-field mixed audio, and the pseudo-label signal is generated; The far-field mixed audio is input into a model to be trained, the pseudo-label signal is used as a supervision target, a loss function is calculated, and model parameters are updated. The loss function is a magnitude spectrum compound loss The specific calculation formula is: Wherein, the mean square error loss The target perceptual cosine similarity loss T represents the total number of time frames, F represents the total number of frequency points, And Respectively represent the STFT coefficients of the model output And the pseudo-label signal Indicates a function, used to determine whether the target signal exists, and takes a value of 1 when the pseudo-label signal is a non-silence segment, otherwise takes a value of 0, As an adjustment factor, used to adjust The proportion in the loss value.​

Images