A target speech extraction method and device based on multi-reference clue fusion

Abstract

The present invention discloses a target speech extraction method based on multi-reference clue fusion, comprising: passing a multi-channel mixed speech to be extracted through a feature extraction module, calculating the complex spectrum and spatial features of the mixed speech, and obtaining a mixed speech feature map; passing the orientation information of a target speaker and a reference speech through an embedding feature extraction module, combining the directional features and the voiceprint features, and obtaining the embedded features of the target speaker; inputting the mixed speech feature map and the embedded features of the target speaker into a main extraction network based on a dual-path Fastformer neural network, and obtaining a high-dimensional feature representation of the speech to be extracted; converting the high-dimensional feature representation into a complex mask for estimating and separating speech through a mask estimation module, and applying the mask to the complex spectrum of the mixed speech signal; reconstructing the time domain waveform of the estimated extracted speech through an inverse short-time Fourier transform; and inputting the extracted speech of the target speaker into a comparative learning module, so that the correct extracted speech can also be output even when the target speaker is absent from the mixed speech.

Description

Technical Field

[0001] The present invention belongs to the field of speech separation and extraction, and in particular relates to a target speech extraction method and device based on multi-reference clue fusion. Background Art

[0002] As the core vehicle for human communication and interaction, voice, with its natural flow, ease of use, and efficient information transmission, has become an indispensable key element in contemporary human-computer interaction. In our daily lives, we are constantly exposed to diverse sound environments, where the sounds we are interested in are often intertwined with unnecessary noise from other surrounding sources and reverberation caused by environmental reflections, making them difficult to distinguish. Therefore, effectively isolating the sound components of interest and making them stand out from the complex background noise is crucial for maintaining clear interpersonal communication and ensuring the accurate transmission of instructions in human-computer interactions.

[0003] Speech separation, a fundamental task in speech front-end signal processing, aims to isolate or extract speech signals of interest from mixed speech from multiple speakers. It plays a vital role in multiple fields and greatly facilitates people's daily lives. Depending on whether reference information about the target speaker is provided, speech separation can be divided into two types: multi-speaker separation and target speech extraction. Since multi-speaker separation, when applied in real-world scenarios, requires solutions to issues such as channel sorting, speaker number conversion, and speaker tracking, target speech extraction algorithms, by contrast, extract the speech of a specific speaker from a mixed speech based on provided reference information. This not only avoids the channel sorting issues inherent in speech separation but also has a wider range of applicability. Therefore, it has gradually become a research hotspot in speech front-end processing.

[0004] In recent years, target speech extraction algorithms based on various types of reference information have been proposed, such as those based on voiceprint features, spatial features, and video. However, most algorithms rely on only a single type of reference information to extract the target speaker's speech. When the target speaker's reference information is highly similar to the interfering speaker's features, target confusion can easily occur, significantly reducing the extraction effect. This is one of the key factors restricting the performance of target speech extraction algorithms. Furthermore, existing target speech extraction algorithms typically assume that the target speaker is continuously active in the mixed speech, resulting in poor suppression of speech segments in which the target speaker is absent, seriously affecting the quality of the extracted speech. Summary of the Invention

[0005] In order to overcome the above-mentioned shortcomings of the prior art, the present invention provides a target speech extraction method and device based on multi-reference clue fusion, which accurately extracts the target speaker's speech by introducing the target speaker's position information and voiceprint information as reference features.

[0006] According to one aspect of the present invention, a method for extracting a target speech based on multi-reference clue fusion is provided, comprising:

[0007] The multi-channel mixed speech to be extracted is passed through the feature extraction module to calculate the complex spectrum and spatial features of the mixed speech and obtain a mixed speech feature map;

[0008] The target speaker's position information and reference speech are passed through the embedding feature extraction module, combined with the direction feature and voiceprint feature to obtain the target speaker's embedded features;

[0009] The mixed speech feature map and the target speaker's embedded features are input into the main extraction network based on the dual-path Fastformer neural network, which outputs a high-dimensional feature representation of the speech to be extracted.

[0010] The output high-dimensional feature representation is converted into a complex mask for estimating speech separation through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal;

[0011] The time domain waveform of the speech is extracted by inverse short-time Fourier transform reconstruction estimation;

[0012] The extracted speech of the target speaker is input into the contrastive learning module so that the correct extracted speech can be output even in mixed speech where the target speaker is absent.

[0013] As a further technical solution, the multi-channel mixed speech to be extracted is passed through a feature extraction module to calculate the complex spectrum and spatial features of the mixed speech, and obtain a mixed speech feature map, including: using short-time Fourier transform to transform the time-domain multi-channel mixed speech to be separated into the time-frequency domain to obtain the complex spectrum of the multi-channel mixed speech; introducing the phase difference between channels as an input feature to calculate the spatial features of the multi-channel mixed speech; and forming a mixed speech feature map based on the complex spectrum and spatial features.

[0014] As a further technical solution, the target speaker's position information and reference speech are embedded in a feature extraction module, combined with directional features and voiceprint features to obtain the target speaker's embedded features, including: calculating a steering vector, and concatenating the real and imaginary parts of the steering vector along the feature dimension as the input directional features; using the pre-trained classic speaker recognition model ECAPA-TDNN as a sound cue encoder to provide the target speaker's voiceprint features; and forming the target speaker's embedded features based on the directional features and voiceprint features.

[0015] As a further technical solution, the main extraction network based on the dual-path Fastformer neural network includes a spectral modeling unit, a sub-band modeling unit and a full-band modeling unit; wherein the spectral modeling unit is used to extract and upgrade the input mixed speech feature map and the embedded features of the target speaker, and use a shuffled convolution attention module to extract spectral local information and spectral spatial information; the sub-band modeling unit is used to fuse the embedded features in each frame of the mixed speech feature map through the convolution-enhanced input fully connected layer, combine the embedded features with the global query vector with global temporal context information in the convolution-enhanced multi-head attention module, and use the cross-linear attention module to fuse the local features obtained by the spectral modeling unit and the sub-band features output by the convolution enhancement module; the full-band modeling unit has the same architecture as the sub-band modeling unit, but does not include embedded feature fusion.

[0016] As a further technical solution, the output high-dimensional feature representation is converted into a complex mask for estimating the separated speech through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal, including: the output of the main extraction network based on the dual-path Fastformer neural network is divided into two paths, and amplitude estimation and phase estimation are performed respectively to obtain the amplitude spectrum and phase spectrum of the mixed speech respectively; based on the amplitude spectrum and phase spectrum, the complex spectrum of the mixed speech is obtained.

[0017] As a further technical solution, the extracted speech of the target speaker is input into a contrastive learning module so that the correct extracted speech can be output even when the target speaker is absent from the mixed speech. This includes: obtaining the output extracted by the target speech extraction network for the target speech, the mixed speech with the target speaker present, and the mixed speech without the target speaker, and constructing training samples; and training based on the training samples using a determined contrastive learning loss function so that the correct extracted speech can be output even when the target speaker is absent from the mixed speech.

[0018] According to one aspect of the present invention, a target speech extraction device based on multi-reference clue fusion is provided, comprising:

[0019] A feature extraction module is used to calculate the complex spectrum and spatial features of the mixed speech according to the multi-channel mixed speech to be extracted, and obtain a mixed speech feature map;

[0020] The embedded feature extraction module is used to obtain the embedded features of the target speaker based on the target speaker's position information and reference speech, combined with the directional features and voiceprint features;

[0021] The network extraction module is used to obtain a high-dimensional feature representation of the speech to be extracted based on the mixed speech feature map and the embedded features of the target speaker, combined with the main extraction network based on the dual-path Fastformer neural network;

[0022] A mask estimation module is used to convert the obtained high-dimensional feature representation into a complex mask for estimating the separated speech, and apply the mask to the complex spectrum of the mixed speech signal;

[0023] A reconstruction estimation module, used for reconstructing and estimating the time domain waveform of the extracted speech through inverse short-time Fourier transform;

[0024] The contrastive learning module is used to input the extracted target speaker's speech into the contrastive learning module so that the correct extracted speech can be output even when the target speaker is absent from the mixed speech.

[0025] According to one aspect of the present invention, there is provided an electronic device comprising: at least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; the memory stores program instructions to be executed by the processor, and the processor calls the program instructions to execute the described method.

[0026] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions cause the computer to execute the method described above.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1) Comprehensively utilizing both orientation features and voiceprint features as reference information. This invention can more comprehensively understand the characteristics of the target speaker, achieve more accurate and stable target speech extraction, and improve the performance and robustness of the target speech extraction system.

[0029] 2) Reduce the probability of target confusion and improve the perception of target speech absence. This invention introduces a contrastive learning training strategy to further suppress target speech absence segments.

[0030] 3) Reduce the number of network parameters and computational complexity. This paper uses lightweight structures such as depthwise separable convolution and multi-head linear attention, while ensuring that the target speaker's voice can be accurately and efficiently separated from multi-channel mixed speech, and shows significant generalization ability on different datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A flowchart of a target speech extraction method based on multi-reference clue fusion provided by an embodiment of the present invention.

[0033] Figure 2 A schematic structural diagram of a spectrum modeling unit provided in an embodiment of the present invention.

[0034] Figure 3 Schematic diagram of the fusion of local and global embedding features of ConFastformer provided in an embodiment of the present invention.

[0035] Figure 4 Schematic diagram of a target speech extraction framework based on contrastive learning provided by an embodiment of the present invention.

[0036] Figure 5 A schematic structural diagram of a target speech extraction device based on multi-reference clue fusion provided by an embodiment of the present invention.

[0037] Figure 6 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that:

[0039] This paper proposes a target speech extraction method and device based on multi-reference cue fusion. By incorporating the target speaker's position information and voiceprint information as reference features, the method accurately extracts the target speaker's speech. To further suppress speech segments in which the target speaker is absent, the paper proposes a contrastive learning loss function for the target speech extraction task, effectively improving the suppression of segments in which the target speaker is absent and reducing the probability of target confusion.

[0040] The terms "including" and "having" and any variations thereof in the description and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be executed in the order described. For example, some operations / steps may be further decomposed, while others may be combined or partially combined, so the actual execution order may vary depending on the actual situation.

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0043] The following combination Figures 1 to 4 The embodiment of the present invention is introduced. A target speech extraction method based on multi-reference clue fusion provided by the embodiment of the present invention includes the following specific steps:

[0044] Step 1: The multi-channel mixed speech to be extracted is passed through the feature extraction module to calculate the complex spectrum and spatial features of the mixed speech and obtain the mixed speech feature map.

[0045] Assuming there are P speakers and M microphones in the recording environment, the mixed signal received by the microphone array can be expressed as:

[0046]

[0047] where s p (n) represents the speech signal of the pth speaker, * represents the convolution operator symbol, h p (n) represents the RIR from the pth speaker to the microphone array, and v(n) represents the additive noise received by the microphone array.

[0048] Use STFT to transform the time-domain multi-channel mixed speech signal y(n) to be separated into the time-frequency domain. At this time, the mixed signal model formula can be restated as:

[0049]

[0050] Where Y(t,f) represents the multi-channel mixed speech spectrum, S p (t,f), and denote the speech signal, direct path signal and reverberation signal spectra of the pth speaker respectively, t and f denote the time and frequency indices respectively, and H(f) denotes the frequency response of RIR.

[0051] In addition, IPD is introduced as an input feature. The calculation method is as follows:

[0052] IPD ij =∠Y i (t, f)-∠Y j (t, f)

[0053] where ∠Y i (t,f) and ∠Y j (t, f) represents the phase of the mixed signal of the i-th channel and the j-th channel respectively. The real part and imaginary part of the mixed speech complex spectrum Y(t, f) are taken as the input spectral features. Since IPD is a periodic function, in order to avoid the problem of feature representation ambiguity, this embodiment uses cosIPD and sinIPD as input spatial features. Combining spectral features and spatial features, the input features of this embodiment can be expressed as:

[0054] F=[Y real ; Y imag ;cosIPD;sinIPD]

[0055] where Y real and Y imag Represent the real and imaginary parts of the mixed signal spectrum Y(t,f), respectively. represents the model input features calculated by the feature extraction module. Cin, T, and F represent the number of feature channels, frames, and frequencies, respectively.

[0056] Step 2: The target speaker's position information and reference speech are passed through the embedding feature extraction module, combined with the direction feature and voiceprint feature to obtain the target speaker's embedding feature;

[0057] In array signal processing, the steering vector refers to the discrete cosine transform of the direct acoustic impulse response of the sound source, which can effectively describe the directionality of the sound source. Its calculation formula is as follows:

[0058]

[0059] where v f =f×f s / F,f∈{0,...,F / 2} represents the frequency index, F represents the number of frequency bands, f s represents the speech sampling rate, r represents the radius of the microphone array, c represents the speed of sound, M represents the number of microphones, q m and q0 represent the distance between the mth microphone and the microphone reference position and the sound source respectively. In this embodiment, an 8-microphone circular microphone array is used to extract the target speech. The microphone reference position is the center of the circular microphone array. At this time, q m The relationship between -q0 and the sound source direction θ is:

[0060] q m -q0=cos(2π(m-1) / M-θ)

[0061] It can be deduced from this that the steering vector of the circular microphone array can be expressed as:

[0062]

[0063] Since the steering vector is a complex number, the real part and the imaginary part of the steering vector are concatenated along the feature dimension as the input direction feature of the model, which is expressed as

[0064] This example uses the pre-trained classic speaker recognition model ECAPA-TDNN as the acoustic cue encoder for the proposed model, providing the network with the target speaker's voiceprint feature information. The dimension of the ECAPA-TDNN output voiceprint feature vector is 1×192. To match the feature channels of the proposed model, this example uses a fully connected layer to map its voiceprint features to the frequency dimension of the proposed model. The number of input channels of this fully connected layer is 192, and the number of output channels is the number of frequency bands F. Therefore, the encoding process of the acoustic cue encoder is expressed as follows:

[0065] s a =FC(ECAPA-TDNN(s aux (t)))

[0066] where s aux (t) represents the reference speech, is the voiceprint feature of the proposed model, FC(·) represents the fully connected layer. Combined with the directional feature d in and voiceprint features a , the target speaker embedding feature of the model proposed in this embodiment can be expressed as:

[0067]

[0068] Where aux represents the embedded features of the target speaker and also indicates the voiceprint and direction information of the target speaker.

[0069] Step 3: Input the mixed speech feature map and the target speaker's embedded features into the main extraction network based on the dual-path Fastformer neural network to obtain a high-dimensional feature representation of the speech to be extracted;

[0070] TSE-DPFNN consists of three modules: spectrum modeling, sub-band modeling and full-band modeling. Its specific structure is shown in Figure 2 As shown in the figure. First, the input feature F is passed through the spectral modeling unit, and then fused and upscaled by the feature encoding module. Then, the SCBAM module is used to extract the local and spatial spectral information. After expanding it along the time dimension with the embedded features, the features are concatenated. Then, the embedded features are fused into the mixed speech feature map of each frame through the input fully connected layer of ConFastformer. At this time, each frame of mixed speech contains the target speaker information provided by the embedded features. This process is expressed as follows:

[0071]

[0072] in represents mixed speech features, expand(·) represents feature expansion calculation, represents the embedded features after expansion along the time dimension, FC(·) represents the fully connected layer, whose input channel is 2M+1+B and output channel is H, and F′ represents the output feature after fusing the embedded information.

[0073] In addition to directly fusing the embedded features in the fully connected layer, the proposed model further combines the embedded features with the global query vector with global temporal context information in the multi-head attention module of ConFastformer, enabling the model to utilize the embedded features from a global perspective. The schematic diagram of the fusion of local and global embedded features of ConFastformer is shown in the figure. Figure 3 The feature fusion implementation process based on the attention mechanism is as follows: First, the input features of the multi-head attention module are The input features are converted into query vector Q, key vector K and value vector V through a linear layer. Then, according to the preset number of attention heads h, Q, K and V are divided into h parts respectively, denoted as {Q i ,K i ,V i}, i=0,...h-1 Based on this, the global query vector q of each attention head is calculated i . The global query vector q of each attention head is iAfter splicing along the feature channel, it is fused with the embedded features and then divided into h parts along the feature channel. At this time, each global query vector attention head is embedded with the target speaker features. The formula is as follows:

[0074] q′=FC([aux;q0;...;q h-1 ])=[q′0;...;q′ h-1 ]

[0075] Next, we use the improved global vector q i , key vector set K i Sum value vector set V i , calculate the output Z of each attention head i All attention head outputs are concatenated along the feature channel and combined with the query vector Q and the multi-scale convolution module output Add together to get the multi-head linear attention output Finally, use the output fully connected layer to att Dimensionality reduction, restoring the original dimension of the feature.

[0076] The ConFastformer module in the full-band modeling stage does not contain the embedded feature fusion step. It first converts the feature dimension into T×B×F, and the remaining calculation steps are consistent with the sub-band modeling process.

[0077] Step 4: The output high-dimensional feature representation is converted into a complex mask for estimating speech separation through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal;

[0078] The output of TSE-DPFNN is divided into two paths, one for amplitude estimation and the other for phase estimation. In the phase estimation process, the output features of TSE-DPFNN are fed into a fully connected layer, and the output is mapped to a range between -1 and 1 through the Tanh activation function as a complex mask for preliminary estimation. The phase mask is then calculated based on the complex mask. The formula is as follows:

[0079]

[0080] in represents the TSE-DPFNN output, Tanh(·) represents the Tanh activation function, and denote the real and imaginary parts of the preliminary estimated complex mask, respectively. Represents the estimated phase mask, and P represents the number of speakers in the mixed speech. The phase mask is multiplied element-by-element by the phase spectrum of the mixed speech of the reference channel to obtain the phase of the separated speech. The formula is as follows:

[0081] S phase =M phase ·Yphase

[0082] in and They represent the phase spectrum of the mixed speech of the reference channel and the phase spectrum of the separated speech, respectively. In the amplitude estimation process, the output of the TSE-DPFNN is fed into another fully connected layer with a Sigmoid activation function to obtain the amplitude mask. The amplitude mask is then multiplied element-by-element by the amplitude spectrum of the mixed speech of the reference channel to obtain the amplitude of the separated speech. The formula is as follows:

[0083]

[0084] in and Represent the reference channel mixed speech amplitude spectrum and the separated speech amplitude spectrum respectively. Combining the amplitude spectrum and phase spectrum of the separated speech, the final separated speech complex spectrum is expressed as follows:

[0085]

[0086] in and Represent the real and imaginary parts of the complex spectrum of the separated speech respectively.

[0087] Step 5: Reconstruct and estimate the time domain waveform of the extracted speech through inverse short-time Fourier transform.

[0088] ISTFT is the inverse process of STFT, which is used to convert the complex spectrum of the signal in the time-frequency domain into a time-domain waveform. First, the IDFT operation is performed on each frame of the complex spectrum. The formula is as follows:

[0089]

[0090] in Represents the time domain signal of the mth frame after IDFT, and then uses the comprehensive window to The purpose of this step is to reduce the adverse effects of truncation and suppress the subjective auditory discontinuity of the frame edge. Commonly used comprehensive windows include the root mean square Hanning window and the Blackman window. Use the comprehensive window z(n) to After weighting, the output is:

[0091]

[0092] Finally Overlap and add to get the final time domain waveform

[0093] Step 6: Input the extracted target speaker's speech into the contrastive learning module so that the correct extracted speech can be output even when the target speaker is absent from the mixed speech.

[0094] The target speech extraction framework based on contrastive learning in the embodiment of the present invention is as follows Figure 4 As shown in Figure 2, the ideal target speech signal in the mixed speech is regarded as the anchor point, the mixed speech containing the target speaker is regarded as the positive sample, and the mixed speech without the target speaker is regarded as the negative sample. The target speech extraction network is regarded as the encoder in contrastive learning. The anchor point is the ideal target speech, and the output of the extraction network is exactly the same as the input. That is, for the target speech I a , extract the network output e a =T a ; For the mixed speech I of the target speaker + , the output of the extraction network should be as close as possible to the target speech, i.e. + →e a , using SDR to measure e + With e a The distance between them is expressed as follows:

[0095]

[0096] For mixed speech with the target speaker absent - , the extraction model output should be as far away from the target speech as possible and close to the silent state, that is, ||e - ||2→0, in order to make the order of magnitude of the negative sample loss function consistent with that of the positive sample loss function, this embodiment uses the similarity between the suppressed speech and the input speech to measure the degree to which the network suppresses the interfering speech in the absence of the target speaker. The formula is as follows:

[0097]

[0098] Combining the loss functions of positive and negative samples, the contrastive learning loss of the model proposed in this embodiment is defined as:

[0099]

[0100] Where γ is a regulation factor used to adjust the degree of suppression of negative samples. A larger value indicates a higher degree of suppression. For a set of input training samples, the proposed model matches the reference cues, mixed speech, and ideal target speech to form positive sample pairs. In the positive sample pairs, the mixed speech contains the target speaker's speech indicated by the reference cues. Subsequently, in order to generate negative sample pairs, the order of the reference cues is randomly disrupted during training. At this time, the mixed speech does not contain the target speaker's speech corresponding to the reference cues. The disrupted reference cues and the mixed speech constitute the negative samples. Therefore, for each set of input mixed speech, the loss function is defined as:

[0101]

[0102] in Representation is the output representation of the positive sample, represents the output representation of negative samples, represents the target speech corresponding to the i-th mixed speech.

[0103] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0104] The implementation of each embodiment of the present invention is based on programmed processing performed by a device with processor functionality. Therefore, in practical engineering, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, and in addition to the aforementioned embodiments, an embodiment of the present invention provides a target speech extraction device based on multi-reference cue fusion, which is used to implement the target speech extraction method based on multi-reference cue fusion described in the aforementioned method embodiments.

[0105] See also Figure 5 The device includes: a feature extraction module, which is used to calculate the complex spectrum and spatial features of the mixed speech according to the multi-channel mixed speech to be extracted, and obtain a mixed speech feature map; an embedded feature extraction module, which is used to obtain the embedded features of the target speaker according to the orientation information of the target speaker and the reference speech, combined with the directional features and the voiceprint features; a network extraction module, which is used to obtain the high-dimensional feature representation of the speech to be extracted according to the mixed speech feature map and the embedded features of the target speaker, combined with the main extraction network based on the dual-path Fastformer neural network; a mask estimation module, which is used to convert the obtained high-dimensional feature representation into a complex mask for estimating the separated speech, and apply the mask to the complex spectrum of the mixed speech signal; a reconstruction estimation module, which is used to reconstruct the time domain waveform of the estimated extracted speech through inverse short-time Fourier transform; and a contrastive learning module, which is used to input the extracted speech of the target speaker into the contrastive learning module, so that the correct extracted speech can be output even when the target speaker is absent from the mixed speech.

[0106] The target speech extraction device based on multi-reference clue fusion provided by the embodiment of the present invention is aimed at the problem that the existing speech extraction method is prone to target confusion and error extraction. Figure 5 Several modules in the proposed method introduce the target speaker's position information and voiceprint information as reference features to achieve accurate extraction of the target speaker's voice. By proposing a contrastive learning loss function for the target speech extraction task, the suppression effect of the target speaker's absence segment is effectively improved and the target confusion probability is reduced.

[0107] It should be noted that the device embodiments provided by the present invention are not only used to implement the methods in the above-mentioned method embodiments, but also used to implement the methods in other method embodiments provided by the present invention. The only difference is the setting of corresponding functional modules. The principles thereof are basically the same as those of the above-mentioned device embodiments provided by the present invention. As long as those skilled in the art refer to the specific technical solutions in other method embodiments on the basis of the above-mentioned device embodiments, obtain corresponding technical means and technical solutions composed of these technical means by combining technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above-mentioned device embodiments to obtain corresponding device-type embodiments, which are used to implement the methods in other method-type embodiments. For example:

[0108] Based on the content of the above device embodiment, as a preferred embodiment, the target speech extraction device based on multi-reference clue fusion provided in the embodiment of the present invention, the feature extraction module is further used to execute the following instructions:

[0109] The multi-channel mixed speech to be extracted is passed through a feature extraction module to calculate the complex spectrum and spatial features of the mixed speech, and obtain a mixed speech feature map, including: using short-time Fourier transform to transform the time-domain multi-channel mixed speech to be separated into the time-frequency domain to obtain the complex spectrum of the multi-channel mixed speech; introducing the phase difference between channels as an input feature to calculate the spatial features of the multi-channel mixed speech; and forming a mixed speech feature map based on the complex spectrum and spatial features.

[0110] Based on the content of the above device embodiment, as a preferred embodiment, the target speech extraction device based on multi-reference clue fusion provided in the embodiment of the present invention, the embedded feature extraction module is further used to execute the following instructions:

[0111] The target speaker's orientation information and reference speech are passed through an embedding feature extraction module, combined with directional features and voiceprint features, to obtain the target speaker's embedded features. This includes: calculating a steering vector and concatenating the real and imaginary parts of the steering vector along the feature dimension as the input directional features; using a pre-trained classic speaker recognition model ECAPA-TDNN as a sound cue encoder to provide the target speaker's voiceprint features; and forming the target speaker's embedded features based on the directional features and voiceprint features.

[0112] Based on the content of the above device embodiment, as a preferred embodiment, the target speech extraction device based on multi-reference clue fusion provided in the embodiment of the present invention, the main extraction network based on the dual-path Fastformer neural network also includes:

[0113] A spectral modeling unit, a sub-band modeling unit and a full-band modeling unit; wherein the spectral modeling unit is used to extract and upgrade the input mixed speech feature map and the embedded features of the target speaker, and use a shuffled convolution attention module to extract spectral local information and spectral spatial information; the sub-band modeling unit is used to fuse the embedded features in each frame of the mixed speech feature map through the convolution-enhanced input fully connected layer, combine the embedded features with the global query vector with global temporal context information in the convolution-enhanced multi-head attention module, and use the cross-linear attention module to fuse the local features obtained by the spectral modeling unit and the sub-band features output by the convolution enhancement module; the full-band modeling unit has the same architecture as the sub-band modeling unit, but does not include embedded feature fusion.

[0114] Based on the content of the above device embodiment, as a preferred embodiment, in the target speech extraction device based on multi-reference clue fusion provided in the embodiment of the present invention, the mask estimation module is further used to execute the following instructions:

[0115] The output high-dimensional feature representation is converted into a complex mask for estimating the separated speech through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal, including: the output of the main extraction network based on the dual-path Fastformer neural network is divided into two paths, and amplitude estimation and phase estimation are performed respectively to obtain the amplitude spectrum and phase spectrum of the mixed speech respectively; and the complex spectrum of the mixed speech is obtained based on the amplitude spectrum and phase spectrum.

[0116] Based on the content of the above device embodiment, as a preferred embodiment, in the target speech extraction device based on multi-reference clue fusion provided in the embodiment of the present invention, the contrastive learning module is further used to execute the following instructions:

[0117] The extracted speech of the target speaker is input into a contrastive learning module so that the correct extracted speech can be output even for mixed speech without the target speaker. The method includes: obtaining outputs extracted by a target speech extraction network for the target speech, the mixed speech with the target speaker, and the mixed speech without the target speaker, and constructing training samples; and training based on the training samples using a determined contrastive learning loss function so that the correct extracted speech can be output even for mixed speech without the target speaker.

[0118] The method of the embodiment of the present invention is implemented by electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 6As shown, the electronic device includes: at least one processor, a communications interface, at least one memory, and a communications bus, wherein the at least one processor, the communications interface, and the at least one memory communicate with each other via the communications bus. The at least one processor invokes logic instructions in the at least one memory to execute all or part of the steps of the methods provided in the aforementioned method embodiments.

[0119] In addition, when the logic instructions in the at least one memory are implemented in the form of a software functional unit and sold or used as an independent product, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution is embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (a personal computer, a server, or a network device) to execute all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, various media for storing program codes.

[0120] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place or distributed across multiple network units. Depending on practical needs, some or all of these modules may be selected to achieve the objectives of this embodiment. Persons of ordinary skill in the art will understand and implement these embodiments without inventive effort.

[0121] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0125] In summary, the present invention discloses a target speech extraction method based on multi-reference clue fusion, which mainly solves the problem that existing speech extraction methods are prone to target confusion and erroneous extraction. The main steps include: 1) passing the multi-channel mixed speech to be extracted through a feature extraction module to calculate the complex spectrum and spatial features of the mixed speech and obtain a mixed speech feature map; 2) passing the orientation information of the target speaker and the reference speech through an embedding feature extraction module, combining the directional features and the voiceprint features, to obtain the embedded features of the target speaker; 3) inputting the mixed speech feature map and the embedded features of the target speaker into a main extraction network based on a dual-path Fastformer neural network to obtain a high-dimensional feature representation of the speech to be extracted; 4) passing the output high-dimensional feature representation through a mask estimation module to convert it into a complex mask for estimating the separated speech, and applying the mask to the complex spectrum of the mixed speech signal; 5) reconstructing the time domain waveform of the estimated extracted speech through an inverse short-time Fourier transform; 6) inputting the extracted target speaker speech into a comparative learning module, so that the correct extracted speech can also be output even when the target speaker is absent from the mixed speech. The speech extraction quality of the present invention is better than that of the extraction model based on a single reference clue, and the target confusion probability is significantly reduced. It can flexibly adapt to different types of mixed speech states and is expected to provide strong technical support and practical reference for solving the problem of target speech extraction.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A target speech extraction method based on multi-reference clue fusion, characterized in that: include: The multi-channel mixed speech to be extracted is passed through the feature extraction module to calculate the complex spectrum and spatial features of the mixed speech and obtain a mixed speech feature map; The target speaker's position information and reference speech are passed through the embedding feature extraction module, combined with the direction feature and voiceprint feature to obtain the target speaker's embedded features; The mixed speech feature map and the target speaker's embedded features are input into the main extraction network based on the dual-path Fastformer neural network, which outputs a high-dimensional feature representation of the speech to be extracted. The output high-dimensional feature representation is converted into a complex mask for estimating speech separation through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal; The time domain waveform of the speech is extracted by inverse short-time Fourier transform reconstruction estimation; The extracted speech of the target speaker is input into the contrastive learning module so that the correct extracted speech can be output even in mixed speech where the target speaker is absent.

2. The target speech extraction method based on multi-reference clue fusion according to claim 1, characterized in that: The multi-channel mixed speech to be extracted is passed through a feature extraction module to calculate the complex spectrum and spatial features of the mixed speech, and obtain a mixed speech feature map, including: using short-time Fourier transform to transform the time-domain multi-channel mixed speech to be separated into the time-frequency domain to obtain the complex spectrum of the multi-channel mixed speech; introducing the phase difference between channels as an input feature to calculate the spatial features of the multi-channel mixed speech; and forming a mixed speech feature map based on the complex spectrum and spatial features.

3. The target speech extraction method based on multi-reference clue fusion according to claim 1, characterized in that: The target speaker's orientation information and reference speech are passed through an embedding feature extraction module, combined with directional features and voiceprint features, to obtain the target speaker's embedded features. This includes: calculating a steering vector and concatenating the real and imaginary parts of the steering vector along the feature dimension as the input directional features; using a pre-trained classic speaker recognition model ECAPA-TDNN as a sound cue encoder to provide the target speaker's voiceprint features; and forming the target speaker's embedded features based on the directional features and voiceprint features.

4. The target speech extraction method based on multi-reference clue fusion according to claim 1, characterized in that: The main extraction network based on the dual-path Fastformer neural network includes a spectral modeling unit, a sub-band modeling unit and a full-band modeling unit; wherein the spectral modeling unit is used to extract and upgrade the input mixed speech feature map and the embedded features of the target speaker, and use a shuffled convolution attention module to extract spectral local information and spectral spatial information; the sub-band modeling unit is used to fuse the embedded features in each frame of the mixed speech feature map through the convolution-enhanced input fully connected layer, combine the embedded features with the global query vector with global temporal context information in the convolution-enhanced multi-head attention module, and use the cross-linear attention module to fuse the local features obtained by the spectral modeling unit with the sub-band features output by the convolution enhancement module; the full-band modeling unit has the same architecture as the sub-band modeling unit, but does not include embedded feature fusion.

5. The target speech extraction method based on multi-reference clue fusion according to claim 1, characterized in that: The output high-dimensional feature representation is converted into a complex mask for estimating the separated speech through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal, including: the output of the main extraction network based on the dual-path Fastformer neural network is divided into two paths, and amplitude estimation and phase estimation are performed respectively to obtain the amplitude spectrum and phase spectrum of the mixed speech respectively; and the complex spectrum of the mixed speech is obtained based on the amplitude spectrum and phase spectrum.

6. The target speech extraction method based on multi-reference clue fusion according to claim 1, characterized in that: The extracted speech of the target speaker is input into a contrastive learning module so that the correct extracted speech can be output even for mixed speech without the target speaker. The method includes: obtaining outputs extracted by a target speech extraction network for the target speech, the mixed speech with the target speaker, and the mixed speech without the target speaker, and constructing training samples; and training based on the training samples using a determined contrastive learning loss function so that the correct extracted speech can be output even for mixed speech without the target speaker.

7. A target speech extraction device based on multi-reference clue fusion, characterized in that: include: A feature extraction module is used to calculate the complex spectrum and spatial features of the mixed speech according to the multi-channel mixed speech to be extracted, and obtain a mixed speech feature map; The embedded feature extraction module is used to obtain the embedded features of the target speaker based on the target speaker's position information and reference speech, combined with the directional features and voiceprint features; The network extraction module is used to obtain a high-dimensional feature representation of the speech to be extracted based on the mixed speech feature map and the embedded features of the target speaker, combined with the main extraction network based on the dual-path Fastformer neural network; A mask estimation module is used to convert the obtained high-dimensional feature representation into a complex mask for estimating the separated speech, and apply the mask to the complex spectrum of the mixed speech signal; A reconstruction estimation module, used for reconstructing and estimating the time domain waveform of the extracted speech through inverse short-time Fourier transform; The contrastive learning module is used to input the extracted speech of the target speaker into the contrastive learning module so that the correct extracted speech can be output even in mixed speech where the target speaker is absent.

8. An electronic device, characterized in that: include: At least one processor, at least one memory, and a communication interface; wherein the processor, memory, and communication interface communicate with each other; The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the method of any one of claims 1 to 6.

Images