Target voice extraction method and device based on multi-reference clue fusion

Abstract

The invention discloses a target voice extraction method based on multi-reference clue fusion, and the method comprises the steps: enabling a to-be-extracted multi-channel mixed voice to pass through a feature extraction module, calculating the complex spectrum and spatial features of the mixed voice, and obtaining a mixed voice feature map; enabling the azimuth information and the reference voice of the target speaker to pass through an embedded feature extraction module, and combining the direction features and the voiceprint features to obtain embedded features of the target speaker; inputting the mixed speech feature map and the embedded features of the target speaker into a main extraction network based on a dual-path Fastform neural network to obtain high-dimensional feature representation of the speech to be extracted; the high-dimensional feature representation is converted into a plurality of masks for estimating the separated voice through a mask estimation module, and the masks act on a plurality of spectrums of the mixed voice signals; estimating and extracting a time domain waveform of the speech through inverse short-time Fourier transform reconstruction; and the extracted voice of the target speaker is input into the comparison learning module, so that the correct extracted voice can be output when the mixed voice is absent by the target speaker.

Description

Technical Field

[0001] The invention belongs to the field of speech separation and extraction, and in particular relates to a target speech extraction method and device based on multi-reference clue fusion. Background Art

[0002] As the core carrier of human communication and interaction, voice has become an indispensable key element in the field of contemporary human-computer interaction due to its natural fluency, simple operation and high efficiency of information transmission. In daily life, we are always in a diverse sound environment, in which the sound we are concerned about is often intertwined with the redundant noise generated by other sound sources around us and the reverberation caused by environmental reflections, making it difficult to distinguish. In view of this, effectively separating the sound components we are interested in and making them stand out from the complex background interference is of great value in maintaining clear interpersonal communication and ensuring the accurate transmission of instructions in human-computer interaction.

[0003] As a basic task in the field of speech front-end signal processing, speech separation aims to separate or extract speech signals of interest from mixed speech of multiple speakers. It plays an important role in many fields and brings great convenience to people's daily lives. Depending on whether reference information of the target speaker is provided, speech separation can be divided into two types: multi-speaker separation and target speech extraction. Since multi-speaker separation also needs to solve problems such as channel sorting, speaker number conversion and speaker tracking when applied to actual scenarios, in contrast, the target speech extraction algorithm extracts the speech of a specific speaker from the mixed speech based on the provided reference information. It not only avoids the channel arrangement problem in speech separation, but also has a wider range of applicable scenarios. Therefore, it has gradually become a research hotspot in the field of speech front-end processing.

[0004] In recent years, target speech extraction algorithms based on various types of reference information have been proposed one after another, such as target speech extraction algorithms based on voiceprint features, target speech extraction algorithms based on spatial features, and target speech extraction algorithms based on video. However, most algorithms rely on only a single type of reference information to extract the target speaker's speech. When the target speaker's reference information has a high similarity with the interfering speaker's features, target confusion is prone to occur, resulting in a significant decrease in the extraction effect. This is also one of the key factors restricting the performance improvement of the target speech extraction algorithm. In addition, existing target speech extraction algorithms usually assume that the target speaker is in a continuously active state in the mixed speech, resulting in poor suppression of speech segments where the target speaker is absent, which seriously affects the quality of the extracted speech. Summary of the invention

[0005] In order to overcome the above-mentioned deficiencies of the prior art, the present invention provides a target speech extraction method and device based on multi-reference clue fusion, which introduces the position information and voiceprint information of the target speaker as reference features to achieve accurate extraction of the target speaker's speech.

[0006] According to one aspect of the present invention, a method for extracting a target speech based on multi-reference clue fusion is provided, comprising:

[0007] The multi-channel mixed speech to be extracted is passed through a feature extraction module to calculate the complex spectrum and spatial features of the mixed speech, and obtain a mixed speech feature map;

[0008] The position information of the target speaker and the reference speech are embedded in the feature extraction module, combined with the direction feature and voiceprint feature to obtain the embedded feature of the target speaker;

[0009] The mixed speech feature map and the target speaker's embedded features are input into the main extraction network based on the dual-path Fastformer neural network, which outputs a high-dimensional feature representation of the speech to be extracted.

[0010] The output high-dimensional feature representation is converted into a complex mask for estimating the separated speech through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal;

[0011] The time domain waveform of the speech is extracted by inverse short-time Fourier transform reconstruction estimation;

[0012] The extracted speech of the target speaker is input into the contrastive learning module, so that the correct extracted speech can be output even in the mixed speech where the target speaker is absent.

[0013] As a further technical solution, the multi-channel mixed speech to be extracted is passed through a feature extraction module to calculate the complex spectrum and spatial features of the mixed speech, and obtain a mixed speech feature map, including: using short-time Fourier transform to transform the time-domain multi-channel mixed speech to be separated into the time-frequency domain to obtain the complex spectrum of the multi-channel mixed speech; introducing the phase difference between channels as an input feature to calculate the spatial features of the multi-channel mixed speech; and forming a mixed speech feature map based on the complex spectrum and spatial features.

[0014] As a further technical solution, the orientation information and reference speech of the target speaker are embedded in a feature extraction module, combined with directional features and voiceprint features to obtain embedded features of the target speaker, including: calculating a steering vector, and concatenating the real and imaginary parts of the steering vector along the feature dimension as the input directional features; using a pre-trained classic speaker recognition model ECAPA-TDNN as a sound cue encoder to provide the voiceprint features of the target speaker; and forming the embedded features of the target speaker based on the directional features and voiceprint features.

[0015] As a further technical solution, the main extraction network based on the dual-path Fastformer neural network includes a spectral modeling unit, a sub-band modeling unit and a full-band modeling unit; wherein the spectral modeling unit is used to extract and upgrade the input mixed speech feature map and the embedded features of the target speaker, and use a shuffled convolutional attention module to extract spectral local information and spectral spatial information; the sub-band modeling unit is used to fuse the embedded features in each frame of the mixed speech feature map through a convolution-enhanced input fully connected layer, combine the embedded features with a global query vector with global temporal context information in a convolution-enhanced multi-head attention module, and use a cross-linear attention module to fuse the local features obtained by the spectral modeling unit with the sub-band features output by the convolution enhancement module; the full-band modeling unit has the same architecture as the sub-band modeling unit, but does not include embedded feature fusion.

[0016] As a further technical solution, the output high-dimensional feature representation is converted into a complex mask for estimating the separated speech through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal, including: the output of the main extraction network based on the dual-path Fastformer neural network is divided into two paths, and amplitude estimation and phase estimation are performed respectively to obtain the amplitude spectrum and phase spectrum of the mixed speech respectively; based on the amplitude spectrum and phase spectrum, the complex spectrum of the mixed speech is obtained.

[0017] As a further technical solution, the extracted speech of the target speaker is input into a contrastive learning module so that the mixed speech without the target speaker can also output the correct extracted speech, including: for the target speech, the mixed speech with the target speaker and the mixed speech without the target speaker, the output extracted by the target speech extraction network is obtained respectively, and training samples are constructed; according to the training samples, training is performed using a determined contrastive learning loss function so that the mixed speech without the target speaker can also output the correct extracted speech.

[0018] According to one aspect of the present invention, there is provided a target speech extraction device based on multi-reference clue fusion, comprising:

[0019] A feature extraction module, used to calculate the complex spectrum and spatial features of the mixed speech according to the multi-channel mixed speech to be extracted, and obtain a mixed speech feature map;

[0020] An embedding feature extraction module is used to obtain the embedding features of the target speaker based on the position information and reference speech of the target speaker, combined with the directional features and the voiceprint features;

[0021] The network extraction module is used to obtain a high-dimensional feature representation of the speech to be extracted based on the mixed speech feature graph and the embedded features of the target speaker, combined with the main extraction network based on the dual-path Fastformer neural network;

[0022] A mask estimation module, used to convert the obtained high-dimensional feature representation into a complex mask for estimating the separated speech, and apply the mask to the complex spectrum of the mixed speech signal;

[0023] A reconstruction estimation module, used for reconstructing and estimating the extracted time domain waveform of speech through inverse short-time Fourier transform;

[0024] The contrastive learning module is used to input the extracted target speaker's speech into the contrastive learning module so that the correct extracted speech can be output even when the target speaker is absent from the mixed speech.

[0025] According to one aspect of the present specification, there is provided an electronic device, comprising: at least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; the memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the described method.

[0026] According to one aspect of the present specification, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to execute the method described above.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] 1) Comprehensively utilize the two features of orientation and voiceprint as reference information. The present invention can more comprehensively understand the characteristics of the target speaker, achieve more accurate and stable target speech extraction, and improve the performance and robustness of the target speech extraction system.

[0029] 2) Reduce the probability of target confusion and improve the perception of target speech absence scenes. The present invention introduces a contrastive learning training strategy to further suppress target speech absence segments.

[0030] 3) Reduce the number of network parameters and computational complexity. The present invention uses lightweight structures such as deep separable convolution and multi-head linear attention, while ensuring that the target speaker's voice can be accurately and efficiently separated from multi-channel mixed speech, and shows significant generalization ability on different data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction is given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0032] Figure 1 A flowchart of a target speech extraction method based on multi-reference clue fusion provided by an embodiment of the present invention.

[0033] Figure 2 A schematic diagram of the structure of a spectrum modeling unit provided in an embodiment of the present invention.

[0034] Figure 3 Schematic diagram of the fusion of local and global embedding features of ConFastformer provided in an embodiment of the present invention.

[0035] Figure 4 A schematic diagram of a target speech extraction framework based on contrastive learning provided in an embodiment of the present invention.

[0036] Figure 5 A schematic diagram of the structure of a target speech extraction device based on multi-reference clue fusion provided by an embodiment of the present invention.

[0037] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0038] It should be noted that:

[0039] The present invention proposes a method and device for extracting target speech based on multi-reference clue fusion, which introduces the position information and voiceprint information of the target speaker as reference features to achieve accurate extraction of the target speaker's speech. In order to further suppress speech segments in which the target speaker is absent, the present invention proposes a contrastive learning loss function for the target speech extraction task, which effectively improves the suppression effect of the target speaker's absent segments and reduces the target confusion probability.

[0040] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, for example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely exemplary and do not necessarily include all the contents and operations / steps, nor do they necessarily have to be executed in the order described. For example, some operations / steps may be further decomposed, while some operations / steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or the structural composition mode, but must be based on the ability of ordinary technicians in this field to achieve. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0043] Combine the following Figures 1 to 4 The embodiment of the present invention is introduced. A method for extracting target speech based on multi-reference clue fusion provided by an embodiment of the present invention comprises the following specific steps:

[0044] Step 1: Pass the multi-channel mixed speech to be extracted through the feature extraction module, calculate the complex spectrum and spatial features of the mixed speech, and obtain the mixed speech feature map.

[0045] Assuming there are P speakers and M microphones in the recording environment, the mixed signal received by the microphone array can be expressed as:

[0046]

[0047] where s p (n) represents the speech signal of the pth speaker, * represents the convolution operator, h p (n) represents the RIR from the pth speaker to the microphone array, and v(n) represents the additive noise received by the microphone array.

[0048] Use STFT to transform the time domain multi-channel mixed speech signal y(n) to be separated into the time-frequency domain. At this time, the mixed signal model formula can be restated as:

[0049]

[0050] Where Y(t,f) represents the multi-channel mixed speech spectrum, S p (t,f), and denote the speech signal, direct path signal and reverberation signal spectra of the pth speaker respectively, t and f denote the time and frequency indexes respectively, and H(f) denotes the frequency response of RIR.

[0051] In addition, IPD is introduced as an input feature. The calculation method is as follows:

[0052] IPD ij =∠Y i (t, f)-∠Y j (t, f)

[0053] Where ∠Y i (t,f) and ∠Y j (t, f) represent the phase of the mixed signal of the i-th channel and the j-th channel respectively. The real part and the imaginary part of the mixed speech complex spectrum Y(t, f) are taken as the input spectrum features. Since IPD is a periodic function, in order to avoid the problem of feature representation ambiguity, this embodiment uses cosIPD and sinIPD as input space features. Combining the spectrum features and space features, the input features of this embodiment can be expressed as:

[0054] F=[Y real ; Y imag ;cosIPD;sinIPD]

[0055] where Y real and Y imag Respectively represent the real and imaginary parts of the mixed signal spectrum Y(t,f), It represents the model input features calculated by the feature extraction module. Cin, T and F represent the number of feature channels, frames and frequencies respectively.

[0056] Step 2: The position information of the target speaker and the reference speech are embedded in the feature extraction module, combined with the direction feature and voiceprint feature to obtain the embedded feature of the target speaker;

[0057] In array signal processing, the steering vector refers to the discrete cosine transform of the direct acoustic impulse response of the sound source, which can effectively describe the directionality of the sound source. Its calculation formula is as follows:

[0058]

[0059] where v f =f×f s / F, f∈{0,...,F / 2} represents the frequency index, F represents the number of frequency bands, and f s represents the speech sampling rate, r represents the radius of the microphone array, c represents the speed of sound, M represents the number of microphones, and q m and q0 represent the distance between the mth microphone and the microphone reference position and the sound source, respectively. In this embodiment, an 8-microphone circular microphone array is used to achieve target speech extraction, and the microphone reference position is the center of the circular microphone array. At this time, q m The relationship between -q0 and the sound source direction θ is:

[0060] q m -q0=cos(2π(m-1) / M-θ)

[0061] It can be deduced that the steering vector of the circular microphone array can be expressed as:

[0062]

[0063] Since the steering vector is a complex number, the real part and the imaginary part of the steering vector are concatenated along the feature dimension as the input direction feature of the model, which is expressed as

[0064] This embodiment selects the pre-trained classic speaker recognition model ECAPA-TDNN as the sound clue encoder of the proposed model to provide the network with the voiceprint feature information of the target speaker. The dimension of the voiceprint feature vector output by ECAPA-TDNN is 1×192. In order to match the feature channel of the proposed model, this embodiment uses a fully connected layer to map its voiceprint features to the frequency dimension of the proposed model. The number of input channels of the fully connected layer is 192, and the number of output channels is the number of frequency bands F. Therefore, the encoding process formula of the sound clue encoder is expressed as follows:

[0065] s a =FC(ECAPA-TDNN(s aux (t)))

[0066] where s aux (t) represents the reference speech, is the voiceprint feature of the proposed model, FC(·) represents the fully connected layer. Combined with the directional feature d in Voiceprint features a , the target speaker embedding feature of the model proposed in this embodiment can be expressed as:

[0067]

[0068] Where aux represents the embedded features of the target speaker and also indicates the voiceprint and direction information of the target speaker.

[0069] Step 3: Input the mixed speech feature map and the embedded features of the target speaker into the main extraction network based on the dual-path Fastformer neural network to obtain a high-dimensional feature representation of the speech to be extracted;

[0070] TSE-DPFNN consists of three modules: spectrum modeling, sub-band modeling and full-band modeling. Its specific structure is shown in Figure 2 As shown. First, the input feature F is passed through the spectrum modeling unit, and then the feature encoding module is used to fuse and upgrade the input features. Then, the SCBAM module is used to extract the local spectrum information and the spectrum space information, which are expanded along the time dimension with the embedded features for feature concatenation. Then, the embedded features are fused in each frame of the mixed speech feature map through the input fully connected layer of ConFastformer. At this time, each frame of the mixed speech contains the target speaker information provided by the embedded features. The formula of this process is expressed as follows:

[0071]

[0072] in represents mixed speech features, expand(·) represents feature expansion calculation, represents the embedded features after expansion along the time dimension, FC(·) represents the fully connected layer, whose input channel is 2M+1+B and output channel is H, and F′ represents the output features after fusing the embedded information.

[0073] In addition to directly fusing the embedded features in the fully connected layer, the proposed model further combines the embedded features with the global query vector with global temporal context information in the multi-head attention module of ConFastformer, so that the model can utilize the embedded features from a global perspective. The schematic diagram of the fusion of local and global embedded features of ConFastformer is shown in the figure. Figure 3 The feature fusion implementation process based on the attention mechanism is as follows: First, the input features of the multi-head attention module are The input features are transformed into query vector Q, key vector K and value vector V through a linear layer. Then, according to the preset number of attention heads h, Q, K and V are divided into h parts respectively, denoted as {Q i ,K i ,V i}, i = 0, ... h-1 Based on this, the global query vector q of each attention head is calculated i . The global query vector q of each attention head is iAfter splicing along the feature channel, it is fused with the embedded features and then cut into h parts along the feature channel. At this time, each global query vector attention head is embedded with the target speaker features. The formula is as follows:

[0074] q′=FC([aux;q0;...;q h-1 ])=[q′0;...;q′ h-1 ]

[0075] Next, we use the improved global vector q i , key vector set K i Sum value vector set V i , calculate the output Z of each attention head i All attention head outputs are concatenated along the feature channels and combined with the query vector Q and the multi-scale convolution module output Add together to get the multi-head linear attention output Finally, use the output fully connected layer to att Dimensionality reduction, restoring the original dimension of the feature.

[0076] The ConFastformer module in the full-band modeling stage does not contain the embedded feature fusion step. It first converts the feature dimension into T×B×F, and the remaining calculation steps are consistent with the sub-band modeling process.

[0077] Step 4: The output high-dimensional feature representation is converted into a complex mask for estimating the separated speech through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal;

[0078] The output of TSE-DPFNN is divided into two paths for amplitude estimation and phase estimation respectively. In the phase estimation process, the output features of TSE-DPFNN are sent to a fully connected layer, and the output is mapped to between -1 and 1 through the Tanh activation function as a complex mask for preliminary estimation. Then, the phase mask is calculated based on the complex mask. The formula is as follows:

[0079]

[0080] in represents the TSE-DPFNN output, Tanh(·) represents the Tanh activation function, and denote the real and imaginary parts of the preliminary estimated complex mask, respectively. Represents the estimated phase mask, and P represents the number of speakers in the mixed speech. The phase mask is multiplied element by element with the phase spectrum of the mixed speech of the reference channel to obtain the phase of the separated speech. The formula is as follows:

[0081] S phase =M phase ·Yphase

[0082] in and They represent the phase spectrum of the mixed speech of the reference channel and the phase spectrum of the separated speech. In the amplitude estimation process, the output of TSE-DPFNN is sent to another fully connected layer with a Sigmoid activation function to obtain the amplitude mask. The amplitude mask is multiplied element by element with the amplitude spectrum of the mixed speech of the reference channel to obtain the amplitude of the separated speech. The formula is as follows:

[0083]

[0084] in and They represent the reference channel mixed speech amplitude spectrum and the separated speech amplitude spectrum respectively. The amplitude spectrum of the separated speech is combined with the phase spectrum, and the final separated speech complex spectrum is expressed as follows:

[0085]

[0086] in and Respectively represent the real and imaginary parts of the complex spectrum of the separated speech.

[0087] Step 5: Reconstruct and estimate the time domain waveform of the extracted speech through inverse short-time Fourier transform.

[0088] ISTFT is the inverse process of STFT, which is used to convert the complex spectrum of the signal in the time-frequency domain into a time-domain waveform. First, perform IDFT operation on each frame of the complex spectrum, and the formula is as follows:

[0089]

[0090] in represents the time domain signal of the mth frame after IDFT, and then uses the comprehensive window to The purpose of this step is to reduce the adverse effects of truncation and suppress the subjective discontinuity of the frame edge. Commonly used comprehensive windows include the RMS Hanning window and the Blackman window. After weighting, the output is:

[0091]

[0092] Finally Overlap and add to get the final time domain waveform

[0093] Step 6: Input the extracted speech of the target speaker into the contrastive learning module so that the correct extracted speech can be output even in the mixed speech where the target speaker is absent.

[0094] The target speech extraction framework based on contrastive learning in the embodiment of the present invention is as follows: Figure 4 As shown in Figure 1, the ideal target speech signal in the mixed speech is regarded as the anchor point, the mixed speech containing the target speaker is regarded as the positive sample, the mixed speech without the target speaker is regarded as the negative sample, and the target speech extraction network is regarded as the encoder in contrastive learning. The anchor point is the ideal target speech, and the output of the extraction network is exactly the same as the input, that is, for the target speech I a , extract the network output e a =T a ; For the mixed speech I of the target speaker + , the output of the extraction network should be as close as possible to the target speech, i.e. + →e a , using SDR to measure e + With e a The distance between them is expressed as follows:

[0095]

[0096] For mixed speech with the target speaker absent I - , the extraction model output should be as far away from the target speech as possible and close to the silent state, that is, ||e - ||2→0, in order to make the negative sample loss function consistent with the positive sample loss function, this embodiment uses the similarity between the suppressed speech and the input speech to measure the degree of suppression of the network on the interfering speech in the absence of the target speaker. The formula is as follows:

[0097]

[0098] Combining the loss functions of positive samples and negative samples, the contrastive learning loss of the model proposed in this embodiment is defined as:

[0099]

[0100] Where γ is a regulation factor used to adjust the degree of suppression of negative samples. The larger the value, the higher the degree of suppression. For a set of input training samples, the proposed model matches the reference cues, mixed speech and ideal target speech to form positive sample pairs. In the positive sample pairs, the mixed speech contains the target speaker's speech indicated by the reference cues. Subsequently, in order to generate negative sample pairs, the order of the reference cues is randomly disrupted during training. At this time, the mixed speech does not contain the target speaker's speech corresponding to the reference cues, and the disrupted reference cues and mixed speech constitute negative samples. Therefore, for each set of input mixed speech, the loss function is defined as:

[0101]

[0102] in Representation is the output representation of the positive sample, represents the output representation of negative samples, Represents the target speech corresponding to the i-th mixed speech.

[0103] It should be understood that parts not elaborated in detail in this specification belong to the prior art.

[0104] The implementation basis of each embodiment of the present invention is to implement programmed processing through a device with a processor function. Therefore, in engineering practice, the technical solutions and functions of each embodiment of the present invention are encapsulated into various modules. Based on this reality, on the basis of the above embodiments, an embodiment of the present invention provides a target speech extraction device based on multi-reference clue fusion, which is used to execute the target speech extraction method based on multi-reference clue fusion in the above method embodiment.

[0105] See also Figure 5 The device includes: a feature extraction module, which is used to calculate the complex spectrum and spatial features of the mixed speech according to the multi-channel mixed speech to be extracted, and obtain the mixed speech feature map; an embedded feature extraction module, which is used to obtain the embedded features of the target speaker according to the position information of the target speaker and the reference speech, combined with the direction features and the voiceprint features; a network extraction module, which is used to obtain the high-dimensional feature representation of the speech to be extracted according to the mixed speech feature map and the embedded features of the target speaker, combined with the main extraction network based on the dual-path Fastformer neural network; a mask estimation module, which is used to convert the obtained high-dimensional feature representation into a complex mask for estimating the separated speech, and apply the mask to the complex spectrum of the mixed speech signal; a reconstruction estimation module, which is used to reconstruct the time domain waveform of the estimated extracted speech through inverse short-time Fourier transform; and a contrast learning module, which is used to input the extracted speech of the target speaker into the contrast learning module, so that the mixed speech without the target speaker can also output the correct extracted speech.

[0106] The target speech extraction device based on multi-reference clue fusion provided by the embodiment of the present invention aims at the problem that the existing speech extraction method is prone to target confusion and error extraction. Figure 5 Several modules in the proposed method introduce the target speaker's position information and voiceprint information as reference features to achieve accurate extraction of the target speaker's voice. By proposing a contrastive learning loss function for the target speech extraction task, the suppression effect of the target speaker's absence segment is effectively improved and the target confusion probability is reduced.

[0107] It should be noted that the device embodiments provided by the present invention are used to implement the methods in the above method embodiments as well as the methods in other method embodiments provided by the present invention. The only difference is that the corresponding functional modules are set, and the principle is basically the same as the principle of the above device embodiments provided by the present invention. As long as the technical personnel in the field refer to the specific technical solutions in other method embodiments on the basis of the above device embodiments, obtain the corresponding technical means and the technical solutions composed of these technical means by combining the technical features, and on the premise of ensuring the practicality of the technical solutions, improve the modules in the above device embodiments to obtain the corresponding device class embodiments, which are used to implement the methods in other method class embodiments. For example:

[0108] Based on the content of the above device embodiment, as a preferred embodiment, the target speech extraction device based on multi-reference clue fusion provided in the embodiment of the present invention, the feature extraction module is also used to execute the following instructions:

[0109] The multi-channel mixed speech to be extracted is passed through a feature extraction module to calculate the complex spectrum and spatial features of the mixed speech, and obtain a mixed speech feature map, including: using short-time Fourier transform to transform the time-domain multi-channel mixed speech to be separated into the time-frequency domain to obtain the complex spectrum of the multi-channel mixed speech; introducing the phase difference between channels as an input feature to calculate the spatial features of the multi-channel mixed speech; and forming a mixed speech feature map according to the complex spectrum and spatial features.

[0110] Based on the content of the above device embodiment, as a preferred embodiment, the target speech extraction device based on multi-reference clue fusion provided in the embodiment of the present invention, the embedded feature extraction module is also used to execute the following instructions:

[0111] The orientation information and reference speech of the target speaker are embedded in a feature extraction module, combined with directional features and voiceprint features to obtain the embedded features of the target speaker, including: calculating a steering vector, and concatenating the real part and the imaginary part of the steering vector along the feature dimension as the input directional features; using a pre-trained classic speaker recognition model ECAPA-TDNN as a sound cue encoder to provide the voiceprint features of the target speaker; and forming the embedded features of the target speaker according to the directional features and voiceprint features.

[0112] Based on the contents of the above device embodiments, as a preferred embodiment, the target speech extraction device based on multi-reference clue fusion provided in the embodiments of the present invention, the main extraction network based on the dual-path Fastformer neural network also includes:

[0113] A spectral modeling unit, a sub-band modeling unit and a full-band modeling unit; wherein the spectral modeling unit is used to extract and upgrade the input mixed speech feature map and the embedded features of the target speaker, and use a shuffled convolutional attention module to extract spectral local information and spectral spatial information; the sub-band modeling unit is used to fuse the embedded features in each frame of the mixed speech feature map through a convolution-enhanced input fully connected layer, combine the embedded features with a global query vector with global temporal context information in a convolution-enhanced multi-head attention module, and use a cross-linear attention module to fuse the local features obtained by the spectral modeling unit with the sub-band features output by the convolution-enhanced module; the full-band modeling unit has the same architecture as the sub-band modeling unit, but does not include embedded feature fusion.

[0114] Based on the content of the above device embodiment, as a preferred embodiment, in the target speech extraction device based on multi-reference clue fusion provided in the embodiment of the present invention, the mask estimation module is further used to execute the following instructions:

[0115] The output high-dimensional feature representation is converted into a complex mask for estimating the separated speech through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal, including: the output of the main extraction network based on the dual-path Fastformer neural network is divided into two paths, and amplitude estimation and phase estimation are performed respectively to obtain the amplitude spectrum and phase spectrum of the mixed speech respectively; based on the amplitude spectrum and phase spectrum, the complex spectrum of the mixed speech is obtained.

[0116] Based on the content of the above device embodiment, as a preferred embodiment, the target speech extraction device based on multi-reference clue fusion provided in the embodiment of the present invention, the contrast learning module is further used to execute the following instructions:

[0117] The extracted speech of the target speaker is input into a contrastive learning module so that the mixed speech without the target speaker can also output the correct extracted speech, including: obtaining the output extracted by the target speech extraction network for the target speech, the mixed speech with the target speaker and the mixed speech without the target speaker, and constructing training samples; according to the training samples, training is performed using a determined contrastive learning loss function so that the mixed speech without the target speaker can also output the correct extracted speech.

[0118] The method of the embodiment of the present invention is implemented by relying on electronic devices, so it is necessary to introduce the relevant electronic devices. Based on this purpose, the embodiment of the present invention provides an electronic device, such as Figure 6As shown, the electronic device includes: at least one processor, a communication interface, at least one memory, and a communication bus, wherein at least one processor, the communication interface, and at least one memory communicate with each other through the communication bus. At least one processor calls the logic instructions in at least one memory to execute all or part of the steps of the method provided by the aforementioned various method embodiments.

[0119] In addition, when the logic instructions in the above-mentioned at least one memory are implemented in the form of a software functional unit and sold or used as an independent product, they are stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or partly in other words, the part that contributes to the prior art or part of the technical solution is embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for enabling a computer device (a personal computer, a server, or a network device) to perform all or part of the steps of the method described in each method embodiment of the present invention. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and various media for storing program codes.

[0120] The system embodiments described above are merely illustrative, wherein the units described as separate components are or are not physically separated, and the components shown as units are or are not physical units, located in one place, or distributed on multiple network units. Some or all of the modules are selected according to actual conditions to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement without creative effort.

[0121] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0123] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] In summary, the present invention discloses a target speech extraction method based on multi-reference clue fusion, which mainly solves the problem that the existing speech extraction method is prone to target confusion and erroneous extraction. The main steps include: 1) passing the multi-channel mixed speech to be extracted through a feature extraction module, calculating the complex spectrum and spatial features of the mixed speech, and obtaining a mixed speech feature map; 2) passing the orientation information of the target speaker and the reference speech through an embedding feature extraction module, combining the directional features and the voiceprint features, to obtain the embedded features of the target speaker; 3) inputting the mixed speech feature map and the embedded features of the target speaker into a main extraction network based on a dual-path Fastformer neural network to obtain a high-dimensional feature representation of the speech to be extracted; 4) passing the output high-dimensional feature representation through a mask estimation module to convert it into a complex mask for estimating the separated speech, and applying the mask to the complex spectrum of the mixed speech signal; 5) reconstructing the time domain waveform of the estimated extracted speech through an inverse short-time Fourier transform; 6) inputting the extracted target speaker speech into a contrast learning module, so that the mixed speech without the target speaker can also output the correct extracted speech. The speech extraction quality of the present invention is better than that of the extraction model based on a single reference clue, and the target confusion probability is significantly reduced. It can flexibly adapt to different types of mixed speech states, and is expected to provide strong technical support and practical reference for solving the problem of target speech extraction.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A target speech extraction method based on multi-reference clue fusion, characterized in that: include: The multi-channel mixed speech to be extracted is passed through a feature extraction module to calculate the complex spectrum and spatial features of the mixed speech, and obtain a mixed speech feature map; The position information of the target speaker and the reference speech are embedded in the feature extraction module, combined with the direction feature and voiceprint feature to obtain the embedded feature of the target speaker; The mixed speech feature map and the target speaker's embedded features are input into the main extraction network based on the dual-path Fastformer neural network, which outputs a high-dimensional feature representation of the speech to be extracted. The output high-dimensional feature representation is converted into a complex mask for estimating the separated speech through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal; The time domain waveform of the speech is extracted by inverse short-time Fourier transform reconstruction estimation; The extracted speech of the target speaker is input into the contrastive learning module, so that the correct extracted speech can be output even in the mixed speech where the target speaker is absent.

2. According to claim 1, a target speech extraction method based on multi-reference clue fusion is characterized in that: The multi-channel mixed speech to be extracted is passed through a feature extraction module to calculate the complex spectrum and spatial features of the mixed speech, and obtain a mixed speech feature map, including: using short-time Fourier transform to transform the time-domain multi-channel mixed speech to be separated into the time-frequency domain to obtain the complex spectrum of the multi-channel mixed speech; introducing the phase difference between channels as an input feature to calculate the spatial features of the multi-channel mixed speech; and forming a mixed speech feature map according to the complex spectrum and spatial features.

3. The target speech extraction method based on multi-reference clue fusion according to claim 1 is characterized in that: The orientation information and reference speech of the target speaker are embedded in a feature extraction module, combined with directional features and voiceprint features to obtain the embedded features of the target speaker, including: calculating a steering vector, and concatenating the real part and the imaginary part of the steering vector along the feature dimension as the input directional features; using a pre-trained classic speaker recognition model ECAPA-TDNN as a sound cue encoder to provide the voiceprint features of the target speaker; and forming the embedded features of the target speaker according to the directional features and voiceprint features.

4. The target speech extraction method based on multi-reference clue fusion according to claim 1 is characterized in that: The main extraction network based on the dual-path Fastformer neural network includes a spectrum modeling unit, a sub-band modeling unit and a full-band modeling unit; wherein the spectrum modeling unit is used to extract and upgrade the input mixed speech feature map and the embedded features of the target speaker, and use a shuffled convolution attention module to extract spectrum local information and spectrum spatial information; the sub-band modeling unit is used to fuse the embedded features in each frame of the mixed speech feature map through a convolution-enhanced input fully connected layer, combine the embedded features with a global query vector with global temporal context information in a convolution-enhanced multi-head attention module, and use a cross-linear attention module to fuse the local features obtained by the spectrum modeling unit with the sub-band features output by the convolution enhancement module; the full-band modeling unit has the same architecture as the sub-band modeling unit, but does not include embedded feature fusion.

5. The target speech extraction method based on multi-reference clue fusion according to claim 1, characterized in that: The output high-dimensional feature representation is converted into a complex mask for estimating the separated speech through a mask estimation module, and the mask is applied to the complex spectrum of the mixed speech signal, including: the output of the main extraction network based on the dual-path Fastformer neural network is divided into two paths, and amplitude estimation and phase estimation are performed respectively to obtain the amplitude spectrum and phase spectrum of the mixed speech respectively; based on the amplitude spectrum and phase spectrum, the complex spectrum of the mixed speech is obtained.

6. The target speech extraction method based on multi-reference clue fusion according to claim 1 is characterized in that: The extracted speech of the target speaker is input into a contrastive learning module so that the mixed speech without the target speaker can also output the correct extracted speech, including: obtaining the output extracted by the target speech extraction network for the target speech, the mixed speech with the target speaker and the mixed speech without the target speaker, and constructing training samples; according to the training samples, training is performed using a determined contrastive learning loss function so that the mixed speech without the target speaker can also output the correct extracted speech.

7. A target speech extraction device based on multi-reference clue fusion, characterized in that: include: A feature extraction module, used to calculate the complex spectrum and spatial features of the mixed speech according to the multi-channel mixed speech to be extracted, and obtain a mixed speech feature map; An embedding feature extraction module is used to obtain the embedding features of the target speaker based on the position information and reference speech of the target speaker, combined with the directional features and the voiceprint features; The network extraction module is used to obtain a high-dimensional feature representation of the speech to be extracted based on the mixed speech feature graph and the embedded features of the target speaker, combined with the main extraction network based on the dual-path Fastformer neural network; A mask estimation module, used to convert the obtained high-dimensional feature representation into a complex mask for estimating the separated speech, and apply the mask to the complex spectrum of the mixed speech signal; A reconstruction estimation module, used for reconstructing and estimating the extracted time domain waveform of speech through inverse short-time Fourier transform; The contrastive learning module is used to input the extracted speech of the target speaker into the contrastive learning module so that the mixed speech without the target speaker can also output the correct extracted speech.

8. An electronic device, characterized in that: include: At least one processor, at least one memory and a communication interface; wherein the processor, memory and communication interface communicate with each other; The memory stores program instructions executed by the processor, and the processor calls the program instructions to execute the method according to any one of claims 1 to 6.

9. A non-transitory computer readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores computer instructions, which cause the computer to execute the method of any one of claims 1 to 6.

Images