Speech extraction method based on feature compression and adaptive fusion of voiceprint information

Abstract

This invention discloses a speech extraction method, apparatus, storage medium, and electronic device based on feature compression and adaptive fusion of voiceprint information. The method includes: extracting target speech using a deep neural network, i.e., acquiring noisy speech and registered speech, wherein the target speech in the registered speech and the noisy speech are different voices; performing a short-time Fourier transform; performing a proposed feature compression based on human auditory features in the time-frequency domain to obtain first and second compressed features; encoding and mapping the first and second compressed features using a noise encoder and a registration encoder, respectively; fusing the encoded and mapped features using a proposed adaptive fusion method of voiceprint information; and obtaining the target speech using a deep neural network-based encoder. This method solves the problems caused by the fixed frequency resolution used in traditional target speech extraction and the dynamic changes in the target object's voice features, thus improving the quality of target speech extracted in existing technologies.

Description

Technical Field

[0001] This invention relates to the field of speech extraction based on feature compression and adaptive fusion of voiceprint information in artificial intelligence. Specifically, it relates to a speech extraction method, apparatus, storage medium, and electronic device based on feature compression and adaptive fusion of voiceprint information. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence and deep neural networks, various industries have been impacted to varying degrees. Intelligent voice technology, in turn, is increasingly being applied to people's daily lives. Target speech extraction, as one of the important tasks of intelligent voice technology, is also shifting towards data-driven methods represented by deep learning, with end-to-end network structures becoming the main technical approach.

[0003] The target speech extraction task originates from the cocktail party effect, which states that in everyday life, even under acoustic conditions with noise, reverberation, or interfering speakers, our brains exhibit a remarkable ability to focus attention and effectively filter out target speech while ignoring other noise. For decades, researchers have dedicated themselves to developing technologies that approximate human hearing capabilities. A key challenge in this task is handling the sounds of interfering speakers, as their similar characteristics to the target speaker complicate their identification / separation. Based on this, target speech extraction techniques have been proposed, utilizing prior registration information that identifies the target speaker to separate the target speaker's speech signal from a mixture of speech from one or more speakers, with or without noise or reverberation. This prior information is typically pre-recorded registration speech of the target speaker, from which their acoustic features can be derived. Target speech extraction is an emerging research field that has received increasing attention in recent years because it provides a suitable approach to the cocktail party problem.

[0004] In current target speech extraction tasks, mainstream methods model mixed speech signals in the short-time Fourier transform domain, utilizing prior information from registered speech to guide target speech extraction. However, the fixed spectral resolution imposes limitations on these methods. On one hand, excessively low resolution makes it difficult to distinguish noise between speech harmonics, leading to performance degradation; on the other hand, excessively high resolution significantly increases computational complexity and the number of model parameters, making it unsuitable for practical applications. Furthermore, because the voice features of the target speaker in noisy speech dynamically change during utterances, traditional methods using a single static prior information struggle to accurately guide the extraction of the target speech portion. Consequently, related technologies exhibit relatively low speech extraction accuracy.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides a speech extraction method, apparatus, storage medium, and electronic device based on feature compression and adaptive fusion of voiceprint information, to at least solve the technical problem of low accuracy in speech extraction based on feature compression and adaptive fusion of voiceprint information in the prior art.

[0007] According to one aspect of the present invention, a speech extraction method based on feature compression and adaptive fusion of speaker information is provided, comprising: acquiring noisy speech of a target object and registered speech, wherein the registered speech and the speech of the target object in the noisy speech are different voices; extracting corresponding first low-frequency features and first mid-to-high-frequency compressed sensing features from the noisy speech, and extracting corresponding second low-frequency features and second mid-to-high-frequency compressed sensing features from the registered speech; encoding the first low-frequency features and the first mid-to-high-frequency compressed sensing features using a noise encoder based on a deep neural network to obtain encoded features; mapping the second low-frequency features and the second mid-to-high-frequency compressed sensing features using a registration encoder based on the deep neural network to obtain mapped features; fusing the encoded features and the mapped features to obtain fused features; and decoding the fused features using a decoder based on the deep neural network to obtain target speech.

[0008] Optionally, fusing the encoded features and the mapped features to obtain fused features includes: linearly projecting the high-dimensional representation of the target object's identity information onto the same feature space as the intermediate features of the noisy speech; calculating the similarity between each frame of the high-dimensional features obtained from the noisy speech through the noise encoder and the projected target object representation; weighting the target object representation with the similarity to obtain the dynamic identity representation of the target object; and fusing the obtained dynamic identity representation of the target object into the high-dimensional representation of the noisy speech to obtain the fused features.

[0009] Optionally, the step of decoding the fused features based on the deep neural network decoder to obtain the target speech includes: reconstructing the low-frequency and high-frequency components of the target speech signal; fusing the obtained low-frequency speech signal and mid-to-high-frequency signal in the frequency domain; and then using inverse Fourier transform to convert the frequency domain signal back to the time domain to obtain the target speech.

[0010] Optionally, the step of extracting the corresponding first low-frequency feature and first mid-to-high-frequency compressed sensing feature from the noisy speech, and extracting the corresponding second low-frequency feature and second mid-to-high-frequency compressed sensing feature from the registered speech, includes: a feature compression method based on human auditory characteristics, dividing both the noisy speech and the registered speech into low-frequency and mid-to-high-frequency parts; performing feature compression on the mid-to-high-frequency parts of the noisy speech and the registered speech respectively to obtain the corresponding first mid-to-high-frequency compressed sensing feature and second mid-to-high-frequency compressed sensing feature respectively.

[0011] Optionally, the step of performing feature compression on the mid-to-high frequency components of the noisy speech and the registered speech to obtain the corresponding first mid-to-high frequency compressed sensing features and second mid-to-high frequency compressed sensing features includes: the feature compression methods for the mid-to-high frequency components of the noisy speech and the registered speech are as follows: taking the square of the modulus of the speech to obtain the energy spectrum; performing nonlinear filtering on the energy spectrum based on the characteristics of human auditory perception; and taking the average value of frequency points within the same bandwidth channel to reduce the feature dimension.

[0012] According to another aspect of the present invention, a speech extraction apparatus based on feature compression and adaptive fusion of voiceprint information is also provided, comprising: an acquisition unit, configured to acquire noisy speech of a target object and registered speech, wherein the registered speech and the speech of the target object in the noisy speech are different voices; an extraction unit, configured to extract corresponding first low-frequency features and first mid-to-high-frequency compressed sensing features from the noisy speech, and extract corresponding second low-frequency features and second mid-to-high-frequency compressed sensing features from the registered speech; a processing unit, configured to encode the first low-frequency features and the first mid-to-high-frequency compressed sensing features based on a noise encoder of a deep neural network to obtain encoded features; and to map the second low-frequency features and the second mid-to-high-frequency compressed sensing features based on a registration encoder of the deep neural network to obtain mapped features; a fusion unit, configured to fuse the encoded features and the mapped features to obtain fused features; and a decoding unit, configured to decode the fused features based on a decoder of the deep neural network to obtain target speech.

[0013] Optionally, the fusion unit includes: a mapping module, used to linearly project the high-dimensional representation of the target object's identity information onto a feature space identical to the intermediate features of the noisy speech; a calculation module, used to calculate the similarity between each frame of the high-dimensional features obtained from the noisy speech through the noise encoder and the projected target object representation; a processing module, used to weight the target object representation based on similarity to obtain the dynamic identity representation of the target object; and a fusion module, used to fuse the obtained dynamic identity representation of the target object into the high-dimensional representation of the noisy speech to obtain the fused feature.

[0014] According to a first aspect of the embodiments of this application, a computer-readable storage medium is provided, characterized in that the storage medium stores a computer program, wherein the computer program is configured to execute the above-described speech extraction method based on feature compression and adaptive fusion of voiceprint information at runtime.

[0015] According to a first aspect of the embodiments of this application, an electronic device is provided, including a memory and a processor, characterized in that the memory stores a computer program, and the processor is configured to run the computer program to perform the above-described speech extraction method based on feature compression and adaptive fusion of voiceprint information.

[0016] In this embodiment of the invention, noisy speech and registered speech of the target object are obtained, wherein the registered speech and the target object's speech in the noisy speech are different. A first low-frequency feature and a first mid-to-high-frequency compressed sensing feature are extracted from the noisy speech, and a second low-frequency feature and a second mid-to-high-frequency compressed sensing feature are extracted from the registered speech. A noise encoder based on a deep neural network encodes the first low-frequency feature and the first mid-to-high-frequency compressed sensing feature to obtain encoded features. A registration encoder based on a deep neural network maps the second low-frequency feature and the second mid-to-high-frequency compressed sensing feature to obtain mapped features. The encoded features and mapped features are fused to obtain fused features. A decoder based on a deep neural network decodes the fused features to obtain the target speech. Target speech is extracted based on the adaptive fusion of feature compression and speaker information. This solves the technical problem of low accuracy in speech extraction based on adaptive fusion of feature compression and speaker information in the prior art. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a mobile terminal based on an optional speech extraction method of feature compression and adaptive fusion of voiceprint information according to an embodiment of the present invention.

[0019] Figure 2 This is a flowchart of an optional speech extraction method based on feature compression and adaptive fusion of speaker information according to an embodiment of the present invention;

[0020] Figure 3 This is an optional flowchart of target speech extraction based on feature compression and adaptive fusion of speaker information according to an embodiment of the present invention;

[0021] Figure 4This is an optional flowchart for extracting compressed features based on human auditory characteristics according to an embodiment of the present invention;

[0022] Figure 5 This is an optional registration voice encoder processing flowchart according to an embodiment of the present invention;

[0023] Figure 6 This is an optional target speaker representation vector adaptive fusion flowchart according to an embodiment of the present invention;

[0024] Figure 7 This is a diagram of an optional speech extraction device based on feature compression and adaptive fusion of voiceprint information according to an embodiment of the present invention. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a sequence of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] The speech extraction method based on feature compression and adaptive fusion of speaker information provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal using a speech extraction method based on feature compression and adaptive fusion of voiceprint information, according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal 10 may include one or more ( Figure 1Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. Optionally, the mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal 10 may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0028] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the speech extraction method based on feature compression and adaptive fusion of voiceprint information in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0029] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the mobile terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0030] This embodiment also provides a speech extraction method based on feature compression and adaptive fusion of speaker information. Figure 2 This is a flowchart of a speech extraction method based on feature compression and adaptive fusion of speaker information according to an embodiment of the present invention, as shown below. Figure 2 As shown, the speech extraction method based on feature compression and adaptive fusion of speaker information includes the following steps:

[0031] Step S202: Obtain the noisy speech and the registered speech of the target object, wherein the registered speech and the speech of the target object in the noisy speech are different voices.

[0032] Step S204: Extract the corresponding first low-frequency feature and first mid-to-high frequency compressed sensing feature from the noisy speech, and extract the corresponding second low-frequency feature and second mid-to-high frequency compressed sensing feature from the registered speech.

[0033] Step S206: The noise encoder based on the deep neural network encodes the first low-frequency feature and the first medium-high frequency compressed sensing feature to obtain the encoded feature; the registration encoder based on the deep neural network maps the second low-frequency feature and the second medium-high frequency compressed sensing feature to obtain the mapped feature.

[0034] Step S208: The encoded features and the mapping features are fused to obtain fused features.

[0035] Step S210: The decoder based on the deep neural network decodes the fused features to obtain the target speech.

[0036] In this embodiment, the speech extraction method based on feature compression and adaptive fusion of speaker information can include, but is not limited to, extracting the speech signal of the target object from noise. This involves the synthesis of multiple scene data for the target speech extraction task.

[0037] The target speech extraction task aims to extract only the target speaker's speech from background noise or interfering human voices. To achieve this, three types of training data need to be synthesized: target speech + interfering human voices; target speech + background noise; and target speech + interfering human voices + background noise. Furthermore, for each scenario, a registered voice recording of the target speaker is prepared as reference information; this registered voice recording is different from the target speaker's voice in the noisy speech.

[0038] In this embodiment, both the noisy speech audio and the target speaker's registered speech audio are extracted using compressed features based on the characteristics of human hearing, resulting in two acoustic features of the original speech: low-frequency features and mid-to-high-frequency compressed sensing features.

[0039] In this embodiment, a high-order abstract representation of input features is extracted using a deep neural network-based encoder. Two encoders are used to model noisy speech and registered speech, respectively. The noisy speech encoder encodes the compressed sensing features of the noisy speech. Its structure is based on a causal convolutional neural network (CNN), which downsamples the input features and, through training, can serve as a high-dimensional feature extractor for noisy speech with strong generalization capabilities. Simultaneously, it satisfies the causal precondition that the CNN only uses the current frame and past frames to calculate the prediction for a given frame, ensuring that the system can be applied in real time. The registered speech encoder encodes the registered speech of the target speaker. The encoder structure mainly consists of a Long Short-Term Memory (LSTM) network, a fully connected network (FC), and a convolutional neural network (CNN). It maps the handcrafted features of the original registered speech to a high-dimensional representation, namely the speaker's voiceprint information, making it easier for downstream extraction neural networks to learn and fit.

[0040] The embodiments provided in this application obtain noisy speech and registered speech of the target object, wherein the registered speech and the target object's speech in the noisy speech are different. First low-frequency features and first mid-to-high-frequency compressed sensing features are extracted from the noisy speech, and second low-frequency features and second mid-to-high-frequency compressed sensing features are extracted from the registered speech. A noise encoder based on a deep neural network encodes the first mid-to-high-frequency compressed sensing features to obtain encoded features. A registration encoder based on a deep neural network maps the second mid-to-high-frequency compressed sensing features to obtain mapped features. The encoded features and mapped features are fused to obtain fused features. A decoder based on a deep neural network decodes the fused features to obtain the target speech. Target speech is extracted based on the adaptive fusion of feature compression and speaker information. This solves the technical problem of low accuracy in speech extraction based on the adaptive fusion of feature compression and speaker information in the prior art.

[0041] It should be noted that the target speech extraction is achieved through a deep neural network, namely, acquiring noisy speech and registered speech, where the target speech in the registered speech and the noisy speech are different voices. A short-time Fourier transform is performed; a proposed feature compression based on human auditory characteristics is applied in the time-frequency domain to obtain first and second compressed features; the first and second compressed features are encoded and mapped using a noise encoder and a registration encoder, respectively; the encoded and mapped features are fused using a proposed adaptive fusion method based on speaker information; and the target speech is obtained using a deep neural network-based encoder. This approach solves the problems caused by the fixed frequency resolution used in traditional target speech extraction and the dynamic changes in the target object's sound features, thus improving the quality of target speech extracted by existing technologies.

[0042] Optionally, fusing the encoded features and the mapped features to obtain fused features includes: linearly projecting the high-dimensional representation of the target object's identity information onto the same feature space as the intermediate features of the noisy speech; calculating the similarity between each frame of the high-dimensional features obtained from the noisy speech through the noise encoder and the projected target object representation; weighting the target object representation with the similarity to obtain the dynamic identity representation of the target object; and fusing the obtained dynamic identity representation of the target object into the high-dimensional representation of the noisy speech to obtain the fused features.

[0043] Optionally, the step of decoding the fused features based on the decoder of the deep neural network to obtain the target speech may include: reconstructing the low-frequency and high-frequency components of the target speech signal; fusing the obtained low-frequency speech signal and mid-to-high-frequency signal in the frequency domain; and then using inverse Fourier transform to convert the frequency domain signal back to the time domain to obtain the target speech.

[0044] Optionally, the step of extracting the corresponding first low-frequency feature and first mid-to-high-frequency compressed sensing feature from the noisy speech, and extracting the corresponding second low-frequency feature and second mid-to-high-frequency compressed sensing feature from the registered speech, may include: a feature compression method based on human auditory characteristics, dividing both the noisy speech and the registered speech into low-frequency and mid-to-high-frequency parts; performing feature compression on the mid-to-high-frequency parts of the noisy speech and the registered speech respectively to obtain the corresponding first mid-to-high-frequency compressed sensing feature and second mid-to-high-frequency compressed sensing feature respectively.

[0045] Optionally, the step of compressing the mid-to-high frequency components of the noisy speech and the registered speech to obtain the corresponding first mid-to-high frequency compressed sensing features and second mid-to-high frequency compressed sensing features may include: compressing the mid-to-high frequency components of the noisy speech and the registered speech in the following ways: taking the square of the modulus of the speech to obtain the energy spectrum; performing nonlinear filtering on the energy spectrum based on the characteristics of human auditory perception; and averaging the frequency points within the same bandwidth channel to reduce the feature dimension.

[0046] As an optional embodiment, this application also proposes a target speech extraction method based on feature compression and adaptive fusion of speaker information. Figure 3 The flowchart shown is a method for extracting target speech based on feature compression and adaptive fusion of voiceprint information.

[0047] In this embodiment, the target speech extraction method based on feature compression and adaptive fusion of voiceprint information mainly includes two key innovations. First, a feature compression method based on human auditory characteristics, which employs a variable-resolution time-frequency conversion approach to adapt to the resolution requirements of different frequency bands. Second, an adaptive fusion method of target speaker voiceprint information, which uses a dynamic voiceprint feature injection mechanism to track changes in the target speaker's speech features in noisy speech in real time.

[0048] First, the feature compression method based on the characteristics of human hearing includes the following:

[0049] In order to reduce the model input size without significantly affecting the performance of the target speech extraction task, and considering the non-linear filtering characteristics of human ear frequency perception of speech, targeted feature compression is performed to reduce data redundancy while ensuring subjective perception quality.

[0050] First, traditional signal processing methods are used to pre-emphasize, frame, window, and perform Fast Fourier Transform on the original audio. Second, filtering is performed to separate the signal into low-frequency and mid-to-high-frequency components. Since the human ear is more sensitive to the low-frequency region of speech signals, this region is preserved. Then, feature compression is performed on the mid-to-high-frequency components. This involves three steps: first, taking the modulus and squaring to obtain the energy spectrum; then, applying nonlinear filtering to the energy spectrum based on the characteristics of human auditory perception; and finally, averaging the frequency points within the same bandwidth channel to reduce feature dimensionality.

[0051] Secondly, the adaptive fusion method for target speaker voiceprint information specifically includes the following:

[0052] Human speakers' vocal characteristics change dynamically during speech, which traditional fixed fusion methods cannot model. To address this issue, an adaptive fusion module for target speaker voiceprint information dynamically models the association between high-dimensional features representing the target speaker's identity and the input noisy speech features. During extraction, the voiceprint information in the target speaker's registered speech is adjusted frame-by-frame to correspond to changes in the target speaker's voice within the noisy speech. This frame-by-frame adjustment allows the fusion to dynamically adapt to changes in the target speaker's vocal features within the corresponding input noisy speech. In the downstream speech extraction network, this adaptive fusion representation is used as conditional information to further improve the system's extraction quality of the target speaker's speech.

[0053] It should be noted that, in this embodiment of the application, a target speech extraction system based on feature compression and adaptive fusion of speaker information mainly includes the following five steps:

[0054] Step 1: Data synthesis for various scenarios in the target speech extraction task.

[0055] The target speech extraction task aims to extract only the target speaker's speech from background noise or interfering human voices. To achieve this, three types of training data need to be synthesized: target speech + interfering human voices; target speech + background noise; and target speech + interfering human voices + background noise. Furthermore, for each scenario, a registered voice recording of the target speaker is prepared as reference information; this registered voice recording is different from the target speaker's voice in the noisy speech.

[0056] Training data is constructed based on clean speech sets, noise sets, reverberation sets, and registered speech sets.

[0057] In real-world scenarios, the target speaker's voice may be affected by background noise, other interfering speakers, and acoustic reverberation. To address this, and to improve the robustness of the compression feature-based and adaptive learning system, three types of datasets are needed for training: target speech + background noise; target speech + interfering speaker speech; and target speech + background noise + interfering speaker speech. Furthermore, some reverberation can be added during data synthesis to simulate a realistic environment. Specifically, an existing clean speech set is used as the source data for synthesizing these three types of noisy speech, and also as the training label. Noise is then added to synthesize noisy speech signals of different types, signal-to-noise ratios, loudnesses, and acoustic environments.

[0058] Step 2: Extract compressed features based on the characteristics of human hearing.

[0059] For both noisy speech audio and the target speaker's registered speech audio, the aforementioned feature compression method based on human hearing characteristics was used for feature extraction, resulting in two acoustic features of the original speech: low-frequency features and mid-to-high-frequency compressed perceptual features.

[0060] Among them, such as Figure 4The flowchart illustrates the extraction of compressed features based on human auditory characteristics. First, the original audio of noisy speech and registered speech in the training data is framed and windowed, then subjected to Fast Fourier Transform (FFT) to obtain time-frequency features—spectral data. Next, the spectrogram features are compressed. On one hand, nonlinear intra-channel filtering based on the nonlinearity of human hearing's resolution at different frequencies is applied to the mid-to-high frequency components of the speech signal. Specifically, the energy spectrum is obtained by taking the modulus and squaring the mid-to-high frequency range of the spectrogram, followed by average pooling within the same channel. The channel length is based on the human ear's acoustic bandwidth measurement method; at lower frequencies, the bandwidth is narrower and the resolution is better. As the frequency increases, the bandwidth widens according to a certain relationship, meaning that the bandwidth at different center frequencies exhibits a nonlinear pattern. On the other hand, complete low-frequency information of the original speech signal is simultaneously provided as model input. This approach ensures that the frequency resolution of key speech bands remains unaffected, allowing the model to fully learn the speech features in the low-frequency region, while also providing an abstract generalization of the entire speech signal frequency band. The combination of these two methods allows the model to enjoy the advantages of both detailed low-frequency modeling and abstract high-frequency modeling.

[0061] Step 3: Use a deep neural network-based encoder to extract a high-order abstract representation of the input features.

[0062] Two encoders are used to model the noisy speech and the registered speech, respectively. The noisy speech encoder encodes the compressed sensing features of the noisy speech. Its structure is based on a causal convolutional neural network (CNN), which downsamples the input features and, through training, can serve as a high-dimensional feature extractor for noisy speech with strong generalization ability. While satisfying the causal precondition, the CNN only uses the current frame and past frames to calculate the prediction for a given frame, ensuring the system can be used in real time. The registered speech encoder encodes the registered speech of the target speaker. The encoder structure mainly consists of a Long Short-Term Memory (LSTM) network, a fully connected network (FC), and a CNN. It maps the handcrafted features of the original registered speech to a high-dimensional representation, namely the speaker's voiceprint information, making it easier for downstream extraction neural networks to learn and fit.

[0063] Specifically, in this embodiment, two encoders are used to model the registered speech and the noisy speech, respectively. The registered speech encoder contains two branches, which process the two types of compressed perceptual speech features respectively. Each branch first uses a Long Short-Term Memory (LSTM) unit to learn temporal features, and then connects to a three-layer convolutional module, such as... Figure 5 The diagram shows the process flow for registering a voice encoder.

[0064] Each convolutional module consists of a 3×3 convolution (3×3 kernel), batch normalization, parameterized corrected linear units, and a 1×1 pointwise convolution (1×1 kernel). The output features from the two branches are then concatenated, passed through a fully connected network, and averaged over time to obtain the speech vector representation of the target speaker. The noisy speech encoder encodes the corresponding compressed sensing features of the input, mapping them to a high-dimensional representation.

[0065] Step 4: Adaptive fusion of the target speaker's voiceprint information.

[0066] Step 4 mainly involves four steps: First, the high-dimensional representation of the target speaker's identity information is linearly projected onto the same feature space as the intermediate features of the noisy speech. Second, the similarity between each frame of the high-dimensional features obtained from the encoder of the noisy speech and the projected target speaker representation is calculated. Third, the target speaker's dynamic identity representation is obtained by weighting the similarity representation obtained in step 2. Fourth, the obtained dynamic identity representation of the target speaker is fused into the high-dimensional representation of the noisy speech and then input into the downstream target speech extraction network to predict the target signal.

[0067] In this embodiment, as Figure 6 The flowchart shown is for the adaptive fusion of target speaker representation vectors.

[0068] The first step is to process the obtained target speaker feature representation into E. i ∈R 1×D1 The feature is mapped to intermediate features through a fully connected module consisting of a single fully connected layer, one-dimensional batch normalization (1D BatchNormalization), and parameterized corrected linear units (PReLU). The same feature space.

[0069]

[0070] The second step is to calculate the similarity between each frame of the target speaker representation and the noisy high-dimensional speech representation, and obtain a weight vector in the time dimension.

[0071]

[0072] The Softmax function scales the similarity score of each frame to the range [0,1]. The representative will E o A row vector is transposed to obtain a column vector.

[0073] The third step is to use the obtained similarity vector W t Reweighted target speaker feature representation E oThis allows for the acquisition of a dynamic, adaptive target speaker representation E for each frame of noisy speech. d .

[0074]

[0075] Finally, the adaptive target speaker representation E is generated by element-wise multiplication. d It is fused with the high-dimensional representation H output by the encoder of the noisy speech.

[0076] H o =H⊙E d

[0077] Step 5: Recover the target speech using a deep neural network-based decoder.

[0078] Two decoders are used as downstream extraction networks to upsample the previously obtained high-dimensional features in order to reconstruct the target speech signal. The process consists of three steps: first, reconstructing the low-frequency portion of the target speech signal; second, reconstructing the mid-to-high-frequency portion of the target speech signal; and third, fusing the obtained low-frequency and mid-to-high-frequency signals in the frequency domain, and then using an inverse Fourier transform to return the frequency domain signal to the time domain, obtaining the final target speech signal.

[0079] Specifically, in this embodiment, two decoders are used to extract the target speech. The first is a low-frequency decoder, which takes the fusion representation obtained in the previous steps as input and predicts the complex-valued mask of the low-frequency part of the target speech signal. This mask is multiplied by the low-frequency part of the noisy speech to extract the low-frequency part of the target speech. Since the operation is only performed on the low-frequency component of the noisy speech at this point, some noise components may still exist in the mid-high frequency region. Therefore, a mid-high frequency decoder is used for further refinement. Its input is still the fusion feature obtained in the previous steps, predicting the mask value of the high-frequency compression feature in the target speech. Then, the mask value in the compression feature domain is first converted to the mid-high frequency domain to obtain the complex-valued mask value of the high-frequency component in the target signal. Then, the mask value is multiplied by the mid-high frequency component of the noisy speech to complete the extraction of the high-frequency component in the target speech. Finally, the two parts are fused in the frequency domain and then an inverse Fourier transform is performed to generate the complete target speech.

[0080] Through the embodiments of this application, firstly, a feature compression method based on human auditory characteristics is proposed. This method employs a variable-resolution time-frequency conversion approach to adapt to the resolution requirements of different frequency bands. Secondly, an adaptive fusion method for target speaker voiceprint information is proposed. This method employs a dynamic voiceprint feature injection mechanism, which can track changes in the target speaker's speech features in noisy speech in real time. These two points significantly improve upon the existing short-time Fourier transform + static fusion framework in terms of speech feature representation and prior information modeling, respectively. They complement each other, jointly improving the modeling capability of the target speech extraction system, which is key to realizing a practically deployable and personalized target speech extraction system. Variable-resolution time-frequency conversion can flexibly balance the enhancement of quality and efficiency, while dynamic voiceprint feature fusion solves the mismatch problem caused by dynamic changes in speech features.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0082] This embodiment also provides a control device for a smart device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0083] Figure 7 This is a structural block diagram of a speech extraction device based on feature compression and adaptive fusion of speaker information according to an embodiment of the present invention, such as... Figure 7 As shown, the speech extraction device based on feature compression and adaptive fusion of speaker information includes:

[0084] The acquisition unit 71 is used to acquire the noisy speech and the registered speech of the target object, wherein the registered speech and the speech of the target object in the noisy speech are different voices.

[0085] Extraction unit 73 is used to extract corresponding first low-frequency features and first mid-to-high frequency compressed sensing features from the noisy speech, and to extract corresponding second low-frequency features and second mid-to-high frequency compressed sensing features from the registered speech.

[0086] The processing unit 75 is used to encode the first low-frequency feature and the first medium-high frequency compressed sensing feature using a noise encoder based on a deep neural network to obtain encoded features; and to map the second low-frequency feature and the second medium-high frequency compressed sensing feature using a registration encoder based on the deep neural network to obtain mapped features.

[0087] The fusion unit 77 is used to fuse the encoded features and the mapping features to obtain fused features.

[0088] Decoding unit 79 is used to decode the fused features based on the deep neural network to obtain the target speech.

[0089] In this embodiment, the acquisition unit 71 acquires the noisy speech and registered speech of the target object, wherein the registered speech and the speech of the target object in the noisy speech are different voices; the extraction unit 73 extracts the corresponding first low-frequency feature and first mid-to-high frequency compressed sensing feature from the noisy speech, and extracts the corresponding second low-frequency feature and second mid-to-high frequency compressed sensing feature from the registered speech; the processing unit 75 encodes the first low-frequency feature and the first mid-to-high frequency compressed sensing feature based on a deep neural network noise encoder to obtain encoded features; the registration encoder based on the deep neural network maps the second low-frequency feature and the second mid-to-high frequency compressed sensing feature to obtain mapped features; the fusion unit 77 fuses the encoded features and the mapped features to obtain fused features; and the decoding unit 79 decodes the fused features based on the deep neural network decoder to obtain the target speech. This method extracts the target speech based on the adaptive fusion of feature compression and voiceprint information. This solves the technical problem of low accuracy in speech extraction based on the adaptive fusion of feature compression and voiceprint information in the prior art.

[0090] Optionally, the fusion unit 77 may include: a mapping module, used to linearly project the high-dimensional representation of the target object's identity information onto a feature space identical to the intermediate features of the noisy speech; a calculation module, used to calculate the similarity between each frame of the high-dimensional features obtained from the noisy speech through the noise encoder and the projected target object representation; a processing module, used to weight the target object representation based on similarity to obtain the dynamic identity representation of the target object; and a fusion module, used to fuse the obtained dynamic identity representation of the target object into the high-dimensional representation of the noisy speech to obtain the fused feature.

[0091] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0092] Embodiments of the present invention also provide a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0093] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0094] S1, acquire the noisy speech and registered speech of the target object, wherein the speech of the target object in the registered speech and the noisy speech are different;

[0095] S2, extract the corresponding first low-frequency feature and first mid-to-high frequency compressed sensing feature from the noisy speech, and extract the corresponding second low-frequency feature and second mid-to-high frequency compressed sensing feature from the registered speech;

[0096] S3, The noise encoder based on the deep neural network encodes the first low-frequency feature and the first medium-high frequency compressed sensing feature to obtain the encoded feature; the registration encoder based on the deep neural network maps the second low-frequency feature and the second medium-high frequency compressed sensing feature to obtain the mapped feature.

[0097] S4, fuse the encoded features and the mapped features to obtain the fused features;

[0098] S5, the target speech is obtained by decoding and fusing features based on a deep neural network decoder.

[0099] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0100] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0101] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0102] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0103] S1, acquire the noisy speech and registered speech of the target object, wherein the speech of the target object in the registered speech and the noisy speech are different;

[0104] S2, extract the corresponding first low-frequency feature and first mid-to-high frequency compressed sensing feature from the noisy speech, and extract the corresponding second low-frequency feature and second mid-to-high frequency compressed sensing feature from the registered speech;

[0105] S3, The noise encoder based on the deep neural network encodes the first low-frequency feature and the first medium-high frequency compressed sensing feature to obtain the encoded feature; the registration encoder based on the deep neural network maps the second low-frequency feature and the second medium-high frequency compressed sensing feature to obtain the mapped feature.

[0106] S4, fuse the encoded features and the mapped features to obtain the fused features;

[0107] S5, the target speech is obtained by decoding and fusing features based on a deep neural network decoder.

[0108] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0109] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0110] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A speech extraction method based on feature compression and adaptive fusion of speaker information, characterized in that, include: Acquire noisy speech and registered speech of the target object, wherein the registered speech and the speech of the target object in the noisy speech are different voices; Extract the corresponding first low-frequency feature and first mid-to-high frequency compressed sensing feature from the noisy speech, and extract the corresponding second low-frequency feature and second mid-to-high frequency compressed sensing feature from the registered speech; A noise encoder based on a deep neural network encodes the first low-frequency feature and the first mid-to-high frequency compressed sensing feature to obtain encoded features; a registration encoder based on the deep neural network maps the second low-frequency feature and the second mid-to-high frequency compressed sensing feature to obtain mapped features. The encoded features and the mapped features are fused to obtain the fused features; The decoder based on the deep neural network decodes the fused features to obtain the target speech; The step of fusing the encoded features and the mapped features to obtain fused features includes: linearly projecting the high-dimensional representation of the target object's identity information onto a feature space identical to the encoded features obtained from the noisy speech through the noise encoder; calculating the similarity between each frame of the high-dimensional features obtained from the noisy speech through the noise encoder and the projected target object representation; weighting the target object representation using the similarity to obtain the dynamic identity representation of the target object; and fusing the obtained dynamic identity representation of the target object into the high-dimensional features of the noisy speech to obtain the fused features. The step of extracting the corresponding first low-frequency feature and first mid-to-high frequency compressed sensing feature from the noisy speech, and extracting the corresponding second low-frequency feature and second mid-to-high frequency compressed sensing feature from the registered speech, includes: a feature compression method based on the characteristics of human hearing, dividing both the noisy speech and the registered speech into low-frequency and mid-to-high frequency parts; performing feature compression on the mid-to-high frequency parts of the noisy speech and the registered speech respectively to obtain the corresponding first mid-to-high frequency compressed sensing feature and second mid-to-high frequency compressed sensing feature respectively; The step of compressing the mid-to-high frequency components of the noisy speech and the registered speech to obtain the corresponding first and second mid-to-high frequency compressed sensing features includes the following methods for compressing the mid-to-high frequency components of the noisy speech and the registered speech: taking the square of the modulus of the speech to obtain the energy spectrum; performing nonlinear filtering on the energy spectrum based on the characteristics of human auditory perception; and averaging the frequency points within the same bandwidth channel to reduce the feature dimension.

2. A speech extraction device based on feature compression and adaptive fusion of speaker information, characterized in that, include: The acquisition unit is used to acquire the noisy speech and the registered speech of the target object, wherein the registered speech and the speech of the target object in the noisy speech are different voices; An extraction unit is configured to extract corresponding first low-frequency features and first mid-to-high-frequency compressed sensing features from the noisy speech, and to extract corresponding second low-frequency features and second mid-to-high-frequency compressed sensing features from the registered speech. The extraction of the first low-frequency features and first mid-to-high-frequency compressed sensing features from the noisy speech, and the extraction of the second low-frequency features and second mid-to-high-frequency compressed sensing features from the registered speech, includes: a feature compression method based on human auditory characteristics, dividing both the noisy speech and the registered speech into low-frequency and mid-to-high-frequency parts; performing feature compression on the mid-to-high-frequency parts of the noisy speech and the registered speech respectively to obtain the corresponding first mid-to-high-frequency compressed sensing features and second mid-to-high-frequency compressed sensing features; and performing feature compression on the mid-to-high-frequency parts of the noisy speech and the registered speech respectively to obtain the corresponding first mid-to-high-frequency compressed sensing features and second mid-to-high-frequency compressed sensing features, including: performing feature compression on the mid-to-high-frequency parts of the noisy speech and the registered speech using the following methods: taking the square of the modulus of the speech to obtain the energy spectrum; performing nonlinear filtering on the energy spectrum based on human auditory perception characteristics; and averaging the frequency points within the same bandwidth channel to reduce the feature dimension. The processing unit is used to encode the first low-frequency feature and the first medium-high frequency compressed sensing feature using a noise encoder based on a deep neural network to obtain encoded features; and to map the second low-frequency feature and the second medium-high frequency compressed sensing feature using a registration encoder based on the deep neural network to obtain mapped features. A fusion unit is used to fuse the encoded features and the mapped features to obtain fused features; A decoding unit is used to decode the fused features based on the deep neural network to obtain the target speech; The fusion unit includes: a mapping module, used to linearly project the high-dimensional representation of the target object's identity information onto a feature space identical to the encoded features obtained by the noisy speech through the noise encoder; a calculation module, used to calculate the similarity between each frame of the high-dimensional features obtained by the noisy speech through the noise encoder and the projected target object representation; a processing module, used to weight the target object representation using the similarity to obtain the dynamic identity representation of the target object; and a fusion module, used to fuse the obtained dynamic identity representation of the target object into the high-dimensional features of the noisy speech to obtain the fused feature.

3. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of claim 1 at runtime.

4. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of claim 1.

Images