Speech enhancement method, speech enhancement model training method, speech processing method and computer program product

Abstract

Provided in the embodiments of the present application are a speech enhancement method, a speech enhancement model training method and a computer program product. The speech enhancement method comprises: acquiring a speech data sample of a target speaker, and performing spectral feature extraction on the speech data sample on the basis of a bidirectional long short-term memory network layer and an attention statistics pooling layer, so as to obtain a spectral feature of the target speaker; inputting the spectral feature into a feature-wise linear modulation layer in a time-frequency narrow-band neural network, and by means of the feature-wise linear modulation layer and on the basis of the spectral feature, adjusting a spectral feature corresponding to mixed speech data extracted by the time-frequency narrow-band neural network, wherein the mixed speech data at least comprises speech data of a plurality of speakers including the target speaker; and on the basis of the adjusted spectral feature, performing speech enhancement processing on the speech data of the target speaker in the mixed speech data, so as to obtain the enhanced speech data of the target speaker.

Description

Speech enhancement and speech enhancement model training methods, speech processing methods, computer program products

[0001] This application claims priority to Chinese Patent Application No. 202410572802.3, filed on May 9, 2024, entitled “Speech Enhancement and Speech Enhancement Model Training Method, Speech Processing Method, Computer Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of speech processing technology, and in particular to a speech enhancement method, a speech enhancement model training method, a speech processing method, and a computer program product. Background Technology

[0003] Speech Enhancement (SE) is a signal processing algorithm that enhances speech components by suppressing ambient noise. It plays a role in traditional audio signal processing, such as echo cancellation, noise reduction, and dereverberation removal.

[0004] However, with the widespread adoption of online audio and video conferencing, extracting only the speech of the target speaker and suppressing the speech of non-target speakers has become a widespread need. This is especially true in one-on-one meeting scenarios (where only one person is present at the near end, and other participants are at a distance), where the speaker may be in a noisy environment with others speaking simultaneously, making it difficult for distant participants to hear clearly. While sound filtering (SE) can remove ambient noise in such scenarios, the similarity in acoustic characteristics between the target and non-target speakers means that SE has no effect on suppressing the speech of non-target speakers.

[0005] Therefore, in scenarios such as audio and video conferencing, especially one-on-one meetings, how to better capture the voice of the target speaker and remove interference from other people's voices has become an urgent problem to be solved. Summary of the Invention

[0006] In view of this, embodiments of this application provide a speech enhancement and model training scheme to at least partially solve the above problems.

[0007] According to a first aspect of the embodiments of this application, a speech enhancement method is provided, comprising: acquiring a speech data sample of a target speaker, and extracting spectral features from the speech data sample based on a bidirectional long short-term memory network layer and an attention statistical pooling layer to obtain spectral features of the target speaker; inputting the spectral features into a feature linear modulation layer in a time-frequency narrowband neural network, and adjusting the spectral features corresponding to the mixed speech data extracted by the time-frequency narrowband neural network based on the spectral features, wherein the mixed speech data includes speech data of at least the target speaker; and performing speech enhancement processing on the speech data of the target speaker in the mixed speech data based on the adjusted spectral features to obtain enhanced speech data of the target speaker.

[0008] According to a second aspect of the embodiments of this application, a method for training a speech enhancement model is provided, comprising: acquiring training samples, the training samples including: mixed speech data samples and speech data samples of a target speaker, wherein the mixed speech data samples are speech data samples including interference signals and speech of multiple speakers including the target speaker; using the training samples, jointly training a time-frequency narrowband neural network and a target speaker feature extraction network included in the speech enhancement model to obtain a target speaker feature extraction network capable of extracting spectral features of the target speaker, and a time-frequency narrowband neural network that obtains the spectral features output by the target speaker feature extraction network through a feature linear modulation layer, and performs speech enhancement for the target speaker on the mixed speech data based on the spectral features.

[0009] According to a third aspect of the embodiments of this application, a speech processing method is provided, comprising: detecting and determining a target speaker by means of a preset detection strategy; acquiring a speech data sample of the target speaker; and sending the speech data sample to a background process, wherein the speech data sample is subjected to spectral feature extraction by at least a bidirectional long short-term memory network layer and an attention statistical pooling layer in the background process, thereby obtaining the spectral features of the target speaker.

[0010] According to a fourth aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the method described in the first, second, or third aspect.

[0011] According to a fifth aspect of the embodiments of this application, a computer program product is provided, including computer instructions that instruct a computing device to perform an operation corresponding to the method described in the first, second, or third aspect.

[0012] According to a sixth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first, second, or third aspect.

[0013] According to the scheme of this application embodiment, when performing speech enhancement for a target speaker, on the one hand, the spectral characteristics of the target speaker are utilized, and the spectral characteristics corresponding to the mixed speech data processed by the time-frequency narrowband neural network are adjusted through a feature linear modulation layer, so that the adjusted spectral characteristics are closer to the spectral characteristics of the target speaker, thereby enhancing the speech data of the target speaker in the mixed speech data and weakening the speech data of non-target speakers. Furthermore, since the spectral characteristics of different speakers' voices differ significantly, even in scenarios such as one-person meetings, the spectral characteristics of the target speaker can be used to effectively distinguish the speech of the target speaker from the interfering voices of other speakers, thereby effectively enhancing the speech of the target speaker. On the other hand, the spectral characteristics of the target speaker are obtained based on a bidirectional long short-term memory network layer and an attention statistical pooling layer independent of the time-frequency narrowband neural network. Compared with other methods of obtaining spectral characteristics, such as using a TDNN (Time Delay Neural Network), the method of this application embodiment has a simpler structure, less computation, effectively reduces the implementation cost of the scheme, and is easy to deploy on resource-limited devices. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 is a schematic diagram of an exemplary system to which the embodiments of this application are applied;

[0016] Figure 2A is a simplified structural block diagram of a speech enhancement model according to an embodiment of this application;

[0017] Figure 2B is an exemplary structural diagram of a speech enhancement model according to an embodiment of this application;

[0018] Figure 3A is a flowchart of the steps of a speech enhancement method according to an embodiment of this application;

[0019] Figure 3B is a diagram of a speech denoising system architecture in the embodiment shown in Figure 3A;

[0020] Figure 4 is a flowchart of a speech processing method according to an embodiment of this application;

[0021] Figure 5 is a flowchart of the steps of a training method for a speech enhancement model according to an embodiment of this application;

[0022] Figure 6 is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0024] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0025] To facilitate understanding of the solutions in the embodiments of this application, some terms involved in the solutions will be explained below.

[0026] Long Short-Term Memory (LSTM) network: a type of recurrent neural network that can effectively learn the dependency information between sequence data. uniLSTM (unidirectional LSTM) can learn the dependency information between sequence data from a single direction, such as the forward direction, while biLSTM (bidirectional LSTM) can learn the dependency information between sequence data from both the forward and reverse directions.

[0027] Attention Statistical Pooling is a neural network method that combines attention mechanisms and statistical pooling techniques. It is typically used to process sequential data, capturing important parts of the sequential data while also considering the overall statistical properties of the sequential data.

[0028] Spectro-Temporal Subband Neural Network (STSubNet): A type of neural network used to process signals in each frequency band in the frequency domain. Each frequency band is processed separately and shares the same set of network parameters, and has the characteristic of having very small network parameters.

[0029] Feature-wise Linear Modulation (FiLM) is a neural network that can be used to conditionally adjust features by scaling and shifting the input features.

[0030] Speech Enhancement (SE): A signal processing algorithm that enhances speech components by suppressing ambient noise.

[0031] Personalized Speech Enhancement (pSE): Based on speech enhancement SE, it extracts only the speech of the target speaker and suppresses the speech of non-target speakers.

[0032] Narrowband (narrow frequency band): Generally refers to sound signals with relatively small bandwidth and concentrated frequency distribution. For example, the main information frequency band of human voice is 300Hz-3400Hz, and this part of the signal can be regarded as a narrowband signal. However, in specific applications, those skilled in the art can define the narrowband range according to actual needs or conventional understanding in the field.

[0033] Full band (full frequency band): usually refers to a sound signal that covers a wide frequency range, which can include information from all frequency bands from low frequency to high frequency.

[0034] Short-Time Fourier Transform (STFT) is a mathematical transform related to the Fourier Transform. It can window long-term non-stationary audio signals for short-time segmentation, making the signal appear stationary within a short period, thus enabling Fourier Transform. By sliding the window function, the Fourier Transform result of the signal within each window can be obtained. These results, stacked and arranged, form a two-dimensional time-frequency domain feature map, representing the time and frequency distribution of the entire audio signal.

[0035] Inverse Short-Time Fourier Transform (iSTFT): It is the inverse operation of STFT and can restore a signal in the frequency domain to a signal in the time domain.

[0036] The following is a detailed description of the solutions in the embodiments of this application.

[0037] Figure 1 illustrates an exemplary system to which an embodiment of this application is applicable. As shown in Figure 1, the system 100 may include a cloud server 102, a communication network 104, and / or one or more user devices 106, with multiple user devices being an example in Figure 1.

[0038] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In this embodiment, the cloud server 102 can serve as the backend for user devices 106 with a voice enhancement solution deployed. In addition to storing the aforementioned content, it can also support online communication between multiple user devices 106, such as online audio and video conferencing.

[0039] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the following: the Internet, an intranet, a wide area network (WAN), a local area network (LAN), a wireless network, a digital subscriber line (DSL) network, a frame relay network, an asynchronous transfer mode (ATM) network, a virtual private network (VPN), and / or any other suitable communication network. The user equipment 106 can be connected to the communication network 104 via one or more communication links (e.g., communication link 112), and the communication network 104 can be linked to the cloud server 102 via one or more communication links (e.g., communication link 114). The communication link can be any communication link suitable for transmitting data between the user equipment 106 and the cloud server 102, such as a network link, a dial-up link, a wireless link, a hardwired link, any other suitable communication link, or any suitable combination of such links.

[0040] User equipment 106 may include sound acquisition devices such as microphones and sound playback devices such as speakers, and is equipped with the speech enhancement scheme of this application embodiment, suitable for any one or more user equipments interacting with users. In some examples, the microphone in user equipment 106 acquires sound from its environment, including the voices of multiple speakers and environmental noise. If the multiple speakers include a target speaker, user equipment 106 can use the scheme of this application embodiment to enhance the voice of the target speaker in the sound, i.e., the speech data of the target speaker, and send the enhanced speech data to other remote user equipments. For example, the user equipment 106 can acquire a speech data sample of the target speaker and extract spectral features from the speech data sample based on a bidirectional long short-term memory network layer and an attention statistical pooling layer to obtain the spectral features of the target speaker. The spectral features are then input into a feature linear modulation layer in a time-frequency narrowband neural network. The feature linear modulation layer adjusts the spectral features corresponding to the mixed speech data extracted by the time-frequency narrowband neural network based on these spectral features. The mixed speech data includes, but is not limited to, the speech data of multiple speakers, including the target speaker, and may also include environmental noise. Based on the adjusted spectral features, speech enhancement processing is performed on the speech data of the target speaker in the mixed speech data to obtain enhanced speech data of the target speaker. Furthermore, the enhanced speech data of the target speaker can be transmitted to other user equipment via the aforementioned communication network 104. In some embodiments, the user equipment 106 may include any suitable type of device. For example, in some embodiments, the user equipment 106 may include a mobile device, tablet computer, laptop computer, desktop computer, wearable computer, game console, media player, vehicle entertainment system, and / or any other suitable type of user equipment.

[0041] To facilitate understanding of the speech enhancement scheme in this application, the model structure involved in the speech enhancement scheme will be described below with reference to Figures 2A and 2B.

[0042] First, as shown in Figure 2A, the speech enhancement model of this application embodiment includes two parts: a target speaker feature extraction network and a time-frequency narrowband neural network.

[0043] The target speaker feature extraction network is a simple, lightweight network used to extract spectral features from speech data. It includes at least a bidirectional long short-term memory (LSTM) network layer and an attention-based statistical pooling layer. The LSM layer extracts spectral features from the input speech data, while the attention-based statistical pooling layer performs attention-based autocorrelation statistical calculations based on the input spectral features. This allows the target speaker feature extraction network to obtain the spectral features of the target speaker that can be output to the feature linear modulation layer of the time-frequency narrowband neural network based on the statistical calculation results.

[0044] Bidirectional Long Short-Term Memory (biLSTM) networks are a type of LSTM network, comprising a forward LSTM and a backward LSTM. The forward LSTM processes the input feature sequence in the forward direction for feature extraction, while the backward LSTM processes the input feature sequence in the reverse direction for feature extraction. The two sets of features are concatenated as the final extracted feature output. Specifically, in the embodiments of this application, the bidirectional long short-term memory network layer is implemented using a biLSTM network. Its forward LSTM and backward LSTM can respectively capture the spectral features of the sound signal on the forward and reverse time axes, thereby gaining a more comprehensive understanding of the sound signal and extracting more accurate and effective spectral features.

[0045] The Attention Statistic Pooling layer is a statistical aggregation layer that incorporates an attention mechanism. It can process data with sequential characteristics, such as the feature sequence corresponding to the speech data sample of the target speaker in this embodiment. It uses the attention mechanism to focus on important information in the feature sequence and employs statistical pooling to extract global features from the feature sequence. For example, when processing the feature sequence data corresponding to the speech data sample, the Attention Statistic Pooling layer first assigns a weight to each element in the feature sequence using the attention mechanism. This weight represents the importance of the element to the entire feature sequence. Then, by statistically pooling these weighted elements (e.g., taking the weighted average, weighted maximum, etc.), it obtains feature data that includes both global sequence features and key features from the speech data sample. This feature data is the spectral feature corresponding to the speech data sample of the target speaker in this embodiment, i.e., the spectral feature of the target speaker.

[0046] An exemplary target speaker feature extraction network may include sequentially connected convolutional layers, biLSTM network layers, fully connected and nonlinear layers, attention-based statistical pooling layers, and a classification output layer. In a specific example, as shown in Figure 2B, the convolutional layer is specifically illustrated as a one-dimensional convolutional layer "Conv1d", the biLSTM network layer as "biLSTM", the fully connected and nonlinear layer as "FC+ReLU", the attention-based statistical pooling layer as "Attentive Stat Pooling", and the classification output layer as "AAM-Softmax".

[0047] In this network, a one-dimensional convolutional layer is connected before the biLSTM network layer. This one-dimensional convolutional layer is used to downsample the input data, and the downsampled result is output to the biLSTM network layer. The use of a one-dimensional convolutional layer not only reduces the dimensionality of the spectral features but also simplifies the network structure, reduces the burden of data computation and processing, and can be matched with the feature linear modulation layer connected to the target speaker feature extraction network in a time-frequency narrowband neural network, facilitating feature transfer and processing.

[0048] In spectral feature extraction of sound signals, the sound signal first undergoes spectrum generation, such as Short Time Fourier Transform (STFT), to convert the time-domain signal into a frequency-domain signal, obtaining the corresponding spectral data. This spectral data describes the contribution of different frequency components in the sound signal and can express important features of the sound signal. Specifically, in the embodiments of this application, the speech data sample of the target speaker, which is the sound signal, also first undergoes STFT to form corresponding spectral data. This spectral data is first input into the one-dimensional convolutional layer, i.e., the Conv1d layer, of the target speaker feature extraction network for feature extraction in a downsampling manner. Then, the extracted features (i.e., the downsampling result) are output to the biLSTM network layer.

[0049] In the biLSTM network layer, these features will be extracted again from both the forward and reverse directions to further output more effective spectral features carrying rich spectral information.

[0050] Furthermore, the spectral features output by the biLSTM network layer are input into the FC+ReLU layer for fully connected and nonlinear activation processing to generate spectral features that can be effectively processed by the subsequent Attentive Stat Pooling layer.

[0051] In the Attentive Stat Pooling layer, the spectral features are subjected to attention-based statistical computation to extract more representative spectral features, which can then be used as input to the time-frequency narrowband neural network.

[0052] However, it should be noted that in this example, a classification output layer, such as an AAM-Softmax layer, is connected after the Attentive Stat Pooling layer. Because in this embodiment, the target speaker feature extraction network is trained together with the time-frequency narrowband neural network during the training phase, to further verify the learning objective of the target speaker feature extraction network and whether this learning objective contributes to the training of the entire speech enhancement model, including the time-frequency narrowband neural network and the target speaker feature extraction network, speaker identification is also used as another learning objective of the speech enhancement model, in addition to the target speaker speech enhancement learning objective, and AAM-Softmax is employed. The AAM-Softmax loss function adds a cosine margin term to the Softmax loss function, thereby maximizing the cosine distance between different categories during optimization, enhancing category separability, and improving classification performance. However, this is not the only possibility; other classification methods and loss functions, such as the Softmax function, can also be applied to the scheme of this embodiment.

[0053] For the time-frequency narrowband neural network STSubNet, it extracts the spectral features of the sound signal as input to the SubNet, based on the narrowband filtering network (SubNet), thereby improving the SubNet's efficiency in noise and reverberation removal. The SubNet uses the spectral features of each frequency band as input and allows each band to share network parameters. Its principle lies in whether the signal features of the frequency bands extracted based on the local spectral pattern can effectively distinguish between speech and non-speech information. Based on this, using spectral features as input can improve the SubNet's ability to distinguish between speech and non-speech, thus improving its efficiency. STSubNet is usually used alone, but in this embodiment, it is used in conjunction with a target speaker feature extraction network. For distinction, the model structure formed by these two networks is called the unified framework STSubNet, denoted as uSTSubNet (unified STSubNet). In this embodiment, the speech enhancement model is also this uSTSubNet.

[0054] In this embodiment, the STSubNet differs from a conventional STSubNet in that it incorporates a FiLM layer. An example of the STSubNet structure from this embodiment is shown in Figure 2B, where the upper left dashed box represents the portion used for initial spectral feature extraction, and the lower left dashed box represents the SubNet portion. The input signals to the STSubNet include the far-end audio signal, the original signal received by the microphone, and the signal processed by the adaptive filter AF (used to eliminate the linear portion of the echo signal in the original signal). The output is the speech enhancement result for the target speaker. As shown in Figure 2B, the target speaker feature extraction network interacts with the FiLM layer in the STSubNet through its Attention Stat Pooling layer to input the extracted spectral features of the target speaker into the FiLM layer.

[0055] In Figure 2B, the part used for preliminary spectral feature extraction includes a one-dimensional convolutional layer (shown as "Conv1d" in Figure 2B), a frequency-time modulation spectrum sliding window layer (shown as "ST window" in Figure 2B), a splicing layer (shown as "concat" in Figure 2B), a two-dimensional convolutional layer (shown as "Conv2d" in Figure 2B), a characteristic linear modulation layer (shown as "FiLM" in Figure 2B), a bidirectional long short-term memory network layer (shown as "biLSTM" in Figure 2B), and a fully connected and nonlinearized layer (shown as "FC+ReLU" in Figure 2B).

[0056] One-dimensional convolutional layers are used to downsample the input data. Using one-dimensional convolutional layers achieves feature extraction while reducing the dimensionality of spectral features, and also simplifies the network structure, reducing the burden of data computation and processing. The features downsampled by the one-dimensional convolutional layers are then input into a frequency-time modulation spectrum sliding window layer, such as the ST window.

[0057] The frequency-time modulation spectrum sliding window layer can sample the spectrum according to a preset number of sampling points, for example, 5 sampling points. Based on the preset number of sampling points, the characteristics of a certain spectrum are determined so that the characteristics carry sufficiently rich spectral information and can more accurately express its characteristics.

[0058] The splicing layer is used to splice the multiple spectral features output from the frequency-time modulation spectrum sliding window layer, as illustrated in Figure 2B, where y(t) (the original signal acquired by the microphone) and x f (t)(far-end audio signal) and y e The spectral characteristics of the three audio signals (t)(AF output signal) are spliced ​​together.

[0059] The spliced ​​spectral features are input into a two-dimensional convolutional layer for further feature extraction, resulting in spectral features that can more effectively represent the characteristics of the sound signal.

[0060] Following this, the spectral features output by the 2D convolutional layer are modulated by the input feature linear modulation layer, namely the FiLM (Feature-wise Linear Modulation) layer. FiLM is a neural network module that can be used to conditionally adjust features. It achieves this adjustment by scaling and shifting the input features. For example, given an input feature x, the FiLM layer first generates two parameters γ' and β' through a fully connected layer or other form of network structure, and then scales and shifts the input feature, i.e., y = γ'*x + β'. Here, γ' and β' are vectors of the same size as the input feature x, and they determine the degree of scaling and shifting of the input feature x, so that the representation of the features it processes is adjusted according to specific conditions (such as spectral features from the target speaker feature extraction network). This allows the time-frequency narrowband neural network to better utilize conditional information (the spectral features of the target speaker), thereby improving target performance, such as speech enhancement performance.

[0061] After adjustment by the FiLM layer, the spectral features are input into the bidirectional long short-term memory network layer, i.e., the biLSTM layer. This biLSTM layer can extract spectral features on the frequency axis. The number of bidirectional LSTMs can be one or more (two or more), as shown in Figure 2B.

[0062] Furthermore, after processing by the biLSTM layer, the spectral features enter the fully connected and nonlinearized layers. Through these fully connected and nonlinearized layers, the time-frequency narrowband neural network can integrate the extracted spectral features of the input audio signal (such as the three audio signals mentioned above) for overall processing.

[0063] After the above processing, the extracted spectral features are processed in the latter half of the time-frequency narrowband neural network, namely the SubNet part. The SubNet part includes a unidirectional long short-term memory network layer formed by one or more unidirectional long short-term memory networks (shown as two "uniLSTM" in Figure 2B), a fully connected and nonlinearized layer (shown as "FC+Tanh" in Figure 2B), and a one-dimensional convolutional layer (shown as "conv1d" in Figure 2B).

[0064] Among them, the uniLSTM layer of the unidirectional long short-term memory network can guide the processing of information in the next frequency band based on the information of the previous frequency band, thereby improving the accuracy of the spectral feature extraction results. Furthermore, through fully connected and nonlinear layers, the extracted spectral features can contain richer information. The one-dimensional convolutional layer can greatly reduce the amount of data that the subsequent time-frequency narrowband neural network needs to process, thereby improving the efficiency of the time-frequency narrowband neural network in speech enhancement processing.

[0065] Furthermore, it should be noted that in speech enhancement processing, if a full-band processing scheme is adopted, it is necessary to directly process the information of the entire band. This results in a large input dimension, requiring numerous parameters and leading to a large model size. Moreover, due to the complexity of the model, many nodes within it may be dependent on various input data. Therefore, as the input dimension increases, the increase in computational load far exceeds the increase in input dimension, leading to excessive computational burden and making it unsuitable for deployment on resource-constrained devices. In contrast, this embodiment employs a time-frequency narrowband neural network. Its model only needs to process features of a single frequency band. For each frequency band, the network model coefficients are shared, effectively reducing the amount of data for model coefficients, lowering computational load, and improving processing efficiency. Furthermore, during training, the model can be trained by sequentially inputting features of individual frequency bands, allowing the model to perform well across different frequency bands, balancing efficiency and accuracy. Because the model size and number of parameters are relatively small, it is suitable for resource-constrained devices, such as end-user devices.

[0066] As can be seen from the above, because both the target speaker feature extraction network and the time-frequency narrowband neural network have relatively simple model structures, they can achieve the effect of small model and low computational cost. This makes the speech enhancement model, which includes these two parts, easier to deploy on devices with limited resources (such as user devices on the terminal). However, those skilled in the art should understand that although the speech enhancement model in this application embodiment includes a time-frequency narrowband neural network, it can also be applied to full-band real-time speech processing.

[0067] Based on the structure of the exemplary speech enhancement model described above, this application also provides a speech enhancement method, which will be described below through embodiments.

[0068] Referring to FIG3A, a flowchart of the steps of a speech enhancement method according to an embodiment of the present application is shown.

[0069] The speech enhancement method of this embodiment can be used in resource-limited end-user devices to achieve real-time speech enhancement of the target speaker's speech. However, those skilled in the art should understand that it is also applicable to server-side devices with abundant resources; that is, the speech enhancement scheme of this application embodiment can be implemented on the server-side device with reference to this embodiment.

[0070] The speech enhancement method in this embodiment includes the following steps:

[0071] Step S302: Obtain speech data samples of the target speaker, and extract spectral features from the speech data samples based on the bidirectional long short-term memory network layer and the attention statistical pooling layer to obtain the spectral features of the target speaker.

[0072] The voice data samples of the target speaker can be obtained in a variety of ways.

[0073] In one feasible approach, the end-user device can provide a human-computer interaction interface, allowing the user to input the voice of the target speaker as a voice data sample. This approach provides users with more autonomous input options, enabling them to input the appropriate voice as a voice data sample as needed.

[0074] In another feasible approach, the end-user device is equipped with a voice duration detection device. If a user's continuous speaking voice is detected to exceed a certain duration, such as 20 seconds, that user is identified as the target speaker, and the resulting speech is used as a voice data sample. This method is imperceptible to the user and can greatly improve the user experience.

[0075] In another feasible approach, with the user's consent, the audio of the end user in multiple meetings can be monitored. For example, if a user is detected to have produced specific audio in multiple consecutive meetings over 5-10 meetings, then the user can be identified as the target speaker, and the speech formed by the user can be used as a speech data sample.

[0076] In another feasible approach, an image acquisition device, such as a camera, can be installed in the end-user device to capture the current face image and perform lip movement detection. If a user's lip movement is detected for a certain duration, such as 10 seconds or 20 seconds, and the continuity of the lip movement is consistent with the continuity of the audio, then the user is considered to be the target speaker. The user's voice is then captured as a voice data sample through a voice acquisition device, such as a microphone.

[0077] Of course, this is not the only method; other methods for identifying the target speaker and obtaining their voice data samples are also applicable to the solutions in this application.

[0078] Speech data samples can be processed through bidirectional long short-term memory network layers such as biLSTM layers and attention-based statistical pooling layers such as Attentive Stat Pooling layers. The biLSTM layer can extract spectral features from the speech data samples in both forward and reverse directions, resulting in richer and more effective spectral features. The Attentive Stat Pooling layer can further extract spectral features from the input spectral features based on an attention mechanism, making the obtained spectral features more representative and better reflecting the spectral characteristics of the target speaker. In practical applications, those skilled in the art can add any appropriate other model layers based on the biLSTM and Attentive Stat Pooling layers to facilitate spectral feature extraction and smooth feature transfer between these two layers, all of which are within the protection scope of the embodiments of this application.

[0079] For example, in this embodiment of the application, a target speaker feature extraction network, including a biLSTM layer and an Attentive Stat Pooling layer as shown in Figure 2B, is used to extract the spectral features of the target speaker. As mentioned above, this target speaker feature extraction network is independent of the time-frequency narrowband neural network. Furthermore, since speech data samples cannot directly express the spectral characteristics of speech, in order to enable the target speaker feature extraction network to extract spectral features more efficiently and accurately, in one feasible approach, the speech data samples can first undergo STFT, and then the result of the STFT, i.e., the spectral data (spectral graph), can be input into the target speaker feature extraction network for spectral feature extraction.

[0080] In the target speaker feature extraction network shown in Figure 2B, on the one hand, to facilitate subsequent processing by the biLSTM layer; and on the other hand, to reduce the data processing and deployment burden of the speech enhancement model, the spectral data is first processed by a one-dimensional convolutional layer in a downsampling manner before being input into the biLSTM layer to obtain preliminary spectral features. That is, the STFT result is downsampled by the one-dimensional convolutional layer in the target speaker feature extraction network to obtain downsampled features.

[0081] However, at this point, the downsampling feature is merely a static spectral representation of the speech data sample, without considering its dynamic changes over time. Therefore, after obtaining the downsampling feature, further spectral feature extraction can be performed on the speech data sample using subsequent biLSTM layers, Attentive Stat Pooling layers, etc. The biLSTM layer can effectively capture and process the contextual information within the spectral features corresponding to the speech data sample—the speech feature sequence. For example, the biLSTM layer can capture the features of the feature sequence on the forward and reverse time axes using both forward and reverse LSTMs, respectively, thus enabling a more comprehensive understanding of the speech data sample and its represented features, and obtaining more accurate and effective spectral features.

[0082] The spectral features output from the biLSTM layer are then processed by the FC+ReLU layer for feature fusion and nonlinearization to extract more representative spectral features. These spectral features are then passed to the subsequent Attentive Stat Pooling layer.

[0083] The Attentive Stat Pooling layer can perform attention-based autocorrelation statistical calculations based on the spectral features output by the FC+ReLU layer. This allows the target speaker feature extraction network to output the final spectral features of the target speaker based on the statistical calculation results. These spectral features accurately represent the spectral characteristics of the target speaker. Because the Attentive Stat Pooling layer uses an attention mechanism to assign different weights to speech frames corresponding to different speech data samples, this means that not all speech frames have the same importance in forming spectral features; rather, weights are dynamically assigned based on the content of the speech frames. Furthermore, unlike traditional statistical pooling layers that simply average across all speech frames to form features, the Attentive Stat Pooling layer in this embodiment calculates both the weighted average and the weighted standard deviation. These two statistical quantities jointly capture the long-term changes in the spectral characteristics of the target speaker, resulting in richer and more comprehensive spectral features of the target speaker.

[0084] At this point, the obtained spectral features of the target speaker can be passed to the time-frequency narrowband neural network. However, as shown in Figure 2B, the target speaker feature extraction network in this embodiment also includes a classification output layer. In one feasible approach, this classification output layer may not be used in the inference stage; that is, the target speaker feature extraction network directly outputs the obtained spectral features without performing subsequent classification operations. This classification output layer is mainly used during the training stage in conjunction with the time-frequency narrowband neural network. In another feasible approach, after obtaining the spectral features of the target speaker during the inference stage, on the one hand, it is output to the time-frequency narrowband neural network; on the other hand, it is output to a classification output layer, such as the AAM-Softmax layer, connected after the Attentive Stat Pooling layer, to classify and identify the spectral features output by the Attentive Stat Pooling layer and output the identification result, i.e., the identifier of the target speaker, to provide richer and more comprehensive target speaker information.

[0085] Step S304: Input the spectral features into the feature linear modulation layer in the time-frequency narrowband neural network, and adjust the spectral features corresponding to the mixed speech data extracted by the time-frequency narrowband neural network based on the spectral features.

[0086] The mixed speech data includes speech data from at least the target speaker and multiple speakers.

[0087] While the target speaker feature extraction network extracts spectral features, the time-frequency narrowband neural network, such as the aforementioned STSubNet, also processes the data input to itself.

[0088] In this embodiment, because the target speaker's voice data is contained within other sound signals, the signal collected by the microphone that includes the voices of at least the target speaker and other speakers is referred to as mixed voice data. Of course, this mixed voice data may also contain environmental noise. For example, mixed voice data may include a far-end sound signal x. f The signal y(t) consists of the original signal y(t) captured by the microphone and the signal y(t) after being filtered by the adaptive filter AF (used to eliminate the linear part of the echo signal). e (t). Here, the original signal y(t) includes the voice signals of multiple speakers, including the target speaker.

[0089] For example, as shown in FIG2B, the above x f (t), y(t) and y e(t) After STFT conversion, spectral data is obtained, which is then input into STSubNet. Compared to the traditional method of using amplitude spectrum for speech enhancement, in this embodiment, x is converted using STFT. f (t), y(t) and y e The complex spectrum (real and imaginary parts) corresponding to (t) can be used as input to STSubNet to make full use of the amplitude and phase of the sound signal, so as to provide a more accurate and information-rich data foundation for subsequent processing.

[0090] x f (t), y(t) and y e The complex spectrum corresponding to (t) is input into STSubNet. As mentioned earlier, it first passes through a one-dimensional convolutional layer, a frequency-time modulation spectrum sliding window layer, a concatenation layer, and a two-dimensional convolutional layer for feature extraction processing, and then is input into the Feature Linear Modulation (FiLM) layer. That is, after performing STFT on the mixed speech data, STSubNet downsamples the STFT result through its one-dimensional convolutional layer to obtain downsampled feature data; then, based on the downsampled feature data, it performs multiple different feature extractions and feature concatenation operations to obtain the spectral features corresponding to the mixed speech data. These spectral features will be used as the input to the FiLM layer.

[0091] In the FiLM layer, the mixed speech data (x) extracted by STSubNet is processed based on the spectral features of the input target speaker. f (t), y(t) and y e (t) The spectral features corresponding to the data formed by the three signals are adjusted according to the conditions described above. For example, the FiLM layer uses the spectral features output by the Attentive Stat Pooling layer as a condition to scale and shift the spectral features obtained from the preceding two-dimensional convolutional layer, illustrated in this example as spectral feature x, to achieve the adjustment of spectral feature x. For example, the FiLM layer first generates two parameters γ' and β', and then scales and shifts the spectral feature x, i.e., y = γ'*x + β'. Here, γ' and β' are vectors of the same size as the spectral feature x, and they determine the degree of scaling and shifting of the spectral feature x. When specifically scaling and shifting the spectral feature x, the FiLM layer uses the spectral features of the target speaker obtained from the Attentive Stat Pooling layer of the target speaker feature extraction network as a condition for adjusting the spectral feature x, in order to retain as many features in the spectral feature x as possible that match the spectral features of the target speaker, while weakening or eliminating those features in the spectral feature x that do not match the spectral features of the target speaker.

[0092] Step S306: Based on the adjusted spectral features, perform speech enhancement processing on the speech data of the target speaker in the mixed speech data to obtain the enhanced speech data of the target speaker.

[0093] After feature adjustment in the FiLM layer, as shown in Figure 2B, the spectral features output by the FiLM layer will be processed by the biLSTM network in the biLSTM layer and the FC+ReLU layer to form spectral features that can be input into the SubNet for processing.

[0094] Furthermore, in the SubNet part, the spectral features output by the aforementioned FC+ReLU layer are sequentially processed by a unidirectional long short-term memory network layer, a fully connected and nonlinearized layer, and a one-dimensional convolutional layer to generate enhanced spectral features for the target speaker. For example, in one feasible approach, as shown in Figure 2B, the output of the one-dimensional convolutional layer can be combined with the filtering result of AF, i.e., y... e (t) Perform a dot product operation to further enhance the features of the target speaker and suppress the features of other non-target speakers. Then, after passing through iSTFT (Inverse Short Time Fourier Transform), it can be converted into enhanced speech data of the target speaker in the time domain.

[0095] It should be noted that in some cases, the input of the target speaker's speech data sample may fail, or the extraction of spectral features from the target speaker's speech data sample may fail. To avoid the impact of such anomalies on speech enhancement processing and ensure the robustness of the scheme, one feasible approach is to use pseudo-voiceprint information as the speech data sample. That is, if the obtained spectral features of the target speaker are unavailable, pseudo-voiceprint information is used as the speech data sample. Spectral features are extracted from the speech data sample based on a bidirectional long short-term memory network layer and an attention statistical pooling layer, so that the spectral features of the obtained pseudo-voiceprint information are used as the spectral features input to the feature linear modulation layer in the time-frequency narrowband neural network. For example, the pseudo-voiceprint information is processed by STFT and then input into the target speaker feature extraction network, as shown in Figure 2B, to extract spectral features through it (especially the bidirectional long short-term memory network layer and the attention statistical pooling layer). Then, the extracted spectral features are passed to the FiLM layer in the time-frequency narrowband neural network for speech enhancement processing. In this case, the speech enhancement model, including the target speaker feature extraction network and the time-frequency narrowband neural network, will change into a non-personalized speech enhancement model.

[0096] In one feasible approach, pseudo-voiceprint information can be implemented as white noise. Using white noise as pseudo-voiceprint information is simple and inexpensive to implement. Furthermore, because white noise has the characteristic of equal energy across all frequency bands, the spectral characteristics obtained from it also help speech enhancement models to more easily identify the target speaker's speech information and perform targeted speech enhancement even without accurate spectral characteristics of the target speaker.

[0097] By using pseudo-voiceprint information as a speech data sample, frame-level switching between ordinary speech enhancement and personalized speech enhancement can be achieved, expanding the application scenarios of the speech enhancement scheme in this application embodiment.

[0098] Furthermore, since the speech enhancement model in this application embodiment mainly targets the target speaker for speech enhancement, environmental noise such as echo and reverberation contained in the mixed speech data can be reduced or eliminated. Therefore, the speech enhancement model can achieve the effect of noise reduction as a whole.

[0099] Furthermore, based on the above speech enhancement model, a speech denoising system applied to real-world scenarios is shown in Figure 3B.

[0100] In Figure 3B, the speech denoising system includes an AF (Adaptive Filter) to eliminate the linear portion of the echo signal in the original signal; a TDE (Time Delay Estimation) module to estimate the time difference between the far-end sound signal and the near-end echo signal to better adapt to the processing of the AF; and uSTSubNet to process the mixed speech data to obtain enhanced speech data for the target speaker. It can be seen that this speech denoising system, especially uSTSubNet, can remove noise, reverberation, residual echoes in the target speaker's environment, as well as interfering speech from non-target speakers.

[0101] As can be seen from Figure 3B, on the proximal side of the real-time communication (RTC), This represents the echo-free sound signal of the target speaker at time t. This represents the echo-free sound signal of the non-target speaker at time t; express The corresponding spatial impulse response between the target speaker and the microphone. express The corresponding spatial impulse response between the non-target speaker and the microphone; n n (t) represents environmental noise; x f(t) represents the far-end audio signal of the RTC; f(·) is the nonlinear representation of the loudspeaker; y(t) represents the raw signal collected by the near-end microphone, y(t) = x n (t)+x o (t))+n n (t) where x n (t) represents the sound signal captured by the microphone, which includes reverberation and the speaker's voice. * indicates a convolution operation, x o (t) represents the echo signal; The registered audio refers to the voice data sample of the target speaker in this embodiment of the application; h f (t) represents the spatial impulse response between the speaker and the microphone.

[0102] Therefore, it can be based on Speech enhancement, including echo cancellation, noise reduction, denoising, and pSE tasks, can be viewed as removing x using the above formula. o (t), inhibit n n (t), recovery and separate extraction The task.

[0103] As shown in Figure 3B, after the TDE module processes the far-end audio signal x... f After processing the original signal y(t) and the signal y(t) acquired by the near-end microphone, the corresponding time delay information delay is obtained. This time delay information delay is sent to the AF for subsequent filtering. The AF adopts a combined linear filtering strategy, which updates the estimated filter... To eliminate echo signal x o The linear component of (t). This combined linear filtering strategy uses a frequency-domain adaptive Kalman filter for updates, combined with a normalized least mean square algorithm as an aid to improve the convergence stability of the filter. Furthermore, in x f (t) and x o There is an unknown delay t between (t) and (t). d Therefore, the TDE in this application embodiment employs a low-complexity TDE based on spectral similarity for this t d To cover the 1-second delay. Based on this, the y-value of the AF output... e (t) can be expressed as:

[0104] Based on this, uSTSubNet uses x f (t), y(t), y e (t) and For the input, speech enhancement processing is performed. Among other things, After passing through STFT, the signal will be fed into the target speaker feature extraction network of uSTSubNet for spectral feature extraction to obtain the spectral features of the target speaker. Meanwhile, x f (t), y(t), y e (t) After passing through STFT, the data is input into the time-frequency narrowband neural network in uSTSubNet, where corresponding spectral feature extraction is performed. Upon reaching the FiLM layer, the FiLM layer receives the target speaker's spectral features output from the Attentive Stat Pooling layer of the target speaker feature extraction network. Using these received spectral features as a condition, the spectral features extracted by the time-frequency narrowband neural network itself are adjusted, and further processing is performed based on the adjusted features until the final spectral features are obtained. These final spectral features, after being converted by iSTFT, become the enhanced speech data for the target speaker.

[0105] As can be seen, through this embodiment, when performing speech enhancement for a target speaker, on the one hand, the spectral characteristics of the target speaker are utilized. A feature linear modulation layer is used to adjust the spectral characteristics of the mixed speech data processed by the time-frequency narrowband neural network, making the adjusted spectral characteristics closer to those of the target speaker. This achieves the effect of enhancing the target speaker's speech data in the mixed speech data while weakening the speech data of non-target speakers. Furthermore, because the spectral characteristics of different speakers' voices vary significantly, even in scenarios such as one-person meetings, the spectral characteristics of the target speaker can be used to effectively distinguish their speech from other interfering voices, thus effectively enhancing the target speaker's speech. On the other hand, the target speaker's spectral characteristics are obtained based on a bidirectional long short-term memory network layer and an attention statistical pooling layer, independent of the time-frequency narrowband neural network. Compared to other methods of obtaining spectral characteristics, such as using a TDNN (Time Delay Neural Network), the method in this embodiment has a simpler structure, requires less computation, effectively reduces the implementation cost, and is easy to deploy on resource-constrained devices.

[0106] Referring to FIG4, a flowchart of the steps of a speech processing method according to an embodiment of the present application is shown. This embodiment focuses on describing the speech processing method from the perspective of determining the target speaker. Those skilled in the art should understand that the method for determining the target speaker shown in this embodiment can be applied to the foregoing embodiments to determine the target speaker in the foregoing embodiments.

[0107] The speech processing method in this embodiment includes the following steps:

[0108] Step S402: Detect and determine the target speaker using a preset detection strategy.

[0109] The detection and determination of the target speaker can be achieved through a preset detection strategy. For example, the target speaker can be detected and determined through at least one of the following preset detection strategies:

[0110] 1) Detect whether a request for the identity of the target speaker has been received through the human-computer interaction interface. If so, identify the speaker requested by the target speaker identity as the target speaker.

[0111] In this approach, the end-user device can provide a human-computer interaction interface (HCI) with corresponding target speaker identity request options. For example, a button or option can be provided for inputting the target speaker's voice. When the button is clicked or the option is selected and confirmed, the target speaker identity request is considered triggered. At this point, a sound acquisition prompt appears, allowing the user to input voice. The input voice can then be considered the target speaker's voice, and the speaker corresponding to that voice is the target speaker. Alternatively, the person who clicked the button or selected the option to trigger the target speaker identity request can be considered the target speaker. For instance, if user A clicks the "Become Target Speaker" button on the HCI, triggering the aforementioned voice acquisition process for user A, then user A is the target speaker, and the voice input by user A can be used as a subsequent voice data sample. This method allows users to make their own selections and clearly identifies the user's target speaker, resulting in high efficiency in acquiring target speaker voice data samples.

[0112] 2) Detect whether the continuous speaking duration of the same speaker collected by the voice acquisition device exceeds the preset duration. If it does, then identify the speaker as the target speaker.

[0113] The preset duration can be appropriately set by those skilled in the art according to actual needs, such as 20 seconds. End-user devices are typically equipped with voice capture devices such as microphones, allowing for the capture of the speaker's voice within the preset duration at the start of an audio or video conference. Since the voice of a target speaker, such as the conference host or current speaker, generally lasts for a certain duration, this can be used as a basis for identifying the target speaker.

[0114] For example, assuming the microphone in the user terminal device captures user B's continuous speech for more than 20 seconds, user B will be identified as the target speaker. Then, user B's speech within this preset duration, such as 20 seconds, can be used as a speech data sample. Of course, it is also possible to continue capturing user B's speech for a longer period and use the captured speech as user B's speech data sample.

[0115] In this way, the target speaker can be identified and their voice data samples can be collected without the user's awareness, thus improving the user experience.

[0116] 3) Detect whether the number of times the same speaker's voice appears in multiple historical audio or video conferences has reached a preset number. If it has, then identify that speaker as the target speaker.

[0117] In a group or organization, the main speaker in an audio or video conference is usually relatively fixed. Based on this, it's possible to detect whether the same speaker's voice appears in multiple historical conferences. If the number of conferences with the same speaker's voice reaches a preset number, then that speaker can be considered the target speaker. The preset number of times can be appropriately set by those skilled in the art according to actual needs; this application embodiment does not impose any limitations on this.

[0118] For example, assuming user C's voice is present in 3, 4, or 5 out of 5 meetings, user C can be identified as the target speaker. Then, user C's voice in these meetings is used as a voice data sample for the current audio or video conference.

[0119] In this way, speech data samples of the target speaker can be predicted and obtained in advance, improving the efficiency of target speaker identification and the efficiency of obtaining speech data samples of the target speaker.

[0120] 4) Acquire facial video images using an image acquisition device and perform lip movement detection on the facial video images. If the duration of the lip movement reaches the preset duration and the continuity of the lip movement is consistent with the continuity of the speech acquired by the speech acquisition device, then the speaker corresponding to the facial video image is identified as the target speaker.

[0121] In this approach, the end-user device is equipped with both a voice acquisition device (such as a microphone) and an image acquisition device (such as a camera). After an audio or video conference begins, the image acquisition device can capture a current facial video image and perform lip movement detection. If the lip movement duration reaches a preset duration, and within that preset duration, the continuity of the lip movement (e.g., the lip movement continues continuously) and the continuity of the speech acquired by the voice acquisition device (e.g., the speech continues continuously within the preset duration) are both confirmed, then the speaker corresponding to the facial video image can be identified as the target speaker. Furthermore, the speech acquired by the voice acquisition device within that preset duration can be used as a speech data sample of the target speaker. The preset duration can be appropriately set by those skilled in the art according to actual needs; this embodiment does not impose any limitations on this.

[0122] By combining lip movements and speech to determine the target speaker, the accuracy of target speaker identification is improved.

[0123] Step S404: Obtain a voice data sample of the target speaker.

[0124] Once the target speaker is identified, a voice data sample of the target speaker can be obtained. The specific method has been explained in step S402 and will not be repeated here.

[0125] Step S406: Extract spectral features from speech data samples by at least a bidirectional long short-term memory network layer and an attention statistical pooling layer to obtain the spectral features of the target speaker.

[0126] The specific implementation of this step can be referred to the description of the relevant parts in the foregoing embodiments, and will not be repeated here.

[0127] Furthermore, the spectral features of the target speaker can be input into the feature linear modulation layer in the time-frequency narrowband neural network. The feature linear modulation layer then adjusts the spectral features corresponding to the mixed speech data extracted by the time-frequency narrowband neural network based on these spectral features. Subsequently, based on the adjusted spectral features, speech enhancement processing can be performed on the target speaker's speech data in the mixed speech data to obtain enhanced speech data of the target speaker. The specific implementation of this process can also refer to the description of the relevant parts in the foregoing embodiments, and will not be repeated here.

[0128] This embodiment enables the effective identification of target speakers in audio or video conferencing. Based on this, the spectral characteristics of the target speaker can be obtained, providing a feature-based basis for subsequent speech enhancement, thereby improving the effectiveness and efficiency of speech enhancement.

[0129] The training process of the speech enhancement model uSTSubNet involved in the above process will be described below. Referring to FIG5, a flowchart of the steps of a speech enhancement model training method according to an embodiment of this application is shown.

[0130] The training method for the speech enhancement model in this embodiment includes the following steps:

[0131] Step S502: Obtain training samples.

[0132] The training samples include: mixed speech data samples and speech data samples of the target speaker. The mixed speech data samples are speech data samples that include interference signals and speech of multiple speakers, including the target speaker.

[0133] In this embodiment, the speech enhancement model can be the model shown in Figure 2B. The speech data samples of the target speaker in the training samples are used as the training input of the target speaker feature extraction network in the speech enhancement model, and the mixed speech data samples are used as the training input of the time-frequency narrowband neural network in the speech enhancement model.

[0134] Step S504: Using training samples, jointly train the time-frequency narrowband neural network and the target speaker feature extraction network included in the speech enhancement model to obtain a target speaker feature extraction network that can extract the spectral features of the target speaker, and a time-frequency narrowband neural network that obtains the spectral features output by the target speaker feature extraction network through a feature linear modulation layer, and performs speech enhancement for the target speaker on the mixed speech data based on the spectral features.

[0135] As mentioned earlier, in the speech enhancement model, the target speaker feature extraction network includes at least a bidirectional long short-term memory network layer and an attention statistical pooling layer. The bidirectional long short-term memory network layer is used to extract spectral features from the speech data samples, and the attention statistical pooling layer is used to perform attention-based autocorrelation statistical calculations based on the input spectral features, so that the target speaker feature extraction network outputs the spectral features of the target speaker to the feature linear modulation layer based on the results of the statistical calculations.

[0136] For example, a target speaker feature extraction network may further include: a one-dimensional convolutional layer connected before the bidirectional long short-term memory network layer, a fully connected and non-linearized layer connected between the bidirectional long short-term memory network layer and the attention statistical pooling layer, and a classification output layer connected after the attention statistical pooling layer. The target speaker feature extraction network is specifically implemented as follows: a one-dimensional convolutional layer is used to downsample the input data (such as the spectral data corresponding to the speech data sample after STFT), and the downsampling result is output to a bidirectional long short-term memory network layer; a bidirectional long short-term memory network layer is used to extract spectral features from the downsampled result output by the one-dimensional convolutional layer again from both the forward and reverse directions; a fully connected and nonlinearized layer is used to fully connect and nonlinearize the spectral features output by the bidirectional long short-term memory network layer, and the processing result is output to an attention statistical pooling layer; an attention statistical pooling layer is used to perform attention-based autocorrelation statistical calculation based on the spectral features output by the fully connected and nonlinearized layers to output the spectral features of the target speaker that can be passed to the time-frequency narrowband neural network; and a classification output layer is used to classify the speech data samples of the target speaker based on the data output by the attention statistical pooling layer, and output the classification result, such as an identifier used to indicate the identity of the target speaker.

[0137] The time-frequency narrowband neural network (TFNN) comprises a part for extracting spectral features from the audio signal and a SubNet part. The spectral feature extraction part includes a one-dimensional convolutional layer, a frequency-time modulation spectrum sliding window layer, a splicing layer, a two-dimensional convolutional layer, a feature linear modulation layer, a bidirectional long short-term memory (LSTM) network layer, and fully connected and nonlinearized layers. The SubNet part includes a unidirectional LSM network layer, fully connected and nonlinearized layers, and a one-dimensional convolutional layer. The TFNN is connected to the attention statistical pooling layer in the target speaker feature extraction network through its feature linear modulation layer, enabling interaction between the TFNN and the target speaker feature extraction network based on the target speaker's spectral features. The feature linear modulation layer adjusts the spectral features of the mixed speech data obtained by the TFNN itself based on the target speaker's spectral features, thereby enhancing the target speaker's spectral features in the mixed speech data and weakening or eliminating the spectral features of non-target speakers.

[0138] The time-frequency narrowband neural network and the target speaker feature extraction network are trained together in a joint training process. Based on this, the loss function used is expressed as follows:

[0139] in:

[0140] The base loss is represented by a constrained signal-to-noise ratio loss as the minimization criterion. The speech enhancement model is trained using the Adam optimizer, as shown below:

[0141] in, It can be seen that s n The value of s(t) depends on whether the input signal to the target speaker feature extraction network is a real speech data sample of the target speaker or pseudo-speaker information. When it is a real speech data sample of the target speaker, s n The value of (t) is When it is pseudo-voiceprint information, s n The value of (t) is This represents the prediction of the speech enhancement model, corresponding to s n The predicted value of (t). β is 10. -3 It should be noted that, to improve the naturalness of the sound after dreverberation, a reference signal with a shortened reverberation time can be selected.

[0142] However, the aforementioned basic losses This is not applicable when the sound signal contains only echoes or noise; in such cases, s n (t) = 0. Therefore, in this embodiment, an auxiliary loss is introduced. Represented as:

[0143] Among them, as mentioned above, y e (t) is the output of AF, used for constraints. Update; β0 takes the value 10 -6 .

[0144] To alleviate the problem of excessive suppression during model training, this application also introduces asymmetric loss in its embodiments. Represented as:

[0145] Where k represents the frequency band, l represents the frame index, and s n (t) after the STFT based on the frequency band k and frame index l, is represented as S n (k,l), similarly, After passing through iSTFT, it is represented as S diff With the aforementioned s n (t) is similar, corresponding to s n (t) can be the value after STFT, which can be The value after STFT, or is The values ​​after STFT are determined. α and γ are both constants, with α potentially taking the value 1.5 and γ potentially taking the value 0.5.

[0146] Furthermore, if the target speaker feature extraction network is trained with the target speaker's identifier as a supervision condition, then the loss function of the speech enhancement model also needs to include a cross-entropy loss component.

[0147] As can be seen, in this embodiment, the loss function used by the speech enhancement model during the training phase includes four parts: the basic loss part used to train the speech enhancement model. The auxiliary loss component when the sound signal contains only echoes or noise (i.e., not human voices). Reduce the asymmetric loss component that addresses the oversuppression of output during speech enhancement model training. The supervised loss part corresponding to the target speaker feature extraction network

[0148] Therefore, unlike traditional speech enhancement models that use a two-stage model training strategy (pre-training - fine-tuning training), the speech enhancement model in this embodiment can be obtained through a single-stage training. Furthermore, both the target speaker feature extraction network and the time-frequency narrowband neural network in this speech enhancement model employ lightweight structures with low complexity (including but not limited to the use of one-dimensional convolutional layers), making the overall speech enhancement model lightweight (e.g., less than 1 million parameters). Although this speech enhancement model uses a time-frequency narrowband neural network, it can also be used to process full-band signals and can be directly used in high-quality RTC applications, exhibiting better applicability and a wider range of application scenarios.

[0149] The trained speech enhancement model can effectively implement the aforementioned speech enhancement scheme and is easy to deploy on devices with limited resources.

[0150] Referring to FIG6, a schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0151] As shown in Figure 6, the electronic device may include: a processor 602, a communications interface 604, a memory 606, and a communications bus 608.

[0152] in:

[0153] The processor 602, communication interface 604, and memory 606 communicate with each other via communication bus 608.

[0154] Communication interface 604 is used for communication with other electronic devices or servers.

[0155] The processor 602 is used to execute program 610, specifically to perform the relevant steps in the above method embodiments.

[0156] Specifically, program 610 may include program code that includes computer operation instructions.

[0157] The processor 602 may be a CPU, a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0158] Memory 606 is used to store program 610. Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0159] Program 610 may include multiple computer instructions. Specifically, program 610 may use multiple computer instructions to cause processor 602 to perform the operation corresponding to any of the methods described in the foregoing multiple method embodiments.

[0160] The specific implementation of each step in program 610 can be found in the corresponding steps and units described in the above method embodiments, and has corresponding beneficial effects, which will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0161] This application also provides a computer storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any of the foregoing method embodiments. The computer storage medium includes, but is not limited to, compact disc read-only memory (CD-ROM), random access memory (RAM), floppy disk, hard disk, or magneto-optical disk.

[0162] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the methods in the above-described multiple method embodiments.

[0163] Furthermore, it should be noted that the user-related information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to sample data used for training the model, data used for speech enhancement processing, data used for analysis, stored data, and displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0164] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0165] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an Application Specific Integrated Circuit (ASIC) or a Field Programmable Gate Array (FPGA)). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., Random Access Memory (RAM), Read-Only Memory (ROM), Flash Memory, etc.) capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the methods shown herein.

[0166] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for specific applications, but such implementations should not be considered beyond the scope of the embodiments of this application.

[0167] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

A speech enhancement method includes: Acquire speech data samples of the target speaker, and extract spectral features from the speech data samples based on a bidirectional long short-term memory network layer and an attention statistical pooling layer to obtain the spectral features of the target speaker; The spectral features are input into the feature linear modulation layer in the time-frequency narrowband neural network. The feature linear modulation layer adjusts the spectral features corresponding to the mixed speech data extracted by the time-frequency narrowband neural network based on the spectral features. The mixed speech data includes speech data of multiple speakers, including the target speaker. Based on the adjusted spectral features, speech enhancement processing is performed on the speech data of the target speaker in the mixed speech data to obtain enhanced speech data of the target speaker. According to the method of claim 1, wherein, The extraction of spectral features from the speech data samples based on a bidirectional long short-term memory network layer and an attention statistical pooling layer includes: The speech data sample is input into a target speaker feature extraction network that is independent of the time-frequency narrowband neural network. The speech data sample is subjected to spectral feature extraction at least through the bidirectional long short-term memory network layer and the attention statistical pooling layer in the target speaker feature extraction network. The bidirectional long short-term memory network layer is used to extract spectral features from the speech data samples, and the attention statistical pooling layer is used to perform attention-based autocorrelation statistical calculations based on the input spectral features, so that the target speaker feature extraction network outputs the spectral features of the target speaker to the feature linear modulation layer based on the results of the statistical calculations. The method according to claim 2, wherein, The step of inputting the speech data samples into a target speaker feature extraction network that is independent of the time-frequency narrowband neural network includes: The speech data samples are subjected to a short-time Fourier transform, and the result of the short-time Fourier transform is input into the target speaker feature extraction network, which is set independently of the time-frequency narrowband neural network. The method according to claim 3, wherein, The step of extracting spectral features from the speech data samples by at least the bidirectional long short-term memory network layer and the attention statistical pooling layer in the target speaker feature extraction network includes: The result of the short-time Fourier transform is downsampled by a one-dimensional convolutional layer in the target speaker feature extraction network to obtain downsampled features; Based on the downsampling features, spectral features are extracted through the bidirectional long short-term memory network layer and the attention statistical pooling layer. The method according to any one of claims 1-4, wherein, The target speaker feature extraction network further includes a classification output layer connected after the attention statistical pooling layer, which is used to classify the results output by the attention statistical pooling layer and output the target speaker identifier corresponding to the speech data sample. The method according to any one of claims 1-5, wherein, The step of adjusting the spectral features corresponding to the mixed speech data extracted by the time-frequency narrowband neural network based on the spectral features through the feature linear modulation layer includes: Through the feature linear modulation layer, the spectral features corresponding to the mixed speech data extracted by the time-frequency narrowband neural network are adjusted according to the input spectral features. The method according to any one of claims 1-6, wherein, The spectral features corresponding to the mixed speech data are obtained through the following methods: The mixed speech data is determined based on the original signal acquired by the microphone, the far-end sound signal, and the filtering result of the adaptive filter used to filter the echo signal in the original signal. The original signal includes the sound signals of multiple speakers, including the target speaker. Perform a short-time Fourier transform on the mixed speech data; The result of the short-time Fourier transform is downsampled through a one-dimensional convolutional layer in the time-frequency narrowband neural network to obtain downsampled feature data. Feature extraction is performed based on the downsampled feature data to obtain the spectral features corresponding to the mixed speech data. The method according to any one of claims 1-7, wherein, The method further includes: If the obtained spectral features of the target speaker are unavailable, then the pseudo-voiceprint information is used as the speech data sample. Based on the bidirectional long short-term memory network layer and the attention statistical pooling layer, the spectral features of the speech data sample are extracted so that the spectral features of the obtained pseudo-voiceprint information can be used as the spectral features of the feature linear modulation layer in the time-frequency narrowband neural network. A method for training a speech enhancement model, comprising: Acquire training samples, which include: mixed speech data samples and speech data samples of the target speaker, wherein the mixed speech data samples are speech data samples that include interference signals and speech of multiple speakers, including the target speaker; Using the training samples, the speech enhancement model includes a time-frequency narrowband neural network and a target speaker feature extraction network, which are jointly trained to obtain a target speaker feature extraction network that can extract the spectral features of the target speaker, and a time-frequency narrowband neural network that obtains the spectral features output by the target speaker feature extraction network through a feature linear modulation layer, and performs speech enhancement on the mixed speech data for the target speaker based on the spectral features. The method according to claim 9, wherein, The step of jointly training the speech enhancement model, comprising a time-frequency narrowband neural network and a target speaker feature extraction network, using the training samples includes: Using the training samples, the time-frequency narrowband neural network and the target speaker feature extraction network of the speech enhancement model are jointly trained based on a preset loss function. The preset loss function includes: a basic loss component for training the speech enhancement model, an auxiliary loss component for processing sound signals that do not include human voices, an asymmetric loss component for reducing the output suppression of the speech enhancement model, and a supervised loss component corresponding to the target speaker feature extraction network. The method according to claim 9 or 10, wherein, The target speaker feature extraction network includes at least: a bidirectional long short-term memory network layer and an attention statistical pooling layer; The bidirectional long short-term memory network layer is used to extract spectral features from the speech data samples, and the attention statistical pooling layer is used to perform attention-based autocorrelation statistical calculations based on the input spectral features, so that the target speaker feature extraction network outputs the spectral features of the target speaker to the feature linear modulation layer based on the results of the statistical calculations. A speech processing method, comprising: The target speaker is identified through a pre-set detection strategy; Obtain a sample of the target speaker's voice data; The spectral features of the target speaker are obtained by extracting spectral features from the speech data samples through at least a bidirectional long short-term memory network layer and an attention statistical pooling layer. The method according to claim 12, wherein, The detection and identification of the target speaker through a preset detection strategy includes at least one of the following: Detect whether a target speaker identity request has been received through the human-computer interaction interface. If so, determine the speaker requested by the target speaker identity as the target speaker. The system detects whether the continuous speaking duration of the same speaker, captured by the voice acquisition device, exceeds a preset duration. If it does, the speaker is identified as the target speaker. The system detects whether the number of times the same speaker's voice appears in multiple historical audio or video conferences has reached a preset number. If so, the speaker is identified as the target speaker. A face video image is acquired by an image acquisition device, and lip movement is detected on the face video image. If the duration of the lip movement reaches a preset duration and the continuity of the lip movement is consistent with the continuity of the speech acquired by the speech acquisition device, then the speaker corresponding to the face video image is identified as the target speaker. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-13. A computer program product includes computer instructions that instruct a computing device to perform an operation corresponding to any one of the methods described in claims 1-13. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of claims 1-13.

Images