Method and apparatus for training speech enhancement network, and storage medium, device and product

Abstract

Disclosed in the present application is a method for training a speech enhancement network, the method comprising: acquiring a training set, wherein the training set comprises a plurality of training samples, and each training sample comprises sample reference speech, sample comparison speech and mixed speech; performing voiceprint extraction on the sample reference speech to obtain a sample voiceprint vector; performing audio feature extraction on the mixed speech to obtain a sample audio feature; performing, by means of a speech enhancement network, enhancement processing on the basis of the sample voiceprint vector and the sample audio feature, and outputting a predicted audio feature for a sample sound-producing object; on the basis of the predicted audio feature and a comparative audio feature corresponding to the sample comparison speech, determining a training loss value of the speech enhancement network; and iteratively updating weight parameters of the speech enhancement network on the basis of the training loss value until a training ending condition is met. The present application is applied to artificial intelligence, and by means of the present application, a speech enhancement network for suppressing interfering human voices can be trained on the basis of sample reference speech and sample comparison speech which originate from the same sound-producing object, thereby effectively improving the quality of speech enhancement.

Description

Training method, device, storage medium, equipment and product for speech enhancement network

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 9, 2023, with application number 2023109993625 and application name “Training method, device, storage medium and computer equipment for speech enhancement network”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and more specifically, to a training method, apparatus, storage medium, computer equipment, and computer program product for a speech enhancement network. Background Art

[0003] Speech enhancement is essentially speech noise reduction. In everyday life, the speech captured by microphones is often "contaminated" with various noises. The main purpose of speech enhancement is to recover the desired clean speech from this "contaminated" noisy speech, effectively suppressing various interfering signals and enhancing the target speech signal, thereby improving speech audio quality. Speech enhancement has applications in areas such as video conferencing and speech recognition, and serves as a preprocessing module for many speech coding and recognition systems.

[0004] In complex speech acquisition environments, existing technologies typically achieve noise reduction by suppressing noise in the captured speech. For example, spectral subtraction is used to estimate the speech spectrum, noise is estimated using a Gaussian mixture model, or a noise reduction neural network is used to learn the spectrum of clean, noise-free speech. However, the enhanced speech obtained by existing speech enhancement techniques can be ineffective. Therefore, improving speech enhancement is a pressing technical issue in related technologies.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide a method, apparatus, storage medium, and computer device for training a speech enhancement network to address the problem of poor speech enhancement results in related technologies.

[0007] On the one hand, an embodiment of the present application provides a training method for a speech enhancement network, the method comprising: obtaining a training set, the training set comprising multiple training samples, a training sample comprising a sample reference speech, a sample comparison speech and a mixed speech; the mixed speech is obtained by mixing interfering human voices, environmental noise and the sample comparison speech; wherein the sample reference speech and the sample comparison speech in the same training sample are derived from the same sample sound-producing object; performing voiceprint extraction on the sample reference speech to obtain a sample voiceprint vector; performing audio feature extraction on the mixed speech to obtain a sample audio feature; performing enhancement processing by the speech enhancement network based on the sample voiceprint vector and the sample audio feature, and outputting predicted audio features for the sample sound-producing object; determining the training loss value of the speech enhancement network based on the predicted audio features and the compared audio features corresponding to the sample comparison speech; and iteratively updating the weight parameters of the speech enhancement network based on the training loss value until the training end condition is met.

[0008] The embodiment provided by the present application can obtain a training set, which includes multiple training samples, and the training samples include sample reference speech, sample comparison speech and mixed speech. Then, voiceprint extraction is performed on the sample reference speech to obtain a sample voiceprint vector, and audio feature extraction is performed on the mixed speech to obtain sample audio features. Furthermore, the speech enhancement network performs enhancement processing based on the sample voiceprint vector and the sample audio features, outputs the predicted audio features for the sample sound object, and determines the training loss value of the speech enhancement network based on the predicted audio features and the comparison audio features corresponding to the sample comparison speech. Therefore, the weight parameters of the speech enhancement network are iteratively updated based on the training loss value until the training end condition is reached.

[0009] In this way, the sample voiceprint vector of the sample sound-making object is added to the input data of the speech enhancement network, so that the speech enhancement network can increase its attention to learning the sound information of the sample sound-making object during the training process, and make the speech enhancement network more focused on enhancing the sound of the sample sound-making object. In addition to removing interfering noise, it can also effectively suppress interfering human voices at the same time, thereby improving the quality and performance of speech enhancement of the trained speech enhancement network.

[0010] On the other hand, an embodiment of the present application provides a speech enhancement method, which includes: obtaining a target voiceprint vector of a target sound-making object; inputting the target voiceprint vector and the audio features of the target speech into a speech enhancement network for enhancement processing to obtain enhanced audio features for the target sound-making object; the speech enhancement network is obtained according to the training method of the speech enhancement network provided in the above embodiment; performing speech reconstruction on the enhanced audio features to obtain enhanced speech corresponding to the target speech.

[0011] On the other hand, an embodiment of the present application also provides a training device for a speech enhancement network, which includes: a sample acquisition module for acquiring a training set, the training set including multiple training samples, a training sample including a sample reference speech, a sample comparison speech and a mixed speech; the mixed speech is obtained by mixing interfering human voice, environmental noise and the sample comparison speech; wherein the sample reference speech and the sample comparison speech in the same training sample come from the same sample sound object; a voiceprint extraction module for performing voiceprint extraction on the sample reference speech to obtain a sample voiceprint vector; a feature extraction module for performing audio feature extraction on the mixed speech to obtain a sample audio feature; a feature prediction module for performing enhancement processing based on the sample voiceprint vector and the sample audio feature by the speech enhancement network, and outputting the predicted audio feature for the sample sound object; a loss determination module for determining the training loss value of the speech enhancement network based on the predicted audio feature and the comparison audio feature corresponding to the sample comparison speech; a parameter update module for iteratively updating the weight parameters of the speech enhancement network based on the training loss value until the training end condition is reached.

[0012] On the other hand, an embodiment of the present application also provides a speech enhancement device, which includes: a vector acquisition module for acquiring a target voiceprint vector of a target sound-making object; a speech enhancement module for inputting the target voiceprint vector and the audio features of the target speech into a speech enhancement network for enhancement processing to obtain enhanced audio features for the target sound-making object; the speech enhancement network is trained according to the training method of the speech enhancement network provided in the above embodiment; and a speech reconstruction module for performing speech reconstruction on the enhanced audio features to obtain enhanced speech corresponding to the target speech.

[0013] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed by a processor, the above-mentioned training method of the speech enhancement network is executed.

[0014] On the other hand, an embodiment of the present application further provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is called by the processor, the computer program executes the above-mentioned training method of the speech enhancement network.

[0015] On the other hand, an embodiment of the present application also provides a computer program product, which includes a computer program stored in a storage medium; a processor of a computer device reads the computer program from the storage medium, and the processor executes the computer program, so that the computer device performs the steps in the above-mentioned training method of the speech enhancement network. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG1 shows a schematic diagram of a system framework provided in an embodiment of the present application.

[0017] FIG2 shows another schematic diagram of a system framework provided in an embodiment of the present application.

[0018] FIG3 shows a flow chart of a method for training a speech enhancement network provided in an embodiment of the present application.

[0019] FIG4 shows a schematic diagram of the architecture of a speech enhancement network provided in an embodiment of the present application.

[0020] FIG5 shows a flow chart of a speech enhancement method provided in an embodiment of the present application.

[0021] FIG6 shows an application scenario diagram provided by an embodiment of the present application.

[0022] FIG7 shows a flow chart of extracting a target voiceprint vector provided in an embodiment of the present application.

[0023] FIG8 shows a flow chart of speech enhancement provided by an embodiment of the present application.

[0024] FIG9 is a module block diagram of a training device for a speech enhancement network provided in an embodiment of the present application.

[0025] FIG10 is a module block diagram of a speech enhancement device provided in an embodiment of the present application.

[0026] FIG11 is a module block diagram of a computer device provided in an embodiment of the present application.

[0027] FIG12 is a module block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application and are not to be construed as limiting the present application.

[0029] Some processes described in the specification, claims, and accompanying figures include multiple steps that appear in a specific order. However, it should be understood that these steps may be performed in a different order or in parallel. Step numbers are used solely to distinguish between different steps and do not inherently indicate an order of execution. Furthermore, terms such as "first" and "second" are used to distinguish similar items and do not necessarily describe a specific order or precedence.

[0030] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0031] It should be noted that in the specific implementation of this application, the sample reference voice, sample comparison voice, registered voice, recorded voice and other related data, when applied to the specific products or technologies of the embodiments of this application, need to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and carry out subsequent data use and processing within the scope of authorization of laws, regulations and personal information subjects.

[0032] The training method of the speech enhancement network proposed in this application involves artificial intelligence (AI) technology, and the speech enhancement network is used for speech processing. The key technologies of speech processing technology (Speech Technology) include automatic speech recognition technology (Automatic Speech Recognition, ASR) and speech synthesis technology (Text To Speech, TTS) and voiceprint recognition technology (VPR). Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction. In the embodiment of the present application, based on the speech enhancement technology in speech processing, the speaker's voiceprint can be extracted from the collected speech and the noise of the speech can be reduced.

[0033] Currently, related technologies typically achieve noise reduction by suppressing noise in the collected speech. For example, speech enhancement methods based on spectral subtraction exploit the fact that additive noise is uncorrelated with speech. Under the assumption that the noise is statistically stationary, the noise spectrum estimated from the non-speech segments replaces the spectrum of the noise during speech. This spectrum is then subtracted from the noisy speech spectrum to obtain an estimated speech spectrum.

[0034] Speech enhancement methods based on Gaussian Mixture Models (GMMs) use GMMs to estimate background noise and spectral subtraction coefficients, then perform spectral subtraction on the noisy speech to restore the clean speech. The noisy speech is preprocessed to obtain its corresponding amplitude and phase. The amplitude is used for noise estimation and spectral subtraction, while the phase is used to restore the time domain signal. Furthermore, the GMM is used to estimate noise parameters and clean speech cepstral features in real time from the noisy speech. Spectral subtraction coefficients are calculated based on the estimated clean speech cepstral features. Spectral subtraction is then performed on the spectrum of the noisy speech to restore the time domain signal and produce the enhanced speech.

[0035] In a speech enhancement method based on a deep neural network (DNN), a noisy speech spectrum is input into a deep neural network, such as a recurrent neural network (RNN) or a convolutional neural network (CNN). The learning target of the network training is a clean speech spectrum. By inputting the collected speech into the trained enhancement network, the enhancement network can directly output a speech spectrum that effectively suppresses stationary and non-stationary noise. However, the above-mentioned speech enhancement method cannot suppress the background interference human voice in the collected speech, and the quality of speech enhancement is low. In order to solve the above problems, the inventor has proposed a training method for the speech enhancement network provided in the embodiment of the present application after research.

[0036] The following first introduces the system architecture of the speech enhancement network training method involved in this application.

[0037] As shown in Figure 1, the training method of the speech enhancement network provided in the embodiment of the present application can be applied in a system 100, and the system 100 can be used for model training. Among them, the data acquisition device 110 is used to obtain a training set, and the training set includes multiple training samples. For the training method of the speech enhancement network in the embodiment of the present application, each training sample may include a sample reference speech, a sample comparison speech, and a mixed speech. Among them, the sample reference speech and the sample comparison speech in the same training sample come from the same sample sound object. The training set can be used to train a target model 101 for speech enhancement of the collected user speech. After acquiring the training data, the data acquisition device 110 can store the training data in the database 120, and the training device 130 can train the target model 101 based on the training set maintained in the database 120.

[0038] Specifically, the training device 130 can train a preset speech enhancement network based on the input training data until the speech enhancement network meets the preset training end conditions, thereby obtaining the trained target model 101, that is, the speech enhancement network of the present application. The training end conditions may be: the loss value of the target loss function (that is, the training loss value) is less than a preset value, the loss value of the target loss function (that is, the training loss value) no longer changes, or the number of training times reaches a preset number, etc. The target model 101 can be used to automatically perform speech enhancement based on the input voiceprint vector and speech audio features of the target user (referred to as the target sound object in this application), and obtain enhanced audio features for the target sound object. The processing involved in the target model 101 may include audio feature extraction, etc. The target model 101 in the embodiment of the present application may be a deep neural network (DNN), wherein the network structure may include a long short-term memory network (LSTM) and a fully connected layer (Fully Connected Layer), a convolutional neural network (CNN), etc., which are not limited here.

[0039] In actual application scenarios, the training data maintained in the database 120 does not necessarily all come from the data acquisition device 110, but may also be received from other devices. For example, the execution device 140 may also serve as a data acquisition terminal, using the acquired data as new training data and storing it in the database 120. In addition, the training device 130 does not necessarily train the preset neural network entirely based on the training data maintained by the database 120, but may also train the preset neural network based on training data obtained from the cloud or other devices. For example, when the execution device 140 is the terminal where the client is located, the collected user voice may be used as training data. The above description should not be regarded as a limitation on the embodiments of the present application.

[0040] The target model 101 obtained by training the training device 130 can be applied to various systems or devices, such as the execution device 140 shown in FIG1 . The training device 130 and the execution device 140 can be servers or terminals, etc. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, blockchain, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, etc.

[0041] During the execution of calculations and other related processing by the processing module 141 of the execution device 140, the execution device 140 may call upon data, programs, etc. in the data storage system 150 for the corresponding calculations, and store the processing results and other data and instructions obtained from the calculations in the data storage system 150. The training device 130 may generate corresponding target models 101 based on different training data for different objectives or tasks. The corresponding target models 101 may be used to complete the training tasks of the corresponding speech enhancement network and the speech enhancement tasks performed using the speech enhancement network.

[0042] For example, the training device 130 in the system 100 shown in FIG1 can be a cloud server deployed by a service provider, and the execution device 140 can be a terminal used by a user (e.g., a smartphone). The cloud server can perform network training based on the training set to obtain a speech enhancement network for performing the speech enhancement task. The speech enhancement network can include a first long short-term memory subnetwork and a first fully connected subnetwork. Furthermore, the terminal can deploy the trained speech enhancement network, i.e., the target model 101, to perform the speech enhancement task.

[0043] For example, in a video conferencing scenario, when a speaker in a conference room speaks, the video conferencing client in that conference room can perform voice enhancement on the collected speaker's speech according to the method of this application. Specifically, the client can obtain the speaker's voiceprint vector and input the voiceprint vector and the audio features of the speech into a speech enhancement network. Then, the speech enhancement network can output the clean speech features after noise reduction for the speaker, and perform speech reconstruction on the clean speech features to obtain the corresponding clean speech after noise reduction, so that the clean speech can be sent to other conference rooms for playback.

[0044] It is worth noting that Figure 1 is merely a schematic diagram of the architecture of a system provided by an embodiment of the present application. The system architecture and application scenarios described in the embodiment of the present application are intended to more clearly illustrate the technical solutions of the embodiment of the present application and do not constitute a limitation on the technical solutions provided by the embodiment of the present application. For example, in other cases, the training device 130 in Figure 1 can also be a terminal. The execution device 140 can also be a cloud server deployed by a service provider.

[0045] As shown in FIG2 , the training method of the speech enhancement network provided in the embodiment of the present application can also be applied in the system 200. For example, the functions and application scenarios of the data acquisition device 210, the database 220, the training device 230 and the database system 250 in the system 200 can be the same as those of the data acquisition device 110, the database 120, the training device 130 and the database system 150 in the system 100. The execution device 240 in the system 200 can be a cloud execution server, which deploys the speech enhancement network trained by the cloud training server (i.e., the training device 230) and can run the speech enhancement network to perform speech enhancement tasks in collaboration with the client device 260. In one possible embodiment, as shown in FIG2 , the execution device 240 can include a processing module 241, which can perform speech enhancement processing on the collected user speech through the target model 201 to complete the speech enhancement task.

[0046] For example, a user can install and use an audio sharing client on a laptop (i.e., client device 260). When the user uses the broadcast function of the audio sharing client on the audio sharing client, the laptop can send the live voice collected by the user when using the broadcast function to the cloud execution server through the network. Furthermore, when the cloud execution server receives the live voice, it uses the voice enhancement network to perform voice enhancement based on the user's voiceprint vector and the audio features of the live voice, and outputs a clean voice after noise reduction. In addition, the cloud execution server can send the clean voice to the audio sharing client of the user listening to the broadcast.

[0047] Please refer to Figure 3, which shows a flow chart of a method for training a speech enhancement network provided by one embodiment of the present application. In this embodiment, the method for training the speech enhancement network can be executed by a server, which has at least storage, computing, and communication functions. As shown in Figure 3, the method for training the speech enhancement network can specifically include the following steps:

[0048] Step S110: Obtain a training set, which includes multiple training samples. A training sample includes a sample reference speech, a sample comparison speech, and a mixed speech; the mixed speech is obtained by mixing interfering human voices, environmental noise, and the sample comparison speech; the sample reference speech and the sample comparison speech in the same training sample come from the same sample sound-producing object.

[0049] In the usage scenarios of speech enhancement technology, the speech to be enhanced that needs to be speech enhanced usually includes interfering human voices. For example, the speaking voice of a classmate collected by the online classroom client, in addition to the environmental noise, also includes the speaking voices of other classmates, that is, interfering human voices. Considering that the relevant technology cannot effectively suppress interfering human voices during the speech enhancement process, for this reason, the present application proposes to train a speech enhancement network based on a sample reference speech and a sample comparison speech with the same sample sound object, so as to use the trained speech enhancement network to perform speech enhancement on the speech to be enhanced that can suppress interfering human voices.

[0050] In the implementation of this application, each training sample may include a sample reference speech, a sample comparison speech (ie, a clean speech), and a mixed speech. The mixed speech may be obtained by mixing interfering human voices, environmental noise, and sample comparison speech in different proportions. Optionally, the mixed speech may be obtained according to the following formula: S mix =αS1+βS2+γS3

[0051] Among them, S mix represents the mixed speech, S1 represents the interfering human voice, S2 represents the ambient noise, and S3 represents the sample comparison speech. α, β, and γ are scaling parameters, with α, β, and γ ∈ (0, 1). It is worth noting that in the same training sample, the speaker of the sample reference speech and the speaker of the sample comparison speech are the same person, while the speakers of the sample reference speech, sample comparison speech, and interfering human voice are different people.

[0052] In some embodiments, the sample reference speech and the sample comparison speech in the training sample can be the speech recorded by the sample sounding object. One goal of the training of the speech enhancement network in this embodiment can be to make the speech audio features predicted by the network based on the sample reference speech and the audio features of the real speech (i.e., the sample comparison speech) as close as possible. In order to ensure the training effect of the speech enhancement network, the duration of the sample reference speech and the sample comparison speech can be set to meet the duration threshold range. The duration threshold range is used to ensure that the sample reference speech and the sample comparison speech can ensure that the duration of the speech is long, thereby ensuring that accurate features can be extracted from the sample reference speech and the sample comparison speech to guide the speech enhancement model to perform speech enhancement. For example, the minimum value of the duration threshold range can be 30 seconds. The duration threshold range can be obtained through experimental calculation based on the actual network training requirements and is not limited here.

[0053] Furthermore, the speech content of the sample reference speech and the sample comparison speech can be different. For example, the content of the sample reference speech and the sample comparison speech can be two different news articles read by the same speaker. The speech content can include as many words with different pronunciations as possible, so that the sample reference speech and the sample comparison speech cover more speech information, improving the accuracy and confidence of network training. Of course, in other embodiments, the speech content of the sample reference speech and the sample comparison speech in the same training sample can also be the same.

[0054] In the implementation of this application, sample reference speech, sample comparison speech, and interfering human voice can be randomly extracted from the speech library, and environmental noise can be randomly extracted from the noise library. Furthermore, the interfering human voice, environmental noise, and sample comparison speech are combined in different proportions to obtain mixed speech, and training samples are formed based on the sample reference speech, sample comparison speech, and mixed speech. In this way, n training samples can be obtained, and then a training set {x 0 ,x 2 ,…,x n}, where n>0&n∈N*. Optionally, the training set is stored in a database.

[0055] As an implementation method, when training the speech enhancement network, the server can obtain the training set {x 0 ,x 2 ,…,x n}.

[0056] Step S120: extracting the voiceprint of the sample reference speech to obtain a sample voiceprint vector.

[0057] As an implementation method, the server can perform time-frequency conversion on the sample reference speech to obtain the frequency domain features of the sample reference speech. Further, the frequency domain features of the sample reference speech are input into the voiceprint extraction network for voiceprint extraction to obtain the sample voiceprint vector.

[0058] For example, after obtaining the sample reference speech, the sample reference speech can be framed and windowed, and then time-frequency conversion can be performed to obtain the corresponding frequency domain features. Specifically, the sample reference speech collected by the microphone is framed and windowed in turn to obtain the speech signal frame of the sample reference speech, and the speech signal frame is subjected to Fast Fourier Transformation (FFT) and the discrete power spectrum after FFT is obtained, and then the obtained discrete power spectrum is logarithmically calculated to obtain the logarithmic power spectrum as the frequency domain feature of the sample reference speech.

[0059] In some embodiments, the voiceprint extraction network may include a second long short-term memory subnetwork, a second fully connected subnetwork, and a pooling subnetwork. Specifically, the frequency domain features of the sample reference speech may be input into the second long short-term memory subnetwork for feature extraction to obtain a first voiceprint feature, and the first voiceprint feature may be input into the second fully connected subnetwork for full connection processing to obtain a second voiceprint feature. Furthermore, the second voiceprint feature may be input into the pooling subnetwork for pooling processing to obtain a sample voiceprint vector. It is worth mentioning that the structure of the voiceprint extraction network is not limited to that listed above. In other embodiments, the voiceprint feature extraction network may also be constructed using other neural networks, such as convolutional neural networks, fully connected neural networks, etc., which are not specifically limited here.

[0060] Step S130: extracting audio features from the mixed speech to obtain sample audio features.

[0061] In the embodiment of the present application, the sample audio features are acoustic features obtained based on mixed speech conversion, such as Logarithmic Power Spectrum (LPS) and Mel-Frequency Cepstral Coefficients (MFCC), etc., which are not limited here.

[0062] Unlike image data, speech data often cannot be directly fed into model training. Its features lack significant long-term temporal variation, making it difficult to learn speech data features. Furthermore, speech data in the time domain typically consists of 16K sampling points, meaning 16,000 samples per second. Directly feeding these time-domain sampling points results in excessively large training data volumes and makes it difficult to achieve meaningful training results. Therefore, in speech enhancement tasks, speech data can be converted into acoustic features as input or output for the network.

[0063] As an implementation method, the server can perform framing, windowing, and Fast Fourier Transform (FFT) on the mixed speech to obtain sample audio features. This transforms the mixed speech from a non-stationary, time-varying signal in the time domain into a stationary signal in the frequency domain, facilitating the training of the speech enhancement network.

[0064] Step S140: The speech enhancement network performs enhancement processing based on the sample voiceprint vector and the sample audio features, and outputs the predicted audio features for the sample sound object. That is, the speech enhancement network performs speech enhancement processing on the sample voiceprint vector and the sample audio features, and outputs the predicted audio features for the sample sound object.

[0065] In actual application scenarios, the voice collected by the microphone includes not only the voice of the target sound-making object and environmental noise, but also the interfering human voices of other sound-making objects. In order to suppress both environmental noise and interfering human voices during the speech enhancement process, this application adds a sample voiceprint vector of the sample sound-making object to the input of the network training to remove environmental noise and interfering human voices other than the voice of the sample sound-making object.

[0066] The speech enhancement network performs speech enhancement processing based on the sample voiceprint vector and the sample audio features, and outputs the predicted audio features for the sample sound object. The predicted audio features can be regarded as the audio features of the mixed speech after suppressing the interfering human voice and environmental noise. In the present application, the sample voiceprint vector is input into the speech enhancement network, and the speech enhancement network can be supervised to separate the features related to the speech of the sample sound object in the sample audio features based on the sample voiceprint vector, thereby suppressing the features of the environmental noise and interfering human voice in the sample audio features, thereby realizing speech enhancement of the mixed speech. In one embodiment, the sample voiceprint vector and the sample audio features can be used as the input of the speech enhancement network, and the predicted audio features for the sample sound object are output after being processed by the speech enhancement network.

[0067] Please refer to Figure 4, which shows a schematic diagram of the architecture of a speech enhancement network. The speech enhancement network may include a first long short-term memory subnetwork (corresponding to the long short-term memory subnetwork of the sample voiceprint vector and sample audio feature input in Figure 4) and a first fully connected subnetwork (corresponding to the fully connected subnetwork in Figure 4). The number of layers in the first long short-term memory subnetwork and the first fully connected subnetwork can be set according to specific training requirements. The speech features have a time series sequence with short-term stability, which is consistent with the long short-term memory capacity of the long short-term memory network, thereby improving the quality of speech enhancement. Optionally, the first long short-term memory subnetwork can also be a bidirectional long short-term memory network (Bi-directional Long-Short Term Memory, Bi-LSTM), which is not limited here.

[0068] As an implementation method, the server can input the sample voiceprint vector and sample audio features into the first long short-term memory sub-network for feature extraction to obtain intermediate features. Furthermore, the intermediate features are input into the first fully connected sub-network for full connection processing to obtain the predicted audio features of the sample sound object.

[0069] Step S150: Determine the training loss value of the speech enhancement network based on the predicted audio features and the compared audio features corresponding to the sample compared speech. The training loss value may also be referred to as the target loss, which is the loss value of the target loss function.

[0070] The comparison audio features corresponding to the sample comparison speech may be obtained by extracting features from the sample comparison speech. In some embodiments, the comparison audio features may be frequency domain features obtained by performing time-frequency conversion on the sample comparison speech. For example, the discrete power spectrum of the sample comparison speech may be obtained by fast Fourier transform.

[0071] In an embodiment of the present application, the learning goal when training the speech enhancement network is to make the predicted audio features output by the speech enhancement network and the comparison audio features corresponding to the sample comparison speech as close as possible in the embedding space, that is, to enable the speech enhancement network to predict predicted audio features that are closer to the clean comparison audio features used as labels.

[0072] The training loss value is used to measure the difference between the predicted audio features and the compared audio features corresponding to the sampled speech. The loss function used to calculate this training loss value can be designed during the speech enhancement network training process. During the training process, the parameters of the speech enhancement network are continuously adjusted using an optimization algorithm (such as gradient descent) to minimize the training loss value output by the loss function, thereby completing the training of the speech enhancement network.

[0073] As an implementation method, the mean square error (MSE) between the predicted audio features and the compared audio features corresponding to the sample comparison speech can be calculated as the training loss value of the speech enhancement network. The calculation formula is as follows:

[0074] Among them, Loss(τ) represents the training loss value of the speech enhancement network, τ represents the weight parameter of the speech enhancement network, and y i represents the i-th training sample (x i , ) corresponding predicted audio features, Represents the comparison audio features of the sample comparison speech corresponding to the i-th training sample.

[0075] In other embodiments, other loss functions (such as cross entropy loss function, absolute value loss function, etc.) can also be used to determine the training loss value of the speech enhancement network based on the predicted audio features and the compared audio features corresponding to the sample compared speech. In some embodiments, at least two different loss functions can be used to calculate the loss under each loss function respectively based on the predicted audio features and the compared audio features corresponding to the sample compared speech. Afterwards, the losses under the at least two different loss functions are weighted, and the weighted processing result is used as the training loss value of the speech enhancement network.

[0076] Step S160: Based on the training loss value, iteratively update the weight parameters of the speech enhancement network until the training end condition is reached.

[0077] In an embodiment of the present application, the training end conditions may include: the loss value of the training loss value is less than a preset value, the loss value of the training loss value no longer changes, or the number of training times reaches a preset number, etc. Optionally, an optimizer can be used to optimize the target loss function, and the learning rate, the Batch Size (batch size) during training, and the Epoch (period) of training can be set based on experimental experience. The training process described in S110 to S150 will obtain a training loss value. If the obtained training loss value meets the training requirements, for example, the training loss value is less than a preset threshold, or the training loss value no longer changes or the change range is very small, then it can be considered that the speech enhancement network in the current situation has been trained. If the training requirements are not met, the weight parameters of the speech enhancement network are iteratively updated until the training end conditions are met.

[0078] As an implementation method, the speech enhancement network is iteratively trained for multiple training cycles based on the training set, where each training cycle can include multiple iterative trainings, and the weight parameters of the speech enhancement network are continuously optimized. The above total loss value becomes smaller and smaller, and finally becomes smaller to a fixed value, or is less than the above preset value. At this time, it means that the speech enhancement network has converged, and the iterative update of the weight parameters of the speech enhancement network is stopped, and the network training is ended.

[0079] Optionally, at the end of network training, the speech enhancement performance of the trained speech enhancement network can be compared with that of an existing deep neural network for speech enhancement. The comparison metric can be the Noise Mean Option Score (NMOS). A larger NMOS value indicates better speech enhancement performance.

[0080] In this embodiment, voiceprint extraction can be performed on the sample reference speech in the training sample to obtain a sample voiceprint vector, and audio feature extraction can be performed on the mixed speech in the training sample to obtain a sample audio feature. The speech enhancement network then performs enhancement processing based on the sample voiceprint vector and the sample audio features, outputting predicted audio features for the sample sound object. Furthermore, the training loss value of the speech enhancement network is determined based on the predicted audio features and the compared audio features corresponding to the sample comparison speech. Based on the training loss value, the weight parameters of the speech enhancement network are iteratively updated until the training end condition is met.

[0081] In this way, the sample voiceprint vector of the sample sound-making object is added to the input data of the speech enhancement network, so that the speech enhancement network can improve its learning attention to the sound information of the sample sound-making object during the training process, and make the speech enhancement network more focused on the sound of the sample enhanced sound-making object. In addition to removing interference noise, it can also effectively suppress interfering human voices, so that the trained speech enhancement network can be used to specifically enhance the speech of the specified sound-making object, effectively suppress environmental noise and interference noise, and improve the quality and performance of speech enhancement.

[0082] In addition, the speech enhancement network in this application uses long short-term memory networks and fully connected networks as the basic structure (Backbone), which effectively reduces the time complexity and space complexity of the entire network structure, making the speech enhancement network more lightweight. In actual speech enhancement application scenarios, it can reduce the consumption of computing resources and space resources.

[0083] In some embodiments, after training, the speech enhancement network can be deployed to the terminal, so that the terminal can use the speech enhancement network to perform speech enhancement on the collected speech (for example, in a voice call scenario, a video call scenario, or a cloud conference scenario) in real time, and transmit the speech-enhanced speech signal to play the speech-enhanced speech on the receiving end, thereby improving the speech call effect. Moreover, in this embodiment, since the speech-enhanced speech signal is transmitted instead of the directly collected speech, it can be understood that when the directly collected speech includes at least one of interfering human voices and environmental noise, the data volume of the directly collected speech is larger than that of the enhanced speech. In this way, transmission after speech enhancement can effectively reduce the amount of data transmission, reduce bandwidth consumption, and improve the utilization rate of network resources.

[0084] Please refer to Figure 5, which shows a flow chart of a speech enhancement method provided by another embodiment of the present application. In this embodiment, the speech enhancement network method can be executed by a terminal, which has at least display, storage, calculation, and communication functions. The speech enhancement network used by the speech enhancement network method can be trained by the server. The speech enhancement network method shown in Figure 5 can be applied to the video conferencing scenario shown in Figure 6.

[0085] In this video conferencing scenario, the cloud server 310 provided by the video conferencing service provider can be used to train the speech enhancement network. After network training is complete, the user can download a video conferencing client with the speech enhancement network from the cloud server 310 and install it on their terminal device. This allows the terminal device to use the speech enhancement network to enhance spoken voices during the video conference. The terminal devices may include a first terminal device 330 and a second terminal device 350. The cloud server 310 establishes a communication connection with the first terminal device 330 and the second terminal device 350 via a network.

[0086] It should be noted that Figure 6 is only an application scenario diagram provided by the embodiment of the present application. The application scenario described in the embodiment of the present application is to more clearly illustrate the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. It is known to those skilled in the art that with the evolution of the system architecture and the emergence of new application scenarios (online voice or live broadcast, etc.), the technical solution provided by the embodiment of the present application is also applicable to solving similar technical problems. As shown in Figure 5, the speech enhancement method may include the following steps:

[0087] Step S210: Obtain the target voiceprint vector of the target sound-making object.

[0088] The target speaker refers to the person whose speech needs to be enhanced. The target speaker can also refer to the current speaker. As shown in Figure 6, in a video conference scenario, user 340, the current speaker, is the target speaker.

[0089] In some embodiments, the target voiceprint vector of the target speaker can be associated and stored with the target speaker's account in the client. This allows the target speaker's target voiceprint vector to be obtained based on the account used to log in to the client after logging in. For example, the target voiceprint vector of the target speaker can be quickly obtained based on the account used to log in to a video conferencing client.

[0090] In other embodiments, before making a voice call or video call, voice collection can be performed on the reference party of the call. The collected voice is used as the registration voice, and the voiceprint vector of the current call participant is extracted through the registration voice. In this case, each participant in the call can be used as the target sound object in this application. In one embodiment, the voiceprint of each sound object is determined by using the registration voice that is separately requested to be collected by the user during registration. This can specifically trigger the sound object to emit a more recognizable sound and obtain a more recognizable voiceprint, which facilitates subsequent voiceprint comparison and obtains voiceprint vectors of different sound objects.

[0091] In some scenarios, considering the possibility that multiple users in a conference room may use the same account, or that multiple users in a group may conduct video or voice calls, since the client of an account includes multiple call participants, in this case, before the call starts, each client can collect the registered voice of each call participant, then perform voiceprint feature extraction on the registered voice of each call participant, and add the extracted voiceprint vector to the voiceprint vector set. Based on this, step S210 can include the following steps A1 to A3:

[0092] Step A1: Extract voiceprint features of the target speech to obtain a reference voiceprint vector.

[0093] The target speech refers to the currently collected speech to be enhanced.

[0094] Step A2: Calculate the voiceprint similarity between the reference voiceprint vector and each voiceprint vector in the voiceprint vector set.

[0095] Step A3: According to the voiceprint similarity, determine from the voiceprint vector set a voiceprint vector that has the highest similarity to the reference voiceprint vector and whose similarity to the reference voiceprint vector exceeds a similarity threshold, as the target voiceprint vector of the target voice object.

[0096] Among them, the similarity threshold can be set according to actual needs. If the similarity between a voiceprint vector and a reference voiceprint vector exceeds the similarity threshold, it indicates that the probability that the voiceprint vector and the reference voiceprint vector are the voiceprints of the same sound-making object is high. In this application, the voiceprint vector with the highest similarity to the reference voiceprint vector in the voiceprint vector set and whose similarity to the reference voiceprint vector exceeds the similarity threshold is determined as the target voiceprint vector of the target sound-making object, which is equivalent to determining the voiceprint vector with the highest probability of corresponding to the same sound-making object as the reference voiceprint vector in the voiceprint vector set as the target voiceprint vector of the target sound-making object, that is, the target voiceprint vector of the sound-making object from which the target speech to be currently enhanced originates. In this way, a more accurate target voiceprint vector can be determined from the voiceprint vector set that includes more accurate voiceprint vectors for each sound-making object, which facilitates subsequent more accurate speech enhancement processing.

[0097] In the above embodiment, the identity of the target sound-making object is confirmed based on the target voice. For example, the second terminal device 350 can perform voice recognition on the target voice in response to the collected target voice to determine the identity of the user 340, that is, whether the user is a registered user, that is, the user from whom the voiceprint vector in the voiceprint vector set comes.

[0098] As an implementation method, the time interval between the target speech acquisition and the last speech enhancement performed can be obtained. If the time interval is greater than a threshold, voiceprint feature extraction can be performed again on the current target speech to match the target voiceprint vector for speech enhancement. If the time interval is less than the threshold, the target voiceprint vector of the target sounding object is directly obtained. In this way, even if the time interval is greater than the threshold, even if the existing sounding object changes, the corresponding target voiceprint vector can still be matched for the new sounding object.

[0099] The target voiceprint vector can be pre-extracted and stored. In some embodiments, a registration voice of the target speaker can be obtained and voiceprint extraction performed on the target speaker's registration voice to obtain the target voiceprint vector of the target speaker. Please refer to Figure 7, which shows a flow chart for extracting a target voiceprint vector. As shown in Figure 7, the target speaker can record a speech segment, i.e., the registration voice, and then perform sound quality testing on the registration voice.

[0100] For example, the registration voice is tested for sound quality to obtain its signal-to-noise ratio (SNR). If the SNR is greater than a threshold, the registration voice can be converted to a time-frequency domain. The frequency domain features of the converted registration voice are then input into a voiceprint extraction network for voiceprint extraction, resulting in a target voiceprint vector for the target sounding object. Obtaining the target voiceprint vector of the target sounding object in the presence of a high SNR can more accurately reflect the voiceprint characteristics of the target sounding object, reduce interference from other sounds, and facilitate subsequent speech enhancement of the target sounding object based on the target voiceprint vector and the audio characteristics of the target speech, thereby obtaining more accurate enhanced audio features for the target sounding object.

[0101] In some embodiments, the registered user corresponding to the account used at the venue is not the same person as user 340. To this end, the recorded voice of the target sounding object can be obtained, and further, the voiceprint extraction network is used to extract the frequency domain features of the recorded voice to obtain the target voiceprint vector of the target sounding object.

[0102] For example, the second terminal device 350 may collect a voice segment of the user 340, that is, record the voice, and then perform voiceprint extraction on the frequency domain features of the recorded voice based on the voiceprint extraction network to obtain the target voiceprint vector of the user 340.

[0103] Step S220: Input the target voiceprint vector and the audio features of the target speech into a speech enhancement network for enhancement processing to obtain enhanced audio features for the target sound object. In one embodiment, the speech enhancement network can suppress interference such as environmental noise and interfering human voices in the target speech by processing the target voiceprint vector and the audio features of the target speech. The obtained enhanced audio features of the target sound object can be regarded as the audio features of the target speech after suppressing interference such as interfering human voices and environmental noise.

[0104] The training process of the speech enhancement network may refer to the contents of steps 110 to 160 in the above embodiment, and will not be described in detail here.

[0105] As an implementation, the speech enhancement network may include a long-short-term memory subnetwork and a fully connected subnetwork. For example, the second terminal device 350 may input the target voiceprint vector of the user 340 and the collected audio features of the target speech into the speech enhancement network. The long-short-term memory subnetwork of the speech enhancement network then extracts features from the target voiceprint vector and the collected audio features of the target speech, generating intermediate features. Furthermore, the second terminal device 350 may input the intermediate features into the fully connected subnetwork for fully connected processing, generating enhanced audio features for the target sounding object.

[0106] Step S230: reconstructing the enhanced audio features to obtain enhanced speech corresponding to the target speech.

[0107] As an implementation method, the acquired enhanced audio features can be reconstructed, the enhanced audio features can be converted from the frequency domain to the time domain, and the enhanced speech after speech enhancement can be calculated. For example, the second terminal device 350 can perform an inverse Fourier transform on the enhanced audio features to obtain the time domain speech after speech enhancement, that is, the enhanced speech. Furthermore, the second terminal device 350 can send the enhanced speech to the first terminal device 330 via the network, and then the user 320 can hear the enhanced speech played by the first terminal device 330.

[0108] For example, please refer to Figure 8, which shows a flow chart of speech enhancement. As shown in Figure 8, when the terminal device obtains the registration voice of the target sounding subject, it can perform time-frequency conversion on the registration voice to obtain corresponding frequency domain features, and then use the voiceprint extraction network to extract the voiceprint of the frequency domain features of the registration voice to obtain the corresponding target voiceprint vector.

[0109] When the terminal device captures the target speech, it can input the target speech's frequency domain features, after time-frequency conversion, and the frequency domain features of the registration speech into a speech enhancement network for speech enhancement. The speech enhancement network then outputs enhanced audio features specific to the target sound source. Furthermore, by performing speech reconstruction on these enhanced audio features, such as an inverse Fourier transform, enhanced speech is obtained, removing interfering human voices and noise.

[0110] Considering that the current target speech includes the speech of multiple main speakers, it is not convenient to perform speech enhancement on multiple main speakers, and only ambient noise can be suppressed. In some embodiments, the speech enhancement method may further include:

[0111] If the similarity between a specified voiceprint vector in the voiceprint vector set and a reference voiceprint vector does not exceed a similarity threshold, the target speech is speech enhanced through a reference speech enhancement network to obtain a reference enhanced audio feature for the target speech; wherein the reference speech enhancement network is obtained by training noisy speech and clean speech corresponding to the noisy speech; the specified voiceprint vector refers to the voiceprint vector in the voiceprint vector set with the highest similarity to the reference voiceprint vector; the reference enhanced audio feature is speech reconstructed to obtain an enhanced speech corresponding to the target speech.

[0112] In some embodiments, the reference speech enhancement network can be trained through the following steps: obtaining a training sample set, the training sample set including sample reference speech, sample comparison speech and interference noise; performing audio feature extraction on the sample reference speech, sample comparison speech and interference noise respectively to obtain corresponding reference audio features, comparison audio features and noise audio features; the reference speech enhancement network performs enhancement processing based on the reference audio features and the comparison audio features, and outputs predicted audio features; determining the training loss value of the reference speech enhancement network based on the predicted audio features and the comparison audio features; based on the training loss value, iteratively updating the weight parameters of the reference speech enhancement network until the training end conditions are met.

[0113] In the above embodiment, two models for speech enhancement can be deployed in the application, namely a speech enhancement network and a reference speech enhancement network. When the voiceprint characteristics of the sound-producing object from which the target speech to be enhanced originates can be determined, or in other words, when the sound-producing object corresponding to the target speech is relatively clear, the speech enhancement network is used to perform speech enhancement according to the process shown in FIG6 , so as to subsequently obtain enhanced speech that removes interfering human voices and environmental noise. If the similarity between a specified voiceprint vector in the voiceprint vector set and the reference voiceprint vector does not exceed the similarity threshold, the possible reason is that the current target speech includes the voices of multiple main sound-producing objects. In this case, the target speech can be speech enhanced by the reference speech enhancement network, thereby avoiding suppressing the voice of any main sound-producing object in the target speech.

[0114] It is worth mentioning that the above voice enhancement method can be executed by the terminal or by a server providing voice services, which is not specifically limited here.

[0115] Please refer to FIG9 , which shows a structural block diagram of a speech enhancement network training device 400 provided in an embodiment of the present application. The speech enhancement network training device 400 includes:

[0116] The sample acquisition module 410 is configured to acquire a training set, wherein the training set includes a plurality of training samples, wherein a training sample includes a sample reference speech, a sample comparison speech, and a mixed speech; the mixed speech is obtained by mixing interfering human voices, environmental noise, and the sample comparison speech; wherein the sample reference speech and the sample comparison speech in the same training sample are derived from the same sample sounding subject;

[0117] A voiceprint extraction module 420 is configured to extract the voiceprint of the sample reference speech to obtain a sample voiceprint vector;

[0118] A feature extraction module 430 is configured to extract audio features from the mixed speech to obtain sample audio features;

[0119] A feature prediction module 440 is configured to perform enhancement processing based on the sample voiceprint vector and the sample audio feature by a speech enhancement network, and output a predicted audio feature for the sample sound object;

[0120] a loss determination module 450 for determining a training loss value of the speech enhancement network based on the predicted audio features and the compared audio features corresponding to the sample compared speech;

[0121] The parameter updating module 460 is used to iteratively update the weight parameters of the speech enhancement network based on the training loss value until the training end condition is reached.

[0122] In some embodiments, the speech enhancement network includes a first long short-term memory subnetwork and a first fully connected subnetwork; the feature prediction module 440 can be specifically used to:

[0123] Inputting the sample voiceprint vector and the sample audio feature into the first long short-term memory sub-network for feature extraction to obtain intermediate features;

[0124] The intermediate features are input into the first fully connected subnetwork for full connection processing to obtain the predicted audio features of the sample sound object.

[0125] In some embodiments, the voiceprint extraction module 420 may include: a time-frequency conversion unit and a voiceprint extraction unit. The time-frequency conversion unit is configured to perform time-frequency conversion on the sample reference speech to obtain frequency domain features of the sample reference speech; and the voiceprint extraction unit is configured to input the frequency domain features of the sample reference speech into a voiceprint extraction network to perform voiceprint extraction and obtain a sample voiceprint vector.

[0126] In some embodiments, the voiceprint extraction network includes a second long short-term memory subnetwork, a second fully connected subnetwork and a pooling subnetwork: the voiceprint extraction unit can be specifically used to: input the frequency domain features of the sample reference speech into the second long short-term memory subnetwork for feature extraction to obtain a first voiceprint feature; input the first voiceprint feature into the second fully connected subnetwork for full connection processing to obtain a second voiceprint feature; input the second voiceprint feature into the pooling subnetwork for pooling processing to obtain a sample voiceprint vector.

[0127] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0128] In several embodiments provided in this application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0129] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0130] In this way, by adding the sample voiceprint vector of the sample sound-making object to the input data of the speech enhancement network, the speech enhancement network can increase its attention to learning the sound information of the sample sound-making object during the training process, allowing the speech enhancement network to be more focused on enhancing the sound of the sample sound-making object. In addition to removing interfering noise, it can also effectively suppress interfering human voices at the same time, thereby improving the quality and performance of speech enhancement of the trained speech enhancement network.

[0131] Please refer to FIG10 , which shows a block diagram of a speech enhancement device 500 provided in an embodiment of the present application. The speech enhancement device 500 includes:

[0132] A vector acquisition module 510 is used to acquire a target voiceprint vector of a target sounding object;

[0133] A speech enhancement module 520 is configured to input the target voiceprint vector and the audio features of the target speech into a speech enhancement network for enhancement processing to obtain enhanced audio features for the target sounding object; the speech enhancement network is trained by the speech enhancement network training device 400 in the above embodiment;

[0134] The speech reconstruction module 530 is configured to perform speech reconstruction on the enhanced audio features to obtain enhanced speech corresponding to the target speech.

[0135] In some embodiments, the vector acquisition module 510 can be specifically used to extract voiceprint features of the target speech to obtain a reference voiceprint vector; calculate the voiceprint similarity between the reference voiceprint vector and each voiceprint vector in the voiceprint vector set; and based on the voiceprint similarity, determine from the voiceprint vector set the voiceprint vector with the highest similarity to the reference voiceprint vector, and whose similarity to the reference voiceprint vector exceeds a similarity threshold, as the target voiceprint vector of the target voice-making object.

[0136] In some embodiments, the vector acquisition module 510 may further include a speech acquisition unit for acquiring the registered speech of the target sound-making object; and a vector generation unit for performing voiceprint extraction on the registered speech of the target sound-making object to obtain a target voiceprint vector of the target sound-making object.

[0137] In some embodiments, the vector generation unit can be specifically used to: perform sound quality detection on the registered voice to obtain the voice signal-to-noise ratio of the registered voice; if the voice signal-to-noise ratio is greater than the signal-to-noise ratio threshold, perform time-to-frequency conversion on the registered voice to obtain the frequency domain features of the registered voice; perform voiceprint extraction on the frequency domain features of the registered voice based on a voiceprint extraction network to obtain the target voiceprint vector of the target sound-making object.

[0138] In some embodiments, the vector acquisition module 510 can also be specifically used to perform speech enhancement on the target speech through a reference speech enhancement network to obtain a reference enhanced audio feature for the target speech if the similarity between the specified voiceprint vector in the voiceprint vector set and the reference voiceprint vector does not exceed the similarity threshold; wherein the reference speech enhancement network is obtained by training the noisy speech and the clean speech corresponding to the noisy speech; the specified voiceprint vector refers to the voiceprint vector in the voiceprint vector set that has the highest similarity with the reference voiceprint vector; and the reference enhanced audio feature is speech reconstructed to obtain the enhanced speech corresponding to the target speech.

[0139] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0140] In several embodiments provided in this application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0141] In addition, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module. The above-mentioned integrated modules may be implemented in the form of hardware or software functional modules.

[0142] As shown in FIG11 , an embodiment of the present application further provides a computer device 600, which includes a processor 610, a memory 620, a power supply 630, and an input unit 640. The memory 620 stores a computer program. When the computer program is called by the processor 610, the various method steps provided in the above embodiments can be executed. It will be understood by those skilled in the art that the structure of the computer device shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently. Among them:

[0143] The processor 610 may include one or more processing cores. The processor 610 utilizes various interfaces and circuits to connect various components within the battery management system. By running or executing instructions, programs, instruction sets, or program sets stored in the memory 620 and accessing data stored in the memory 620, the processor 610 performs various functions of the battery management system and processes data, as well as various functions of the computer device and processes data, thereby providing overall control of the computer device. Optionally, the processor 610 may be implemented in the form of at least one of a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 610 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing display content; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 610 and may be implemented separately via a communication chip.

[0144] The memory 620 may include a random access memory 620 (Random Access Memory, RAM), and may also include a read-only memory 620 (Read-Only Memory). The memory 620 may be used to store instructions, programs, instruction sets, or program sets. The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the various method embodiments described above, etc. The data storage area may also store data created by the computer device during use (such as a phone book and audio and video data), etc. Accordingly, the memory 620 may also include a memory controller to provide the processor 610 with access to the memory 620.

[0145] The power supply 630 can be logically connected to the processor 610 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. The power supply 630 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0146] The input unit 640 can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0147] Although not shown, the computer device 600 may further include a display unit, etc., which will not be described in detail herein. Specifically, in this embodiment, the processor 610 in the computer device loads the executable files corresponding to one or more computer program processes into the memory 620 according to the following instructions. The processor 610 then executes the data stored in the memory 620, such as the phone book and audio and video data, thereby implementing the various method steps provided in the aforementioned embodiments.

[0148] As shown in FIG12 , an embodiment of the present application further provides a computer-readable storage medium 700 , in which a computer program 710 is stored. The computer program 710 can be called by a processor to execute various method steps provided in the embodiment of the present application.

[0149] The computer-readable storage medium may be an electronic memory such as a flash memory, an EEPROM (Electrically Erasable Programmable Read-Only Memory), an EPROM, a hard disk, or a ROM. Alternatively, the computer-readable storage medium includes a non-transitory computer-readable storage medium. The computer-readable storage medium 700 has storage space for a computer program that executes any of the method steps in the above embodiments. These computer programs can be read from or written to one or more computer program products. The computer program can be compressed in an appropriate form.

[0150] According to one aspect of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the various method steps provided in the above embodiments.

[0151] The above is only a preferred embodiment of the present application and does not constitute any form of limitation to the present application. Although the present application has been disclosed as above with preferred embodiments, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present application. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present application without departing from the content of the technical solution of the present application are still within the scope of the technical solution of the present application.

Claims

1. A method for training a speech enhancement network, characterized in that: include: Acquire a training set, the training set includes a plurality of training samples, each of the plurality of training samples includes a sample reference speech, a sample comparison speech, and a mixed speech; the mixed speech is obtained by mixing interfering human voice, environmental noise, and the sample comparison speech; wherein the sample reference speech and the sample comparison speech in the same training sample are derived from the same sample sound-producing object; Extracting voiceprints from the sample reference speech to obtain sample voiceprint vectors; Extracting audio features from the mixed speech to obtain sample audio features; Performing speech enhancement processing on the sample voiceprint vector and the sample audio feature using a speech enhancement network, and outputting predicted audio features for the sample sound object; Determining a training loss value of the speech enhancement network according to the predicted audio features and the compared audio features corresponding to the sample compared speech; Based on the training loss value, the weight parameters of the speech enhancement network are iteratively updated until a training end condition is reached.

2. The method according to claim 1, characterized in that The speech enhancement network includes a first long short-term memory subnetwork and a first fully connected subnetwork; The using a speech enhancement network to perform speech enhancement processing on the sample voiceprint vector and the sample audio feature, and outputting a predicted audio feature for the sample sound object, comprises: Inputting the sample voiceprint vector and the sample audio feature into the first long short-term memory subnetwork for feature extraction to obtain intermediate features; The intermediate features are input into the first fully connected subnetwork for full connection processing to obtain the predicted audio features of the sample sound object.

3. The method according to claim 1 or 2, characterized in that: The step of extracting voiceprint from the sample reference speech to obtain a sample voiceprint vector includes: Performing time-frequency conversion on the sample reference speech to obtain frequency domain features of the sample reference speech; The frequency domain features of the sample reference speech are input into a voiceprint extraction network for voiceprint extraction to obtain a sample voiceprint vector.

4. The method according to any one of claims 1 to 3, characterized in that: The voiceprint extraction network includes a second long short-term memory sub-network, a second fully connected sub-network and a pooling sub-network; The step of inputting the frequency domain features of the sample reference speech into a voiceprint extraction network to extract the voiceprint and obtain a sample voiceprint vector comprises: Inputting the frequency domain features of the sample reference speech into the second long short-term memory subnetwork for feature extraction to obtain a first voiceprint feature; Inputting the first voiceprint feature into the second fully connected subnetwork for full connection processing to obtain a second voiceprint feature; The second voiceprint feature is input into the pooling subnetwork for pooling processing to obtain a sample voiceprint vector.

5. A speech enhancement method, characterized in that: The method comprises: Obtain the target voiceprint vector of the target sounding object; Inputting the target voiceprint vector and the audio features of the target speech into a speech enhancement network for speech enhancement processing to obtain enhanced audio features for the target sound object; the speech enhancement network is trained according to the method according to any one of claims 1 to 4; The enhanced audio features are subjected to speech reconstruction to obtain enhanced speech corresponding to the target speech.

6. The method according to claim 5, characterized in that The step of obtaining a target voiceprint vector of a target sounding object includes: Extract voiceprint features of the target speech to obtain a reference voiceprint vector; Calculate the voiceprint similarity between the reference voiceprint vector and each voiceprint vector in the voiceprint vector set; According to the voiceprint similarity, a voiceprint vector having the highest similarity to the reference voiceprint vector and a similarity to the reference voiceprint vector exceeding a similarity threshold is determined from the voiceprint vector set as a target voiceprint vector of the target voice utterance object.

7. The method according to claim 5 or 6, characterized in that: Before obtaining the target voiceprint vector of the target sound-making object, the method further includes: Get the registered voice of the target sounding object; Voiceprint extraction is performed on the registered speech of the target sound-making object to obtain a target voiceprint vector of the target sound-making object.

8. The method according to any one of claims 5 to 7, characterized in that: The step of extracting the voiceprint of the target sound-making object's registered voice to obtain a target voiceprint vector of the target sound-making object includes: Performing sound quality detection on the registered voice to obtain a voice signal-to-noise ratio of the registered voice; If the speech signal-to-noise ratio is greater than the signal-to-noise ratio threshold, performing time-frequency conversion on the registered speech to obtain frequency domain features of the registered speech; The voiceprint extraction network is used to extract the frequency domain features of the registered speech to obtain a target voiceprint vector of the target sounding object.

9. The method according to any one of claims 5 to 8, characterized in that: The method further comprises: If the similarity between the specified voiceprint vector in the voiceprint vector set and the reference voiceprint vector does not exceed the similarity threshold, the target speech is voice-enhanced by a reference speech enhancement network to obtain a reference enhanced audio feature for the target speech; wherein the reference speech enhancement network is obtained by training a noisy speech and a clean speech corresponding to the noisy speech; the specified voiceprint vector refers to a voiceprint vector in the voiceprint vector set that has the highest similarity with the reference voiceprint vector; The reference enhanced audio features are subjected to speech reconstruction to obtain enhanced speech corresponding to the target speech.

10. A training device for a speech enhancement network, characterized in that: The device comprises: A sample acquisition module is used to acquire a training set, wherein the training set includes a plurality of training samples, each of which includes a sample reference speech, a sample comparison speech, and a mixed speech; the mixed speech is obtained by mixing interfering human voice, environmental noise, and the sample comparison speech; wherein the sample reference speech and the sample comparison speech in the same training sample are derived from the same sample sound-producing object; A voiceprint extraction module, used to extract the voiceprint of the sample reference speech to obtain a sample voiceprint vector; A feature extraction module, used to extract audio features from the mixed speech to obtain sample audio features; A feature prediction module, configured to perform speech enhancement processing on the sample voiceprint vector and the sample audio feature using a speech enhancement network, and output a predicted audio feature for the sample sound object; A loss determination module, configured to determine a training loss value of the speech enhancement network based on the predicted audio features and the compared audio features corresponding to the sample compared speech; A parameter updating module is used to iteratively update the weight parameters of the speech enhancement network based on the training loss value until the training end condition is reached.

11. A speech enhancement device, characterized in that: The device comprises: A vector acquisition module is used to obtain a target voiceprint vector of a target sounding object; A speech enhancement module, used to input the target voiceprint vector and the audio features of the target speech into a speech enhancement network for enhancement processing to obtain enhanced audio features for the target sound object; the speech enhancement network is according to any one of claims 1 to 4 obtained through method training; The speech reconstruction module is used to perform speech reconstruction on the enhanced audio features to obtain enhanced speech corresponding to the target speech.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 9.

13. A computer device, characterized in that: include: Memory; One or more processors coupled to the memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications configured to execute the method according to any one of claims 1 to 9.

14. A computer program product comprising a computer program, characterized in that The processor of the computer device executes the computer program, so that the computer device performs the steps in the training method of the speech enhancement network as claimed in any one of claims 1 to 9.

Images