Speech enhancement method and device and electronic equipment

Abstract

The embodiment of the invention discloses a speech enhancement method and device and electronic equipment, and relates to the technical field of computers, and the method comprises the steps: carrying out the feature extraction of a received to-be-processed speech signal, and obtaining a to-be-processed speech feature; inputting the to-be-processed speech features into a pre-constructed speech enhancement network model, and obtaining target gain information corresponding to the to-be-processed speech features by using the speech enhancement network model; wherein the speech enhancement network model comprises seven gated loop network layers, the input data volumes of the first three gated loop network layers are sequentially increased, the input data volumes of the fourth layer and the fifth layer are both smaller than the input data volume of the third gated loop network layer, and the input data volumes of the latter three gated loop network layers are sequentially increased; and obtaining a speech enhancement signal according to the target gain information and the to-be-processed speech signal. By using the technical scheme provided by the invention, the rapid enhancement processing of the voice signal in the voice call is realized, and the quality of the voice call is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech enhancement method, device and electronic device. Background Art

[0002] With the development of computer Internet technology, smart devices have greatly enriched people's lives. Using smart devices to make voice calls has become a necessary way for people's daily life or work. However, when there is noise in the environment, the quality of voice calls may be affected. Generally, voice enhancement technology is needed to reduce the interference of noise on voice calls.

[0003] Speech enhancement technology refers to the technology that extracts useful speech signals from the noise background and suppresses and reduces noise interference when the speech signal is interfered with or even drowned by various noises. Generally, when using speech enhancement technology, a VAD (Voice Activity Detection) detection result may be calculated to assist in the subsequent speech enhancement calculation. However, VAD results have a large misjudgment (noise segments are easily misjudged as speech segments), which may affect the accuracy of speech enhancement results.

[0004] How to accurately extract pure speech from speech signals and improve the quality of voice calls is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0005] The embodiments of this specification provide a speech enhancement method, apparatus, device, storage medium, and computer program product, which can perform speech enhancement processing on speech signals in a voice call scenario to improve the quality of the voice call.

[0006] On the one hand, an embodiment of this specification provides a method for speech enhancement, the method comprising:

[0007] receiving a voice signal to be processed;

[0008] Extracting features of the speech signal to be processed to obtain speech features of the speech signal to be processed;

[0009] Input the speech feature to be processed into a pre-built speech enhancement network model, and use the speech enhancement network model to obtain the target gain information corresponding to the speech feature to be processed; wherein the speech enhancement network model includes seven gated recurrent network layers, the input data amounts of the first three gated recurrent network layers increase in sequence, the input data amounts of the fourth gated recurrent network layer and the fifth gated recurrent network layer are both smaller than the input data amount of the third gated recurrent network layer, and the input data amounts of the last three gated recurrent network layers increase in sequence;

[0010] A speech enhancement signal is obtained according to the target gain information and the speech signal to be processed.

[0011] Another aspect provides a speech enhancement device, the device comprising:

[0012] A signal receiving module, used for receiving a voice signal to be processed;

[0013] A feature extraction module, used to extract features of the speech signal to be processed to obtain speech features of the speech signal to be processed;

[0014] A gain calculation module, used for inputting the speech feature to be processed into a pre-built speech enhancement network model, and using the speech enhancement network model to obtain target gain information corresponding to the speech feature to be processed; wherein the speech enhancement network model includes seven gated recurrent network layers, the input data amounts of the first three gated recurrent network layers increase in sequence, the input data amounts of the fourth gated recurrent network layer and the fifth gated recurrent network layer are both smaller than the input data amount of the third gated recurrent network layer, and the input data amounts of the last three gated recurrent network layers increase in sequence;

[0015] A signal enhancement module is used to obtain a speech enhancement signal according to the target gain information and the speech signal to be processed.

[0016] Another aspect provides an electronic device, including: a processor;

[0017] a memory for storing instructions executable by the processor;

[0018] The processor is configured to execute the instructions to implement any of the above-mentioned speech enhancement methods.

[0019] On the other hand, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can perform any of the above-mentioned speech enhancement methods.

[0020] On the other hand, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the speech enhancement method provided in the above various optional implementations.

[0021] The speech enhancement method, apparatus, device, storage medium, and computer program product provided in the embodiments of this specification have the following technical effects:

[0022] The speech enhancement method provided in the embodiment of this specification pre-trains and constructs a speech enhancement network model including seven layers of GRU. The input data volume of the 7-layer GRU network is a structure that first increases, then decreases, and then increases again, so that after the to-be-processed speech features of the to-be-processed speech signal are input into the pre-constructed speech enhancement network model, the model can first perform deep learning on the input features, and the input data volume of the middle layer is appropriately reduced, which can improve the data processing speed, while the input data volume of the subsequent network layer is appropriately increased, and the input features continue to be deeply learned. Not only can the input feature information be deeply learned to obtain the gain information corresponding to each to-be-processed speech feature, but also the data processing volume and complexity are reduced, and the balance between the complexity and effect of the neural network structure is achieved, thereby improving the data processing efficiency and accuracy of the speech enhancement processing, and can be applied to real-time voice call scenarios to meet the needs of real-time voice calls. In addition, the entire process does not require VAD detection, which avoids the problem of inaccurate speech enhancement results caused by VAD detection misjudgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions and advantages in the embodiments of this specification or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 is a schematic diagram of an application environment of a speech enhancement method provided in an embodiment of this specification;

[0025] Figure 2 is a flowchart of a speech enhancement method provided by an embodiment of this specification;

[0026] Figure 3 It is a schematic diagram of the principle of a speech enhancement network model in one embodiment of this specification;

[0027] Figure 4 is a schematic diagram of an original noisy speech signal in one embodiment of the present specification;

[0028] Figure 5 is a schematic diagram of a speech signal after speech enhancement processing in one embodiment of the present specification;

[0029] Figure 6 is a structural schematic diagram of a speech enhancement device provided in an embodiment of this specification;

[0030] Figure 7 is a block diagram of a speech enhancement electronic device provided by an embodiment of this specification;

[0031] Figure 8 This is another block diagram of a speech enhancement electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this specification.

[0033] It should be noted that the terms "first", "second", etc. in the description and claims of the embodiments of this specification and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of this specification described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0034] See also Figure 1 , Figure 1 1 is a schematic diagram of an application environment of a speech enhancement method provided in an embodiment of the present specification, and the application environment may at least include a server 100 and a terminal 200.

[0035] In an optional embodiment, server 100 can be used to perform voice enhancement processing. Server 100 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0036] In an optional embodiment, the terminal 200 may be a client that sends a voice signal or a voice enhancement request to the server 100. Specifically, the terminal 200 may include, but is not limited to, electronic devices such as smart phones, desktop computers, tablet computers, laptop computers, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, vehicle terminals, smart TVs, etc.; it may also be software running on the above electronic devices, such as applications, applets, etc. The operating system running on the electronic device in the embodiments of this specification may include, but is not limited to, Android, IOS, Linux, Windows, etc.

[0037] In addition, it should be noted that Figure 1 What is shown is only an application environment of the speech enhancement method, and the embodiments of this specification are not limited to the above.

[0038] In the embodiments of this specification, the server 100 and the terminal 200 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of this specification.

[0039] Voice calls are a technology that people often use in their daily lives and work, such as real-time voice conferences or video conferences. However, in some scenarios, users may be in a noisy environment, and the collected voice signals may contain a lot of noise. For example, during a voice conference, the user may be in an environment with many people. When the user speaks, the voices of other people may also be collected, and the voice of the main speaker cannot be accurately identified, resulting in poor voice call quality. Voice enhancement refers to the technology of extracting useful voice signals from the noisy background and suppressing and reducing noise interference when the voice signal is interfered with or even drowned by various noises. In other words, it is to extract the purest original voice from the noisy voice. Therefore, voice enhancement is a very important technical means in voice call technology, which can greatly improve the voice call effect.

[0040] The following describes a method for speech enhancement in an embodiment of this specification. Figure 2This is a flow chart of a speech enhancement method provided by an embodiment of this specification. This specification provides method operation steps such as the embodiment or flow chart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only execution order. When the actual system or server product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment). The method can be applied to terminal devices such as computers, tablet computers, and smart phones, and can also be used in servers. Specifically, Figure 2 As shown, the method may include:

[0041] S202: Receive a voice signal to be processed.

[0042] In the specific implementation process, generally, in a voice call scenario, the voice call device will be equipped with a voice collection device such as a microphone. The voice collection device can be used to collect the voice signal to be processed during the voice call, that is, the audio data of the voice call. The voice signal to be processed may contain noise such as environmental noise.

[0043] S204: Extract features of the speech signal to be processed to obtain speech features of the speech signal to be processed.

[0044] In a specific real-time process, after obtaining the speech signal to be processed, feature extraction can be performed on the speech signal to be processed, such as dividing the speech signal to be processed into certain audio frequency bands, and then extracting the features in each divided frequency band to obtain the speech feature to be processed. The specific method of feature extraction can be selected according to actual business needs, such as selecting a speech feature extraction algorithm to extract features from the speech signal to be processed, which is not specifically limited in the embodiments of this specification. The speech feature to be processed can characterize the relevant attributes of the speech signal to be processed, such as changes in timbre, frequency, pitch, and sound intensity, which are not specifically described in the embodiments of this specification.

[0045] In some embodiments of the present specification, feature extraction is performed on the speech signal to be processed to obtain the speech feature to be processed of the speech signal to be processed, including:

[0046] Performing sparse processing on the speech signals with a frequency greater than a preset frequency in the speech signals to be processed, and obtaining a sparse frequency band corresponding to the speech signals to be processed;

[0047] Feature extraction is performed on the sparse frequency band to obtain the features of the speech to be processed.

[0048] In the specific implementation process, the speech enhancement method in the embodiment of this specification is mainly a processing method for broadband speech signals. Broadband speech signals usually refer to signals with a speech signal sampling rate of 16000kHz (bandwidth is half of the sampling rate 8000Hz), which are commonly used in mobile phone communications, wired headphones and unlimited Bluetooth headphones connected to the computer side, and some low-end computer hands-free communications. After the collected voice signal to be processed, in the embodiment of this specification, the voice signal greater than the preset frequency in the collected voice signal to be processed is first sparsely processed, such as: the voice signal greater than 1000Hz can be sparsely processed, and the sparse processing method can be selected according to actual needs, such as: the voice signal greater than 1000Hz can be divided into multiple sparse frequency bands by equal or unequal divisions of 200 / 400 / 600Hz. Of course, according to actual needs, other sparse processing means can also be selected, such as: selecting a suitable algorithm to sparsely process the speech signal, which is not specifically limited in the embodiment of this specification. After performing sparse processing on speech signals with a frequency greater than a preset frequency to obtain a plurality of sparse frequency bands, speech features in each sparse frequency band may be extracted respectively to obtain speech features to be processed.

[0049] Among them, the value of the preset frequency can be set or adjusted according to actual needs, and the embodiments of this specification do not make specific limitations. The embodiments of this specification only perform sparse processing on the voice signals greater than the preset frequency in the voice signals to be processed and then extract voice features. Under normal circumstances, low-frequency signals are more important than high-frequency signals, and more important audio information may be retained in low-frequency signals, such as: the voice information of the main speaker in the low-frequency signal may be relatively large. In the embodiments of this specification, not all voice signals are sparsely processed, and low-frequency signals are not sparsely processed, which better ensures the integrity of low-frequency voice information, ensures that low-frequency voice signals can be not damaged, and ensures that important voice signals in low-frequency signals are not damaged, thereby ensuring the subsequent voice enhancement processing effect and improving the quality of voice calls.

[0050] In some embodiments of the present specification, sparse processing is performed on speech signals larger than a preset bandwidth in the speech signals to be processed to obtain a sparse frequency band corresponding to the speech signals to be processed, including:

[0051] Perform Bark domain sparse processing on the bandwidth greater than a preset frequency in the speech signal to be processed to obtain a specified number of sparse frequency bands;

[0052] Extract features from sparse frequency bands to obtain speech features to be processed, including:

[0053] According to a specified number of sparse frequency bands, respectively obtain the Bark frequency cepstral coefficients corresponding to each sparse frequency band, and obtain a specified number of Bark domain features;

[0054] Obtain a preset number of auxiliary features respectively according to the Bark frequency cepstral coefficients corresponding to the specified sparse frequency band;

[0055] A specified number of Bucky domain features and a preset number of auxiliary features are used as speech features to be processed.

[0056] In the specific implementation process, when performing speech feature extraction in the embodiment of this specification, the bandwidth greater than the preset frequency in the speech signal to be processed can be first subjected to Bark domain sparse processing to obtain a specified number of sparse frequency bands, namely bark frequency bands, such as: the frequency band of 1000-8000Hz can be divided into bark domains, and the division method can be increasingly sparse, that is, the divided bandwidth is from small to large. The specific sparse division method is not specifically limited in the embodiment of this specification, and the number of divisions can also be determined according to actual needs. In some scene examples of this specification, the speech signal to be processed can be divided into 56 bark bands. Then the corresponding Bark frequency cepstral coefficients, namely bfcc (Bark-frequency cepstral coefficients) parameters are calculated for the obtained sparse frequency bands to obtain the Bark domain features corresponding to each sparse frequency band. For example: if 56 bark bands are obtained during sparse processing, the corresponding bfcc parameters can be calculated in these 56 bark bands respectively to obtain 56 bark domain parameter features, namely 56 Bark domain features. At the same time, the obtained BFCC parameters can also be used to calculate a preset number of auxiliary features, such as performing differential processing on the specified BFCC parameters, etc. The specific processing method is not specifically limited in the embodiments of this specification. The obtained Bark domain features or auxiliary features are used as the speech features to be processed of the speech signal to be processed.

[0057] Among them, the Bark domain can be understood as an audio processing scale, a sound psychology scale, with Hz as the unit, mapping the frequency to 24 critical frequency bands of psychoacoustics, the 25th critical frequency band occupies about: 16K ~ 20kHz frequency, the width of a critical frequency band is equal to one Bark, simply put, the Bark scale is to convert physical frequency to psychoacoustic frequency. Bark domain sparse processing of the speech signal to be processed can be understood as converting the speech signal greater than the preset frequency into the frequency corresponding to the Bark domain. The specific conversion method can be selected according to actual needs, and the embodiments of this specification are not specifically limited.

[0058] The embodiments of this specification perform Bark domain coefficient processing on speech signals greater than a preset frequency to obtain a specified number of bark frequency bands, and then calculate BFCC parameters on the bark frequency bands to obtain Bark domain features. At the same time, a preset number of auxiliary features are calculated based on the specified BFCC parameters, and the auxiliary features are used to assist in the processing of speech enhancement to improve the effect of speech enhancement processing.

[0059] In some embodiments of the present specification, a preset number of auxiliary features are obtained according to the Bark frequency cepstral coefficients corresponding to the specified sparse frequency band, including:

[0060] According to the bandwidth of the sparse frequency band, a specified number of sparse frequency bands are sorted, and the sparse frequency band sorted at the specified position is used as the specified sparse frequency band;

[0061] Obtaining a first auxiliary feature according to a first-order difference coefficient of a Bark frequency cepstral coefficient corresponding to a specified sparse frequency band;

[0062] Obtaining a second auxiliary feature according to the second-order difference coefficients of the Bark frequency cepstral coefficients corresponding to the specified sparse frequency band;

[0063] Obtaining a third auxiliary feature according to a discrete cosine transform value of a cross-correlation coefficient of a specified sparse frequency band;

[0064] The pitch period, pitch stability, first auxiliary feature, second auxiliary feature and third auxiliary feature of the speech signal to be processed are used as auxiliary features of the speech signal to be processed.

[0065] In the specific implementation process, when extracting auxiliary features, the sparse frequency bands can be sorted first, and the frequency bands sorted at the specified position can be used as designated sparse frequency bands, such as: in some scenario examples in this specification, the first 1-6 bark bands can be used as designated sparse frequency bands. Of course, according to actual needs, other frequency bands can also be selected as designated sparse frequency bands, and this specification embodiment does not make specific limitations. Calculate the first-order difference sparseness and second-order difference sparseness of the bfcc parameters of the first 1-6 bark bands to obtain 6 first auxiliary features and 6 second auxiliary features respectively. Then calculate the discrete cosine transform value, i.e., the DCT transform value, of the mutual correlation coefficient of the first 1-6 bark bands to obtain 6 third auxiliary features. The first auxiliary feature, the second auxiliary feature and the third auxiliary feature are combined with the fundamental pitch period and fundamental pitch stability of the speech signal to be processed as auxiliary features of the speech signal to be processed.

[0066] Among them, general sounds are composed of a series of vibrations with different frequencies and amplitudes emitted by the sound-producing body. Among these vibrations, there is a vibration with the lowest frequency. The sound emitted by it is the fundamental tone, and the rest are overtones. The fundamental tone period is a detection method for recording the time wavelength of the fundamental tone. The fundamental tone period is the time of a complete wavelength of a mechanical wave. The gene period of the speech signal to be processed can be obtained by pitch analysis processing (such as pitch tracking). The specific calculation method is not specifically limited in the embodiments of this specification. Gene stability can be understood as an energy parameter of speech. It can be obtained by performing some energy calculations (such as filtering, energy cepstral coefficient calculation, etc.) on a specified sparse frequency band or on the speech signal to be processed. The specific calculation method is not specifically limited in the embodiments of this specification.

[0067] The embodiments of this specification provide a specific method for extracting auxiliary features of a speech signal to be processed. The auxiliary features can be used to characterize some specific attributes of the speech signal, thereby improving the accuracy of subsequent speech enhancement processing.

[0068] S206: Input the speech features to be processed into a pre-built speech enhancement network model, and use the speech enhancement network model to obtain target gain information corresponding to the speech features to be processed; wherein the speech enhancement network model includes seven gated recurrent network layers, the input data amounts of the first three gated recurrent network layers increase sequentially, the input data amounts of the fourth gated recurrent network layer and the fifth gated recurrent network layer are both smaller than the input data amount of the third gated recurrent network layer, and the input data amounts of the last three gated recurrent network layers increase sequentially.

[0069] In the specific implementation process, a speech enhancement network model capable of speech enhancement can be trained in advance through speech sample data. The network structure of the speech enhancement model in the embodiment of this specification selects a neural network model, and is a neural network model based on a gated recurrent network GRU (Gated Recurrent Unit) module. The speech enhancement model in the embodiment of this specification includes 7 layers of GRU, and the input data volume of the first three layers of GRU increases successively, and the input data volume of the last three layers of GRU decreases successively, while the input data volume of the fourth and fifth layers of GRU in the middle is less than that of the third layer of GRU. From this, it can be seen that the input data volume of the 7-layer GRU network in the speech enhancement network model in the embodiment of this specification is a structure that increases first, then decreases, and then increases again, so that the complexity of the model structure also increases first and then decreases. Through this network structure, data is passed down layer by layer to achieve the purpose of deep training and learning, and at the same time, the computational complexity and effect are balanced.

[0070] After receiving the speech signal to be processed and extracting the speech features to be processed of the speech signal to be processed, the extracted speech features to be processed can be input into the speech enhancement network model of the training number, and the speech enhancement network model can be used to obtain the target gain information corresponding to each speech feature to be processed.

[0071] S208: Obtain a speech enhancement signal according to the target gain information and the speech signal to be processed.

[0072] In a specific implementation process, after obtaining the target gain information of each speech feature to be processed, a speech enhancement signal can be obtained based on the target gain information and the speech signal to be processed. For example, the target gain information can be multiplied by the power spectrum of the noisy speech signal to be processed, and then the final speech enhancement signal can be obtained by inverse transformation using FFT (fast Fourier transform).

[0073] The speech enhancement method provided in the embodiment of this specification pre-trains and constructs a speech enhancement network model including seven layers of GRU. The input data volume of the 7-layer GRU network is a structure that first increases, then decreases, and then increases again, so that after the to-be-processed speech features of the to-be-processed speech signal are input into the pre-constructed speech enhancement network model, the model can first perform deep learning on the input features, and the input data volume of the middle layer is appropriately reduced, which can improve the data processing speed, while the input data volume of the subsequent network layer is appropriately increased, and the input features continue to be deeply learned. Not only can the input feature information be deeply learned to obtain the gain information corresponding to each to-be-processed speech feature, but also the data processing volume and complexity are reduced, and the balance between the complexity and effect of the neural network structure is achieved, thereby improving the data processing efficiency and accuracy of the speech enhancement processing, and can be applied to real-time voice call scenarios to meet the needs of real-time voice calls. In addition, the entire process does not require VAD detection, which avoids the problem of inaccurate speech enhancement results caused by VAD detection misjudgment.

[0074] In some embodiments of this specification, the input data volume of the seven-layer gated recurrent network layer in the speech enhancement network model is set to:

[0075] The input and output of the first gated recurrent network layer are used as the input of the second gated recurrent network layer, the input and output of the first gated recurrent network layer and the output of the second gated recurrent network layer are used as the input of the third gated recurrent network layer, the output of the third gated recurrent network layer is used as the input of the fourth gated recurrent network layer, the output of the fourth gated recurrent network layer is used as the input of the fifth gated recurrent network layer, the output of the fourth gated recurrent network layer and the output of the fifth gated recurrent network layer are used as the input of the sixth gated recurrent network layer, and the output of the fourth gated recurrent network layer, the output of the fifth gated recurrent network layer and the output of the sixth gated recurrent network layer are used as the input of the seventh gated recurrent network layer.

[0076] In the specific implementation process, Figure 3 is a schematic diagram of the principle of a speech enhancement network model in an embodiment of this specification, such as Figure 3 As shown, the speech enhancement network model in the embodiment of this specification includes 7 layers of GRU, wherein the input of the first layer of GRU is the original speech feature, the input of the second layer of GRU is the original speech feature (i.e., the input of the first layer of GRU) and the output of the first layer of GRU, and the input of the third layer of GRU is the input and output of the first layer of GRU and the output of the second layer of GRU. It can be seen that the input data volume of the first three layers of GRU increases successively. The input of the fourth layer of GRU is the output of the third layer, and the input of the fifth layer of GRU is the output of the fourth layer. It can be seen that the input data volume of the fourth layer of GRU and the fifth layer of GRU in the middle is reduced compared to the input data volume of the third layer of GRU, that is, the complexity of the intermediate network layer is relatively low relative to the complexity of the third layer. The input of the sixth layer of GRU is the output of the fourth layer of GRU and the output of the fifth layer of GRU. The input of the seventh layer of GRU is the output of the fourth layer of GRU, the output of the fifth layer of GRU, and the output of the sixth layer of GRU. It can be seen that the amount of input data of the last three layers of GRU network increases successively, that is, after the complexity of the network structure of the fourth and fifth layers decreases, the complexity of the network structure of the sixth and seventh layers gradually increases. After experimental verification, this network structure can not only improve the speed of data processing and meet the requirements of real-time voice communication, but also ensure the accuracy of voice enhancement, taking into account the complexity of the network model and the effect of voice enhancement.

[0077] In some embodiments of this specification, the number of nodes in the seven-layer gated recurrent network layer in the speech enhancement network model is set to:

[0078] The number of nodes in the first three gated recurrent network layers increases successively, the number of nodes in the fourth and fifth gated recurrent network layers are both smaller than the number of nodes in the third gated recurrent network layer, and the number of nodes in the last three gated recurrent network layers increases successively.

[0079] In the specific implementation process, each layer of the neural network model can contain multiple nodes, each node can represent a neuron, and the connection between nodes corresponds to the weight value. When the input variable passes through the neuron, the activation function will run to weight and bias the input value, and the output will be passed to the next layer of neurons. The weight and bias of the neural network are constantly modified during the training process. Figure 3 As shown, the number after each layer of GRU can represent the number of nodes in the GRU layer. It can be seen that in the embodiment of this specification, the number of nodes in the first three layers of GRU increases successively, and the number of nodes in the last three layers of GRU decreases successively, while the number of nodes in the fourth and fifth layers of GRU in the middle are both smaller than the number of nodes in the third layer of GRU. The specific value of the number of nodes in each layer of GRU can be set according to actual needs, and this specification embodiment does not make specific limitations. Figure 3 As shown, in some scenario instances of this specification, the node data volume of the first layer GRU can be set to 70, the node number of the second layer GRU can be set to 90, the node data volume of the third layer GRU can be set to 150, the node data volume of the fourth layer GRU can be set to 70, the node data volume of the fifth layer GRU can be set to 70, the node data volume of the sixth layer GRU can be set to 90, and the node data volume of the seventh layer GRU can be set to 150.

[0080] It can be seen that the number of nodes in each layer of the network in the speech enhancement network model of the embodiment of this specification and the change in the amount of input data in each layer of the speech enhancement network model are the same. The more nodes there are, the more complex the network structure of this layer will be and the stronger the computing power will be. By setting the number of nodes in this way, the stability of the structure of the speech enhancement network model can be ensured.

[0081] In addition, if Figure 3 As shown, in some embodiments of this specification, the activation function of each layer of GRU in the speech enhancement network model can be different, and a sigmoid activation function or a tanh hyperbolic tangent activation function can be selected. Of course, according to actual needs, other activation functions such as ReLU activation function can also be selected, and this specification embodiment does not make specific limitations. Of course, each layer of GRU can also select the same activation function, which can be determined according to actual conditions.

[0082] The speech enhancement method provided in the embodiment of this specification can be applied to real-time communication conference software. When the call environment is noisy, the background environmental noise can be effectively removed, thereby ensuring the quality and intelligibility of the voice call. When the microphone is turned on, the sound signal is collected by the microphone, and then the signal is processed using the speech enhancement method provided in the embodiment of this specification to obtain a relatively "clean" voice signal, which is regarded as an enhanced voice signal, effectively retaining the voice of the main speaker and removing excess background noise, thereby improving the quality of real-time voice calls.

[0083] Combine the following Figure 3 The process of the speech enhancement method in the embodiment of this specification is specifically introduced:

[0084] Speech sample data can be collected for training and constructing a speech enhancement model. The speech sample data can include noise sample data and corresponding clean speech data. According to the noise sample data and the corresponding clean speech data, the real gain information corresponding to the noise sample data can be calculated. At the same time, feature extraction can be performed on the noise sample data to obtain sample features corresponding to the noise sample data. When extracting speech features, the bark domain sparsification processing can be omitted for signals with a frequency band below 1000Hz, and only the bark domain division can be performed for the frequency band of 1000-8000Hz, with a total of 56 frequency bands divided. Then the corresponding BFCC parameters, i.e., the bark domain parameter features, are calculated in these 56 bark bands. Then the first-order differential coefficients of the BFCC in the first 1-6 bark bands and the second-order differential coefficients in the first 1-6 bark bands, the DCT transform values ​​of the signal correlation coefficients in the first 1-6 bark bands, the pitch (i.e., gene period) and energy parameters (i.e., gene stability) of each dimension are calculated to obtain 6+6+6+1+1=20-dimensional auxiliary features. Therefore, in some embodiments of this specification, feature extraction can extract 76-dimensional features. Then construct the model structure of the speech enhancement network model, input the sample features into the speech enhancement network model, perform model training, obtain the sample gain information corresponding to the noise sample data, compare the sample gain information with the true gain information by regression cost function, and adjust the model parameters of the speech enhancement network model until the sample gain information matches the true gain information. The specific model training process can be referred to as follows:

[0085] 1. Input training speech data, i.e. noise sample data;

[0086] 2. Extract 76-dimensional features from each frame of speech signal;

[0087] 3. Input the proposed 76-dimensional features into the first layer of GRU sigmoid network;

[0088] 4. The output of the first layer of GRU sigmoid and the original 76-dimensional input features are passed to the second layer of GRU sigmoid network for calculation;

[0089] 5. The output of the second layer GRU is combined with the output of the first layer GRU and the initial 76-dimensional features, and passed to the third layer GRU sigmoid for calculating network parameters;

[0090] 6. The output of the third layer of GRU sigmoid is then input to the fourth layer of GRU sigmoid;

[0091] 7. Input the output of the fourth layer GRU tanh directly into the fifth layer GRU tanh;

[0092] 8. Pass the output of the fifth layer GRU tanh and the 70-dimensional output of the fourth layer together to the sixth layer GRU tanh network for calculation;

[0093] 9. The output of the sixth layer GRU tanh is combined with the output of the fifth layer GRU and the 70-dimensional output features of the fourth layer, and passed to the seventh layer GRU tanh for calculating network parameters;

[0094] 10. The seventh layer of GRU tanh is activated by the dense sigmoid function to obtain the gain information Gain outputs of the corresponding dimension, such as Figure 3 As shown, a 56-dimensional gain can be obtained, which means that the auxiliary features have no corresponding gain information;

[0095] 11. The estimated 56-dimensional gain is compared with the actual gain (calculated by clean speech and noisy speech signals) by using the MSE (Mean Squared Error) cost function to obtain the model parameters of the entire network and train the speech enhancement network model.

[0096] After the speech enhancement network model is trained, it can be used for online enhancement processing. When speech enhancement processing is required, the corresponding features of the noisy speech signal can be extracted first (consistent with the above features), and then the features are input into the trained speech enhancement network model. The speech enhancement network model will perform adaptive parameter adaptation to obtain the optimal gain information. Finally, the gain is multiplied by the power spectrum of the noisy speech signal, and the final enhanced speech signal can be obtained by using the inverse FFT transform.

[0097] Figure 4 is a schematic diagram of an original noisy speech signal in one embodiment of this specification, Figure 5 is a schematic diagram of a speech signal after speech enhancement processing in one embodiment of this specification, such as Figure 4 , Figure 5 As shown, it can be seen that the speech enhancement method provided in the embodiment of this specification has a good enhancement processing effect. In the speech segment and the non-speech segment, the noise removed by the speech enhancement method provided in the embodiment of this specification is relatively clean. In addition, the parameter amount of the speech enhancement method provided in the embodiment of this specification is about 500,000 GRU parameters, which meets the requirements of real-time communication and has a relatively fast data processing speed.

[0098] Based on the speech enhancement method described above, one or more embodiments of this specification also provide a client and a server for speech enhancement processing. The client and the server may include a device (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiment of this specification and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware for predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0099] It can be seen from the technical solutions provided in the above embodiments of this specification that the embodiments of this specification also provide a speech enhancement device. Figure 6 is a structural diagram of a speech enhancement device provided in an embodiment of this specification, such as Figure 6 As shown, the above device comprises:

[0100] A signal receiving module 610, configured to receive a speech signal to be processed;

[0101] The feature extraction module 620 is used to extract features of the speech signal to be processed to obtain speech features of the speech signal to be processed;

[0102] The gain calculation module 630 is used to input the speech feature to be processed into a pre-built speech enhancement network model, and use the speech enhancement network model to obtain the target gain information corresponding to the speech feature to be processed; wherein the speech enhancement network model includes seven gated recurrent network layers, the input data amounts of the first three gated recurrent network layers increase in sequence, the input data amounts of the fourth gated recurrent network layer and the fifth gated recurrent network layer are both less than the input data amount of the third gated recurrent network layer, and the input data amounts of the last three gated recurrent network layers increase in sequence;

[0103] The signal enhancement module 640 is configured to obtain a speech enhancement signal according to the target gain information and the speech signal to be processed.

[0104] In some embodiments of the present specification, the device further includes a speech enhancement model construction module, and the speech enhancement model construction module is specifically used to set the input data volume of the seven-layer gated recurrent network layer in the speech enhancement network model to:

[0105] The input and output of the first gated recurrent network layer are used as the input of the second gated recurrent network layer, the input and output of the first gated recurrent network layer and the output of the second gated recurrent network layer are used as the input of the third gated recurrent network layer, the output of the third gated recurrent network layer is used as the input of the fourth gated recurrent network layer, the output of the fourth gated recurrent network layer is used as the input of the fifth gated recurrent network layer, the output of the fourth gated recurrent network layer and the output of the fifth gated recurrent network layer are used as the input of the sixth gated recurrent network layer, and the output of the fourth gated recurrent network layer, the output of the fifth gated recurrent network layer and the output of the sixth gated recurrent network layer are used as the input of the seventh gated recurrent network layer.

[0106] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be elaborated in detail here. The device in the above embodiment may also include other implementation methods according to the description of the method embodiment, and the specific implementation method can refer to the description of the relevant method embodiment, which will not be repeated here one by one.

[0107] Figure 7 is a block diagram of an electronic device for speech enhancement provided by an embodiment of this specification. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The electronic device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a voice enhancement method is implemented. The display screen of the electronic device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the housing of the electronic device, or an external keyboard, touchpad or mouse, etc.

[0108] Figure 8 is a block diagram of another electronic device for speech enhancement provided by an embodiment of this specification. The electronic device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The electronic device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech enhancement method is implemented.

[0109] Those skilled in the art will understand that Figure 7 or Figure 8 The structure shown in the figure is merely a block diagram of a portion of the structure related to the embodiment scheme of this specification, and does not constitute a limitation on the electronic device to which the embodiment scheme of this specification is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0110] In an exemplary embodiment, an electronic device is also provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the speech enhancement method as described in the embodiments of this specification.

[0111] In an exemplary embodiment, a computer-readable storage medium is also provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the speech enhancement method in the embodiment of this specification.

[0112] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the speech enhancement method provided in the above various optional implementations.

[0113] It is understandable that in the specific implementation of this specification, user-related data is involved. When the above embodiments of this specification are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this specification can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] Those skilled in the art will readily appreciate other implementations of the embodiments of the present specification after considering the specification and practicing the invention disclosed herein. The present specification is intended to cover any variations, uses, or adaptations of the embodiments of the present specification, which follow the general principles of the embodiments of the present specification and include common knowledge or customary technical means in the art that are not disclosed in the embodiments of the present specification. The description and embodiments are to be considered merely as exemplary, and the true scope and spirit of the embodiments of the present specification are indicated by the following claims.

[0116] It should be understood that the embodiments of the present specification are not limited to the precise structures described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the embodiments of the present specification is limited only by the appended claims.

Claims

1. A speech enhancement method, characterized in that: The method comprises: receiving a voice signal to be processed; Extracting features of the speech signal to be processed to obtain speech features of the speech signal to be processed; Input the speech feature to be processed into a pre-built speech enhancement network model, and use the speech enhancement network model to obtain the target gain information corresponding to the speech feature to be processed; wherein the speech enhancement network model includes seven gated recurrent network layers, the input data amounts of the first three gated recurrent network layers increase in sequence, the input data amounts of the fourth gated recurrent network layer and the fifth gated recurrent network layer are both smaller than the input data amount of the third gated recurrent network layer, and the input data amounts of the last three gated recurrent network layers increase in sequence; A speech enhancement signal is obtained according to the target gain information and the speech signal to be processed.

2. The method according to claim 1, characterized in that: The input data volume of the seven-layer gated recurrent network layer in the speech enhancement network model is set as: The input and output of the first gated recurrent network layer are used as the input of the second gated recurrent network layer, the input and output of the first gated recurrent network layer and the output of the second gated recurrent network layer are used as the input of the third gated recurrent network layer, the output of the third gated recurrent network layer is used as the input of the fourth gated recurrent network layer, the output of the fourth gated recurrent network layer is used as the input of the fifth gated recurrent network layer, the output of the fourth gated recurrent network layer and the output of the fifth gated recurrent network layer are used as the input of the sixth gated recurrent network layer, and the output of the fourth gated recurrent network layer, the output of the fifth gated recurrent network layer and the output of the sixth gated recurrent network layer are used as the input of the seventh gated recurrent network layer.

3. The method according to claim 1, characterized in that The number of nodes in the seven-layer gated recurrent network layer in the speech enhancement network model is set to: The number of nodes in the first three gated recurrent network layers increases successively, the number of nodes in the fourth gated recurrent network layer and the fifth gated recurrent network layer are both smaller than the number of nodes in the third gated recurrent network layer, and the number of nodes in the last three gated recurrent network layers increases successively.

4. The method according to claim 1, characterized in that: The extracting features of the speech signal to be processed to obtain speech features of the speech signal to be processed includes: Performing sparse processing on the speech signals with a frequency greater than a preset frequency in the speech signals to be processed to obtain a sparse frequency band corresponding to the speech signals to be processed; Feature extraction is performed on the sparse frequency band to obtain the speech feature to be processed.

5. The method according to claim 4, characterized in that The sparse processing of the speech signals with a frequency greater than a preset frequency in the speech signals to be processed to obtain the sparse frequency bands corresponding to the speech signals to be processed includes: Performing Bark domain sparse processing on the bandwidth greater than the preset frequency in the speech signal to be processed to obtain a specified number of sparse frequency bands; The step of extracting features from the sparse frequency band to obtain the speech features to be processed includes: According to the specified number of sparse frequency bands, respectively obtain the Bark frequency cepstral coefficient corresponding to each sparse frequency band, and obtain the specified number of Bark domain features; Obtain a preset number of auxiliary features respectively according to the Bark frequency cepstral coefficients corresponding to the specified sparse frequency band; The specified number of Bucky domain features and the preset number of auxiliary features are used as the speech features to be processed.

6. The method according to claim 5, characterized in that The step of obtaining a preset number of auxiliary features according to the Bark frequency cepstral coefficients corresponding to the specified sparse frequency band comprises: According to the bandwidth of the sparse frequency band, a specified number of sparse frequency bands are sorted, and the sparse frequency band sorted at the specified position is used as the specified sparse frequency band; Obtaining a first auxiliary feature according to a first-order difference coefficient of a Bark frequency cepstral coefficient corresponding to the specified sparse frequency band; Obtaining a second auxiliary feature according to the second-order difference coefficients of the Bark frequency cepstral coefficients corresponding to the specified sparse frequency band; Obtaining a third auxiliary feature according to a discrete cosine transform value of the cross-correlation coefficient of the specified sparse frequency band; The pitch period, pitch stability, and the first auxiliary feature, the second auxiliary feature, and the third auxiliary feature of the speech signal to be processed are used as auxiliary features of the speech signal to be processed.

7. A speech enhancement device, characterized in that: The device comprises: A signal receiving module, used for receiving a voice signal to be processed; A feature extraction module, used to extract features of the speech signal to be processed to obtain speech features of the speech signal to be processed; A gain calculation module, used for inputting the speech feature to be processed into a pre-built speech enhancement network model, and using the speech enhancement network model to obtain target gain information corresponding to the speech feature to be processed; wherein the speech enhancement network model includes seven gated recurrent network layers, the input data amounts of the first three gated recurrent network layers increase in sequence, the input data amounts of the fourth gated recurrent network layer and the fifth gated recurrent network layer are both smaller than the input data amount of the third gated recurrent network layer, and the input data amounts of the last three gated recurrent network layers increase in sequence; A signal enhancement module is used to obtain a speech enhancement signal according to the target gain information and the speech signal to be processed.

8. The device according to claim 7, characterized in that The device also includes a speech enhancement model construction module, which is specifically used to set the input data volume of the seven-layer gated recurrent network layer in the speech enhancement network model to: The input and output of the first gated recurrent network layer are used as the input of the second gated recurrent network layer, the input and output of the first gated recurrent network layer and the output of the second gated recurrent network layer are used as the input of the third gated recurrent network layer, the output of the third gated recurrent network layer is used as the input of the fourth gated recurrent network layer, the output of the fourth gated recurrent network layer is used as the input of the fifth gated recurrent network layer, the output of the fourth gated recurrent network layer and the output of the fifth gated recurrent network layer are used as the input of the sixth gated recurrent network layer, and the output of the fourth gated recurrent network layer, the output of the fifth gated recurrent network layer and the output of the sixth gated recurrent network layer are used as the input of the seventh gated recurrent network layer.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the speech enhancement method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the speech enhancement method as claimed in any one of claims 1 to 6.

Images