Speech enhancement method, device and equipment based on feature augmentation and storage medium

Abstract

The application relates to a speech enhancement method, device and equipment based on feature augmentation and a storage medium, and the method comprises the following steps: obtaining to-be-processed speech data; obtaining to-be-processed speech audio features according to the to-be-processed speech data; inputting the to-be-processed speech audio features into a preset feature augmentation network to output a to-be-processed speech audio augmented feature tensor; wherein the to-be-processed speech audio augmented feature tensor is obtained by combining a to-be-recognized speech signal and a noise signal through a mask and a mask bit inversion; inputting the to-be-processed speech audio augmented feature tensor into the speech enhancement network to output a processed speech audio feature tensor; and obtaining non-noisy speech according to the processed speech audio feature tensor. The application solves the problem that existing speech enhancement training samples can damage original pure speech samples and reduce the discriminability of a speech enhancement model.

Description

Technical Field

[0001] This invention relates to the field of intelligent speech synthesis, specifically to speech enhancement methods, apparatus, devices, and storage media based on feature augmentation. Background Technology

[0002] The purpose of speech enhancement is to separate noise from the speech signal when speech is interfered with by noise, thereby eliminating noise, restoring clean speech, and improving speech quality and intelligibility. With the development of artificial intelligence, neural network models are widely used in the field of speech enhancement, such as SEGAN (Speech Enhancement Generative Adversarial Network) and the audio processing network WaveNet.

[0003] The training samples for speech enhancement in neural networks generally require the collection of a large number of clean speech signals and noise signals to generate new noisy samples, or the synthesis of clean speech signals and noise signals based on noise degradation factors to generate noisy samples.

[0004] Existing speech enhancement training samples can corrupt the original clean speech samples, reducing the discriminative power of speech enhancement models. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to provide a speech enhancement method, apparatus, device, and storage medium based on feature augmentation. This method primarily utilizes a feature augmentation network to calculate a speech augmentation feature tensor from speech features, inputting the speech augmentation feature tensor to enhance the discriminative power of the speech enhancement model.

[0006] According to a first aspect of the present disclosure, a speech enhancement method based on feature augmentation is provided, comprising: obtaining speech data to be processed; obtaining speech audio features to be processed based on the speech data to be processed; outputting a speech audio augmentation feature tensor to be processed based on a preset feature augmentation network, taking the speech audio features to be processed as input; wherein the speech audio augmentation feature tensor to be processed is obtained by combining a speech signal to be identified and a noise signal through a mask and bitwise inversion of the mask; outputting a processed speech audio feature tensor based on the speech enhancement network, taking the speech audio augmentation feature tensor to be processed as input; and obtaining noisy speech based on the processed speech audio feature tensor.

[0007] Furthermore, the method further includes: obtaining noise data and clean speech data; mixing the noise data and the clean speech data to obtain comprehensive speech data; extracting features from the comprehensive speech data to obtain comprehensive speech-audio features; training the feature augmentation network with the comprehensive speech-audio features as input to obtain a comprehensive speech-audio augmented feature tensor; and training the speech enhancement network with the comprehensive speech-audio augmented feature tensor as training samples.

[0008] Further, the feature augmentation network comprises: a first convolutional branch, a second convolutional branch, and an augmentation module; wherein the first convolutional branch and the second convolutional branch have the same structure but different parameters; training the feature augmentation network comprises: using the comprehensive speech-audio features as training samples, training the first convolutional branch to extract features corresponding to the clean speech data from the comprehensive speech-audio features; using the comprehensive speech-audio features as training samples, training the second convolutional branch to extract features corresponding to the noise data from the comprehensive speech-audio features; using the features corresponding to the noise data and the features corresponding to the clean speech data as inputs, training the augmentation module to generate a comprehensive speech-audio augmented feature tensor based on a binary mask, a bit-inverted binary mask, the features corresponding to the noise data, and the features corresponding to the clean speech data.

[0009] Furthermore, the first convolutional branch includes modules 1 and 2 connected in series. Module 1 includes three convolutional layers with a kernel size of 3×3 and one average pooling layer. Module 2 includes a learnable convolutional kernel of size K×N×3×3, where K is the number of output samples for feature augmentation and N is the number of input feature samples. The input of module 1 is connected to the output of module 2 via a skip connection unit. The second convolutional branch includes modules 3 and 4 connected in series. Module 3 includes three convolutional layers with a kernel size of 3×3 and one average pooling layer. Module 4 includes a learnable convolutional kernel of size K×N×3×3, where K is the number of output samples for feature augmentation and N is the number of input feature samples. The input of module 3 is connected to the output of module 4 via a skip connection unit.

[0010] Furthermore, the training of the augmentation module generates a comprehensive speech-audio augmentation feature tensor based on a binary mask, a bit-inverted binary mask, features corresponding to the noise data, and features corresponding to the clean speech data, including: calculating the comprehensive speech-audio augmentation feature tensor using the following formula: ⊙ ⊙ ;in, ( ) represents the comprehensive speech audio augmentation feature tensor. ( Let C be the tensor of the processed speech audio features, and C be the number of convolution channels. ⊙ represents convolution calculation, and ⊙ represents matrix dot product. For learnable convolutional kernels of size K×N×3×3, For learnable convolutional kernels of size K×N×3×3, Let N be the binary mask for the speech signal corresponding to N input feature samples. It is a binary mask for the noise signal corresponding to N input feature samples.

[0011] Furthermore, based on a preset feature augmentation network, the speech audio features to be processed are used as input to output an augmented speech audio feature tensor. The preset feature augmentation network includes a first convolutional branch, a second convolutional branch, and an augmentation module. The first and second convolutional branches have the same structure but different parameters. Using the speech audio features to be processed as input, the first convolutional branch extracts features corresponding to the clean speech data from the speech audio features to be processed. Using the speech audio features to be processed as input, the second convolutional branch extracts features corresponding to the noise data from the speech audio features to be processed. Using the features corresponding to the noise data and the features corresponding to the clean speech data as input, the augmentation module generates an augmented speech audio feature tensor based on a binary mask, a bit-inverted binary mask, the features corresponding to the noise data, and the features corresponding to the clean speech data.

[0012] According to a second aspect of the present disclosure, a speech enhancement apparatus based on feature augmentation is provided, comprising: a data acquisition module for acquiring speech data to be processed; The data processing module is used to obtain the audio features of the speech to be processed based on the speech data to be processed; The data augmentation module is used to output the augmented feature tensor of the speech audio to be processed, based on a preset feature augmentation network and taking the speech audio features to be processed as input. The audio augmentation feature tensor to be processed is obtained by combining the speech signal and noise signal through a mask and bitwise inversion of the mask; The speech enhancement module is used to take the augmented feature tensor of the speech audio to be processed as input and output the processed speech audio feature tensor based on the speech enhancement network. The voice output module is used to obtain noisy voice based on the processed voice audio feature tensor.

[0013] Furthermore, the device further includes: The network training module is used to obtain noisy data and clean speech data; The noise data and the clean speech data are used as the combined speech data; Based on the comprehensive speech data, comprehensive speech audio features are obtained; The feature augmentation network is trained using the comprehensive speech and audio features as input. The speech enhancement network is trained using the comprehensive speech audio augmentation feature tensor as input.

[0014] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; and a processor for reading executable instructions from the memory and executing the instructions to implement the feature-augmented speech enhancement method provided in the first aspect of the present disclosure.

[0015] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the feature-enhanced speech enhancement method provided in the first aspect of the present disclosure.

[0016] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects: This application achieves speech recognition by combining a feature augmentation network and a speech enhancement network. The feature augmentation network serves three purposes: 1. During the training phase, it augments the features of the comprehensive speech data, obtaining a large number of training samples for the speech enhancement network, thus reducing the time required to collect training samples; 2. The audio feature tensor of the speech to be processed is obtained from the comprehensive speech data features, which are part of the real sound signal, therefore the feature augmentation network does not destroy the sound signal; 3. Tensors have multiple dimensions, and in this application, the dimension refers to the combination of noise and pure tone signals. Utilizing the multi-dimensional nature of tensors, multiple combination methods are input as a whole feature into the speech enhancement network, allowing the network to recognize the sound signal from multiple perspectives, thereby improving the recognition accuracy of the speech enhancement network. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in one or more embodiments of this specification or in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a feature-enhanced speech enhancement method according to an exemplary embodiment; Figure 2 This is a flowchart illustrating the training process of the augmented network in Example 1 according to an exemplary embodiment; Figure 3 This is a feature augmentation network construction diagram illustrating a feature augmentation-based speech enhancement method according to an exemplary embodiment; Figure 4 This is a schematic diagram of an apparatus for a feature-enhanced speech enhancement method according to an exemplary embodiment. Detailed Implementation

[0019] To enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of this document.

[0020] Speech enhancement (SE) transforms noisy audio into clean audio. This invention extracts speech audio features, which are then processed by an augmentation network to derive an audio feature augmentation tensor, increasing the accuracy of speech enhancement. Existing speech enhancement network training samples are generally generated using two methods: one is direct synthesis from clean speech and noise signals, and the other is machine synthesis based on noise degradation factors. Existing technologies using direct synthesis from clean and noise signals require collecting a large amount of both noise and clean speech signals, which is time-consuming. For example, using rain sounds as noise requires waiting for rainy days to collect the noise; at other times, one simply has to wait. Machine-synthesized training samples are susceptible to degradation factors that corrupt the original clean speech samples, reducing the discriminative power of the speech enhancement model. In this feature augmentation network, convolutional branches extract pure tone features and noise features separately. Preprocessing the speech audio before inputting it into the speech enhancement network helps separate noise.

[0021] Exemplary Method 1 Figure 1 This is a flowchart illustrating a feature-enhanced speech enhancement method according to an exemplary embodiment, comprising the following steps: In step S101, the voice data to be processed is obtained.

[0022] In step S102, the audio features of the speech to be processed are obtained based on the speech data to be processed.

[0023] The extracted features of the speech data to be processed include at least one of amplitude spectrum, Mel spectrogram, and spectrogram; in this embodiment, amplitude spectrum is selected as the feature.

[0024] Amplitude spectrum: The Fourier transform converts a time-domain signal into a frequency-domain signal. In the frequency domain, the signal can be represented as an amplitude and phase spectrum. The amplitude spectrum represents the magnitude of the signal's amplitude at different frequencies.

[0025] Mel-spectrogram: In 1937, Stevens, Volkmann, and Newmann proposed a unit of pitch to make equal pitch intervals sound equal to the listener; this is called the Mel scale. Mathematical operations are performed on frequencies to convert them to the Mel scale. The Mel-spectrogram is a spectrum of frequencies converted to the Mel scale.

[0026] Spectrogram: The horizontal axis of a spectrogram represents time, the vertical axis represents frequency, and the values ​​at each coordinate point represent the energy of the speech data. Because it uses a two-dimensional plane to represent three-dimensional information, the energy value is represented by color; a darker color indicates stronger speech energy at that point.

[0027] In step S103, based on a preset feature augmentation network, the speech audio features to be processed are used as input to output the speech audio augmentation feature tensor to be processed.

[0028] The audio augmentation feature tensor to be processed is obtained by combining the speech signal to be identified and the noise signal through a mask and bitwise inversion of the mask.

[0029] Figure 3 This is a feature augmentation network construction diagram illustrating a feature augmentation-based speech enhancement method according to an exemplary embodiment.

[0030] The speech audio features to be processed are taken as input and fed into a preset feature augmentation network. The network passes through a first convolutional branch As and a second convolutional branch An. The first convolutional branch As extracts the features corresponding to the clean speech data from the speech audio features to be processed. The second convolutional branch An extracts the features corresponding to the noisy data from the speech audio features to be processed.

[0031] First convolutional branch As: The first convolutional branch As includes a first module 301 and a second module 302 connected in series. The first module 301 includes three convolutional layers with a kernel size of 3×3 and an average pooling layer. The second module 302 includes a learnable convolutional kernel with a size of K×N×3×3, where K is the number of feature augmentation output samples and N is the number of input feature samples.

[0032] The input of the first module 301 is connected to the output of the second module 302 through a jump connection unit.

[0033] The second convolutional branch includes a third module 303 and a fourth module 304 connected in series. The third module 303 includes three convolutional layers with a kernel size of 3×3 and an average pooling layer. The third module 303 includes a learnable convolutional kernel with a size of K×N×3×3, where K is the number of feature augmentation output samples and N is the number of input feature samples.

[0034] The input of the third module 303 is connected to the output of the fourth module 304 through a jump connection unit.

[0035] The features corresponding to the clean speech data, the features corresponding to the noise data, the speech audio features to be processed, and the augmentation module 305 calculate the augmented feature tensor of the speech audio to be processed based on the formula.

[0036] A convolutional kernel can be viewed as a weighted summation of a local area; it corresponds to local perception. Its principle is that when observing an object, we cannot observe every pixel or the whole at once, but rather begin to perceive it from the local perspective, which corresponds to convolution. In the embodiments of this application, convolutional kernels are used to construct the augmented feature tensor of the speech audio to be processed.

[0037] Specifically, the augmented feature tensor of the speech audio to be processed includes various combinations of noise and pure tone. In order to obtain various combinations of noise and pure tone, this application first extracts various noise features and various pure tone features from the speech audio features to be processed based on two convolution kernels, and then combines these noise features and pure tone features to obtain various combinations of noise and pure tone, that is, to obtain the augmented feature tensor of the speech audio to be processed.

[0038] Convolutional neural networks (CNNs) extract local features from audio through convolutional kernels, generate neurons, and then construct them through deep connections.

[0039] A convolutional layer is the sum of the products obtained by multiplying the input matrices at the same positions with the convolution kernel matrix. In this embodiment, the convolutional layer compresses the extracted features.

[0040] A pooling layer is used to compress data; it reduces each element of the input submatrix by 1 / 2 1 / 3 1 / 4 ...3 1 / 4 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 1 / 3 One element becomes one element.

[0041] The augmentation module 305 includes a binary mask for the speech signal and a binary mask for the noise signal.

[0042] The formula is as follows: ⊙ ⊙ ; in, ( ) represents the augmented feature tensor of the speech audio to be processed. ( Let C be the tensor of the processed speech audio features, and C be the number of convolution channels. ⊙ represents convolution calculation, and ⊙ represents matrix dot product. For learnable convolutional kernels of size K×N×3×3, For learnable convolutional kernels of size K×N×3×3, Let N be the binary mask for the speech signal corresponding to N input feature samples. It is a binary mask for the noise signal corresponding to N input feature samples.

[0043] A mask typically refers to a special type of tensor used to control how certain inputs or outputs are processed. Specifically, a mask is a binary tensor with the same shape as the input or output tensor, where each element is either 1 or 0. A 1 in the mask indicates that the corresponding input or output needs to be included in the calculation, while a 0 indicates that the input or output at that position should be ignored. In this embodiment, a pure tone feature is introduced into the tensor using a mask Ms, and noise features are introduced into the tensor using a mask 1-Ms, thereby achieving the fusion of noise and pure tone features and obtaining various combinations of noise and pure tone.

[0044] In step S104, based on the speech enhancement network, the processed speech audio feature tensor is obtained by taking the speech audio augmentation feature tensor to be processed as input.

[0045] Speech enhancement networks, including but not limited to convolutional neural networks, recurrent neural networks, or long short-term memory networks, have their network parameters jointly optimized with the feature augmentation network through gradient backpropagation to extract noisy speech features. In this embodiment, a long short-term memory network is used as the speech enhancement network.

[0046] Convolutional Neural Networks (CNNs) are a type of feedforward neural network that incorporates convolutional computations and has a deep structure. CNNs are inspired by the biological mechanism of receptive fields. They are specifically designed to process data with a grid-like structure.

[0047] Recurrent Neural Network (RNN): It takes sequence data as input, recursively moves in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain.

[0048] Long Short-Term Memory (LSTM) network: a type of temporal recurrent neural network suitable for processing and predicting important events with relatively long intervals and delays in time series.

[0049] In step S105, the noisy speech is obtained based on the processed speech audio feature tensor.

[0050] The processed speech features are reconstructed into a speech waveform to obtain noisy speech.

[0051] Exemplary Method Two Figure 2 This is a flowchart illustrating the training process of the augmented network in Example 1 according to an exemplary embodiment.

[0052] In step S201, noise data and clean speech data are obtained.

[0053] In this embodiment of the application, 5,000 clean speech data points and 5,000 noisy speech data points are collected as training datasets.

[0054] In step S202, the noise data and the clean speech data are mixed to obtain comprehensive speech data.

[0055] In step S203, feature extraction is performed on the integrated speech data to obtain integrated speech audio features.

[0056] The extracted comprehensive speech audio features include at least one of amplitude spectrum, Mel spectrogram, and spectrogram; in this embodiment, amplitude spectrum is selected as the feature.

[0057] In step S204, the feature augmentation network is trained using the comprehensive speech and audio features as input to obtain the comprehensive speech and audio augmentation feature tensor.

[0058] The feature augmentation network includes: a first convolutional branch, a second convolutional branch, and an augmentation module; wherein the first convolutional branch and the second convolutional branch have the same structure but different parameters.

[0059] Training the feature augmentation network includes: Using the comprehensive speech and audio features as training samples, the first convolutional branch As is trained to extract the features corresponding to the clean speech data from the comprehensive speech and audio features; Using the comprehensive speech and audio features as training samples, the second convolutional branch An is trained to extract the features corresponding to the noise data from the comprehensive speech and audio features; Using the features corresponding to the noise data and the features corresponding to the clean speech data as input, the augmentation module is trained to generate a comprehensive speech audio augmentation feature tensor based on a binary mask, a bit-inverted binary mask, the features corresponding to the noise data, and the features corresponding to the clean speech data.

[0060] First convolutional branch As: The first convolutional branch As includes a first module 301 and a second module 302 connected in series. The first module 301 includes three convolutional layers with a kernel size of 3×3 and an average pooling layer. The second module 302 includes a learnable convolutional kernel with a size of K×N×3×3, where K is the number of feature augmentation output samples and N is the number of input feature samples.

[0061] The input of the first module 301 is connected to the output of the second module 302 through a jump connection unit.

[0062] The second convolutional branch includes a third module 303 and a fourth module 304 connected in series. The third module 303 includes three convolutional layers with a kernel size of 3×3 and an average pooling layer. The third module 303 includes a learnable convolutional kernel with a size of K×N×3×3, where K is the number of feature augmentation output samples and N is the number of input feature samples.

[0063] The input of the third module 303 is connected to the output of the fourth module 304 through a jump connection unit.

[0064] The comprehensive speech audio augmentation feature tensor is calculated using the following formula: ⊙ ⊙ ; in, ( ) represents the comprehensive speech audio augmentation feature tensor. ( Let C be the tensor of the processed speech audio features, and C be the number of convolution channels. ⊙ represents convolution calculation, and ⊙ represents matrix dot product. For learnable convolutional kernels of size K×N×3×3, For learnable convolutional kernels of size K×N×3×3, Let N be the binary mask for the speech signal corresponding to N input feature samples. It is a binary mask for the noise signal corresponding to N input feature samples.

[0065] In this embodiment, N=10 and K=20 are selected. The augmenting network generates a large number of comprehensive speech and audio augmentation feature tensors to achieve the goal of collecting only a limited number of data samples (5000 clean speech data and 5000 noisy speech data) to obtain a large number of training samples (the specific number is determined by the value of K; in this embodiment, 20,000 training samples are used. If you want to continue to increase the number of training samples, you can increase the value of K).

[0066] In step S205, the speech enhancement network is trained using the comprehensive speech audio augmentation feature tensor as training samples.

[0067] Exemplary device Figure 4 This is a schematic diagram of an apparatus for a feature-enhanced speech enhancement method according to an exemplary embodiment.

[0068] In this embodiment of the application, the data acquisition module 401 is used to acquire the voice data to be processed; Data processing module 402 is used to obtain audio features of the speech to be processed based on the speech data to be processed; Data augmentation module 403 is used to output a speech audio augmentation feature tensor based on a preset feature augmentation network, taking the speech audio features to be processed as input; wherein the speech audio augmentation feature tensor to be processed is obtained by combining the speech signal and noise signal through a mask and bitwise inversion of the mask; The speech enhancement module 404 is used to take the augmented feature tensor of the speech audio to be processed as input and output the processed speech audio feature tensor based on the speech enhancement network. The voice output module 405 is used to obtain noisy voice based on the processed voice audio feature tensor.

[0069] The network training module 406 is used to obtain noise data and clean speech data; use the noise data and the clean speech data as comprehensive speech data; obtain comprehensive speech audio features based on the comprehensive speech data; train the feature augmentation network using the comprehensive speech audio features as input; and train the speech enhancement network using the comprehensive speech audio augmentation feature tensor as input.

[0070] Exemplary electronic devices This embodiment proposes an electronic device, including: one or more processors, an internal memory, and an external memory. The internal memory stores instructions, which, when executed by the one or more processors, cause the one or more processors to execute a comprehensive speech and audio augmentation feature tensor method as described in any of the preceding embodiments.

[0071] The processor is used to execute all or part of the steps in the comprehensive speech audio augmentation feature tensor method as described in the embodiments. The memory is used to store various types of data, which may include, for example, instructions for any application or method in the electronic device, as well as application-related data.

[0072] The processor may be implemented as an Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor, or other electronic components, and is used to execute a comprehensive speech and audio augmentation feature tensor as described in the embodiments.

[0073] Computer storage media A computer program is stored on the computer-readable storage medium, which, when executed by one or more processors, implements a method for synthesizing speech and audio augmentation feature tensors as described in any of the foregoing embodiments.

[0074] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0075] In the 1930s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement to the methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0076] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0077] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0078] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.

[0079] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0084] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0085] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0086] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0087] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0088] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0089] The above description is merely an embodiment of this document and is not intended to limit the scope of this document. Various modifications and variations can be made to this document by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this document should be included within the scope of the claims of this document.

Claims

1. A speech enhancement method based on feature augmentation, characterized in that, include: Obtain the voice data to be processed; Based on the speech data to be processed, the audio features of the speech to be processed are obtained; Based on a preset feature augmentation network, the speech audio features to be processed are taken as input, and the speech audio augmentation feature tensor to be processed is output; wherein, the speech audio augmentation feature tensor to be processed is obtained by combining the speech signal to be recognized and the noise signal through a mask and bitwise inversion of the mask; Based on the speech enhancement network, the speech audio augmentation feature tensor to be processed is taken as input, and the processed speech audio feature tensor is output. Based on the processed speech audio feature tensor, the noisy speech is obtained; The method further includes: Obtain noise data and clean speech data; The noise data and the clean speech data are mixed to obtain composite speech data; Feature extraction is performed on the comprehensive speech data to obtain comprehensive speech audio features; The feature augmentation network is trained using the comprehensive speech and audio features as input to obtain the comprehensive speech and audio augmentation feature tensor; The speech enhancement network is trained using the comprehensive speech audio augmentation feature tensor as training samples; The feature augmentation network includes: a first convolutional branch, a second convolutional branch, and an augmentation module; wherein the first convolutional branch and the second convolutional branch have the same structure but different parameters; Training the feature augmentation network includes: Using the comprehensive speech and audio features as training samples, the first convolutional branch is trained to extract the features corresponding to the clean speech data from the comprehensive speech and audio features; Using the comprehensive speech and audio features as training samples, the second convolutional branch is trained to extract the features corresponding to the noise data from the comprehensive speech and audio features; Using the features corresponding to the noise data and the features corresponding to the clean speech data as input, the augmentation module is trained to generate a comprehensive speech audio augmentation feature tensor based on a binary mask, a bit-inverted binary mask, the features corresponding to the noise data, and the features corresponding to the clean speech data. The training of the augmentation module generates a comprehensive speech audio augmentation feature tensor based on a binary mask, a bit-inverted binary mask, features corresponding to the noise data, and features corresponding to the clean speech data, including: The comprehensive speech audio augmentation feature tensor is calculated using the following formula: ⊙ ⊙ ； in, ( ) represents the comprehensive speech audio augmentation feature tensor. ( Let C be the tensor of the processed speech audio features, and C be the number of convolution channels. ⊙ represents convolution calculation, and ⊙ represents matrix dot product. For learnable convolutional kernels of size K×N×3×3, For learnable convolutional kernels of size K×N×3×3, Let N be the binary mask for the speech signal corresponding to N input feature samples. It is a binary mask for the noise signal corresponding to N input feature samples.

2. The method according to claim 1, characterized in that, The first convolutional branch includes module 1 and module 2 connected in series. Module 1 includes three convolutional layers with a kernel size of 3×3 and an average pooling layer. Module 2 includes a learnable convolutional kernel with a size of K×N×3×3, where K is the number of feature augmentation output samples and N is the number of input feature samples. The input of module 1 is connected to the output of module 2 via a jump connection unit; The second convolutional branch includes modules 3 and 4 connected in series. Module 3 includes three convolutional layers with a kernel size of 3×3 and an average pooling layer. Module 4 includes a learnable convolutional kernel with a size of K×N×3×3, where K is the number of feature augmentation output samples and N is the number of input feature samples. The input of module 3 is connected to the output of module 4 via a jump connection unit.

3. The method according to claim 1, characterized in that, The output speech audio augmentation feature tensor to be processed includes: Based on a preset feature augmentation network, the speech audio features to be processed are taken as input, and the speech audio augmentation feature tensor to be processed is output. The preset feature augmentation network includes: a first convolutional branch, a second convolutional branch, and an augmentation module; wherein the first convolutional branch and the second convolutional branch have the same structure but different parameters; Using the audio features to be processed as input, the first convolutional branch extracts the features corresponding to the clean speech data from the audio features to be processed; Using the speech and audio features to be processed as input, the second convolutional branch extracts the features corresponding to the noise data from the speech and audio features to be processed; Using the features corresponding to the noise data and the features corresponding to the clean speech data as input, the augmentation module generates an augmentation feature tensor for the speech audio to be processed based on a binary mask, a bit-inverted binary mask, the features corresponding to the noise data, and the features corresponding to the clean speech data.

4. A speech enhancement device based on feature augmentation, characterized in that, include: The data acquisition module is used to acquire the voice data to be processed; The data processing module is used to obtain the audio features of the speech to be processed based on the speech data to be processed; The data augmentation module is used to output the augmented feature tensor of the speech audio to be processed, based on a preset feature augmentation network and taking the speech audio features to be processed as input. The audio augmentation feature tensor to be processed is obtained by combining the speech signal and noise signal through a mask and bitwise inversion of the mask; The speech enhancement module is used to take the augmented feature tensor of the speech audio to be processed as input and output the processed speech audio feature tensor based on the speech enhancement network. The voice output module is used to obtain noisy voice based on the processed voice audio feature tensor; The device further includes: The network training module is used to obtain noisy data and clean speech data; The noise data and the clean speech data are used as the combined speech data; Based on the comprehensive speech data, comprehensive speech audio features are obtained; The feature augmentation network is trained using the comprehensive speech and audio features as input. The speech enhancement network is trained using the comprehensive speech audio augmentation feature tensor as input; The feature augmentation network includes: a first convolutional branch, a second convolutional branch, and an augmentation module; wherein the first convolutional branch and the second convolutional branch have the same structure but different parameters; Training the feature augmentation network includes: Using the comprehensive speech and audio features as training samples, the first convolutional branch is trained to extract the features corresponding to the clean speech data from the comprehensive speech and audio features; Using the comprehensive speech and audio features as training samples, the second convolutional branch is trained to extract the features corresponding to the noise data from the comprehensive speech and audio features; Using the features corresponding to the noise data and the features corresponding to the clean speech data as input, the augmentation module is trained to generate a comprehensive speech audio augmentation feature tensor based on a binary mask, a bit-inverted binary mask, the features corresponding to the noise data, and the features corresponding to the clean speech data. The training of the augmentation module generates a comprehensive speech audio augmentation feature tensor based on a binary mask, a bit-inverted binary mask, features corresponding to the noise data, and features corresponding to the clean speech data, including: The comprehensive speech audio augmentation feature tensor is calculated using the following formula: ⊙ ⊙ ； in, ( ) represents the comprehensive speech audio augmentation feature tensor. ( Let C be the tensor of the processed speech audio features, and C be the number of convolution channels. ⊙ represents convolution calculation, and ⊙ represents matrix dot product. For learnable convolutional kernels of size K×N×3×3, For learnable convolutional kernels of size K×N×3×3, Let N be the binary mask for the speech signal corresponding to N input feature samples. It is a binary mask for the noise signal corresponding to N input feature samples.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the feature augmentation-based speech enhancement method as described in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the feature augmentation-based speech enhancement method as described in any one of claims 1 to 3.

Images