Audio data processing method and device
By using BFCC and Pitch features for feature superposition processing in real-time communication scenarios, the problem of high computational complexity in the prior art is solved, and the effect of rapidly suppressing noise and improving speech quality is achieved.
Patent Information
- Application Number
- CN202110362289.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-02
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-04-02
AI Technical Summary
The prior art has poor suppression effect on transient noise due to high computational complexity in real-time communication scenarios, which affects voice quality.
By extracting the Buck frequency cepspectral coefficient (BFCC) and the fundamental pitch Pitch features, the feature dimension is reduced, and feature superposition processing is performed in a three-layer recurrent neural network to reduce the amount of model parameters and calculation complexity, and noise is quickly suppressed.
It realizes rapid suppression of noise in real-time communication scenarios, improves voice quality, obtains clearer and pure vocal signals, and reduces the computational complexity.
Smart Images

Figure CN113096679B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the technical field of audio processing, and in particular, to a method and device for processing audio data. Background Art
[0002] In recent years, the development of network communication technologies such as voice communication and video communication has greatly facilitated people's communication needs. However, during the communication process, the audio data is often affected by noise such as environmental noise and noise in the transmission channel, resulting in unclear voice and affecting the call quality. Therefore, there is an urgent need for a speech enhancement method to suppress the noise part in noisy speech, while retaining the pure speech part in the audio data as much as possible to improve the voice quality.
[0003] In the related technology, the commonly used deep learning-based speech enhancement method is the Ideal Ratio Mask (IRM) technology based on deep learning. The time domain speech data is first transformed into a two-dimensional time-frequency expression, and the energy ratio between the speech signal and the noise is calculated for each time-frequency unit. Then, the weight is determined according to the calculated energy ratio to change the energy size of the time-frequency unit, and the degree of noise suppression on each time-frequency unit is adjusted to improve the speech quality.
[0004] However, the existing methods often require a large number of neurons and weights to calculate the energy of time-frequency units, resulting in high computational complexity of the entire process. In real-time communication scenarios, transient noise needs to be suppressed, and a large number of complex calculation processes waste a lot of time, resulting in poor noise suppression effects. Summary of the invention
[0005] The embodiments of the present disclosure provide an audio data processing method and device to overcome the problem in the prior art that transient noise needs to be suppressed in real-time communication scenarios, a large number of complex calculation processes waste a lot of time, and thus lead to poor noise suppression effect.
[0006] In a first aspect, an embodiment of the present disclosure provides an audio data processing method, including:
[0007] Get time domain audio data;
[0008] Extracting a first acoustic feature and a second acoustic feature from the time-domain audio data respectively, wherein dimensions of the first acoustic feature and the second acoustic feature are smaller than a preset dimension;
[0009] Concatenating the first acoustic feature and the second acoustic feature to obtain an input feature;
[0010] Inputting the input features into a pre-established speech enhancement model to perform feature superposition processing between network layers of the speech enhancement model to obtain mask features corresponding to the audio data;
[0011] The mask feature is expanded to the frequency domain and then multiplied with the amplitude spectrum feature of the time domain audio data to obtain a human voice prediction result in the audio data.
[0012] In a second aspect, an embodiment of the present disclosure provides an audio data processing device, including:
[0013] A data acquisition module, used for acquiring time domain audio data;
[0014] A feature extraction module, configured to extract a first acoustic feature and a second acoustic feature from the time-domain audio data, respectively, wherein dimensions of the first acoustic feature and the second acoustic feature are smaller than a preset dimension;
[0015] A feature concatenation module, used for concatenating the first acoustic feature and the second acoustic feature to obtain an input feature;
[0016] A feature processing module, used for inputting the input features into a pre-established speech enhancement model to perform feature superposition processing between network layers of the speech enhancement model to obtain mask features corresponding to the audio data;
[0017] The human voice prediction module is used to expand the mask feature into the frequency domain and then multiply it with the amplitude spectrum feature of the time domain audio data to obtain the human voice prediction result in the audio data.
[0018] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;
[0019] The memory stores computer-executable instructions;
[0020] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the audio data processing method described in the first aspect and various possible designs of the first aspect.
[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the audio data processing method described in the first aspect and various possible designs of the first aspect is implemented.
[0022] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the audio data processing method described in the first aspect and various possible designs of the first aspect.
[0023] The audio data processing method and device provided by the embodiment of the present disclosure first obtain time domain audio data, then extract the first acoustic feature and the second acoustic feature from the time domain audio data respectively; concatenate the first acoustic feature and the second acoustic feature to obtain input features; input the input features into a pre-established speech enhancement model to perform feature superposition processing between the network layers of the speech enhancement model to obtain mask features corresponding to the audio data; finally, the mask features are expanded to the frequency domain and multiplied with the amplitude spectrum features of the time domain audio data to obtain the human voice prediction result in the audio data. Since the dimensions of the first acoustic feature and the second acoustic feature are smaller than the preset dimension, and the dimension of the input feature obtained by splicing the two is also smaller, after the input feature obtained by splicing the two is input into the speech enhancement model, the speech enhancement model performs feature fusion between the network layers in the form of feature superposition, and can further reduce the feature dimension while suppressing the noise in the audio data, thereby greatly reducing the number of parameters and the computational complexity of the model, thereby avoiding the waste of more time due to the complex computation process caused by the large feature dimension, and even in real-time communication scenarios, the noise in the speech data can be quickly suppressed to obtain a clearer and purer human voice signal. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0025] Figure 1 An application scenario diagram of the audio data processing method provided in an embodiment of the present disclosure;
[0026] Figure 2 The audio data processing method provided in the present disclosure is shown in FIG. Figure 1 ;
[0027] Figure 3 The audio data processing method provided in the present disclosure is shown in FIG. Figure 2 ;
[0028] Figure 4 The audio data processing method provided in the present disclosure is shown in FIG. Figure 3 ;
[0029] Figure 5 A schematic diagram of the structure of a speech enhancement model provided in an embodiment of the present disclosure;
[0030] Figure 6A schematic diagram of the structure of an audio data processing device provided in an embodiment of the present disclosure;
[0031] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0033] In recent years, the development of network communication technologies such as voice communication and video communication has greatly facilitated people's communication needs. However, during the communication process, the audio data is often affected by noise such as environmental noise and noise in the transmission channel, resulting in unclear voice and affecting the call quality. Therefore, there is an urgent need for a speech enhancement method to suppress the noise part in noisy speech, while retaining the pure speech part in the audio data as much as possible to improve the voice quality.
[0034] In the related art, the commonly used speech enhancement method based on deep learning is the technology of ideal ratio mask (IRM) based on deep learning. First, the time domain speech data is transformed into a two-dimensional time-frequency expression, and the energy ratio between the speech signal and the noise is calculated for each time-frequency unit. Then, the weight is determined according to the calculated energy ratio to change the energy size of the time-frequency unit, and the degree of noise suppression on each time-frequency unit is adjusted, thereby improving the speech quality. Alternatively, the time domain speech data is first subjected to a short-time Fourier transform (STFT) to obtain the frequency domain characteristic spectrum of the speech data, and the amplitude spectrum of the pure speech signal is directly predicted based on the frequency domain characteristic spectrum of the speech data. However, the existing above-mentioned methods often require a large number of neurons and weights for calculating the energy of the time-frequency unit, resulting in a high computational complexity of the entire process. However, since transient noise needs to be suppressed in real-time communication scenarios, a large number of complex calculation processes need to waste a lot of time, resulting in poor noise suppression effect.
[0035] To address this defect, the technical concept of the disclosed embodiment mainly includes: designing a speech enhancement model of a three-layer recurrent neural network, after acquiring the time domain audio data, first converting the time domain audio data to the complex frequency domain through a short-time Fourier transform, then converting the frequency domain features with larger dimensions into Bark frequency cepstral coefficients BFCC features with smaller dimensions, and extracting the fundamental pitch Pitch features from the audio data; concatenating the BFCC features and the Pitch features to obtain input features; then inputting the input features into the speech enhancement model of the three-layer recurrent neural network to perform feature superposition processing between the network layers of the speech enhancement model to obtain mask features corresponding to the audio data; finally, expanding the mask features to the frequency domain and multiplying them with the amplitude spectrum features of the time domain audio data to obtain the human voice prediction results in the audio data. Since the dimensions of the BFCC feature and the fundamental pitch Pitch feature are small, and the dimension of the input feature obtained by splicing the two is also small, and the BFCC feature has better human voice harmonic characteristics and the Pitch feature can well reflect the personal voice characteristics, after the input features obtained by splicing the two are input into the speech enhancement model, the speech enhancement model performs feature fusion between network layers in a feature superposition manner, which can further reduce the feature dimension while suppressing the noise in the audio data, thereby greatly reducing the number of parameters and computational complexity of the model. Even in real-time communication scenarios, it can quickly suppress the noise in the voice data and obtain a clearer and purer human voice signal.
[0036] Figure 1 This is an application scenario diagram of the audio data processing method provided in an embodiment of the present disclosure.
[0037] like Figure 1 As shown, the scenario provided by this embodiment is a scenario in which user 1 and user 2 use communication devices to conduct a real-time conversation. In this scenario, the first communication device collects voice data of user 1 in real time, and then sends the voice data to the second communication device through a wireless channel. User 2 receives the voice data in real time through the second communication device. Similarly, the voice data sent by user 2 is also sent to the first communication device in real time. Since there may be environmental noise in the environment where user 1 or user 2 is located, the voice data collected by the communication device is also mixed with noise, and the voice data is also interfered by the channel noise during transmission over the wireless channel, resulting in the voice data received by the other user containing noise, thereby reducing the voice quality.
[0038] Figure 2 The audio data processing method provided in the present disclosure is shown in FIG. Figure 1 The execution subject of the method provided in this embodiment may be Figure 1The communication device in the application scenario shown may also be a server with a voice processing function or other device with a voice processing function, which is not specifically limited in this embodiment.
[0039] like Figure 2 As shown, the method provided in this embodiment may include the following steps.
[0040] S201, obtaining time domain audio data.
[0041] Specifically, the voice acquisition device collects the audio data of the user during the communication process, and transmits the collected audio data to the audio processing device (such as the server) frame by frame, wherein one frame of audio data can be 10 milliseconds (ms) time domain audio data, or 20ms audio data, etc., which is not specifically limited here.
[0042] Among them, the voice collection device can be a mobile phone, tablet computer, walkie-talkie or other device with communication function.
[0043] S202: extracting a first acoustic feature and a second acoustic feature from the time-domain audio data respectively, wherein dimensions of the first acoustic feature and the second acoustic feature are smaller than a preset dimension.
[0044] The first acoustic feature may be a Bark Frequency Cepstral Coefficients (BFCC) feature, the second acoustic feature may be a Pitch feature, and the preset dimension may be 40 dimensions.
[0045] In this embodiment, the Bark Frequency Cepstral Coefficients (BFCC) feature has better vocal harmonic characteristics, the Pitch feature can well reflect the personal voice characteristics, and the dimensions of the BFCC feature and the Pitch feature are both small. The BFCC feature includes a 22-dimensional cepstral feature, a 6-dimensional first-order feature and a 6-dimensional second-order difference feature, a total of 34 dimensions, and the Pitch feature is an 8-dimensional vector.
[0046] Specifically, the time domain audio data is transformed into the complex frequency domain by short-time Fourier transform, and the amplitude spectrum features of the complex frequency domain are obtained. Then, the frequency domain features with larger dimensions are converted into BFCC features with smaller dimensions. Then, the audio data is calculated through the buffer to obtain the most significant pitch feature within a period of time.
[0047] The buffer is a buffer used to store a section of audio data. For example, the voice acquisition device transmits each frame of 10ms audio data to the audio processing device. The audio processing device stores the 10ms audio data frame by frame in the buffer. A buffer stores approximately 4 frames of 40ms audio data. The pitch features within the 40ms audio data stored in the buffer are calculated.
[0048] It should be noted that the detailed extraction process of the BFCC feature and the fundamental pitch Pitch feature will be described in the following embodiments.
[0049] S203: Concatenate the first acoustic feature and the second acoustic feature to obtain an input feature.
[0050] Specifically, both the BFCC feature and the Pitch feature are vectors, and the input feature can be obtained by concatenating the two vectors into one vector.
[0051] For example, the BFCC feature is a 34-dimensional vector, and the Pitch feature is an 8-dimensional vector. The two are concatenated to obtain an input feature with a dimension of 42.
[0052] S204: Input the input features into a pre-established speech enhancement model to perform feature superposition processing between network layers of the speech enhancement model to obtain mask features corresponding to the audio data.
[0053] Specifically, the speech enhancement model includes multiple network layers. The input features obtained in step S203 are input into the speech enhancement model, and feature superposition processing is performed between the network layers of the speech enhancement model. The purpose of superposition is to fuse features. Feature fusion can be performed while keeping the dimension unchanged by addition, and finally the mask features corresponding to the audio data are obtained. The mask features are used to eliminate noise in the audio data.
[0054] S205 , expanding the mask feature to the frequency domain and then multiplying the mask feature with the amplitude spectrum feature of the time domain audio data to obtain a human voice prediction result of the audio data.
[0055] Specifically, the obtained mask features are expanded to the frequency domain and then directly multiplied with the frequency domain amplitude spectrum features of the audio data to obtain a denoised amplitude spectrum, through which clearer and purer human voice information can be restored.
[0056] In the disclosed embodiment, time domain audio data is acquired, and then Bark frequency cepstral coefficient BFCC features and fundamental pitch Pitch features are respectively extracted from the time domain audio data; the BFCC features and Pitch features are concatenated to obtain input features; the input features are input into a pre-established speech enhancement model to perform feature superposition processing between network layers of the speech enhancement model to obtain mask features corresponding to the audio data; finally, the mask features are expanded to the frequency domain and multiplied with the amplitude spectrum features of the time domain audio data to obtain the human voice prediction result in the audio data. Since the dimensions of BFCC features and fundamental pitch Pitch features are small, and the dimensions of the input features obtained by splicing the two are also small, and the BFCC features have better human voice harmonic characteristics and the Pitch features can well reflect personal voice characteristics, the input features obtained by splicing the two are input into the speech enhancement model, and the speech enhancement model performs feature fusion between network layers in a feature superposition manner, which can further reduce the feature dimension while suppressing the noise in the audio data, thereby greatly reducing the number of parameters and computational complexity of the model. Even in real-time communication scenarios, the noise in the voice data can be quickly suppressed to obtain a clearer and purer voice signal.
[0057] Figure 3 The audio data processing method provided in the present disclosure is shown in FIG. Figure 2 In this embodiment, Figure 2 Based on the illustrated embodiment, the complete process of the audio data processing method is described in detail.
[0058] like Figure 3 As shown, the method provided in this embodiment may include the following steps.
[0059] S301, obtaining time domain audio data.
[0060] Specifically, the voice acquisition device collects the audio data of the user during the communication process, and transmits the collected audio data to the audio processing device (such as the server) frame by frame, wherein one frame of audio data can be 10 milliseconds (ms) time domain audio data, or 20ms audio data, etc., which is not specifically limited here.
[0061] S302: extracting Bark frequency cepstral coefficient BFCC features and fundamental pitch Pitch features from the time domain audio data.
[0062] In a possible embodiment, the process of extracting BFCC features is: performing short-time Fourier transform on the time domain audio data to obtain amplitude spectrum features of the audio data in the complex frequency domain; determining the power spectrum of the audio data based on the amplitude spectrum features; mapping the power spectrum to the Bark domain to obtain bark domain power spectrum data; taking the logarithm of the bark domain power spectrum data, and performing discrete cosine transform on the logarithm to obtain the BFCC features.
[0063] It should be noted that the dimension of the frequency domain features obtained by performing short-time Fourier transform on the time domain audio data is often above 161 dimensions. If the frequency domain features are used directly to predict the human voice, a very large amount of calculation is required. Therefore, in this embodiment, the frequency domain features with larger dimensions are converted into BFCC features with smaller dimensions. The BFCC features include 22-dimensional inverse spectrum features, a 6-dimensional first-order and a 6-dimensional second-order difference feature, totaling 34 dimensions.
[0064] In a possible embodiment, the pitch feature of the audio data can be extracted based on a fundamental pitch extraction algorithm, and the most significant pitch feature within a period of time is calculated by using an autocorrelation function on the audio data through a buffer, and the dimension of the pitch feature is 8. The buffer is a buffer used to store a section of audio data. For example, a voice acquisition device transmits each frame of 10ms audio data collected to an audio processing device. After receiving the 10ms audio data frame by frame, the audio processing device stores it in a buffer. The audio data stored in a buffer is approximately 4 frames of 40ms audio data. The pitch feature within the 40ms is obtained by calculating the 40ms audio data stored in the buffer.
[0065] In some embodiments, the fundamental pitch extraction algorithm includes three categories: time domain estimation method, transformation method and hybrid method. Among them, the time domain estimation method is to estimate the fundamental pitch period directly from the speech waveform, usually including autocorrelation function method, parallel processing method, average amplitude difference method, data reduction method, etc.; the transformation method is a method of transforming the speech signal into the frequency domain or time domain to estimate the fundamental pitch period. First, the influence of the vocal tract is eliminated by the homomorphic analysis method to obtain the information belonging to the excitation part, and then the fundamental pitch period is obtained. The most commonly used method is the inverse spectrum method. The disadvantage of this method is that the algorithm is relatively complicated, but the effect of fundamental pitch estimation is better; the hybrid method first extracts the signal vocal tract model parameters, and then uses the vocal tract model parameters to filter the signal to obtain the sound source sequence, and finally uses the autocorrelation method or the average amplitude difference method to obtain the fundamental pitch period.
[0066] S303, inputting the input feature obtained by concatenating the BFCC feature and the fundamental pitch Pitch feature into the speech enhancement model, performing feature fusion processing between the network layers of the speech enhancement model, and obtaining the mask feature output by the speech enhancement model.
[0067] Specifically, both the BFCC feature and the Pitch feature are vectors. The BFCC feature in this embodiment is a vector with a dimension of 34, and the Pitch feature is a vector with a dimension of 8. The two vectors are concatenated into one vector to obtain an input feature with a dimension of 42.
[0068] In one or more possible embodiments, the speech enhancement model includes a three-layer gated recurrent unit (GRU) network and three fully connected layers and two adders, and the model is as follows: Figure 5 As shown, the first fully connected layer is connected to the first GRU network, and the first GRU network is connected to the second GRU network through the first adder, and the input of the first adder includes the output of the first fully connected layer and the output of the first GRU network; the second GRU network is connected to the third GRU network through the second adder, and the input of the second adder includes the output of the first adder and the output of the GRU network; the third GRU network is also connected to the second fully connected layer, and the first GRU network is also connected to the third fully connected layer.
[0069] Specifically, the input features are input into the first fully connected layer in the speech enhancement model, and feature superposition and other processing are performed between the recurrent neural network and the fully connected network in the speech enhancement model. The purpose of superposition is to fuse features. Feature fusion can be performed while keeping the dimension unchanged by addition, and finally the mask features corresponding to the audio data are obtained, and the mask features are used to eliminate noise in the audio data.
[0070] S304: Post-process the mask features to obtain target mask features.
[0071] Specifically, based on the triangular filtering algorithm, the mask features are subjected to triangular filtering to obtain filtered mask features. By performing triangular filtering on the mask features, residual noise between resonance peaks can be reduced.
[0072] Furthermore, based on the envelope post-processing algorithm, envelope post-processing is performed on the filtered mask features. The envelope post-processing can further enhance the mask features and further enhance the model's denoising capability for transient noise.
[0073] S305 , expanding the target mask feature into the frequency domain and then multiplying the result with the amplitude spectrum feature of the time domain audio data to obtain a human voice prediction result in the audio data.
[0074] Specifically, the target mask features obtained through post-processing are expanded to the frequency domain and then directly multiplied with the frequency domain amplitude spectrum features of the audio data to obtain a denoised amplitude spectrum, through which clearer and purer human voice information can be restored.
[0075] In the disclosed embodiment, since the dimensions of the BFCC feature and the fundamental pitch Pitch feature are small, and the dimensions of the input features obtained by splicing the two are also small, and the BFCC feature has better human voice harmonic characteristics and the Pitch feature can well reflect the personal voice characteristics, the input features obtained by splicing the two are input into the speech enhancement model, and the speech enhancement model performs feature fusion between network layers in the form of feature superposition, which can further reduce the feature dimension while suppressing the noise in the audio data, thereby greatly reducing the number of parameters and computational complexity of the model, and even in real-time communication scenarios, it can quickly suppress the noise in the voice data to obtain a clearer and purer human voice signal. Furthermore, by post-processing the mask features, the model's ability to reduce transient noise is further enhanced, so that it can run smoothly in low-end models and meet the needs of real-time communication scenarios.
[0076] In order to better understand this application, Figure 4 The feature processing method flow chart and Figure 5 The speech enhancement model shown further describes the feature processing process in detail.
[0077] It should be noted that the GRU network is a recurrent neural network, and the network can be used to model in time series. The speech enhancement model in this embodiment uses a three-layer GRU network structure for modeling and processing, and features are captured through a multi-layer GRU network, so that the mask features of the final output can better suppress noise.
[0078] In a possible case of the present embodiment, the method for processing input features includes: inputting the input features into the first fully connected layer and the first GRU network in sequence, and performing a first feature superposition processing on the features respectively output by the first fully connected layer and the first GRU network; inputting the features obtained by the first superposition processing into the second GRU network, and performing a second feature superposition processing on the features output by the second GRU network; inputting the features obtained by the second feature superposition processing into the third GRU network and the second fully connected layer in sequence to obtain output features; and mapping the output features into mask features.
[0079] Specifically, Figure 4 As shown, the method for processing input features specifically includes the following steps.
[0080] S401, input the input feature to a first fully connected layer, the first fully connected layer maps the input feature to a first intermediate feature, and the dimension of the first intermediate feature is lower than the dimension of the input feature.
[0081] Specifically, the function of the first fully connected layer is to map features and reduce feature dimensions. In this step, the 42-dimensional input features obtained by concatenating the 34-dimensional BFCC features and the 8-dimensional Pitch features are input into the first fully connected layer, and the first fully connected layer maps the 42-dimensional input features into 32-dimensional first intermediate features.
[0082] S402: Input the first intermediate feature into a first GRU network to obtain a second intermediate feature output by the first GRU network.
[0083] Among them, the second intermediate feature output by the first GRU network has the same dimension as the first intermediate feature, both of which are 32 dimensions.
[0084] In one possible embodiment, see Figure 5 The second intermediate feature output by the first GRU network can be input to the third fully connected layer, and the third fully connected layer is used to map the second intermediate feature of 32 dimensions into a voice probability output (Voice Activity Detection, VAD) feature of dimension 2, and map the value of the VAD feature to a preset numerical range through a softmax function, and the preset numerical range is [0, 1]. The voice probability output feature is used to determine whether there is human voice information in the audio data.
[0085] S403: Superimpose the first intermediate feature and the second intermediate feature to obtain a third intermediate feature.
[0086] Specifically, the first intermediate feature and the second intermediate feature are added by the first adder in the speech enhancement model to obtain a new third intermediate feature as the input of the second GRU network. The fusion of the first intermediate feature and the second intermediate feature by adding does not increase the feature dimension, and the dimension of the obtained third intermediate feature is the same as that of the first intermediate feature and the second intermediate feature, both of which are 32 dimensions.
[0087] S404: Input the third intermediate feature into the second GRU network to obtain a fourth intermediate feature output by the second GRU network.
[0088] Among them, the dimension of the fourth intermediate feature output by the second GRU network is the same as the third intermediate feature, which is 32 dimensions.
[0089] S405, superimposing the third intermediate feature and the fourth intermediate feature to obtain a fifth intermediate feature.
[0090] For details, see Figure 5 , the third intermediate feature is added to the fourth intermediate feature through the second adder to obtain a new fifth intermediate feature as the input of the third GRU network.
[0091] Similarly, fusing the third intermediate feature and the fourth intermediate feature by adding them will not increase the feature dimension, and the fifth intermediate feature obtained is still 32 dimensions.
[0092] S406: Input the fifth intermediate feature into the third GRU network to obtain a sixth intermediate feature output by the third GRU network.
[0093] Among them, the sixth intermediate feature of the output is the same as the fifth intermediate feature of the input, which is 32 dimensions.
[0094] S407: Input the sixth intermediate feature to a second fully connected layer, and the second fully connected layer maps the sixth intermediate feature to an output feature, wherein the dimension of the output feature is smaller than the dimension of the sixth intermediate feature.
[0095] Specifically, the second fully connected layer is used to reduce the dimension of the sixth intermediate feature and map the 32-dimensional sixth intermediate feature to a 22-dimensional output feature.
[0096] In this step, the feature dimension is further reduced through the second fully connected layer, which reduces the amount of calculation.
[0097] S408: Map the output features into mask features.
[0098] Specifically, the output feature can be mapped to a mask feature with a value range of [0, 1] through a sigmoid function.
[0099] In this embodiment, based on the speech enhancement model of the three-layer GRU network, the fusion method of GRU features greatly reduces the number of parameters and calculation dimensions of the model, while also ensuring the human voice prediction effect.
[0100] Figure 6 A schematic diagram of the structure of an audio data processing device provided in an embodiment of the present disclosure.
[0101] like Figure 6As shown, the device provided in this embodiment includes: a data acquisition module 601, a feature extraction module 602, a feature splicing module 603, a feature processing module 604 and a human voice prediction module 605; wherein the data acquisition module 601 is used to acquire time domain audio data; the feature extraction module 602 is used to extract the first acoustic feature and the second acoustic feature respectively from the time domain audio data, and the dimension of the first acoustic feature and the second acoustic feature is less than the preset dimension; the feature splicing module 603 is used to splice the first acoustic feature and the second acoustic feature to obtain input features; the feature processing module 604 is used to input the input features into a pre-established speech enhancement model to perform feature superposition processing between the network layers of the speech enhancement model to obtain mask features corresponding to the audio data; the human voice prediction module 605 is used to expand the mask features to the frequency domain and then multiply them with the amplitude spectrum features of the time domain audio data to obtain the human voice prediction results in the audio data.
[0102] Furthermore, the first acoustic feature is a Bark frequency cepstral coefficient BFCC feature, and the feature extraction module is specifically used for:
[0103] Performing a short-time Fourier transform on the time-domain audio data to obtain amplitude spectrum characteristics of the audio data in the complex frequency domain;
[0104] Determining a power spectrum of the audio data according to the amplitude spectrum feature;
[0105] Mapping the power spectrum to the Bark domain to obtain Bark domain power spectrum data;
[0106] The logarithm of the Bark domain power spectrum data is taken, and discrete cosine transform is performed on the logarithm to obtain the BFCC feature.
[0107] Furthermore, the second acoustic feature is a fundamental pitch Pitch feature, and the feature extraction module is specifically used to extract the Pitch feature of the audio data based on a fundamental pitch extraction algorithm.
[0108] Furthermore, the speech enhancement model includes a three-layer gated recurrent unit GRU network and three fully connected layers, and the feature processing module is specifically used for:
[0109] Inputting the input features into the first fully connected layer and the first GRU network in sequence, and performing a first feature superposition process on the features respectively output by the first fully connected layer and the first GRU network;
[0110] Input the features obtained by the first superposition process into the second GRU network, and perform a second feature superposition process on the features output by the second GRU network;
[0111] Input the features obtained by the second feature superposition process into the third GRU network and the second fully connected layer in sequence to obtain output features;
[0112] The output features are mapped to mask features.
[0113] Furthermore, the feature processing module is specifically used for:
[0114] Inputting the input feature into a first fully connected layer, where the first fully connected layer maps the input feature into a first intermediate feature, where the dimension of the first intermediate feature is lower than the dimension of the input feature;
[0115] Inputting the first intermediate feature into a first GRU network to obtain a second intermediate feature output by the first GRU network;
[0116] The first intermediate feature and the second intermediate feature are superimposed to obtain a third intermediate feature.
[0117] Furthermore, the feature processing module is specifically used for:
[0118] Inputting the third intermediate feature into the second GRU network to obtain a fourth intermediate feature output by the second GRU network;
[0119] The third intermediate feature is superimposed on the fourth intermediate feature to obtain a fifth intermediate feature.
[0120] Furthermore, the feature processing module is specifically used for:
[0121] Inputting the fifth intermediate feature into the third GRU network to obtain a sixth intermediate feature output by the third GRU network;
[0122] The sixth intermediate feature is input into a second fully connected layer, and the second fully connected layer maps the sixth intermediate feature into an output feature, where a dimension of the output feature is smaller than a dimension of the sixth intermediate feature.
[0123] Furthermore, the feature processing module is also used to:
[0124] The second intermediate feature is input into the third fully connected layer, and the third fully connected layer maps the second intermediate feature into a speech probability output VAD feature with a dimension of two, and maps the value of the VAD feature into a preset value range, which is [0, 1].
[0125] The speech probability output feature is used to determine whether there is human voice information in the audio data.
[0126] Furthermore, the apparatus provided in this embodiment further includes: a post-processing module 606, which is used to perform triangular filtering post-processing on the mask features based on a triangular filtering algorithm to obtain filtered mask features.
[0127] Furthermore, the post-processing module is also used to: perform envelope processing on the filtered mask features based on an envelope post-processing algorithm.
[0128] Furthermore, the speech enhancement model also includes a first adder and a second adder, wherein:
[0129] The first fully connected layer is connected to the first GRU network, the first GRU network is connected to the second GRU network through the first adder, and the input of the first adder includes the output of the first fully connected layer and the output of the first GRU network;
[0130] The second GRU network is connected to the third GRU network via a second adder, and the input of the second adder includes the output of the first adder and the output of the GRU network;
[0131] The third GRU network is also connected to the second fully connected layer, and the first GRU network is also connected to the third fully connected layer.
[0132] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.
[0133] refer to Figure 7 , which shows a schematic diagram of the structure of an electronic device 700 suitable for implementing the embodiment of the present disclosure, and the electronic device 700 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0134] like Figure 7As shown, the electronic device 700 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0135] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 708 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 709. The communication devices 709 may allow the electronic device 700 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 The electronic device 700 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0136] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0137] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0138] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.
[0139] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.
[0140] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0141] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0142] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".
[0143] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0145] In a first aspect, according to one or more embodiments of the present disclosure, a method for processing audio data is provided, comprising:
[0146] Get time domain audio data;
[0147] Extracting a first acoustic feature and a second acoustic feature from the time-domain audio data respectively, wherein dimensions of the first acoustic feature and the second acoustic feature are smaller than a preset dimension;
[0148] Concatenating the first acoustic feature and the second acoustic feature to obtain an input feature;
[0149] Inputting the input features into a pre-established speech enhancement model to perform feature superposition processing between network layers of the speech enhancement model to obtain mask features corresponding to the audio data;
[0150] The mask feature is expanded to the frequency domain and then multiplied with the amplitude spectrum feature of the time domain audio data to obtain a human voice prediction result in the audio data.
[0151] According to one or more embodiments of the present disclosure, the first acoustic feature is a Bark frequency cepstral coefficient (BFCC) feature, and extracting the first acoustic feature from the time domain audio data includes:
[0152] Performing a short-time Fourier transform on the time-domain audio data to obtain amplitude spectrum characteristics of the audio data in the complex frequency domain;
[0153] Determining a power spectrum of the audio data according to the amplitude spectrum feature;
[0154] Mapping the power spectrum to the Bark domain to obtain Bark domain power spectrum data;
[0155] The logarithm of the Bark domain power spectrum data is taken, and discrete cosine transform is performed on the logarithm to obtain the BFCC feature.
[0156] According to one or more embodiments of the present disclosure, the second acoustic feature is a fundamental pitch feature, and extracting the second acoustic feature from the time domain audio data includes:
[0157] Based on a fundamental pitch extraction algorithm, a Pitch feature of the audio data is extracted.
[0158] According to one or more embodiments of the present disclosure, the speech enhancement model includes a three-layer gated recurrent unit GRU network and three fully connected layers, and the input feature is input into the pre-established speech enhancement model to perform feature superposition processing between the network layers of the speech enhancement model to obtain the mask feature corresponding to the audio data, including:
[0159] Inputting the input features into the first fully connected layer and the first GRU network in sequence, and performing a first feature superposition process on the features respectively output by the first fully connected layer and the first GRU network;
[0160] Input the features obtained by the first superposition process into the second GRU network, and perform a second feature superposition process on the features output by the second GRU network;
[0161] Input the features obtained by the second feature superposition process into the third GRU network and the second fully connected layer in sequence to obtain output features;
[0162] The output features are mapped to mask features.
[0163] According to one or more embodiments of the present disclosure, the step of sequentially inputting the input features into the first fully connected layer and the first GRU network, and performing a first feature superposition process on the features respectively output by the first fully connected layer and the first GRU network, includes:
[0164] Inputting the input feature into a first fully connected layer, where the first fully connected layer maps the input feature into a first intermediate feature, where the dimension of the first intermediate feature is lower than the dimension of the input feature;
[0165] Inputting the first intermediate feature into a first GRU network to obtain a second intermediate feature output by the first GRU network;
[0166] The first intermediate feature and the second intermediate feature are superimposed to obtain a third intermediate feature.
[0167] According to one or more embodiments of the present disclosure, the step of inputting the features obtained by the first superposition process into a second GRU network, and performing a second feature superposition process on the features output by the second GRU network, includes:
[0168] Inputting the third intermediate feature into the second GRU network to obtain a fourth intermediate feature output by the second GRU network;
[0169] The third intermediate feature is superimposed on the fourth intermediate feature to obtain a fifth intermediate feature.
[0170] According to one or more embodiments of the present disclosure, the features obtained by the second feature superposition process are sequentially input into the third GRU network and the second fully connected layer to obtain output features, including:
[0171] Inputting the fifth intermediate feature into the third GRU network to obtain a sixth intermediate feature output by the third GRU network;
[0172] The sixth intermediate feature is input into a second fully connected layer, and the second fully connected layer maps the sixth intermediate feature into an output feature, where a dimension of the output feature is smaller than a dimension of the sixth intermediate feature.
[0173] According to one or more embodiments of the present disclosure, the method further includes:
[0174] The second intermediate feature is input into the third fully connected layer, and the third fully connected layer maps the second intermediate feature into a speech probability output VAD feature with a dimension of two, and maps the value of the VAD feature into a preset value range, which is [0, 1].
[0175] The speech probability output feature is used to determine whether there is human voice information in the audio data.
[0176] According to one or more embodiments of the present disclosure, after extending the mask feature to the frequency domain and before multiplying the mask feature with the amplitude spectrum feature of the time domain audio data, the method further includes:
[0177] Based on the triangular filtering algorithm, the mask features are subjected to triangular filtering post-processing to obtain filtered mask features.
[0178] According to one or more embodiments of the present disclosure, the method further includes:
[0179] Based on an envelope post-processing algorithm, envelope processing is performed on the filtered mask features.
[0180] According to one or more embodiments of the present disclosure, the speech enhancement model further includes a first adder and a second adder, wherein:
[0181] The first fully connected layer is connected to the first GRU network, the first GRU network is connected to the second GRU network through the first adder, and the input of the first adder includes the output of the first fully connected layer and the output of the first GRU network;
[0182] The second GRU network is connected to the third GRU network via a second adder, and the input of the second adder includes the output of the first adder and the output of the GRU network;
[0183] The third GRU network is also connected to the second fully connected layer, and the first GRU network is also connected to the third fully connected layer.
[0184] In a second aspect, according to one or more embodiments of the present disclosure, there is provided an audio data processing device, comprising: a data acquisition module, a feature extraction module, a feature splicing module, a feature processing module and a human voice prediction module; wherein the data acquisition module is used to acquire time domain audio data; the feature extraction module is used to extract a first acoustic feature and a second acoustic feature from the time domain audio data, respectively, and the dimensions of the first acoustic feature and the second acoustic feature are less than a preset dimension; the feature splicing module is used to splice the first acoustic feature and the second acoustic feature to obtain an input feature; the feature processing module is used to input the input feature into a pre-established speech enhancement model to perform feature superposition processing between the network layers of the speech enhancement model to obtain a mask feature corresponding to the audio data; the human voice prediction module is used to expand the mask feature to the frequency domain and then multiply it with the amplitude spectrum feature of the time domain audio data to obtain a human voice prediction result in the audio data.
[0185] According to one or more embodiments of the present disclosure, the first acoustic feature is a Bark frequency cepstral coefficient BFCC feature, and the feature extraction module is specifically used for:
[0186] Performing a short-time Fourier transform on the time-domain audio data to obtain amplitude spectrum characteristics of the audio data in the complex frequency domain;
[0187] Determining a power spectrum of the audio data according to the amplitude spectrum feature;
[0188] Mapping the power spectrum to the Bark domain to obtain Bark domain power spectrum data;
[0189] The logarithm of the Bark domain power spectrum data is taken, and discrete cosine transform is performed on the logarithm to obtain the BFCC feature.
[0190] According to one or more embodiments of the present disclosure, the second acoustic feature is a fundamental pitch Pitch feature, and the feature extraction module is specifically used to extract the Pitch feature of the audio data based on a fundamental pitch extraction algorithm.
[0191] According to one or more embodiments of the present disclosure, the speech enhancement model includes a three-layer gated recurrent unit GRU network and three fully connected layers, and the feature processing module is specifically used for:
[0192] Inputting the input features into the first fully connected layer and the first GRU network in sequence, and performing a first feature superposition process on the features respectively output by the first fully connected layer and the first GRU network;
[0193] Input the features obtained by the first superposition process into the second GRU network, and perform a second feature superposition process on the features output by the second GRU network;
[0194] Input the features obtained by the second feature superposition process into the third GRU network and the second fully connected layer in sequence to obtain output features;
[0195] The output features are mapped to mask features.
[0196] According to one or more embodiments of the present disclosure, the feature processing module is specifically used to:
[0197] Inputting the input feature into a first fully connected layer, where the first fully connected layer maps the input feature into a first intermediate feature, where the dimension of the first intermediate feature is lower than the dimension of the input feature;
[0198] Inputting the first intermediate feature into a first GRU network to obtain a second intermediate feature output by the first GRU network;
[0199] The first intermediate feature and the second intermediate feature are superimposed to obtain a third intermediate feature.
[0200] According to one or more embodiments of the present disclosure, the feature processing module is specifically used to:
[0201] Inputting the third intermediate feature into the second GRU network to obtain a fourth intermediate feature output by the second GRU network;
[0202] The third intermediate feature is superimposed on the fourth intermediate feature to obtain a fifth intermediate feature.
[0203] According to one or more embodiments of the present disclosure, the feature processing module is specifically used to:
[0204] Inputting the fifth intermediate feature into the third GRU network to obtain a sixth intermediate feature output by the third GRU network;
[0205] The sixth intermediate feature is input into a second fully connected layer, and the second fully connected layer maps the sixth intermediate feature into an output feature, where a dimension of the output feature is smaller than a dimension of the sixth intermediate feature.
[0206] According to one or more embodiments of the present disclosure, the feature processing module is further configured to:
[0207] The second intermediate feature is input into the third fully connected layer, and the third fully connected layer maps the second intermediate feature into a speech probability output VAD feature with a dimension of two, and maps the value of the VAD feature into a preset value range, which is [0, 1].
[0208] The speech probability output feature is used to determine whether there is human voice information in the audio data.
[0209] According to one or more embodiments of the present disclosure, the apparatus provided in this embodiment further includes: a post-processing module, configured to perform triangular filtering post-processing on the mask features based on a triangular filtering algorithm to obtain filtered mask features.
[0210] According to one or more embodiments of the present disclosure, the post-processing module is further used to: perform envelope processing on the filtered mask features based on an envelope post-processing algorithm.
[0211] According to one or more embodiments of the present disclosure, the speech enhancement model further includes a first adder and a second adder, wherein:
[0212] The first fully connected layer is connected to the first GRU network, the first GRU network is connected to the second GRU network through the first adder, and the input of the first adder includes the output of the first fully connected layer and the output of the first GRU network;
[0213] The second GRU network is connected to the third GRU network via a second adder, and the input of the second adder includes the output of the first adder and the output of the GRU network;
[0214] The third GRU network is also connected to the second fully connected layer, and the first GRU network is also connected to the third fully connected layer.
[0215] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;
[0216] The memory stores computer-executable instructions;
[0217] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the audio data processing method described in the first aspect and various possible designs of the first aspect.
[0218] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the audio data processing method described in the first aspect and various possible designs of the first aspect is implemented.
[0219] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program. When the computer program is executed by a processor, it implements the audio data processing method described in the first aspect and various possible designs of the first aspect.
[0220] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0221] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0222] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. An audio data processing method, characterized in that: Methods include: Get time domain audio data; Extracting a first acoustic feature and a second acoustic feature from the time-domain audio data respectively, wherein dimensions of the first acoustic feature and the second acoustic feature are smaller than a preset dimension; Concatenating the first acoustic feature and the second acoustic feature to obtain an input feature; Inputting the input features into a pre-established speech enhancement model to perform feature superposition processing between network layers of the speech enhancement model to obtain mask features corresponding to the audio data, wherein the mask features are used to eliminate noise in the audio data; Expanding the mask feature into the frequency domain and then multiplying the mask feature with the amplitude spectrum feature of the time domain audio data to obtain a human voice prediction result of the audio data; Among them, the speech enhancement model includes a three-layer gated recurrent unit GRU network and three fully connected layers. Accordingly, the input features are input into the pre-established speech enhancement model to perform feature superposition processing between the network layers of the speech enhancement model to obtain mask features corresponding to the audio data, including: inputting the input features into the first fully connected layer and the first GRU network in sequence, and performing a first feature superposition processing on the features output by the first fully connected layer and the first GRU network respectively; inputting the features obtained by the first feature superposition processing into the second GRU network, and performing a second feature superposition processing on the features output by the second GRU network; inputting the features obtained by the second feature superposition processing into the third GRU network and the second fully connected layer in sequence to obtain output features; and mapping the output features into mask features.
2. The method according to claim 1, characterized in that The first acoustic feature is a Bark frequency cepstral coefficient BFCC feature, and the extracting the first acoustic feature from the time domain audio data includes: Performing a short-time Fourier transform on the time-domain audio data to obtain amplitude spectrum characteristics of the audio data in the complex frequency domain; Determining a power spectrum of the audio data according to the amplitude spectrum feature; Mapping the power spectrum to the Bark domain to obtain Bark domain power spectrum data; The logarithm of the Bark domain power spectrum data is taken, and discrete cosine transform is performed on the logarithm to obtain the BFCC feature.
3. The method according to claim 1, characterized in that The second acoustic feature is a fundamental pitch feature, and the extracting the second acoustic feature from the time domain audio data includes: Based on a fundamental pitch extraction algorithm, a Pitch feature of the audio data is extracted.
4. The method according to claim 1, characterized in that: The step of sequentially inputting the input features into the first fully connected layer and the first GRU network, and performing a first feature superposition process on the features respectively output by the first fully connected layer and the first GRU network, comprises: Inputting the input feature into a first fully connected layer, where the first fully connected layer maps the input feature into a first intermediate feature, where the dimension of the first intermediate feature is lower than the dimension of the input feature; Inputting the first intermediate feature into a first GRU network to obtain a second intermediate feature output by the first GRU network; The first intermediate feature and the second intermediate feature are superimposed to obtain a third intermediate feature.
5. The method according to claim 4, characterized in that The step of inputting the features obtained by the first feature superposition process into a second GRU network, and performing a second feature superposition process on the features output by the second GRU network, comprises: Inputting the third intermediate feature into the second GRU network to obtain a fourth intermediate feature output by the second GRU network; The third intermediate feature is superimposed on the fourth intermediate feature to obtain a fifth intermediate feature.
6. The method according to claim 5, characterized in that The features obtained by the second feature superposition process are sequentially input into the third GRU network and the second fully connected layer to obtain output features, including: Inputting the fifth intermediate feature into the third GRU network to obtain a sixth intermediate feature output by the third GRU network; The sixth intermediate feature is input into a second fully connected layer, and the second fully connected layer maps the sixth intermediate feature into an output feature, where a dimension of the output feature is smaller than a dimension of the sixth intermediate feature.
7. The method according to any one of claims 4 to 6, characterized in that: Also includes: The second intermediate feature is input into the third fully connected layer, and the third fully connected layer maps the second intermediate feature into a speech probability output VAD feature with a dimension of two, and maps the value of the VAD feature into a preset numerical range, and the preset numerical range is [0, 1]. The speech probability output feature is used to determine whether there is human voice information in the audio data.
8. The method according to any one of claims 1 to 6, characterized in that: After the mask feature is extended to the frequency domain and before the mask feature is multiplied by the amplitude spectrum feature of the time domain audio data, the method further includes: Based on the triangular filtering algorithm, the mask features are subjected to triangular filtering post-processing to obtain filtered mask features.
9. The method according to claim 8, characterized in that Also includes: Based on an envelope post-processing algorithm, envelope processing is performed on the filtered mask features.
10. The method according to claim 7, characterized in that The speech enhancement model also includes a first adder and a second adder, wherein: The first fully connected layer is connected to the first GRU network, the first GRU network is connected to the second GRU network through the first adder, and the input of the first adder includes the output of the first fully connected layer and the output of the first GRU network; The second GRU network is connected to the third GRU network via a second adder, and the input of the second adder includes the output of the first adder and the output of the GRU network; The third GRU network is also connected to the second fully connected layer, and the first GRU network is also connected to the third fully connected layer.
11. An audio data processing device, characterized in that: include: A data acquisition module, used for acquiring time domain audio data; A feature extraction module, configured to extract a first acoustic feature and a second acoustic feature from the time-domain audio data, respectively, wherein dimensions of the first acoustic feature and the second acoustic feature are smaller than a preset dimension; A feature concatenation module, used for concatenating the first acoustic feature and the second acoustic feature to obtain an input feature; A feature processing module, used for inputting the input features into a pre-established speech enhancement model to perform feature superposition processing between network layers of the speech enhancement model to obtain mask features corresponding to the audio data, wherein the mask features are used to eliminate noise in the audio data; A human voice prediction module, used for expanding the mask feature into the frequency domain and then multiplying the mask feature with the amplitude spectrum feature of the time domain audio data to obtain a human voice prediction result in the audio data; Among them, the speech enhancement model includes a three-layer gated recurrent unit GRU network and three fully connected layers. Accordingly, the feature processing module inputs the input features into the pre-established speech enhancement model to perform feature superposition processing between the network layers of the speech enhancement model to obtain mask features corresponding to the audio data, including: inputting the input features into the first fully connected layer and the first GRU network in sequence, and performing a first feature superposition processing on the features output by the first fully connected layer and the first GRU network respectively; inputting the features obtained by the first feature superposition processing into the second GRU network, and performing a second feature superposition processing on the features output by the second GRU network; inputting the features obtained by the second feature superposition processing into the third GRU network and the second fully connected layer in sequence to obtain output features; and mapping the output features into mask features.
12. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the audio data processing method according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the processor executes the computer-executable instructions, the audio data processing method according to any one of claims 1 to 10 is implemented.
Citation Information
Patent Citations
Voice recognizing method and device, as well as neural network training method and device
CN110600018A
Speech enhancement method of hopping connection deep neural network
CN111192598A
Voice signal noise reduction processing method, microphone and electronic equipment
CN111223493A
Speech denoising method and device thereof
CN111261183A
Single-channel speech enhancement method and system for neural network sub-band modeling, and storage medium
CN111986660A
