Method and apparatus for constructing noise suppression module, electronic device, and storage medium

Abstract

The present disclosure provides a method for constructing a noise suppression module, which can include: training an original feature extractor with mixed speech samples to obtain a mask extractor for a target mask; performing a data volume compression process on the mask extractor to obtain a target extractor, wherein the data volume of the target extractor is less than that of the mask extractor; and generating a noise suppression module with a target data volume based on the target extractor, wherein the target data volume is less than or equal to a storage capacity threshold of a target device in which the noise suppression module is embedded. The present disclosure also provides a device for constructing a noise suppression module, a noise suppression method, an electronic device, and a storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of speech enhancement technology, and in particular to a method, apparatus, electronic device, and storage medium for constructing a noise suppression module. Background Technology

[0002] Noise suppression technology is mainly used to eliminate background noise, improve the signal-to-noise ratio of speech signals, and obtain clear human voice signals, thereby improving speech clarity and call experience during conversations. Related technologies typically employ low-power Wiener filtering to extract the speech signal from noisy signals. Alternatively, neural networks can be used to extract the features of the human voice signal and the noise from noisy signals. An amplitude modulator is then obtained based on the human voice feature set and the noise feature set, and this modulator is applied to the noisy signal to obtain a clear human voice signal.

[0003] However, while low-computational-power noise suppression techniques like Wiener filtering require relatively little memory, their noise suppression effectiveness is limited, and they struggle to suppress unstable noise, resulting in musical noise. Neural network noise suppression techniques, on the other hand, require millions of network parameters, demanding significant computational power and making them unsuitable for deployment in small embedded devices such as Bluetooth headsets. Therefore, it is difficult to achieve a noise suppression technique that integrates small memory requirements with the ability to output clear human voice signals in related technologies. Summary of the Invention

[0004] To address at least one of the aforementioned technical problems, this disclosure provides a method for constructing a noise suppression module, a construction apparatus, a noise suppression method, an electronic device, and a storage medium.

[0005] One aspect of this disclosure provides a method for constructing a noise suppression module, which may include: training an original feature extractor using mixed speech samples to obtain a mask extractor for a target mask; performing data compression processing on the mask extractor to obtain a target extractor, wherein the data size of the target extractor is less than that of the mask extractor; and generating a noise suppression module with a target data size based on the target extractor, wherein the target data size is less than or equal to a storage capacity threshold of the target device into which the noise suppression module is embedded.

[0006] In some implementations, training an original feature extractor using mixed speech samples to obtain a mask extractor for generating a target mask may include: inputting the mixed speech samples into the original feature extraction layer of the original feature extractor, and having the original feature extraction layer output estimated human voice features corresponding to the mixed speech samples, wherein the mixed speech samples include noise data and human voice data; comparing the estimated human voice features with the expected human voice features to determine a feature loss value used to characterize the difference between the two; adjusting the weights of each target feature matrix in the original feature extraction layer in response to a judgment result that the feature loss value exceeds an error threshold, to obtain a human voice feature extraction layer that makes the feature loss value less than or equal to the error threshold; and integrating the various human voice feature extraction layers to generate a mask extractor containing multiple human voice feature extraction layers, wherein the mask extractor is used to generate a target mask based on the target human voice features output by the human voice feature extraction layers.

[0007] In some implementations, data compression processing is performed on the mask extractor to obtain a target extractor. This may include: quantizing each feature extraction matrix of the human voice feature extraction layer to generate a target human voice feature extraction layer with a target extraction matrix, wherein the data size of the target extraction matrix is ​​smaller than the data size of the feature extraction matrix; merging the batch normalization layer and the human voice feature extraction layer in the mask extractor to form an equivalent target feature layer, wherein the data size of the target feature layer is smaller than the sum of the data sizes of the batch normalization layer and the target human voice feature extraction layer; and integrating the various target feature layers, the target extractor.

[0008] In some implementations, the mask extractor is subjected to data compression processing to obtain the target extractor, and may further include: setting a precision restoration area for restoring the process features output by the target feature layer to the target human voice features.

[0009] In some implementations, before training the original feature extractor with the mixed speech samples to obtain a mask extractor for generating a target mask, the process may further include: processing the acquired noise data and human voice data to generate mixed speech samples, wherein the noise data includes ambient noise data and room impact response data.

[0010] In some implementations, processing the collected noise data and human voice data to generate mixed speech samples may include: performing a convolution operation on the room impact response data in the noise data and the human voice data to generate first mixed data; merging the first mixed data with the environmental noise data in the noise data to generate second mixed data; and preprocessing the second mixed data to generate mixed speech samples containing the speech spectrum of the second mixed data.

[0011] Another aspect of this disclosure provides an apparatus for constructing a noise suppression module, which may include: a training unit for training an original feature extractor using mixed speech samples to obtain a mask extractor for generating a target mask; a data compression unit for compressing the data of the mask extractor to obtain a target extractor, wherein the data size of the target extractor is less than that of the mask extractor; and a module generation unit for generating a noise suppression module with a target data size based on the target extractor, wherein the target data size is less than or equal to a storage capacity threshold of the target device into which the noise suppression module is embedded.

[0012] Another aspect of this disclosure provides a noise suppression method, which may include: preprocessing input noisy speech to obtain a target spectrum of the noisy speech; inputting the target spectrum to a noise suppression module, whereby a target extractor in the noise suppression module processes the target spectrum to generate a human voice mask, wherein the noise suppression module is constructed using the construction method of the noise suppression module described in any of the above embodiments; covering the target spectrum with the human voice mask by the noise suppression module to extract the human voice spectrum in the target spectrum; and postprocessing the human voice spectrum by the noise suppression module to generate a target human voice signal.

[0013] Another aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a method for constructing a noise suppression module as described in any of the above embodiments.

[0014] Another aspect of this disclosure provides a readable storage medium storing a computer program adapted for loading by a processor to execute a method for constructing a noise suppression module as described in any of the above embodiments. Attached Figure Description

[0015] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0016] Figure 1 This is a flowchart illustrating a method for constructing a noise suppression module according to an exemplary embodiment of this disclosure;

[0017] Figure 2 This is a flowchart illustrating the training process of a target extractor according to an exemplary embodiment of this disclosure;

[0018] Figure 3 This is a schematic diagram of the target extractor structure according to an exemplary embodiment of the present disclosure;

[0019] Figure 4This is a schematic diagram of a bridging unit according to an exemplary embodiment of this disclosure;

[0020] Figure 5 This is a flowchart of a data processing procedure according to an exemplary embodiment of the present disclosure;

[0021] Figure 6 A block diagram of a construction apparatus for a noise suppression module according to an exemplary embodiment of the present disclosure; and

[0022] Figure 7 This is a flowchart of a noise suppression method according to an exemplary embodiment of the present disclosure.

[0023] Explanation of reference numerals in the attached figures

[0024] Construction apparatus for 1000 noise suppression modules

[0025] 1002 training units

[0026] 1004 Data Compression Units

[0027] 1006 Module Generation Unit

[0028] 1100 bus

[0029] 1200 processor

[0030] 1300 memory

[0031] 1400 Other Circuits Detailed Implementation

[0032] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0033] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0035] The terminology used herein is for the purpose of describing particular embodiments and is not restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0036] Figure 1 This is a flowchart illustrating a method for constructing a noise suppression module according to an exemplary embodiment of this disclosure; Figure 2 1 is a flowchart of the target extractor training process according to an exemplary embodiment of the present disclosure; 2 is a schematic diagram of the target extractor structure according to an exemplary embodiment of the present disclosure; Figure 4 This is a schematic diagram of a bridging unit according to an exemplary embodiment of this disclosure; Figure 5 This is a flowchart of a data processing procedure according to an exemplary embodiment of this disclosure. The following will be combined with... Figures 1 to 5 This section details the specific implementation of each step in the construction method S100 for the noise suppression module.

[0037] Step S102: Train the original feature extractor using mixed speech samples to obtain a mask extractor for generating the target mask.

[0038] The mixed speech samples consist of noise data and human voice data, used as training samples for the original feature extractor. The noise data in the mixed speech samples is further divided into environmental noise data and RIR (Room Impulse Response) data. During the training of the original feature extractor, one environmental noise data point, one room impulse response data point, and one human voice data point are randomly selected and mixed and concatenated according to certain rules to obtain the mixed speech sample data. Specifically, the environmental noise data can be various environmental noises with a sampling rate of 16kHz, including wind noise, knocking sounds, vehicle noise, shopping mall noise, subway operation noise, etc.; the RIR data sampling rate can be 16kHz, such as simulating the reverberation field of a room. When the human voice data is concatenated with the RIR data, the human voice data becomes richer and more realistic. The human voice data can be recordings of various languages ​​and age groups.

[0039] The original feature extractor is a neural network that has not been trained with mixed speech samples. It has five regions: logarithmic operation region, Conv2D (Convolution 2D) region, Conv2DTrans (Convolution 2D Transpose) region, GRU (Gated Recurrent Unit) region, and Reshape (reshaping function) region. The Conv2D region and Conv2DTrans region are composed of multiple original feature extraction layers, batch normalization layers, and ELU (Exponential Linear Unit) layers, respectively.

[0040] The target mask is a weighted vector used to filter the human voice spectrum on the speech spectrum of the mixed speech sample. Each frame of the mixed speech sample corresponds to a target mask of length 129 with a value between 0 and 1.

[0041] The mask extractor is the result of training each partition of the original feature extractor using a large number of mixed speech samples. After training the original feature extractor with a large number of mixed speech samples, the weight values ​​of each partition in the original feature extractor can be adjusted so that it can output a target mask that can accurately filter the human voice spectrum.

[0042] like Figure 2 As shown, environmental noise data n(t) is randomly sampled from the environmental noise dataset, RIR data r(t) is randomly sampled from the RIR dataset, and human voice data x(t) is randomly sampled from the human voice dataset. These three data points are then concatenated to obtain a mixed speech sample y(t). The specific method for obtaining y(t) will be described in later implementations. The mixed speech sample is input into the original feature extractor, and each partition of the original feature extractor is trained. The loss function calculates the mask difference between the estimated mask output by the original feature extractor and the desired mask. Finally, the optimizer adjusts the weight parameters of each partition in the original feature extractor to generate a mask extractor that outputs a mask difference less than the mask error threshold.

[0043] The training process for the original feature extraction layer in the mask extractor is described in detail below, and may include: inputting mixed speech samples into the original feature extraction layer of the original feature extractor, and having the original feature extraction layer output estimated human voice features corresponding to the mixed speech samples; comparing the estimated human voice features with the expected human voice features to determine a feature loss value used to characterize the difference between the two; in response to the judgment result that the feature loss value exceeds an error threshold, adjusting the weights of each target feature matrix in the original feature extraction layer to obtain a human voice feature extraction layer in which the feature loss value is less than or equal to the error threshold. Finally, integrating the various human voice feature extraction layers generates a mask extractor containing multiple human voice feature extraction layers, wherein the mask extractor is used to generate a target mask based on the target human voice features output by the human voice feature extraction layers.

[0044] Step S104: Perform data compression processing on the mask extractor to obtain the target extractor.

[0045] The Conv2D and Conv2DTrans regions of the mask extractor are composed of multiple original feature extraction layers, batch normalization layers, and ELU layers, respectively. These multi-layered Conv2D and Conv2DTrans regions undoubtedly generate a large amount of data, requiring more storage space. Furthermore, the feature extraction matrices of the voice feature extraction layers in the mask extractor are single-precision floating-point numbers, which also contribute to the large data volume.

[0046] Based on the above, the mask extractor needs to be compressed so that the data size of the target extractor is smaller than that of the mask extractor, thus enabling it to be deployed in small devices, such as Bluetooth headsets.

[0047] Specifically, the feature extraction matrices of each voice feature extraction layer are quantized to generate a target voice feature extraction layer with a target extraction matrix, wherein the data size of the target extraction matrix is ​​smaller than that of the feature extraction matrix; the target voice feature extraction layer and the batch normalization layer in the mask extractor are merged to form an equivalent target feature layer, wherein the data size of the target feature layer is smaller than the sum of the data sizes of the batch normalization layer and the target voice feature extraction layer; and the various target feature layers are integrated to generate a target extractor.

[0048] The voice feature extraction layer has multiple feature extraction matrices. In the process of quantizing each feature extraction matrix, 8-bit character data (char) is used instead of single-precision floating-point numbers, which can save 75% of the space.

[0049] For example, the feature extraction matrix can be represented as follows when it is not quantized:

[0050] A=[a(i,j,k)i=1,2,…I;j=1,2,…J;k=1,2,…K],

[0051] Where A represents the unquantized feature extraction matrix, a(i,j,k) represents the weight element of dimension (i,j,k) in the feature extraction matrix A, i is the length of the current weight element, I is the total length of the matrix; j is the width of the current weight element, J is the total width of the matrix; k is the channel number of the current weight element, and K is the total number of channels.

[0052] The quantized feature extraction matrix is ​​obtained as follows:

[0053]

[0054] Where b(i, j, k) are the weight elements of dimension (i, j, k) in the feature extraction matrix B, and abs is the function for calculating the absolute value of the feature extraction matrix A. This indicates rounding down to the nearest integer.

[0055] Feature extraction matrix B is the result of quantizing feature extraction matrix A, and can be represented as:

[0056] B=[b(i,j,k)i=1,2,…I;j=1,2,…J;k=1,2,…K],

[0057] At this point, all weight elements in the feature extraction matrix B are certificates with an absolute value less than or equal to 127, which can be stored in a char array.

[0058] It is obvious that the process features obtained using feature extraction matrix B are 127 / max(abs(A)) times less accurate than the target human voice features output by feature extraction matrix A. Therefore, a precision restoration region can be set to restore the process features output by the target feature layer to the target human voice features, that is, dividing the process features by 127 / max(abs(A)) will yield the target human voice features. In this way, the accuracy of the convolution results can be guaranteed while saving a significant amount of storage space.

[0059] Furthermore, most operations in the GRU region of the mask extractor are matrix and vector multiplication operations. For example, the GRU region can contain 89,784 parameters, each corresponding to a parameter matrix. Therefore, the parameter matrix can be quantized using the method described above to save a significant amount of storage space. It should be noted that matrix multiplication and convolution operations are linear operations, and therefore conform to the distributive law of multiplication.

[0060] In this method, the batch normalization layer and the voice feature extraction layer in the mask extractor are merged, effectively forming a target feature layer. Specifically, each Conv2D region and each Conv2DTrans region contains multiple batch normalization layers and voice feature extraction layers. Each batch normalization layer accelerates the feature extraction efficiency of its corresponding Conv2D region or Conv2DTrans region. The convolution operations in the Conv2D and Conv2DTrans regions are linear operations, and the batch normalization layer acts as a linear activation function during inference. Therefore, when the voice feature extraction layer and the batch normalization layer are directly connected, they can be considered equivalent to a single target feature layer, mapping the parameters in the batch normalization layer to the voice feature extraction layer. This method reduces the storage and computation of all batch normalization layers, decreasing the data volume and saving storage space. Furthermore, since the parameters in the batch normalization layer are the results of training, accuracy is guaranteed.

[0061] Step S106: Based on the target extractor, generate a noise suppression module with a target data volume, wherein the target data volume is less than or equal to the storage capacity threshold of the target device into which the noise suppression module is embedded.

[0062] The target extractor outputs an accurate target mask with minimal storage requirements. The noise suppression module includes the target extractor and other processing modules for overlaying the target mask onto the mixed speech samples to output a human voice spectrum; it also performs post-processing on the human voice spectrum to obtain the target human voice signal. The target human voice signal contains only clear human voices, effectively filtering out noise signals.

[0063] The noise suppression module is used to embed in small devices with limited storage capacity and only able to support a small amount of computation, i.e., the target device. Therefore, after the above processing steps, the noise suppression module retains the advantage of the neural network in accurately shielding noise, while also reducing its computational and data volume, thus enabling its use in the target device.

[0064] In some implementations, prior to step S102, the process may include collecting various noise data and human voice data to generate an environmental noise dataset, an RIR dataset, and a human voice dataset.

[0065] The total duration of the environmental noise dataset is approximately 200 hours, and the total duration of the human voice dataset is approximately 50 hours.

[0066] In some implementations, before step S102, the method may further include: processing the collected noise data and human voice data to generate mixed speech samples, wherein the noise data includes environmental noise data and room impact response data.

[0067] Specifically, the RIR data in the noise data is convolved with the human voice data to generate the first mixed data; the first mixed data is merged with the environmental noise data in the noise data to generate the second mixed data; and the second mixed data is preprocessed to generate a mixed speech sample containing the speech spectrum of the second mixed data.

[0068] For example, first, randomly select a noise signal n(t) from the environmental noise dataset, where t is the sampling time; randomly select an RIR data r(t); and randomly select a human voice data x(t).

[0069] Therefore, the mixed speech samples are:

[0070] y(t)=αn(t)+βr(t)*x(t)

[0071] In this diagram, * represents the convolution operator, α is the signal-to-noise ratio weight, and β is the volume weight. The values ​​of α and β are arbitrary values ​​between 0 and 1. Based on α and β, a noisy signal with random signal-to-noise ratio and random volume can be generated from the mixed speech sample y(t).

[0072] In some implementations, such as Figure 5 As shown, the second mixed data is a noisy speech signal in the time domain, while the target extractor can only process the spectrum of the noisy speech signal. Therefore, the second mixed data needs to be preprocessed.

[0073] Preprocessing consists of three steps: frame segmentation, windowing, and fast Fourier transform.

[0074] In the frame segmentation process, a framing scheme with 50% frame overlap is used. The Fast Fourier Transform algorithm operates at its most efficient state, employing a frame length of 256 sampling points, resulting in a frame shift of 128 points. Therefore, the second mixed data is segmented into the following form:

[0075]

[0076] Where Y is the speech signal intensity matrix corresponding to each sampling point in the second mixed data; y(n) is the intensity signal of each sampling point, and n is the frame number.

[0077] The nth row of matrix Y represents the temporal data of the nth frame, that is:

[0078] Y n =[y(128(n-1)+1)y(128(n-1)+2)…y(128(n+1)].

[0079] In the windowing process, windowing operations are performed on the input of frame 1, frame 2, frame 3, up to frame N, where N represents the length of the matrix. In this disclosure, a Vorbis window is used, which is a window function that satisfies the Princen-Bradley condition. A Vorbis window with 256 sampling points has the following form:

[0080]

[0081] Among them, w vorbis For the windowing weight vector, N is 256 in this disclosure.

[0082] The result of windowing the nth frame of data is: Y′ n =Y⊙w vorbis , where ⊙ represents the Hadama multiplier operator.

[0083] For the windowed result Y′ n Performing a Fast Fourier Transform can transform Y′ n Transform to the frequency domain to obtain a frequency domain complex vector: Among them, H n Given a complex vector of length 129, FFT is the Fast Fourier Transform function.

[0084] In some implementations, after obtaining the human voice spectrum using the target extractor, the human voice spectrum needs to be post-processed to convert it from a frequency domain signal to a time domain signal.

[0085] Specifically, the target extractor infers a target mask in each frame, and the target mask output from the inference in the nth frame is labeled as m. n m n Given a weight vector of length 129 with values ​​between 0 and 1, the noise-suppressed human voice spectrum of the nth frame is:

[0086] H n =H n ⊙m n .

[0087] Use the fast inverse Fourier transform to transform H n Transform to the time domain to obtain the noise-suppressed human voice spectrum in the nth frame: Here, IFFT stands for Inverse Fast Fourier Transform. At this point, Y′ n It is a real number vector of length 256.

[0088] Y′ n Windowing needs to be applied again, still using a 256-point Vorbis window, to obtain the time-domain value of the human voice spectrum after noise suppression in the nth frame. The time-domain windowed result is: Yn =w vorbis ⊙Y′ n Finally, the outputs of the first, second, third, and Nth frames after windowing are combined into the time-domain signal with a 50% overlap to complete the post-processing of the human voice spectrum and obtain the target human voice signal.

[0089] In some implementations, such as Figure 3 As shown, the target extractor includes five partitions: the logarithmic operation area, the Conv2D area, the Conv2DTrans area, the GRU area, and the Reshape area.

[0090] Specifically, in the log operation area, for any complex number z in the input speech spectrum, the following operation is performed: log 10 (conj(z)*z+∈), where ∈=1.0*10 -12 It is a very small quantity, and generally its impact on the calculation result is negligible. Its function is to prevent logarithmic errors. 10 (0) leads to numerical anomalies in the training and inference processes.

[0091] The target extractor has four Conv2D regions, namely Conv2D_1, Conv2D_2, Conv2D_3, and Conv2D_4, which are connected in series to form the encoder in the target extractor. Each Conv2D region contains a target feature layer equivalent to the target human voice feature extraction layer and the batch normalization layer, as well as an ELU layer.

[0092] The target extractor has four Conv2DTrans regions, namely Conv2DTrans_1, Conv2DTrans_2, Conv2DTrans_3, and Conv2DTrans_4, which are concatenated in reverse order to form the decoder in the target extractor. Each Conv2DTrans region also contains a target feature layer, which is equivalent to the target human voice feature extraction layer and the batch normalization layer, as well as an ELU layer.

[0093] The target extractor has two GRU regions, namely GRU_1 and GRU_2. Each GRU region uses the standard GRU implementation provided by the deep learning framework Keras. The activation function at the output is the hyperbolic tangent function, and the activation function used at the loop time step is the sigmoid function.

[0094] The purpose of the Reshape region is to connect the Conv2D region and the GRU region, because the data processed by the Conv2D region has an additional dimension compared to the GRU region and the input / output layers of the object extractor.

[0095] In addition, such as Figure 4As shown, the target extractor also includes bridging units to connect each Conv2D region to its corresponding Conv2DTrans region. Specifically, Conv2D_1 is bridged with Conv2DTrans_1, Conv2D_2 with Conv2DTrans_2, Conv2D_3 with Conv2DTrans_3, and Conv2D_4 with Conv2DTrans_4. The outputs of the Conv2D regions and the intermediate layers are designed to be the same size. After bridging by the bridging units, the outputs of the Conv2D regions and the intermediate layers are combined into a larger tensor, which is then input to the corresponding Conv2DTrans region of the Conv2D region.

[0096] In some implementations, a loss function is set up to assist in judging and estimating the human voice spectrum before generating the target human voice signal. The spectral error between the estimated human voice spectrum and the desired human voice spectrum S. The goal of the target extractor is to make the estimated human voice spectrum obtained using the target mask approximate the spectrum S of the human voice signal βr(t)*x(t) without ambient noise, and the loss function is used to evaluate... The spectral difference between S and S.

[0097] The loss function can be:

[0098]

[0099] λ is the proportionality coefficient, and c is the power-law compression ratio. The values ​​of λ and c can both be set to 0.3. S and S are represented respectively. The argument of , ∑ represents the summation symbol, and b represents the imaginary unit.

[0100] In some implementations, an optimizer is provided after the spectral differences are obtained. The optimizer adjusts the weights of each partition of the original feature extractor based on the spectral differences to obtain a target extractor such that the spectral difference between the estimated human voice spectrum obtained using the target mask generated therefrom and the desired human voice spectrum is less than or equal to a spectral error threshold. The optimizer can be an Adam optimizer with a learning rate of 0.01.

[0101] In some implementations, the time step of the target extractor is set to T, and the parameter list for each partition in the target extractor can be found in the following table:

[0102] Table 1

[0103] Module Name hyperparameters Input tensor information Output tensor information Number of parameters loq10 none T×129compeex T×129rea 0 reshape_1 none T×129real T×129×1real 0 Conv2D_1 2×3，(1，2)，4 T×129×1real T×64×4real 28+16 Conv2D_2 2×3，(1，2)，8 T×64×4real T×31×8real 200+32 Conv2D_3 2×3，(1，2)，16 T×31×8real T×15×16real 784+64 Conv2D_4 2×3，(1，2)，16 T×15×16real T×7×16real 1552+64 resnape_2 none T×7×16real T×112real 0 GRU_1 60 T×112real T×60real 31320 GRU_2 112 T×60real T×112real 58464 reshape_3 none T×112real T×7×16real 0 Concat_1 none (T×7×16,T×7×16)real T×7×32real 0 Conv2DTrans_4 2×3，(1，2)，16 T×7×32real T×15×16real 3088+64 ConCat_2 none (T×15×16T×15×16)real T×15×32real 0 Conv2DTrans_3 2×3，(1，2)，8 T×15×32real T×31×8real 1544+32 Concat_3 none (T×31×8T×31×8)real T×31×16real 0 Conv2DTrans_2 2×3，(1，2)，4 T×31×16real T×64×4real 388+16 Concat_4 none (T×64×4T×64×4)real T×64×8real 0 Conv2DTrans_1 2×3，(1，2)，1 T×64×8real T×129×1real 49+4 resnape_4 none T×129×1real T×129real 0

[0104] Table 1 lists the parameters for each partition in the target extractor, mainly showing the hyperparameters, input tensor information, output tensor information, and number of parameters for partitions named log10, reshape_1, Conv2D_1, Conv2D_2, Conv2D_3, Conv2D_4, reshape_2, GRU_1, GRU_2, reshape_3, Concat_1, Conv2DTrans_4, Concat_2, Conv2DTrans_3, Concat_3, Conv2DTrans_2, Concat_4, Conv2DTrans_1, and reshape_4. For example, log10 has no hyperparameters, its input tensor information is T×129complex (complex being a complex number), its output tensor information is T×129real (real being a real number), and its parameter value is 0. The parameters for the other partitions are not listed individually.

[0105] The hyperparameters of the Conv2D and Conv2DTrans regions specify the size, stride, and number of feature extraction matrices, respectively. The input and output tensor information represents the tensor size and data type (real or complex) of the corresponding partition. The "Number of Parameters" column specifies the total number of parameters for the corresponding partition. The total number of parameters for the Conv2D and Conv2DTrans regions consists of two parts: the parameters of the voice feature extraction layer and the parameters of the batch normalization layer within the module.

[0106] In some implementations, SIMD (Single Instruction Multiple Data) technology is also incorporated, enabling a single instruction to process multiple data items. In this disclosure, over 80% of the operations are vector dot product operations. Matrix multiplication can essentially be decomposed into multiple vector dot product operations. Dot product operations have high parallelism, which can improve computational efficiency.

[0107] The proposed method for constructing a noise suppression module reduces the data volume and computational load of the noise suppression module by quantizing each feature extraction matrix in the human voice feature extraction layer and merging the human voice feature extraction layer and the batch normalization layer, thus enabling it to be deployed in small devices while ensuring the effect of the target human voice signal.

[0108] Figure 6 This is a block diagram of an apparatus for constructing a noise suppression module according to an exemplary embodiment of the present disclosure. Figure 6As shown, another aspect of this disclosure provides a noise suppression module construction apparatus 1000, which may include: a training unit 1002 for training an original feature extractor using mixed speech samples to obtain a mask extractor for generating a target mask; a data compression unit 1004 for compressing the data of the mask extractor to obtain a target extractor, wherein the data size of the target extractor is less than that of the mask extractor; and a module generation unit 1006 for generating a noise suppression module with a target data size based on the target extractor, wherein the target data size is less than or equal to a storage capacity threshold of the target device into which the noise suppression module is embedded.

[0109] The device 1000 may include corresponding units that perform one or more steps in the flowchart described above. Therefore, each or more steps in the flowchart can be performed by a corresponding unit, and the device 1000 may include one or more of these units. A unit may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable storage medium for implementation by a processor, or implemented through some combination thereof.

[0110] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application of the hardware and bus design constraints. Bus 1100 will connect various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0111] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, this diagram uses only one connection line, but it does not imply that there is only one bus or one type of bus.

[0112] Any process or method description in the flowcharts or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure may be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0113] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0114] The noise suppression module construction device proposed in this disclosure reduces the data volume and computational load of the noise suppression module by quantizing each feature extraction matrix in the human voice feature extraction layer and merging the human voice feature extraction layer and the batch normalization layer, so that it can be deployed in small devices while ensuring the effect of the target human voice signal.

[0115] Figure 7 This is a flowchart of a noise suppression method according to an exemplary embodiment of the present disclosure. Figure 7 As shown, another aspect of this disclosure provides a noise suppression method S200, which may include: step S202, preprocessing the input noisy speech to obtain the target spectrum of the noisy speech; step S204, inputting the target spectrum to a noise suppression module, whereby the target extractor in the noise suppression module processes the target spectrum to generate a human voice mask, wherein the noise suppression module is constructed using the construction method of the noise suppression module described in any of the above embodiments; step S206, whereby the noise suppression module covers the human voice mask onto the target spectrum to extract the human voice spectrum from the target spectrum; and step S208, whereby the noise suppression module performs post-processing on the human voice spectrum to generate a target human voice signal.

[0116] The noise suppression method proposed in this disclosure uses a noise suppression module to process noisy speech and generate a target human voice signal. This target human voice signal is then processed by a neural network to obtain a human voice mask, which has a good noise filtering effect and solves the problem that small devices cannot support neural network denoising.

[0117] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0118] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0119] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware, and the program can be stored in a readable storage medium. When executed, the program includes one or a combination of the steps of the method implementation.

[0120] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0121] This disclosure also provides an electronic device, including: a memory storing execution instructions; and a processor or other hardware module executing the execution instructions stored in the memory, causing the processor or other hardware module to perform the above-described method.

[0122] This disclosure also provides a readable storage medium storing execution instructions, which, when executed by a processor, are used for a method of constructing a noise suppression module. The method may include: training an original feature extractor using mixed speech samples to obtain a mask extractor for a target mask; compressing the data volume of the mask extractor to obtain a target extractor, wherein the data volume of the target extractor is less than that of the mask extractor; and generating a noise suppression module with a target data volume based on the target extractor, wherein the target data volume is less than or equal to a storage capacity threshold of the target device into which the noise suppression module is embedded.

[0123] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0124] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this disclosure, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0125] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A method for constructing a noise suppression module, characterized in that, include: The original feature extractor is trained using mixed speech samples to obtain a mask extractor for generating a target mask; the mixed speech samples include noise data and human voice data. The mask extractor is subjected to data compression processing to obtain a target extractor, wherein the data volume of the target extractor is smaller than that of the mask extractor; Based on the target extractor, a noise suppression module with a target data volume is generated, wherein the target data volume is less than or equal to the storage capacity threshold of the target device into which the noise suppression module is embedded. The step of compressing the data volume of the mask extractor to obtain the target extractor includes: quantizing each feature extraction matrix of the human voice feature extraction layer to generate a target human voice feature extraction layer with a target extraction matrix, wherein the data volume of the target extraction matrix is ​​less than the data volume of the feature extraction matrix; merging the batch normalization layer in the mask extractor with the human voice feature extraction layer to form an equivalent target feature layer, wherein the data volume of the target feature layer is less than the sum of the data volumes of the batch normalization layer and the target human voice feature extraction layer; and integrating each of the target feature layers to generate the target extractor. The step of performing data compression processing on the mask extractor to obtain the target extractor further includes: setting a precision restoration area for restoring the process features output by the target feature layer to the target human voice features.

2. The method for constructing the noise suppression module according to claim 1, characterized in that, The step of training the original feature extractor using mixed speech samples to obtain a mask extractor for generating the target mask includes: The mixed speech sample is input into the original feature extraction layer of the original feature extractor, and the original feature extraction layer outputs the estimated human voice features corresponding to the mixed speech sample; The estimated human voice features and the expected human voice features are compared to determine the feature loss value used to characterize the difference between the two. In response to the determination that the feature loss value exceeds the error threshold, the weights of each target feature matrix in the original feature extraction layer are adjusted to obtain a voice feature extraction layer where the feature loss value is less than or equal to the error threshold; and The various voice feature extraction layers are integrated to generate a mask extractor that includes multiple voice feature extraction layers, wherein the mask extractor is used to generate the target mask based on the target voice features output by the voice feature extraction layers.

3. The method for constructing the noise suppression module according to claim 1, characterized in that, Before training the original feature extractor using mixed speech samples to obtain a mask extractor for generating the target mask, the method further includes: The collected noise data and human voice data are processed to generate the mixed speech sample, wherein the noise data includes environmental noise data and room impact response data.

4. The method for constructing the noise suppression module according to claim 3, characterized in that, The process of processing the collected noise data and human voice data to generate the mixed speech sample includes: The room impact response data in the noise data is convolved with the human voice data to generate the first mixed data. The first mixed data is combined with the environmental noise data in the noise data to generate the second mixed data; and The second mixed data is preprocessed to generate the mixed speech sample containing the speech spectrum of the second mixed data.

5. A device for constructing a noise suppression module, characterized in that, include: The training unit is used to train the original feature extractor using mixed speech samples to obtain a mask extractor for generating the target mask. The mixed speech samples include noise data and human voice data; A data compression unit is used to compress the data of the mask extractor to obtain a target extractor, wherein the data volume of the target extractor is smaller than the data volume of the mask extractor. as well as A module generation unit is used to generate a noise suppression module with a target data amount based on the target extractor, wherein the target data amount is less than or equal to the storage capacity threshold of the target device into which the noise suppression module is embedded. The step of compressing the data volume of the mask extractor to obtain the target extractor includes: quantizing each feature extraction matrix of the human voice feature extraction layer to generate a target human voice feature extraction layer with a target extraction matrix, wherein the data volume of the target extraction matrix is ​​less than the data volume of the feature extraction matrix; merging the batch normalization layer in the mask extractor with the human voice feature extraction layer to form an equivalent target feature layer, wherein the data volume of the target feature layer is less than the sum of the data volumes of the batch normalization layer and the target human voice feature extraction layer; and integrating each of the target feature layers to generate the target extractor. The step of performing data compression processing on the mask extractor to obtain the target extractor further includes: setting a precision restoration area for restoring the process features output by the target feature layer to the target human voice features.

6. A noise suppression method, characterized in that, include: The input noisy speech is preprocessed to obtain the target spectrum of the noisy speech; The target spectrum is input to the noise suppression module, and the target extractor in the noise suppression module processes the target spectrum to generate a human voice mask, wherein the noise suppression module is constructed using the construction method of the noise suppression module according to any one of claims 1 to 4; The noise suppression module covers the target spectrum with the human voice mask to extract the human voice spectrum from the target spectrum; as well as The noise suppression module performs post-processing on the human voice spectrum to generate the target human voice signal.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a method for constructing a noise suppression module as described in any one of claims 1 to 4.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program adapted for loading by a processor to execute the method for constructing a noise suppression module as described in any one of claims 1 to 4.

Images