Neural network training method, speech noise reduction method, device and storage medium

Abstract

The embodiments of the present application provide a neural network training method, a speech denoising method, a computer program product, a device and a storage medium. When training a neural network, the network parameters of the neural network can be subjected to bit width truncation processing, and the accuracy of the denoising parameters determined by the network parameters after bit width truncation processing and the influence of the bit width truncation processing on the network parameters themselves are used as constraints in the neural network training process, thereby minimizing the influence of the bit width truncation processing on the data accuracy, and training to obtain a lightweight and high-performance speech denoising neural network, so that the lightweight speech denoising neural network can be deployed in low-performance speech acquisition equipment.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a neural network training method, a speech noise reduction method, a computer program product, a device and a storage medium. Background Art

[0002] In some voice collection scenarios, the voice data collected by the voice collection device may be mixed with a lot of environmental noise, and the sound collection effect is very poor, so the collected voice data needs to be processed for noise reduction. Traditional voice noise reduction technology either achieves noise reduction by improving hardware circuits or by signal processing. However, the method of achieving noise reduction by improving hardware circuits can only suppress noise in a specific frequency range, while the method of achieving noise reduction by signal processing can only suppress steady-state noise, and cannot suppress some instantaneous and sudden noises (such as car horns, etc.).

[0003] With the development of AI technology, this technology has gradually been applied to the field of speech noise reduction. At present, speech noise reduction through pre-trained neural networks can achieve very good noise reduction effects. This method can suppress various noises (for example, steady-state, transient, and noise related to the speech itself) and retain the information of the speech itself to a great extent. However, due to the complex network structure of the neural network itself, there are many network parameters, and for fields such as recording that require high voice quality, the requirements for audio streams are also higher (usually sampling rate 48 kHz, bit width 24bit), which leads to the use of neural networks in the process of noise reduction of speech data. The amount of data involved is very large, requiring a large amount of memory, and the amount of calculation is large, and the hardware performance requirements of the processing equipment are high, making it difficult to apply the speech noise reduction method based on neural networks to some low-performance speech acquisition equipment. Summary of the invention

[0004] In view of this, the present application provides a neural network training method, a speech noise reduction method, a computer program product, a device and a storage medium.

[0005] According to a first aspect of the present application, a method for training a speech denoising neural network is provided, the method comprising:

[0006] Acquire noisy speech data, and perform feature extraction on the noisy speech data to obtain a first target speech feature, wherein the noisy speech data is obtained by mixing clean speech data with noise;

[0007] Performing bit width truncation processing on current network parameters of a preset neural network, and determining predicted noise reduction parameters for performing noise reduction on the noisy speech data based on the network parameters after the bit width truncation processing and the first target speech feature;

[0008] Determining a first loss according to a difference between the predicted noise reduction parameter and the actual noise reduction parameter; wherein the actual noise reduction parameter is determined based on the noisy speech data and the clean speech data;

[0009] Determine a second loss according to a difference between the network parameter after the bit width truncation processing and the current network parameter;

[0010] Determine a target loss according to the first loss and the second loss, and adjust the current network parameters based on the target loss;

[0011] Repeat the above steps until the target loss meets the preset conditions, and perform the bit width truncation processing on the current network parameters obtained in the current round of adjustment as the network parameters of the trained speech denoising neural network.

[0012] According to a second aspect of the present application, a speech noise reduction method is provided, the method comprising:

[0013] Obtaining the voice data to be de-noised;

[0014] Extracting features of the speech data to be denoised to obtain second target speech features;

[0015] Inputting the second target speech feature into the speech denoising neural network trained by the method mentioned in the first aspect above, so as to predict the denoising parameters for denoising the speech data to be denoised by the speech denoising neural network;

[0016] The noise reduction parameter is used to perform noise reduction processing on the speech data to be noise reduced.

[0017] According to a third aspect of the present application, an electronic device is provided, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the processor implements the method mentioned in the first aspect and / or the second aspect when executing the computer program.

[0018] According to a fourth aspect of the present application, a computer program product is provided, wherein the computer program product comprises a computer program, and when the computer program is executed, the method mentioned in the first aspect and / or the second aspect is implemented.

[0019] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed, the method mentioned in the first aspect and / or the second aspect is implemented.

[0020] By applying the solution provided in the present application, in order to allow the speech denoising neural network to be deployed in low-performance speech collection equipment, when training the neural network, the network parameters of the neural network can be subjected to bit width truncation processing, and the accuracy of the denoising parameters determined by the network parameters after the bit width truncation processing, as well as the influence of the bit width truncation processing on the network parameters themselves, are used as constraints in the neural network training process, thereby minimizing the influence of the bit width truncation processing on the data accuracy, and training to obtain a lightweight and high-performance speech denoising neural network, so that the lightweight speech denoising neural network can be deployed in low-performance speech collection equipment.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 It is a schematic diagram of a neural network training method according to an embodiment of the present application.

[0024] Figure 2 It is a flowchart of a neural network training method according to an embodiment of the present application.

[0025] Figure 3 It is a schematic diagram of extracting a first target speech feature from noisy speech data according to an embodiment of the present application.

[0026] Figure 4 It is a structural diagram of a speech denoising neural network according to an embodiment of the present application.

[0027] Figure 5 It is a schematic diagram of the logical structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] In some voice collection scenarios, the voice data collected by the voice collection device may be mixed with a lot of environmental noise, and the sound collection effect is very poor, so the collected voice data needs to be processed for noise reduction. For example, taking a wireless lavalier microphone as an example, since it uses an omnidirectional microphone, it basically has no additional attenuation for sounds in all directions. Therefore, when used in a noisy environment, the user's voice is easily damaged by environmental noise, and voice noise reduction processing is required before it can be used.

[0030] Traditional speech noise reduction technology either achieves noise reduction by improving hardware circuits or by signal processing. However, the method of improving hardware circuits can only suppress noise in a specific frequency range and cannot cover all types of noise. The method of signal processing can only suppress steady-state noise and cannot suppress some instantaneous and sudden noise (such as car horns, etc.).

[0031] With the development of AI technology, this technology has gradually been applied to the field of speech noise reduction. At present, speech noise reduction through pre-trained neural networks can achieve very good noise reduction effects. This method can not only suppress various noises (for example, steady-state, transient, and noise related to the speech itself), but also retain the information of the speech itself to a great extent with low distortion. However, due to the complex network structure of the neural network itself, the large number of network parameters, and the large amount of data, and for fields such as recording that require high voice quality, in addition to high intelligibility, high fidelity (high restoration, no jamming, etc.) is also required, so the requirements for audio streams are also higher (usually sampling rate 48 kHz, bit width 24bit), resulting in the use of neural networks in the process of noise reduction of speech data. The amount of data involved is very large, requiring a large amount of memory, and the amount of calculation is large, and the hardware performance requirements of the processing equipment are high, making it difficult to apply the speech noise reduction method based on neural networks to some low-performance voice acquisition equipment (for example, wireless lavalier microphones).

[0032] Considering the complex structure of the neural network and the large number of network parameters, the network parameters often need to occupy a large amount of memory and the amount of calculation is also large. In order to allow this neural network-based speech denoising method to be applied to embedded speech acquisition devices with poor performance, the embodiment of the present application thinks that the network parameters of the neural network can be processed by bit width truncation to reduce the amount of data, thereby saving memory and reducing the amount of calculation. However, the bit width truncation of the network parameters will inevitably sacrifice the data accuracy. In order to make the data accuracy lost as small as possible during the bit width truncation process, the present application can use the influence of the bit width truncation process on the network parameters themselves and the accuracy of the noise reduction parameters determined by the network parameters after the bit width truncation process as constraints in the neural network training process when training the neural network, so that the valid data in the network parameters obtained by training is concentrated in the first few bits as much as possible, and then the data accuracy lost after the network parameters are truncation can be minimized. For example, taking the network parameters as 32bit as an example, the network parameters can be continuously adjusted during the training process to make the valid data in the network parameters as concentrated as possible in the first 16 bits, so that even if the last 16 bits are truncated, the impact on the network parameters is relatively small. This can reduce the memory occupied by the network parameters of the neural network and reduce the amount of calculation, so that the neural network can be deployed in devices with poor performance. At the same time, the accuracy of the speech noise reduction results can be guaranteed.

[0033] Based on the above-mentioned inventive concept, an embodiment of the present application provides a training method for a speech denoising neural network, which can be implemented by various devices such as mobile phones, computers, cloud servers, server clusters, etc.

[0034] The following combination Figure 1 and Figure 2 Introducing the training method. Figure 1 is the flowchart of the training method. Figure 2 is a schematic diagram of the training method. Figure 1 As shown, the training method includes the following steps:

[0035] S102, acquiring noisy speech data, and performing feature extraction on the noisy speech data to obtain a first target speech feature; wherein the noisy speech data is obtained by mixing clean speech data with noise;

[0036] Before training the speech denoising neural network, a large amount of sample data for training the speech denoising neural network can be collected, wherein each set of sample data includes clean speech data and noisy speech data obtained by mixing the clean speech data with environmental noise. Since the clean speech data corresponding to the noisy speech data is known, appropriate noise reduction parameters can be determined based on the two, hereinafter referred to as actual noise reduction parameters, as labels for training the neural network.

[0037] In step S102, noisy speech data may be first obtained, and feature extraction may be performed on the noisy speech data according to some pre-set feature extraction strategies to obtain speech features, which are hereinafter referred to as first target speech features. The extracted speech features may be various types of speech features such as logarithmic power spectrum, Mel-frequency cepstral coefficients (MFCC), and Gammatone coefficients, which are not limited in the embodiments of the present application. There may also be multiple ways to extract features. For example, the noisy speech data may be first divided into multiple sub-bands, and then feature extraction may be performed on the sub-bands, or feature extraction may be performed directly on the noisy speech data, which is not limited in the embodiments of the present application.

[0038] The first target speech feature may include multiple types of features, or multiple features, which may be represented by a vector or a matrix.

[0039] S104, performing bit width truncation processing on current network parameters of a preset neural network, and determining predicted noise reduction parameters for performing noise reduction on the noisy speech data based on the network parameters after the bit width truncation processing and the first target speech feature;

[0040] In step S104, in order to reduce the amount of data, the current network parameters of the preset neural network can be subjected to bit width truncation processing. Among them, the bit width truncation processing refers to truncating the last few bits in the network parameters. For example, assuming that the original network parameters of the neural network are 32 bits, the last 16 bits can be removed and only the first 16 bits are retained, thereby greatly reducing the amount of data. Of course, the number of bits truncated by the bit width truncation processing can be flexibly set based on the demand for data accuracy, for example, it can be the last 1 bit, 2 bits, 4 bits, 8 bits, 16 bits, 24 bits, etc., which is not limited in the embodiments of the present application.

[0041] Then, the noise reduction parameters for noise reduction of the noisy speech data can be determined according to the network parameters after the bit width truncation processing and the first target speech features, which are hereinafter referred to as predicted noise reduction parameters. Among them, the noise reduction parameters can be filter coefficients, and based on the filter coefficients, a filter suitable for noise reduction processing of the speech data to be noise reduced can be designed. Of course, the noise reduction parameter can also be a gain, and the gain can be used to increase the strength of the effective speech signal to make it more prominent in the background noise. Among them, the type of noise reduction parameter can be flexibly set based on actual needs, and the embodiments of the present application are not limited.

[0042] S106, determining a first loss according to a difference between the predicted noise reduction parameter and the actual noise reduction parameter; wherein the actual noise reduction parameter is determined based on the noisy speech data and the clean speech data;

[0043] In step S106, after determining the predicted noise reduction parameters, the actual noise reduction parameters corresponding to the noisy speech data can be obtained, and the actual noise reduction parameters can be determined based on the noisy speech data and its corresponding clean speech data. Then the first loss can be determined according to the difference between the predicted noise reduction parameters and the actual noise reduction parameters. The difference between the predicted noise reduction parameters and the actual noise reduction parameters can be characterized by various parameters that measure the difference. For example, the mean square error between the predicted noise reduction parameters and the actual noise reduction parameters can be determined as the first loss. Alternatively, the absolute error between the two can be directly used as the first loss. The first loss is used to measure the accuracy of the noise reduction parameters determined based on the network parameters after bit width truncation processing. Training the neural network with the first loss as a constraint can ensure the accuracy of the neural network prediction results.

[0044] S108, determining a second loss according to a difference between the network parameter after the bit width truncation processing and the current network parameter;

[0045] In step S108, in order to minimize the impact of the bit width truncation processing on the prediction results of the neural network, the second loss can be determined according to the difference between the network parameters after the bit width truncation processing and the network parameters before the bit width truncation processing. The second loss is used to measure the difference between the network parameters before and after the bit width truncation processing. Using the second loss as another constraint in the neural network training process can minimize the impact of the bit width truncation processing on the network parameters, that is, the effective data of the adjusted network parameters can be concentrated to the first few bits as much as possible, that is, the truncated bit width is as much as possible 0.

[0046] S110, determining a target loss according to the first loss and the second loss, and adjusting the current network parameters based on the target loss;

[0047] In step S110, after determining the first loss and the second loss, a target loss can be determined based on the two. For example, the sum of the two can be used as the target loss, or a certain weight can be given to each of the two and then added to obtain the target loss. Then, the current network parameters of the neural network can be adjusted based on the target loss to obtain new current network parameters. For example, the original 32-bit network parameters of the neural network can be adjusted to obtain new current network parameters for the next round of training.

[0048] S112, repeat the above steps until the target loss meets the preset conditions, and perform the bit width truncation processing on the current network parameters obtained in the current round of adjustment as the network parameters of the trained speech denoising neural network.

[0049] Step S102-step S110 can be repeated continuously until the target loss meets certain conditions, such as the target loss converges, or the target loss is less than a preset threshold. At this time, the bit width truncation processing can be performed on the network parameters currently adjusted as the network parameters of the trained speech denoising model. For example, when the target loss converges, a 32-bit network parameter can be obtained at this time, and then the 32-bit network parameter can be truncated into a 16-bit network parameter as the network parameter of the trained speech denoising neural network, thereby obtaining a lightweight speech denoising neural network.

[0050] In an embodiment of the present application, when training a neural network, the network parameters of the neural network can be subjected to bit width truncation processing, and the impact of the bit width truncation processing on the network parameters themselves, as well as the accuracy of the noise reduction parameters determined by the network parameters after the bit width truncation processing are used as constraints in the neural network training process, thereby minimizing the impact of the bit width truncation processing on data accuracy, and training a lightweight and high-performance speech denoising neural network.

[0051] Usually, when extracting speech features from speech data, the speech data can be first sub-band divided, and the speech data can be divided into multiple frequency ranges, and then the speech features of different frequency ranges (i.e., different sub-bands) can be analyzed. In the prior art, a fixed sub-band division method based on acoustic experiments is usually used to divide the speech data into sub-bands. The sub-band type obtained by this fixed sub-band division method is relatively single, which cannot fully reflect the speech features, is not flexible enough, and the optimal sub-band division method is different for different types of speech data. Based on this, the embodiment of the present application no longer adopts a fixed sub-band division method, but can set a variety of feasible sub-band division strategies, and select the optimal sub-band division strategy from a variety of feasible sub-band division strategies to divide the speech data, so that the sub-bands obtained by the division can more accurately reflect the speech features. Specifically, a sub-band division strategy pool can be pre-set, and the sub-band division strategy pool includes a variety of sub-band division strategies. In order to ensure the effectiveness of the sub-band division strategy, these sub-band division strategies must meet certain conditions, that is, the multiple sub-bands obtained by sub-band division of the speech data according to these sub-band division strategies must meet the following conditions at the same time:

[0052] (1) The low-frequency part of any of the multiple sub-bands obtained by division is denser than the high-frequency part.

[0053] (2) If adjacent sub-bands among the multiple sub-bands obtained by division overlap, the overlapping frequency range accounts for the same proportion in the frequency ranges of the two adjacent sub-bands.

[0054] That is, when dividing the sub-bands, adjacent sub-bands may overlap or may not overlap. If adjacent sub-bands overlap, in order to ensure that the overlapping part has the same impact on the two sub-bands, it is possible to ensure that the overlapping frequency range has the same proportion in the frequency range of the two adjacent sub-bands.

[0055] (3) The total frequency range obtained by superimposing the frequency ranges of the multiple sub-bands obtained by division covers the frequency range of the noisy speech data, that is, these sub-bands can cover the entire frequency range of the noisy speech data, avoiding omission of some frequency ranges.

[0056] After setting up multiple subband division strategies in the subband division strategy pool, when training a neural network, a subband division strategy can be selected from the subband division strategy to perform subband division processing on the noisy speech data to obtain multiple subbands, and then feature extraction can be performed on each subband to obtain the speech features of each subband, and then the first target speech feature can be obtained based on the speech features of the multiple subbands. For example, the first target speech feature can be a feature matrix composed of the speech features of the multiple subbands, or it can be a speech feature obtained by splicing, merging or various processing of the speech features of the multiple subbands.

[0057] In some embodiments, in order to select the optimal partitioning strategy from the subband partitioning strategy pool for use in the neural network inference stage. In the neural network training stage, each subband partitioning strategy in the subband partitioning strategy pool can be selected one by one for subband partitioning of noisy speech data to extract the first target speech feature for training the neural network. In addition, for each subband partitioning strategy in the subband partitioning strategy pool, the convergence loss when the noisy speech data is subband partitioned using the seed band partitioning strategy to train the speech denoising neural network can be recorded, wherein the convergence loss refers to the loss when the loss of the neural network converges, and then the subband partitioning strategy with the smallest convergence loss can be used as the optimal partitioning strategy for the inference stage of the speech denoising neural network.

[0058] Considering that speech has a certain continuity, there is often a certain correlation between adjacent sub-bands. In order to obtain more comprehensive speech features, in some embodiments, such as Figure 3As shown, when extracting features from noisy speech data to obtain the first target speech feature, the noisy speech data can be processed by sub-band division to obtain multiple sub-bands (such as sub-band 1 to sub-band n in the figure), and then feature extraction is performed on each of the multiple sub-bands to obtain the speech feature of each sub-band (speech feature 1 to speech feature n in the figure). For each sub-band, the speech feature of the sub-band can be spliced ​​with the speech feature of the associated sub-band of the sub-band to obtain the spliced ​​feature as the first target speech feature, wherein the associated sub-band is the adjacent sub-band of the sub-band. By splicing the features of adjacent sub-bands with a certain association, speech features that reflect the connection between adjacent sub-bands can be reconstructed, so that the speech features finally extracted are more diverse.

[0059] In some embodiments, when determining the second loss based on the difference between the network parameters after the bit width truncation processing and the network parameters before the bit width truncation, the mean square error of the network parameters after the bit width truncation processing and the network parameters before the bit width truncation can be determined, and the second loss is determined based on the mean square error.

[0060] Considering that in the process of training the neural network, the model trained with sample data that is more in line with the actual application scenario performs better in the inference stage. In the past, when collecting sample data for training the speech denoising neural network, some fixed speech acquisition devices were usually used to collect noisy speech data as sample data. These sample data cannot reflect the characteristics of the speech data collected by the speech acquisition device that actually needs noise reduction. For example, taking the speech acquisition device that needs noise reduction as a wireless lavalier microphone as an example, during the operation of the wireless lavalier microphone, the hardware structure, material, and distance between the wireless lavalier microphone and the mouth of the wireless lavalier microphone itself will affect the collected user voice and environmental noise. If other devices are used to collect sample data, for example, noisy speech data including car horn noise is also collected, there is still a certain difference between it and the noisy speech data collected in the actual working scene of the wireless microphone. In order to make the trained neural network have better performance in the inference stage, if the speech denoising neural network is used to perform noise reduction on the speech data collected by the target speech acquisition device, when collecting the noisy speech data for training the neural network, the noise in the noisy speech data can be obtained by simulating the actual working scene of the target speech acquisition device. For example, taking a wireless lavalier microphone as an example, the actual working process of the wireless lavalier microphone can be simulated, and various types of noise that may exist in the collection during this working process can be used to simulate the actual working conditions of the wireless lavalier microphone.

[0061] In some embodiments, Figure 4As shown, the speech denoising neural network includes a noise estimation subnetwork and a noise reduction parameter determination subnetwork. After a series of pre-processing is performed on the noisy speech data to obtain the first target speech feature, the noise estimation subnetwork can determine the characteristics of the noise in the noisy speech data based on the first target speech feature, and the noise reduction parameter determination subnetwork can determine the noise reduction parameters based on the characteristics of the noise. Among them, both the noise estimation subnetwork and the noise reduction parameter determination subnetwork can use a deep neural network (DNN), that is, a two-stage DNN network is used to respectively perform noise estimation and noise reduction parameter determination to improve accuracy.

[0062] In addition, the embodiment of the present application also provides a method for speech denoising, which can use the speech denoising neural network trained by the neural network training method provided in any of the above embodiments to perform denoising on the denoised speech data. Among them, since the network parameters of the speech denoising neural network trained by the neural network training method provided in any of the above embodiments occupy a small bit width, that is, a small amount of data, the occupation of the device memory and the amount of calculation in the process of denoising the speech data can be reduced, so that the speech denoising neural network can be deployed in some embedded speech acquisition devices with poor performance (such as wireless lavalier microphones). In addition, since the network parameters are optimized during the training of the speech denoising neural network, the data accuracy loss caused by the bit width truncation of the network parameters is reduced to a minimum, so that the prediction results of the trained lightweight speech denoising neural network still have a high accuracy.

[0063] The speech denoising method of the embodiment of the present application can be adapted to various devices deployed with the above-mentioned speech denoising neural network. For example, the speech denoising neural network can be deployed in some low-performance speech acquisition devices, and the method can be executed by these speech acquisition devices. Alternatively, the speech denoising neural network can also be deployed in a server, a server cluster or other devices with better performance, and the method can be executed by these devices with better performance.

[0064] Specifically, the speech data to be denoised can be obtained first, and the speech to be denoised can be speech data collected by a speech acquisition device. Then, feature extraction can be performed on the speech data to be denoised to obtain speech features, hereinafter referred to as second target speech features. Then, the second target speech features can be input into the speech denoising neural network trained by the training method provided in any of the above embodiments, so as to predict the denoising parameters for denoising the speech data to be denoised through the speech denoising neural network, and then the denoising parameters can be used to denoise the speech data to be denoised. For example, the denoising parameters can be filter coefficients, and based on the filter coefficients, a filter suitable for denoising the speech data to be denoised can be designed.

[0065] In some embodiments, the speech denoising neural network can be deployed in an embedded platform. In order to improve the processing speed in the speech denoising process, an embedded platform including multiple computing units (e.g., Tensilica HiFiDSP) can be selected so that in the process of predicting the denoising parameters, multiple computing units can calculate the denoised speech data and network parameters in parallel to obtain the denoising parameters. At the same time, in order to achieve parallel computing, the denoised speech data and network parameters of the speech denoising neural network can be stored in a preset storage format, which is a format that can meet the parallel computing requirements of the embedded platform. For example, it can be stored in the form of a high-dimensional tensor, so that the data that needs to be used at the same time can be stored continuously, so that when the embedded platform uses the denoised speech data and network parameters for computing, it can load multiple data needed in the computing process in parallel, so that the multiple computing units can process the multiple data in parallel.

[0066] In addition, considering that in the process of predicting noise reduction parameters using a speech noise reduction neural network, the access to the calculation data (including the speech data to be noise reduced and the network parameters) is very time-consuming, in order to achieve rapid data storage and improve the processing efficiency of the neural network, in the embodiment of the present application, the speech data to be noise reduced and the network parameters of the speech noise reduction neural network can be stored in the form of high-dimensional tensors, and then a four-dimensional array for quickly accessing data from the memory can be pre-designed based on the data storage structure. In the process of calculating the noise reduction speech data and network parameters, the data required for the calculation process can be accessed from the memory according to the four-dimensional array. Among them, the format of the four-dimensional array is as follows:

[0067]

[0068] in, Indicates the storage address of the data to be read from the memory, Indicates the storage address of the data to be stored in the memory, Indicates the dimension of the data to be read from memory, The dimensions of the data to be stored in memory.

[0069] For example, in some embodiments, the embedded platform may include an input pointer orientation module, a logic operation module, and an output pointer orientation module. The input pointer orientation module may be configured to output a pointer according to the input pointer of the four-dimensional array. and Determine the multiple data to be read from the memory at one time, and then load the multiple data from the memory based on the parallel bit width of the embedded platform (for example, taking Tensilica HiFi DSP as an example, the parallel bit width is 128 bits), and send the multiple data to the logic operation module. Since the data is stored continuously, according to and We can also know the address of the next data to be read from the memory, so that we can continue to read data from the memory. For example, assuming that the dimension of the data to be read is 8, By offsetting 8 addresses, the address of the data to be read next time can be obtained. Similarly, by offsetting 8 bits based on the address of the data to be read next time, the address of the data to be read next time can be obtained. If repeated, data can be read continuously.

[0070] The logic operation module can perform the same operation on multiple loaded data in parallel to obtain the calculation result, which is the data to be stored in the memory. Determine the storage address of the data to be stored in the memory (i.e., the calculation result) and store it in the memory. The dimensions of the data to be stored in memory can be determined based on The storage address of the next data to be stored in the memory can be determined, so that data can be continuously stored in the memory. For example, assuming that the dimension of the data to be stored is 8, By offsetting 8 addresses, the address of the data to be stored next time can be obtained. Similarly, by offsetting 8 bits based on the address of the data to be stored next time, the address of the data to be stored next time can be obtained. If repeated, data can be stored continuously.

[0071] In the embodiment of the present application, the speech data to be denoised and the network parameters are stored in the format of a high-dimensional tensor, so that the embedded platform can load data in parallel during the calculation process to achieve parallel calculation and improve the processing speed. At the same time, by designing a four-dimensional array for accessing data based on the data storage structure, the data access efficiency in the calculation process can be accelerated, greatly improving the processing efficiency of using the speech denoising neural network to denoise the denoised speech data.

[0072] In some embodiments, when extracting features from the denoised speech data to obtain the second target speech feature, the denoised speech data can be sub-divided based on a predetermined target sub-band division strategy to obtain multiple sub-bands, and then feature extraction is performed on each of the multiple sub-bands, and the second target speech feature is obtained based on the speech features extracted from each sub-band. The target sub-band division strategy can be determined during the training phase of the speech denoising neural network. In order to extract speech features that more accurately and comprehensively reflect the characteristics of the denoised speech data, an optimal sub-band division strategy can be determined during the training phase of the speech denoising neural network to divide the sub-bands to determine the second target speech feature. Specifically, during the training process of the speech denoising neural network, the noisy speech data used to train the speech denoising neural network can be sub-divided according to the preset multiple sub-band division strategies one by one, so as to extract speech features from the divided sub-bands for training the speech denoising neural network. At the same time, the convergence loss of the speech denoising neural network corresponding to each sub-band division strategy can be recorded, and the sub-band division strategy with the smallest convergence loss can be used as the target sub-band division strategy for the reasoning process.

[0073] Considering that speech has a certain continuity, adjacent subbands often store certain associations. In order to obtain more comprehensive speech features, in some embodiments, when extracting features from the denoised speech data to obtain the second target speech features, the denoised speech data may be divided into subbands to obtain multiple subbands, and then feature extraction is performed on each of the multiple subbands to obtain speech features of each subband. For each subband, the speech features of the subband may be concatenated with the speech features of the associated subband of the subband to obtain the second target speech features, wherein the associated subband is the adjacent subband of the subband.

[0074] Among them, the solutions of the above-mentioned embodiments can be freely combined to obtain new solutions when there is no conflict. Due to space constraints, they will not be listed one by one here.

[0075] In addition, an embodiment of the present application further provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed, the method described in any of the above embodiments is implemented.

[0076] Furthermore, the present application also provides an electronic device, such as Figure 5 FIG. 1 is a hardware structure diagram of an electronic device 50 according to an embodiment of the present application, except Figure 5In addition to the processor 51 and memory 52 shown, the device may also include other hardware, such as a forwarding chip responsible for processing messages; from the perspective of hardware structure, the device may also be a distributed device, which may include multiple interface cards to expand message processing at the hardware level. The memory 52 stores computer instructions, and when the processor 51 executes the computer instructions, the image processing method mentioned in any of the above embodiments is implemented.

[0077] Correspondingly, an embodiment of the present application further provides a computer storage medium, in which a program is stored, and when the program is executed by a processor, the method in any of the above embodiments is implemented.

[0078] The embodiments of the present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be a computer-readable instruction, a data structure, a module of a program, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0079] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0080] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0081] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprises a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0082] The method and device provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of the present application should not be understood as a limitation on the present application.

Claims

1. A method for training a speech denoising neural network, characterized in that: The method comprises: Acquire noisy speech data, and perform feature extraction on the noisy speech data to obtain a first target speech feature, wherein the noisy speech data is obtained by mixing clean speech data with noise; Performing bit width truncation processing on current network parameters of a preset neural network, and determining predicted noise reduction parameters for performing noise reduction on the noisy speech data based on the network parameters after the bit width truncation processing and the first target speech feature; Determining a first loss according to a difference between the predicted noise reduction parameter and the actual noise reduction parameter; wherein the actual noise reduction parameter is determined based on the noisy speech data and the clean speech data; Determine a second loss according to a difference between the network parameter after the bit width truncation processing and the current network parameter; Determine a target loss according to the first loss and the second loss, and adjust the current network parameters based on the target loss; Repeat the above steps until the target loss meets the preset conditions, and perform the bit width truncation processing on the current network parameters obtained in the current round of adjustment as the network parameters of the trained speech denoising neural network.

2. The method according to claim 1, characterized in that Extracting features from the noisy speech data to obtain a first target speech feature includes: Selecting a subband division strategy from a subband division strategy pool to perform subband division processing on the noisy speech data to obtain a plurality of subbands; Performing feature extraction on each of the multiple sub-bands respectively, and obtaining the first target speech feature according to the speech feature extracted from each sub-band; The multiple subbands obtained by dividing according to any division strategy in the subband division strategy pool meet the following conditions: A low frequency portion of any subband of the plurality of subbands is denser than a high frequency portion; If adjacent sub-bands in the plurality of sub-bands overlap, the overlapping frequency ranges have the same proportion in the frequency ranges of the two adjacent sub-bands; The total frequency range obtained by superimposing the frequency ranges of the multiple sub-bands covers the frequency range of the noisy speech data.

3. The method according to claim 2, characterized in that The method further comprises: For each sub-band division strategy in the sub-band division strategy pool, recording the convergence loss when the noisy speech data is subjected to sub-band division processing by using the seed band division strategy to train the speech denoising neural network; The sub-band division strategy with the minimum convergence loss is used as the optimal division strategy for the inference stage of the speech denoising neural network.

4. The method according to claim 1, characterized in that Extracting features from the noisy speech data to obtain a first target speech feature includes: Performing sub-band division processing on the noisy speech data to obtain a plurality of sub-bands; Extracting features from each of the multiple sub-bands to obtain speech features of each sub-band; For each sub-band, the speech feature of the sub-band is concatenated with the speech feature of the associated sub-band of the sub-band, and the concatenated speech feature is used as the first target speech feature, wherein the associated sub-band is a sub-band adjacent to the sub-band.

5. The method according to claim 1, characterized in that Determining the second loss according to the difference between the network parameter after the bit width truncation processing and the current network parameter includes: Determine a mean square error between the network parameter after the bit width truncation processing and the current network parameter, and determine the second loss according to the mean square error.

6. The method according to claim 1, characterized in that The speech denoising neural network is used to perform denoising processing on speech data collected by a target speech collection device, and the noise in the noisy speech data is obtained by simulating an actual working scenario of the target speech collection device.

7. The method according to claim 1, characterized in that The speech denoising neural network includes a noise estimation subnetwork and a denoising parameter determination subnetwork. The noise estimation subnetwork is used to determine the characteristics of the noise in the noisy speech data based on the first target speech characteristics; The noise reduction parameter determination subnetwork is used to determine the noise reduction parameter based on the characteristics of the noise.

8. A speech noise reduction method, characterized in that: The method comprises: Obtaining the voice data to be de-noised; Extracting features of the speech data to be denoised to obtain second target speech features; Inputting the second target speech feature into a speech denoising neural network trained by the method according to any one of claims 1 to 7, so as to predict, by the speech denoising neural network, denoising parameters for denoising the speech data to be denoised; The noise reduction processing is performed on the speech data to be noise reduced based on the noise reduction parameters.

9. The method according to claim 8, characterized in that The speech denoising neural network is deployed in an embedded platform, the embedded platform includes a plurality of computing units, the speech data to be denoised and the network parameters of the speech denoising neural network are stored in a preset storage format, wherein the preset storage format is a format that meets the parallel computing requirements of the embedded platform, and the embedded platform predicts the denoising parameters for denoising the speech data to be denoised through the speech denoising neural network, including: Acquire the speech data to be denoised and network parameters of the speech denoising neural network; The multiple computing units are used to calculate the speech data to be denoised and the network parameters in parallel to obtain the denoising parameters.

10. The method according to claim 9, characterized in that The speech data to be denoised and the network parameters of the speech denoising neural network are stored in the form of high-dimensional tensors. When the embedded platform uses the multiple computing units to calculate the speech data to be denoised and the network parameters in parallel, it includes: In the process of calculating the speech data to be denoised and the network parameters, the data required for the calculation process is retrieved from the memory of the embedded platform according to a preset four-dimensional array, and the format of the four-dimensional array is as follows: in, Indicates the storage address of the data to be read from the memory, Indicates the storage address of the data to be stored in the memory, Indicates the dimension of the data to be read from memory, The dimensions of the data to be stored in memory.

11. The method according to claim 10, characterized in that The embedded platform includes an input pointer orientation module, a logic operation module, and an output pointer orientation module; The input pointer orientation module is used to and Determine a plurality of data to be read from the memory of the embedded platform at a time, read the plurality of data based on the parallel bit width of the embedded platform, and send the plurality of data to the logic operation module; The logic operation module is used to perform parallel calculation on the multiple data, obtain calculation results, and send them to the output pointer orientation module; The output pointer orientation module is used to output the pointer according to the and Determine a storage address of the calculation result in the memory of the embedded platform, and store the calculation result in the memory based on the storage address.

12. The method according to claim 8, characterized in that Extracting features of the speech data to be denoised to obtain second target speech features includes: Performing sub-band division processing on the speech data to be denoised based on a predetermined target sub-band division strategy to obtain a plurality of sub-bands; Performing feature extraction on each of the multiple sub-bands respectively, and obtaining the second target speech feature according to the speech feature extracted from each sub-band; The target subband division strategy is determined based on the following method: During the training process of the speech denoising neural network, the noisy speech data is sub-band divided according to the preset multiple sub-band division strategies one by one to train the speech denoising neural network, and the convergence loss of the speech denoising neural network corresponding to each sub-band division strategy is recorded, and the sub-band division strategy with the smallest convergence loss is used as the target sub-band division strategy.

13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed, the method according to any one of claims 1 to 7 or 8 to 12 is implemented.

14. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and computer instructions stored in the memory, and the processor implements the method according to any one of claims 1-7 or 8-12 when executing the computer instructions.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed, implements the method according to any one of claims 1-7 or 8-12.

Images