Model training method and device, electronic equipment and nonvolatile readable storage medium
Through audio data noise reduction and feature data repair processing, the data dimension and computing power requirements are reduced, the problem of low training efficiency of big data models in the existing technology is solved, and the wake-up capability of sweepers in noisy environments is improved.
Patent Information
- Application Number
- CN202411170137.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-11-08
AI Technical Summary
In the prior art, the training efficiency of big data models is low, especially in noisy environments, and the problem of sweepers being difficult to wake up.
By acquiring audio data, first perform noise reduction processing, extract feature data, and then perform feature data repair and training to wake up the model, reducing data dimensions and computing power requirements, and improving model training efficiency.
It improves model training efficiency and wake-up probability, reduces the demand for data, and enhances the wake-up capability of cleaning equipment in noisy environments.
Smart Images

Figure CN118918882A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence algorithm training technology, and in particular to a model training method, device, electronic device and non-volatile readable storage medium. Background Art
[0002] When a sweeping machine is working, the noise is very loud and the signal-to-noise ratio is very low. At this time, it is difficult for users to wake up the sweeping machine. In order to solve the above problems, an algorithm model is usually deployed in the sweeping machine to process voice data through the model to achieve wake-up. In related technologies, model training usually adopts an end-to-end mode. However, the end-to-end solution has a strong dependence on data and a large data dimension, which affects the training efficiency of the model. Summary of the invention
[0003] The present invention aims to at least solve or improve the technical problem of low training efficiency of big data models in the prior art.
[0004] To this end, the first aspect of the present invention proposes a model training method.
[0005] A second aspect of the present invention provides a model training device.
[0006] A third aspect of the present invention provides an electronic device.
[0007] A fourth aspect of the present invention provides a non-volatile readable storage medium.
[0008] A fifth aspect of the present invention provides a computer program product.
[0009] In view of this, according to a first aspect of the present invention, the present invention proposes a model training method, including: obtaining first audio data; inputting the first audio data into an audio data denoising model, and obtaining second audio data output by the audio data denoising model; extracting first feature data from the second audio data; inputting the first feature data into a feature data repair model, and obtaining second feature data output by the feature data repair model; and inputting the second feature data into a wake-up model to train the wake-up model.
[0010] The model training method proposed in the present invention includes: first determining first audio data, inputting the first audio data into an audio data denoising model to denoise the first audio data, and the audio data denoising model outputs second audio data.
[0011] The second audio data output by the noise reduction model is obtained, and features are extracted on the second audio data to extract first feature data from the second audio data, thereby reducing the size of the data.
[0012] The first feature data is input into the feature data repair model. The feature data repair model is used to repair the distortion of the first feature data, remove residual noise, and output the second feature data. Since the noise reduction of the noise reduction model will cause the second audio data to be distorted, the distortion of the data can be repaired to a certain extent through the repair model.
[0013] The second feature data output by the feature data repair model is obtained, and the second feature data is input into the wake-up model to train the wake-up model.
[0014] As mentioned above, the present invention will perform a series of processing on the first audio data, and finally train the wake-up model through the second feature data. The dimension of the second feature data is small, which can compress the size of the model and reduce the computing power requirements, thereby improving the training efficiency of the model.
[0015] In addition, the model training method in the above technical solution provided by the present invention may also have the following additional technical features:
[0016] In some embodiments, optionally, the first feature data is input into a feature data repair model to obtain second feature data output by the feature data repair model, including: inputting the first feature data into the feature data repair model to train the feature data repair model; after the feature data repair model training is completed, obtaining the second feature data output by the feature data repair model.
[0017] In this embodiment, the first feature data is input into the feature data repair model, and the second feature data output by the feature data repair model is obtained, including: inputting the first feature data into the feature data repair model to realize the training of the feature data repair model, after the feature data repair model training is completed, obtaining the second feature data output by the feature data repair model to train the awakening model through the second feature data, the feature data repair model and the awakening model in the present invention are jointly trained to reduce the demand for data and improve the training efficiency of the model.
[0018] In some embodiments, optionally, the first feature data is input into a feature data repair model to train the feature data repair model, including: extracting third feature data from original audio data, the original audio data corresponding to the first audio data; inputting the first feature data into the feature data repair model; obtaining second feature data output by the feature data repair model; performing loss calculation using the second feature data and the third feature data; and adjusting parameters of the feature data repair model based on the calculation results.
[0019] In this embodiment, the first feature data is input into the feature data repair model to train the feature data repair model, including: extracting the third feature data in the original audio data, the original audio data corresponds to the first audio data, and the original audio data can be considered as the standard data before the noise is mixed.
[0020] The first feature data is input into the feature data repair model, and the feature data repair model outputs the second feature data, the second feature data is obtained, loss calculation is performed through the second feature data and the third feature data, and the parameters of the feature data repair model are adjusted according to the calculation results, so as to realize the training of the feature data repair model. Moreover, since the third feature data is obtained through the original audio data, the efficiency of the feature data repair model training can be improved through the loss calculation of the third feature data and the second feature data.
[0021] In some embodiments, optionally, the audio data noise reduction model, the feature data repair model and the wake-up model are deployed on the cleaning device; the first audio data is the audio data output by the sound receiving component of the cleaning device; and the original audio data is the audio data received by the sound receiving component.
[0022] In this embodiment, the audio data denoising model, the feature data repair model and the wake-up model are deployed on the cleaning device, the original audio data is the audio data received by the sound receiving component, and the first audio data is the audio data output by the sound receiving component of the cleaning device, that is, the original audio data is played by the user or the player, the sound receiving component of the cleaning device receives the original audio data, and then outputs the first audio data. After the first audio data is processed by the audio data denoising model and the feature data repair model, it enters the wake-up model, thereby improving the wake-up probability of the cleaning device.
[0023] The first audio data is audio data output by the sound receiving component of the cleaning device, that is, the cleaning device is first controlled to work and play the original audio data, or the original audio data is formed by manual reading. After the sound receiving component receives the original audio data, it outputs the first audio data. The first audio data conforms to the situation in the actual application process. Since the cleaning device will generate a lot of noise when working, there will be a lot of noise information in the first audio data, which needs to be processed by the audio data denoising model and the feature data repair model.
[0024] In some embodiments, optionally, the first feature data, the second feature data and the third feature data are filter bank feature data and / or Mel-frequency cepstral coefficient feature data.
[0025] In this embodiment, the first feature data, the second feature data and the third feature data are filter group feature data and / or Mel-frequency cepstral coefficient feature data. The filter group feature data or Mel-frequency cepstral coefficient feature data have a smaller dimension, which can reduce the parameter amount and computing power requirements of the repair model.
[0026] In some embodiments, optionally, inputting the first audio data into an audio data denoising model and obtaining the second audio data output by the audio data denoising model includes: inputting the first audio data into the audio data denoising model to train the audio data denoising model; and obtaining the second audio data output by the audio data denoising model.
[0027] In this embodiment, first audio data is input into an audio data denoising model, and second audio data output by the audio data denoising model is obtained, including: inputting the first audio data into the audio data denoising model to train the audio data denoising model, and the audio data denoising model processes the first audio data and outputs the second audio data. The audio data denoising model, feature data repair model and wake-up model in the present invention are jointly trained to reduce the demand for data and improve the training efficiency of the model.
[0028] In some embodiments, optionally, after the feature data repair model training is completed, second feature data output by the feature data repair model is obtained, including: after the feature data repair model training is completed, freezing the parameters of the audio data denoising model and the feature data repair model; obtaining the second feature data output by the feature data repair model.
[0029] In this embodiment, after the feature data repair model training is completed, the second feature data output by the feature data repair model is obtained, including: after the feature data repair model training is completed, the parameters of the audio data denoising model and the feature data repair model are frozen, thereby saving the audio data denoising model and the feature data repair model.
[0030] Then, the first audio data is input into the audio data denoising model to obtain the second audio data, the first feature data of the second audio data is extracted, the first feature data is input into the feature data repair model to obtain the second feature data, and the second feature data is obtained for training the wake-up model.
[0031] In some embodiments, optionally, the first audio data is multi-channel data, and the second audio data is single-channel data.
[0032] In this embodiment, the first audio data is multi-channel data, and the second audio data is single-channel data, that is, the multi-channel first audio data is input to the audio data denoising model, and the audio data denoising model outputs the single-channel second audio data. The single-channel second audio data can reduce the training difficulty and improve the training efficiency when the audio feature repair model and the wake-up model are subsequently trained.
[0033] According to the second aspect of the present invention, the present invention proposes a model training device, including: a first acquisition module, used to acquire first audio data; a second acquisition module, used to input the first audio data into an audio data denoising model, and acquire second audio data output by the audio data denoising model; an extraction module, used to extract first feature data from the second audio data; a third acquisition module, used to input the first feature data into a feature data repair model, and acquire second feature data output by the feature data repair model; a training model, used to input the second feature data into a wake-up model to train the wake-up model.
[0034] The model training device proposed in the present invention includes a first acquisition module, a second acquisition module, an extraction module, a third acquisition module, and a training model.
[0035] The first acquisition module, the second acquisition module, the extraction module, the third acquisition module, and the training model are used to first determine the first audio data, input the first audio data into the audio data denoising model to denoise the first audio data, and the audio data denoising model outputs the second audio data.
[0036] The second audio data output by the noise reduction model is obtained, and features are extracted on the second audio data to extract first feature data from the second audio data, thereby reducing the size of the data.
[0037] The first feature data is input into the feature data repair model. The feature data repair model is used to repair the distortion of the first feature data, remove residual noise, and output the second feature data. Since the noise reduction of the noise reduction model will cause the second audio data to be distorted, the distortion of the data can be repaired to a certain extent through the repair model.
[0038] The second feature data output by the feature data repair model is obtained, and the second feature data is input into the wake-up model to train the wake-up model.
[0039] As mentioned above, the present invention will perform a series of processing on the first audio data, and finally train the wake-up model through the second feature data. The dimension of the second feature data is small, which can compress the size of the model and reduce the computing power requirements, thereby improving the training efficiency of the model.
[0040] According to the third aspect of the present invention, the present invention proposes an electronic device, including a processor and a memory, the memory storing programs or instructions that can be run on the processor, and the programs or instructions, when executed by the processor, implement the steps of the model training method provided in the embodiment of the first aspect.
[0041] The electronic device proposed in the present invention includes a memory that stores programs or instructions that implement the steps of the model training method provided in the first aspect embodiment when executed by a processor. Therefore, it has all the beneficial effects of the model training method provided in the first aspect embodiment, which are no longer stated one by one here.
[0042] According to a fourth aspect of the present invention, the present invention proposes a non-volatile readable storage medium, which stores a program or instruction. When the program or instruction is executed, the steps of the model training method provided in the first aspect embodiment are performed.
[0043] The non-volatile readable storage medium proposed in the present invention stores a program or instruction that implements the steps of the model training method provided in the first aspect embodiment when executed. Therefore, it has all the beneficial effects of the model training method provided in the first aspect embodiment, which will not be stated one by one here.
[0044] According to a fifth aspect of the present invention, the present invention proposes a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the model training method provided in the embodiment of the first aspect.
[0045] The computer program product proposed in the present invention includes computer instructions that implement the steps of the model training method provided in the first aspect embodiment when executed by a processor. Therefore, it has all the beneficial effects of the model training method provided in the first aspect embodiment, which are no longer listed one by one here.
[0046] Additional aspects and advantages of the present invention will become apparent from the following description or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0048] Figure 1 A flow chart of a model training method provided by an embodiment of the present invention is shown;
[0049] Figure 2 A framework diagram of a denoising model, a repair model and a wake-up model training method in a model training method provided by an embodiment of the present invention is shown;
[0050] Figure 3A framework diagram of a repair model training method in a model training method provided by an embodiment of the present invention is shown;
[0051] Figure 4 A structural block diagram of a model training device provided by an embodiment of the present invention is shown;
[0052] Figure 5 A structural block diagram of an electronic device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION
[0053] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0055] Refer to the following Figures 1 to 5 To describe a model training method, device, electronic device and non-volatile readable storage medium provided according to some embodiments of the present invention.
[0056] According to a first aspect of the present invention, the present invention provides a model training method. Figure 1 FIG. 1 is a flow chart showing a model training method provided by an embodiment of the present invention. Figure 1 As shown, the process of the model training method provided by one embodiment of the present invention is as follows:
[0057] Step 102: Acquire first audio data.
[0058] Specifically, first audio data is acquired first, and the first audio data may contain noise information.
[0059] Step 104: input the first audio data into an audio data denoising model, and obtain second audio data output by the audio data denoising model.
[0060] Specifically, the first audio data is input into the audio data denoising model, the audio data denoising model processes the first audio data to achieve denoising of the first audio data, and then outputs the denoised second audio data.
[0061] Step 106: Extract the first feature data from the second audio data.
[0062] Specifically, the first feature data in the second audio data is extracted, thereby reducing the data dimension of subsequent processing and reducing computing pressure.
[0063] Step 108: input the first feature data into the feature data repair model to obtain the second feature data output by the feature data repair model.
[0064] Specifically, the first feature data is input into the feature data repair model, the feature data repair model processes the first feature data to repair the first feature data, and then outputs the repaired second feature data.
[0065] Step 110: Input the second feature data into the wake-up model to train the wake-up model.
[0066] Specifically, the wake-up model is trained by the second feature data. Since the second feature data has been repaired by the feature data repair model, it has a better training effect, thereby improving the training efficiency of the wake-up model.
[0067] The model training method proposed in the present invention includes: first determining first audio data, inputting the first audio data into an audio data denoising model to denoise the first audio data, and the audio data denoising model outputs second audio data.
[0068] The second audio data output by the noise reduction model is obtained, and features are extracted on the second audio data to extract first feature data from the second audio data, thereby reducing the size of the data.
[0069] The first feature data is input into the feature data repair model. The feature data repair model is used to repair the distortion of the first feature data, remove residual noise, and output the second feature data. Since the noise reduction of the noise reduction model will cause the second audio data to be distorted, the distortion of the data can be repaired to a certain extent through the repair model.
[0070] The second feature data output by the feature data repair model is obtained, and the second feature data is input into the wake-up model to train the wake-up model.
[0071] As mentioned above, the present invention will perform a series of processing on the first audio data, and finally train the wake-up model through the second feature data. The dimension of the second feature data is small, which can compress the size of the model and reduce the computing power requirements, thereby improving the training efficiency of the model.
[0072] like Figure 2 As shown, the present invention performs a series of processing on the first audio data, and finally trains the wake-up model through the second feature data. The dimension of the second feature data is small, which can compress the size of the model and reduce the computing power requirement, thereby improving the training efficiency of the model.
[0073] In some embodiments, optionally, the first feature data is input into a feature data repair model to obtain second feature data output by the feature data repair model, including: inputting the first feature data into the feature data repair model to train the feature data repair model; after the feature data repair model training is completed, obtaining the second feature data output by the feature data repair model.
[0074] In this embodiment, the first feature data is input into the feature data repair model, and the second feature data output by the feature data repair model is obtained, including: inputting the first feature data into the feature data repair model to realize the training of the feature data repair model, after the feature data repair model training is completed, obtaining the second feature data output by the feature data repair model to train the awakening model through the second feature data, the feature data repair model and the awakening model in the present invention are jointly trained to reduce the demand for data and improve the training efficiency of the model.
[0075] In some embodiments, optionally, the first feature data is input into a feature data repair model to train the feature data repair model, including: extracting third feature data from original audio data, the original audio data corresponding to the first audio data; inputting the first feature data into the feature data repair model; obtaining second feature data output by the feature data repair model; performing loss calculation using the second feature data and the third feature data; and adjusting parameters of the feature data repair model based on the calculation results.
[0076] In this embodiment, the first feature data is input into the feature data repair model to train the feature data repair model, including: extracting the third feature data in the original audio data, the original audio data corresponds to the first audio data, and the original audio data can be considered as the standard data before the noise is mixed.
[0077] The first feature data is input into the feature data repair model, and the feature data repair model outputs the second feature data, the second feature data is obtained, loss calculation is performed through the second feature data and the third feature data, and the parameters of the feature data repair model are adjusted according to the calculation results, so as to realize the training of the feature data repair model. Moreover, since the third feature data is obtained through the original audio data, the efficiency of the feature data repair model training can be improved through the loss calculation of the third feature data and the second feature data.
[0078] Specifically, the step of extracting the third feature data from the original audio data may be performed before or after the step of inputting the first feature data into the feature data repair model, or both may be performed in parallel.
[0079] In some embodiments, optionally, the audio data noise reduction model, the feature data repair model and the wake-up model are deployed on the cleaning device; the first audio data is the audio data output by the sound receiving component of the cleaning device; and the original audio data is the audio data received by the sound receiving component.
[0080] In this embodiment, the audio data denoising model, the feature data repair model and the wake-up model are deployed on the cleaning device, the original audio data is the audio data received by the sound receiving component, and the first audio data is the audio data output by the sound receiving component of the cleaning device, that is, the original audio data is played by the user or the player, the sound receiving component of the cleaning device receives the original audio data, and then outputs the first audio data. After the first audio data is processed by the audio data denoising model and the feature data repair model, it enters the wake-up model, thereby improving the wake-up probability of the cleaning device.
[0081] The first audio data is audio data output by the sound receiving component of the cleaning device, that is, the cleaning device is first controlled to work and play the original audio data, or the original audio data is formed by manual reading. After the sound receiving component receives the original audio data, it outputs the first audio data. The first audio data conforms to the situation in the actual application process. Since the cleaning device will generate a lot of noise when working, there will be a lot of noise information in the first audio data, which needs to be processed by the audio data denoising model and the feature data repair model.
[0082] Specifically, since the cleaning equipment generates noise when it is working, the first audio data is also data with noise information. Since the cleaning equipment generates a lot of noise when it is working, most of the noise can be eliminated by the audio data denoising model. The audio data denoising model is trained by the first audio data, and the audio data denoising model can reduce the noise in the first audio data, thereby obtaining the second audio data.
[0083] The second audio data output by the audio data denoising model is obtained, and features are extracted on the second audio data to extract first feature data from the second audio data, thereby reducing the size of the data.
[0084] The feature data repair model is trained through the first feature data. The second audio data is the noise-reduced data, which is distorted. Therefore, the first feature data is also distorted. The feature data repair model can repair and improve the distortion of the first feature data and remove residual noise, thereby outputting the second feature data.
[0085] In some embodiments, optionally, the first feature data, the second feature data and the third feature data are filter bank (Filter Bank, Fbank) feature data and / or Mel Frequency Cepstral Coefficient (Mel Frequency Cepstral Coefficient, MFCC) feature data.
[0086] In this embodiment, the first feature data, the second feature data and the third feature data are filter group feature data and / or Mel-frequency cepstral coefficient feature data. The filter group feature data or Mel-frequency cepstral coefficient feature data have a smaller dimension, which can reduce the parameter amount and computing power requirements of the repair model.
[0087] Specifically, the first Fbank feature data or the first MFCC feature data of the second audio data after noise reduction is extracted, and the first Fbank feature data or the first MFCC feature data is input into the repair model. The repair model repairs the distortion of the first Fbank feature data or the first MFCC feature data, removes residual noise, and outputs the second Fbank feature data or the second MFCC feature data. The dimension of the Fbank feature data or the MFCC feature data is small, which can effectively reduce the parameter amount and computing power requirement of the repair network.
[0088] FBank feature is a commonly used feature extraction method in speech signal processing. It is designed based on the auditory characteristics of the human ear. In the FBank feature extraction process, the speech signal is first divided into frames, and each frame is windowed. Then, each frame is subjected to Fast Fourier Transform (FFT) to obtain a spectrum diagram. The spectrum diagram is then filtered through a set of triangular filters to obtain the energy of each filter. The energy of each filter is then logarithmically transformed to obtain the FBank feature. The FBank is subjected to Discrete Cosine Transform (DCT) to obtain the MFCC feature. FBank features and MFCC features are widely used in tasks such as speech recognition and speaker recognition.
[0089] like Figure 3 As shown, the training method of the repair model is: first extract the first feature data of the second audio data, input the first feature data into the repair model, the repair model outputs the second feature data, and the loss is calculated for the second feature data and the third feature data extracted from the original audio data. The parameters of the repair model are updated by back-propagation of the gradient of the loss. The loss can be the mean square error (MSE) loss.
[0090] In one embodiment, the restoration model includes 5 layers of downsampling convolutions, gated recurrent units (Gated Recurrent Units, GRU) and 5 layers of transposed convolutions.
[0091] In one embodiment, the downsampling convolution includes a two-dimensional convolution (Conv2D), a batch normalization (BatchNorm, BN) function, and an exponential linear unit (Exponential Linear Unit, ELU) activation function.
[0092] In one embodiment, the transposed convolution includes a two-dimensional transposed convolution (Trans Conv2D), a batch normalization (BatchNorm, BN) function, and an exponential linear unit (Exponential Linear Unit, ELU) activation function.
[0093] The repair model reduces the feature dimensions of the input and output, compresses the model size and computing power requirements. The parameters of the repair model can be reduced to 77k, and the amount of calculation per second can reach 19M, which means that the effect of using less computing power to improve audio quality is achieved.
[0094] In some embodiments, optionally, inputting the first audio data into an audio data denoising model and obtaining the second audio data output by the audio data denoising model includes: inputting the first audio data into the audio data denoising model to train the audio data denoising model; and obtaining the second audio data output by the audio data denoising model.
[0095] In this embodiment, the first audio data is input into the audio data denoising model, and the second audio data output by the audio data denoising model is obtained, including: the first audio data is input into the audio data denoising model to train the audio data denoising model, and the audio data denoising model processes the first audio data and outputs the second audio data. The audio data denoising model, the feature data repair model and the wake-up model in the present invention are jointly trained to reduce the demand for data and improve the training efficiency of the model. In addition, the training and use of the model are the same data processing process, which greatly improves the wake-up rate of the wake-up model.
[0096] In addition, the audio data denoising model, feature data repair model and wake-up model in the present invention are jointly trained to reduce the demand for data and further improve the training efficiency of the model. The wake-up rate of the cleaning equipment in a high-noise environment can be improved through joint training.
[0097] In some embodiments, optionally, after the feature data repair model training is completed, second feature data output by the feature data repair model is obtained, including: after the feature data repair model training is completed, freezing the parameters of the audio data denoising model and the feature data repair model; obtaining the second feature data output by the feature data repair model.
[0098] In this embodiment, after the feature data repair model training is completed, the second feature data output by the feature data repair model is obtained, including: after the feature data repair model training is completed, the parameters of the audio data denoising model and the feature data repair model are frozen, thereby saving the audio data denoising model and the feature data repair model.
[0099] Then, the first audio data is input into the audio data denoising model to obtain the second audio data, the first feature data of the second audio data is extracted, the first feature data is input into the feature data repair model to obtain the second feature data, and the second feature data is obtained for training the wake-up model.
[0100] That is, after the training of the audio data denoising model and the feature data repair model is completed, the parameters of the audio data denoising model and the feature data repair model are frozen, and then the second feature data repaired by the feature data repair model is used to jointly train the wake-up model. At this time, the input data obtained by the wake-up model has a high signal-to-noise ratio and low speech distortion, which effectively improves the wake-up rate of the cleaning equipment when it is working while maintaining a low false wake-up.
[0101] The present invention uses a pre-trained audio data denoising model to denoise the first audio data, and then uses a feature data repair model to repair the quality of the first feature data of the denoised second audio data, while eliminating the residual noise that the denoising model has not eliminated, and finally freezes the audio data denoising model and the feature data repair model, and trains the wake-up model.
[0102] In some embodiments, optionally, the first audio data is multi-channel data, and the second audio data is single-channel data.
[0103] In this embodiment, the first audio data is multi-channel data, and the second audio data is single-channel data, that is, the multi-channel first audio data is input to the audio data denoising model, and the audio data denoising model outputs the single-channel second audio data. The single-channel second audio data can reduce the training difficulty and improve the training efficiency when the audio feature repair model and the wake-up model are subsequently trained.
[0104] Specifically, the first audio data is input into the audio data denoising model to train the audio data denoising model, including: inputting the multi-channel first audio data into the audio data denoising model to train the audio data denoising model; and obtaining the single-channel second audio data output by the audio data denoising model.
[0105] The present invention is based on a multi-channel audio data denoising model, outputs a single-channel second audio data, performs quality repair on the second audio data through a feature data repair model, and eliminates residual noise at the same time, and then combines the training of the wake-up model to achieve joint training of the audio data denoising model, the feature data repair model and the wake-up model, thereby effectively reducing the difficulty of training the wake-up model and improving the wake-up rate of the cleaning equipment when it is working. The present invention uses a pre-trained audio data denoising model to denoise the multi-channel first audio data, and then uses a feature data repair model to perform quality repair on the denoised second audio data, and eliminates residual noise that the audio data denoising model has not eliminated. Finally, the audio data denoising model and the feature data repair model are frozen, and the wake-up model is trained.
[0106] Specifically, after the audio data denoising model performs denoising on the multi-channel first audio data, it outputs a single-channel second audio data. However, the audio data denoising model can only eliminate most of the noise from the cleaning equipment. Some noise from the cleaning equipment will still remain in the second audio data, and the second audio data will have speech distortion.
[0107] The above methods may be implemented in various ways according to specific features and / or example applications. For example, these methods may be implemented by a combination of hardware, firmware, and / or software. For example, in a hardware implementation, the processor may be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, electronic devices, other equipment units for performing the above functions, and / or combinations thereof.
[0108] like Figure 4 As shown, according to the second aspect of the present invention, the present invention provides a model training device 400, including: a first acquisition module 402, used to acquire first audio data; a second acquisition module 404, used to input the first audio data into an audio data denoising model, and acquire second audio data output by the audio data denoising model; an extraction module 406, used to extract first feature data from the second audio data; a third acquisition module 408, used to input the first feature data into a feature data repair model, and acquire second feature data output by the feature data repair model; a training model 410, used to input the second feature data into a wake-up model to train the wake-up model.
[0109] The model training device provided by the present invention comprises a first acquisition module, a second acquisition module, an extraction module, a third acquisition module, and a training model.
[0110] The first acquisition module, the second acquisition module, the extraction module, the third acquisition module, and the training model are used to first determine the first audio data, input the first audio data into the audio data denoising model to denoise the first audio data, and the audio data denoising model outputs the second audio data.
[0111] The second audio data output by the noise reduction model is obtained, and features are extracted on the second audio data to extract first feature data from the second audio data, thereby reducing the size of the data.
[0112] The first feature data is input into the feature data repair model. The feature data repair model is used to repair the distortion of the first feature data, remove residual noise, and output the second feature data. Since the noise reduction of the noise reduction model will cause the second audio data to be distorted, the distortion of the data can be repaired to a certain extent through the repair model.
[0113] The second feature data output by the feature data repair model is obtained, and the second feature data is input into the wake-up model to train the wake-up model.
[0114] As mentioned above, the present invention will perform a series of processing on the first audio data, and finally train the wake-up model through the second feature data. The dimension of the second feature data is small, which can compress the size of the model and reduce the computing power requirements, thereby improving the training efficiency of the model.
[0115] In some embodiments, optionally, the third acquisition module includes: a first training submodule, used to input the first feature data into the feature data repair model to train the feature data repair model; and a first acquisition submodule, used to obtain the second feature data output by the feature data repair model after the feature data repair model training is completed.
[0116] In some embodiments, optionally, the first training submodule includes: an extraction unit, used to extract third feature data from original audio data, the original audio data corresponding to the first audio data; a first input unit, used to input the first feature data into a feature data repair model; a first acquisition unit, used to acquire second feature data output by the feature data repair model; a calculation unit, used to perform loss calculation using the second feature data and the third feature data; and an adjustment unit, used to adjust the parameters of the feature data repair model according to the calculation results.
[0117] In some embodiments, optionally, the audio data noise reduction model, the feature data repair model and the wake-up model are deployed on the cleaning device; the first audio data is the audio data output by the sound receiving component of the cleaning device; and the original audio data is the audio data received by the sound receiving component.
[0118] In some embodiments, optionally, the first feature data, the second feature data and the third feature data are filter bank feature data and / or Mel-frequency cepstral coefficient feature data.
[0119] In this embodiment, the first feature data, the second feature data and the third feature data are filter group feature data and / or Mel-frequency cepstral coefficient feature data. The filter group feature data or Mel-frequency cepstral coefficient feature data have a smaller dimension, which can reduce the parameter amount and computing power requirements of the repair model.
[0120] In some embodiments, optionally, the second acquisition module includes: a second input submodule, used to input the first audio data into the audio data denoising model to train the audio data denoising model; and a second acquisition submodule, used to acquire the second audio data output by the audio data denoising model.
[0121] In some embodiments, optionally, the first acquisition submodule includes: a freezing unit, used to freeze the parameters of the audio data denoising model and the feature data repair model after the feature data repair model training is completed; and a second acquisition unit, used to acquire the second feature data output by the feature data repair model.
[0122] In some embodiments, optionally, the first audio data is multi-channel data, and the second audio data is single-channel data.
[0123] like Figure 5 As shown, according to the third aspect of the present invention, the present invention proposes an electronic device 500, including a processor 502 and a memory 504, the memory 504 stores programs or instructions that can be run on the processor 502, and when the program or instructions are executed by the processor 502, the steps of the model training method provided in the first aspect embodiment are implemented.
[0124] The electronic device proposed in the present invention includes a memory that stores programs or instructions that implement the steps of the model training method provided in the first aspect embodiment when executed by a processor. Therefore, it has all the beneficial effects of the model training method provided in the first aspect embodiment, which are no longer stated one by one here.
[0125] According to a fourth aspect of the present invention, the present invention proposes a non-volatile readable storage medium, which stores a program or instruction. When the program or instruction is executed, the steps of the model training method provided in the first aspect embodiment are performed.
[0126] The non-volatile readable storage medium proposed in the present invention stores a program or instruction that implements the steps of the model training method provided in the first aspect embodiment when executed. Therefore, it has all the beneficial effects of the model training method provided in the first aspect embodiment, which will not be stated one by one here.
[0127] According to a fifth aspect of the present invention, the present invention proposes a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the model training method provided in the embodiment of the first aspect.
[0128] The computer program product proposed in the present invention includes computer instructions that implement the steps of the model training method provided in the first aspect embodiment when executed by a processor. Therefore, it has all the beneficial effects of the model training method provided in the first aspect embodiment, which are no longer listed one by one here.
[0129] A storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer storage medium may be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above devices, but is not limited thereto. A non-exhaustive list of more specific examples of computer storage media includes: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital universal disk (DVD), memory card, floppy disk, encoding mechanical device (such as a punch card or a groove with a raised structure with instructions recorded) and any suitable combination of the above devices. The computer storage medium used herein should not be understood as a transmission signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media, or electrical signals transmitted through wires.
[0130] In the present invention, the terms "first", "second", and "third" are used for descriptive purposes only and should not be understood as indicating or implying relative importance; the term "plurality" refers to two or more, unless otherwise clearly defined. The terms "installed", "connected", "connected", "fixed", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; "connected" can be a direct connection or an indirect connection through an intermediate medium. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0131] In the description of the present invention, it is necessary to understand that the directions or positional relationships indicated by the terms "up", "down", "left", "right", "front", "back", etc. are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the components or units referred to must have a specific direction, be constructed and operated in a specific orientation, and therefore, should not be understood as limiting the present invention.
[0132] In the description of this specification, the description of the terms "one embodiment", "some embodiments", "specific embodiments", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.
[0133] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A model training method, It is characterized in that include: Acquire first audio data; Inputting the first audio data into an audio data denoising model, and obtaining second audio data output by the audio data denoising model; extracting first feature data from the second audio data; Inputting the first feature data into a feature data repair model, and obtaining second feature data output by the feature data repair model; The second feature data is input into a wake-up model to train the wake-up model.
2. The model training method according to claim 1, It is characterized in that Inputting the first feature data into a feature data repair model, and obtaining second feature data output by the feature data repair model, comprises: Inputting the first feature data into the feature data repair model to train the feature data repair model; After the feature data repair model is trained, the second feature data output by the feature data repair model is obtained.
3. The model training method according to claim 2, It is characterized in that Inputting the first feature data into the feature data repair model to train the feature data repair model includes: Extracting third feature data from original audio data, where the original audio data corresponds to the first audio data; inputting the first feature data into the feature data repair model; Acquire the second feature data output by the feature data repair model; Calculate the loss by using the second characteristic data and the third characteristic data; According to the calculation results, the parameters of the feature data repair model are adjusted.
4. The model training method according to claim 3, It is characterized in that The audio data noise reduction model, the feature data repair model and the wake-up model are deployed on a cleaning device; The first audio data is audio data output by a sound receiving component of the cleaning device; The original audio data is the audio data received by the sound receiving component.
5. The model training method according to claim 3, It is characterized in that The first feature data, the second feature data and the third feature data are filter bank feature data and / or Mel-frequency cepstral coefficient feature data.
6. The model training method according to any one of claims 2 to 5, It is characterized in that Inputting the first audio data into the audio data denoising model, and obtaining second audio data output by the audio data denoising model, comprises: Inputting the first audio data into the audio data denoising model to train the audio data denoising model; Acquire second audio data output by the audio data noise reduction model.
7. The model training method according to claim 6, It is characterized in that After the feature data repair model is trained, obtaining the second feature data output by the feature data repair model includes: After the feature data repair model training is completed, freezing the parameters of the audio data denoising model and the feature data repair model; The second feature data output by the feature data repair model is obtained.
8. The model training method according to any one of claims 1 to 5, It is characterized in that The first audio data is multi-channel data, and the second audio data is single-channel data.
9. A model training device, It is characterized in that include: A first acquisition module, used to acquire first audio data; A second acquisition module, used for inputting the first audio data into an audio data denoising model, and acquiring second audio data output by the audio data denoising model; An extraction module, used for extracting first feature data from the second audio data; A third acquisition module, used for inputting the first feature data into a feature data repair model, and acquiring second feature data output by the feature data repair model; A training model is used to input the second feature data into a wake-up model to train the wake-up model.
10. An electronic device, It is characterized in that It includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the model training method as described in any one of claims 1 to 8 are implemented.
11. A non-volatile readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the model training method according to any one of claims 1 to 8 are implemented.
12. A computer program product comprising computer instructions, It is characterized in that When the computer instructions are executed by a processor, the steps of the model training method as described in any one of claims 1 to 8 are implemented.
Citation Information
Patent Citations
Voice wake-up method and device, processor, sound box and television
CN111354341A
Microphone array-oriented channel attention weighted speech enhancement method
CN112151059A
Voice wake-up method and device, storage medium and electronic equipment
CN113327611A
Voice noise reduction method and system, electronic equipment and storage medium
CN114267372A
Construction method and device of noise suppression module, electronic equipment and storage medium
CN115662454A
