VPU-based noise reduction processing method and device, intelligent wearable device and medium

Abstract

The application discloses a VPU-based noise reduction processing method and device, an intelligent wearable device and a medium, and relates to the technical field of noise reduction processing. The method comprises the following steps: converting and merging a preset microphone voice signal and a microphone noise signal according to a preset conversion rule between the microphone signal and a target VPU signal, so as to obtain a VPU mixed signal; converting the VPU mixed signal into a first frequency domain signal, and performing model training on a preset neural network model according to the first frequency domain signal; and performing VPU signal noise reduction processing according to the neural network model after the model training is completed. The application realizes noise reduction processing on the VPU signal in the intelligent wearable device.

Description

Technical Field

[0001] This application relates to the field of noise reduction technology, and in particular to noise reduction methods, apparatus, smart wearable devices and media based on VPU. Background Technology

[0002] In smart wearable devices, call functionality is crucial, primarily achieved through microphones and speakers. However, the audio signal captured by the microphone is often unclear due to noise interference. Therefore, an increasing number of smart wearable devices are adopting VPU (bone conduction) for call assistance. However, even in noisy environments, the VPU signal is still susceptible to noise interference. Therefore, achieving noise reduction for the VPU signal in smart wearable devices has become a pressing issue.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main objective of this application is to provide a VPU-based noise reduction processing method, apparatus, smart wearable device, and medium, wherein the medium is a computer-readable storage medium, aiming to solve the technical problem of how to perform noise reduction processing on VPU signals in smart wearable devices.

[0005] To achieve the above objectives, this application proposes a VPU-based noise reduction method, which includes the following steps:

[0006] Based on the preset conversion rules between the microphone signal and the target VPU signal, the preset microphone voice signal and microphone noise signal are converted and merged to obtain the VPU mixed signal;

[0007] The VPU mixed signal is converted into a first frequency domain signal, and a preset neural network model is trained based on the first frequency domain signal.

[0008] VPU signal denoising is performed based on the neural network model after training.

[0009] Optionally, the step of converting and merging a preset microphone voice signal and a microphone noise signal to obtain a VPU mixed signal according to a preset conversion rule between the microphone signal and the target VPU signal includes:

[0010] When the conversion rule includes a first transfer function corresponding to the microphone voice signal, the microphone voice signal is converted into a VPU voice signal according to the first transfer function;

[0011] When the conversion rule includes the second transfer function corresponding to the microphone noise signal, the microphone noise signal is converted into a VPU noise signal according to the second transfer function;

[0012] The VPU speech signal and the VPU noise signal are mixed according to a preset signal-to-noise ratio to obtain a VPU mixed signal.

[0013] Optionally, the step of training a preset neural network model based on the first frequency domain signal includes:

[0014] If the training task of the neural network model is determined to be a first target task for instructing VPU signal denoising processing, then the neural network model is trained according to the first target task and the first frequency domain signal until the preset training iteration conditions are met, and the neural network model after training is obtained.

[0015] Optionally, the step of training the neural network model for the first objective based on the first objective task and the first frequency domain signal includes:

[0016] Based on the neural network model and the first target task, the first frequency domain signal is denoised to generate a second frequency domain signal;

[0017] Perform a Fourier transform on the second frequency domain signal to obtain a clean first VPU speech signal;

[0018] The VPU speech signal corresponding to the first VPU mixed signal and the first VPU speech signal are input into a preset noise reduction processing loss function for calculation to obtain the first prediction loss;

[0019] The neural network model is updated based on the first prediction loss to train the neural network model for the first objective.

[0020] Optionally, the step of training a preset neural network model based on the first frequency domain signal further includes:

[0021] If the training task of the neural network model is determined to be a first target task for instructing VPU signal noise reduction processing and a second target task for instructing VPU signal speech activity detection processing, then the first target of the neural network model is trained according to the first target task and the first frequency domain signal, and the second target of the neural network model is trained according to the second target task and the first frequency domain signal, until the preset training iteration conditions are met, and the neural network model after model training is obtained.

[0022] Optionally, the step of training the neural network model for the second objective based on the second objective task and the first frequency domain signal includes:

[0023] Based on the neural network model and the second target task, speech activity detection is performed on the first frequency domain signal to obtain speech activity detection results;

[0024] The speech activity detection result and the speech activity label corresponding to the first VPU mixed signal are input into a preset speech activity detection loss function for calculation to obtain the second prediction loss;

[0025] The neural network model is updated based on the second prediction loss to train the neural network model for the second objective.

[0026] Optionally, the step of performing VPU signal denoising processing based on the neural network model after model training includes:

[0027] If an actual VPU signal is acquired, the actual VPU signal is converted into a third frequency domain signal, and the third frequency domain signal is input into the neural network model after model training is completed, so that the third frequency domain signal is denoised according to the neural network model after training to generate a fourth frequency domain signal, and the fourth frequency domain signal is converted into a clean VPU speech signal corresponding to the actual VPU signal.

[0028] Furthermore, to achieve the above objectives, this application also proposes a noise reduction processing device based on a VPU, comprising:

[0029] The conversion module is used to convert and merge the preset microphone voice signal and microphone noise signal according to the preset conversion rules between the microphone signal and the target VPU signal to obtain the VPU mixed signal.

[0030] The model training module is used to convert the VPU mixed signal into a first frequency domain signal and train a preset neural network model based on the first frequency domain signal.

[0031] The noise reduction module is used to perform VPU signal noise reduction processing based on the neural network model after model training.

[0032] In addition, to achieve the above objectives, this application also proposes a smart wearable device, which includes: a VPU, a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the VPU-based noise reduction processing method described above.

[0033] In addition, to achieve the above objectives, this application also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the VPU-based noise reduction processing method described above.

[0034] In this application, a VPU hybrid signal is obtained by converting and merging microphone speech signals and microphone noise signals according to the conversion rules between microphone signals and target VPU signals. This allows for the acquisition of the required VPU hybrid signal using easily acquired microphone signals (i.e., microphone speech signals and microphone noise signals), avoiding the inability to obtain sufficient training samples due to the difficulty in directly acquiring VPU signals. The VPU hybrid signal is then converted into a first frequency domain signal, and a neural network model is trained. Finally, VPU signal denoising is performed based on the trained neural network model. This enables denoising of the VPU signal in smart wearable devices based on the neural network model, preventing interference from noise signals and thus avoiding unclear signals. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the first embodiment of the VPU-based noise reduction processing method of this application.

[0038] Figure 2 This is a schematic diagram of the neural network model architecture in the first embodiment of the VPU-based noise reduction processing method of this application;

[0039] Figure 3 This is a schematic diagram of the VPU noise reduction process provided in the second embodiment of the VPU-based noise reduction method of this application;

[0040] Figure 4 This is a schematic diagram of the module architecture of the VPU-based noise reduction processing device of this application;

[0041] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the noise reduction processing method based on VPU in the embodiments of this application.

[0042] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0043] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0044] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0045] It should be noted that the executing entity of this embodiment can be a wearable device or smart wearable device with VPU functionality, such as bone conduction headphones, head-mounted displays, etc. Alternatively, it can be a smart wearable device capable of achieving the aforementioned functions. The following uses a smart wearable device as an example to illustrate this embodiment and the subsequent embodiments. A neural network model to be trained can be deployed in the smart wearable device. This neural network model can be used to perform noise reduction processing on the VPU signal. The neural network model can be a U-Net model with an Encode-Decode structure. Optionally, the VPU signal can be any type of VPU signal in any of the following embodiments, such as a mixed VPU signal, an actual VPU signal, etc. In this embodiment, VPU refers to bone conduction technology.

[0046] Based on this, embodiments of this application provide a noise reduction processing method based on VPU, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the noise reduction processing method based on VPU in this application.

[0047] In this embodiment, the noise reduction processing method based on VPU includes steps S10 to S30.

[0048] Step S10: Based on the preset conversion rules between the microphone signal and the target VPU signal, the preset microphone voice signal and microphone noise signal are converted and merged to obtain the VPU mixed signal.

[0049] Optionally, the conversion rule can be conversion logic used to convert the microphone signal's signal format into a VPU signal of the target format. The target VPU signal can be a VPU signal of the target format.

[0050] Optionally, since different manufacturers use different formats for VPU signals, while microphone signals are in a conventional, known format, different conversion rules can be set for smart wearable devices with VPU functionality from different manufacturers to convert microphone signals into VPU signals of different formats.

[0051] Optionally, the VPU mixed signal may include a VPU speech signal and a VPU noise signal. The VPU speech signal may be speech information generated by the user speaking as collected by the VPU sensor in the smart wearable device. The VPU noise signal may be noise signal collected by the VPU sensor in the smart wearable device.

[0052] Optionally, since VPU signals are relatively difficult to acquire, and VPU signals differ significantly from microphone signals, with different companies using different VPU signal formats, a mapping can be established in quiet environments. This can be achieved by comparing the VPU signal acquired by the VPU sensor with the microphone signal acquired by the microphone when the user speaks while wearing a smart wearable device (such as headphones). This mapping allows for the determination of a conversion rule between the acquired microphone signal and the acquired VPU signal. Optionally, the conversion rule can include a transfer function. Optionally, the transfer function can be determined using spectral analysis. Optionally, a Fast Fourier Transform can be performed on the acquired microphone signal and the acquired VPU signal to obtain their spectra, and the ratio between these two spectra can be determined to obtain the transfer function.

[0053] Optionally, when the acquired microphone signal is a microphone speech signal and the acquired VPU signal is a VPU speech signal, the transfer function is the first transfer function corresponding to the microphone speech signal. When the acquired microphone signal is a microphone noise signal and the acquired VPU signal is a VPU noise signal, the transfer function is the second transfer function corresponding to the microphone noise signal. Optionally, the first transfer function and the second transfer function can be the same or different.

[0054] Optionally, the preset microphone voice signal and microphone noise signal can be converted and merged according to the transfer function to obtain the VPU mixed signal.

[0055] Optionally, the microphone speech signal and microphone noise signal can be converted and processed separately according to the transfer function to convert them into VPU speech signal and VPU noise signal in the target format, and then merged to obtain VPU mixed signal.

[0056] In one feasible embodiment, step S10, which involves converting and merging a preset microphone voice signal and a microphone noise signal according to a preset conversion rule between the microphone signal and the target VPU signal to obtain a VPU mixed signal, includes steps a10-a30.

[0057] Step a10: When the conversion rule includes the first transfer function corresponding to the microphone voice signal, the microphone voice signal is converted into a VPU voice signal according to the first transfer function;

[0058] Optionally, the microphone voice signal can be the voice signal generated when the user speaks, collected by the microphone in the smart wearable device. Optionally, there can be multiple microphone voice signals.

[0059] Optionally, when it is necessary to convert the microphone speech signal, a first transfer function corresponding to the microphone speech signal is obtained. The microphone speech signal is then converted into a VPU speech signal based on the first transfer function. For example, the microphone speech signal is input into the first transfer function for calculation, and the VPU speech signal is output.

[0060] Optionally, in one embodiment, a model with conversion rules can be set up. The microphone voice signal is input into the model, and a VPU voice signal is output. Alternatively, the VPU voice signal corresponding to the microphone voice signal can be determined according to a pre-set mapping relationship. Etc.

[0061] Step a20: When the conversion rule includes the second transfer function corresponding to the microphone noise signal, the microphone noise signal is converted into a VPU noise signal according to the second transfer function;

[0062] Optionally, the microphone noise signal can be an ambient noise signal collected by the microphone in the smart wearable device. Optionally, there can be multiple microphone noise signals.

[0063] Optionally, when it is necessary to convert the microphone noise signal, a second transfer function corresponding to the microphone noise signal is obtained. The microphone noise signal is then converted into a VPU noise signal based on the second transfer function. For example, the microphone noise signal is input into the second transfer function for calculation, and the VPU noise signal is output.

[0064] Optionally, in one embodiment, a model with conversion rules can be set up, the microphone noise signal is input into the model, and a VPU noise signal is output. Alternatively, the VPU noise signal corresponding to the microphone noise signal can be determined according to a pre-set mapping relationship. Etc.

[0065] Step a30: Mix the VPU speech signal and the VPU noise signal according to the preset signal-to-noise ratio to obtain the VPU mixed signal.

[0066] Optionally, the signal-to-noise ratio (SNR) can be pre-set according to different scenario requirements. Optionally, due to the strong anti-interference capability of the VPU, the SNR can be selected within the range of (-5, 20) dB.

[0067] Optionally, the smart wearable device can mix and merge the individual VPU speech signals and VPU noise signals according to the signal-to-noise ratio to obtain the synthesized VPU speech signal, and use it as the VPU mixed signal.

[0068] In this embodiment, the microphone voice signal is converted and processed according to the first transfer function corresponding to the microphone voice signal to obtain the VPU voice signal, and the microphone noise signal is converted and processed according to the second transfer function corresponding to the microphone noise signal to obtain the VPU noise signal. Then, the VPU voice signal and the VPU noise signal are mixed and processed according to a preset signal-to-noise ratio to obtain the VPU mixed signal, thereby ensuring the effectiveness of the obtained VPU mixed signal.

[0069] Step S20: Convert the VPU mixed signal into a first frequency domain signal, and train the preset neural network model based on the first frequency domain signal;

[0070] It should be noted that the neural network model is used to denoise the first frequency domain signal to generate the second frequency domain signal, and then convert the second frequency domain signal into a clean VPU speech signal.

[0071] Optionally, a neural network model can be pre-defined. The input and output of this model can be single-channel input and single-channel output, or multi-channel input and multi-channel output. There are no restrictions here. For example, ... Figure 2 As shown, the model architecture of a neural network model can include an encoder, a GPU (Gated Recurrent Unit), and a decoder. The encoder has 5 conv layers, and the decoder has 5 deconv layers.

[0072] Optionally, the input to the preset neural network can be a Fourier transform of the VPU mixed signal (with a frame length of 256 sampling points) to obtain a first frequency domain signal. The dimension of the first frequency domain signal is 2x129 (the imaginary dimension is counted as 2). The output can be a Fourier transform of the clean VPU speech signal corresponding to the VPU mixed signal (with a frame length of 256 points), resulting in a second frequency domain signal with a dimension of 2×129 (the imaginary dimension is counted as 2).

[0073] Optionally, the VPU mixed signal can be subjected to frequency domain signal conversion processing to obtain a first frequency domain signal corresponding to the VPU mixed signal. Optionally, during the frequency domain signal conversion processing, the VPU mixed signal can be subjected to Fourier transform processing (with a frame length of 256 sampling points) to obtain the first frequency domain signal. The dimension of the first frequency domain signal is 2x129 (the imaginary dimension is counted as 2).

[0074] Optionally, a neural network model can be used to train the first frequency domain signal. During model training, the neural network model performs noise reduction on the first frequency domain signal to generate a second frequency domain signal. Then, a Fourier transform is performed on the second frequency domain signal to obtain a clean VPU speech signal. That is, the clean VPU speech signal at this point is the VPU speech signal after noise reduction. The model parameters of the neural network model are then optimized and updated based on this clean VPU speech signal until certain conditions are met, completing the model training of the neural network model.

[0075] Step S30: Perform VPU signal denoising processing based on the neural network model after model training is completed.

[0076] Optionally, after the neural network model has been trained, subsequent VPU signal denoising processing can be performed based on the trained neural network model.

[0077] For example, the VPU sensor in a smart wearable device collects VPU signals (including speech signals generated when the user speaks and environmental noise generated by the environment in which the smart wearable device is located). Then, based on the neural network model after model training, the collected VPU signals are subjected to VPU signal noise reduction processing to obtain the corresponding clean VPU speech signal.

[0078] In this embodiment, the microphone speech signal and microphone noise signal are converted and merged according to the conversion rules between the microphone signal and the target VPU signal to obtain a VPU mixed signal. This allows the acquisition of the required VPU mixed signal using easily acquired microphone signals (i.e., microphone speech signal and microphone noise signal), avoiding the inability to obtain sufficient training samples due to the difficulty in directly acquiring VPU signals. The VPU mixed signal is then converted into a first frequency domain signal, and a neural network model is trained. Based on the trained neural network model, VPU signal noise reduction is performed. This allows for noise reduction of the VPU signal in the smart wearable device based on the neural network model, preventing interference from noise signals and thus avoiding unclear signals.

[0079] Based on the first embodiment of this application, a second embodiment of this application is proposed. In this second embodiment, content that is the same as or similar to the above embodiment can be referred to the above description and will not be repeated hereafter. Based on this, step S20, the step of training a preset neural network model based on the first frequency domain signal, includes step b10.

[0080] Step b10: If the training task of the neural network model is determined to be the first target task for instructing VPU signal denoising processing, then the first target of the neural network model is trained according to the first target task and the first frequency domain signal until the preset training iteration conditions are met, and the neural network model after the model training is completed is obtained.

[0081] In this embodiment, during the model training phase of the neural network model, the training task of the neural network model can be determined first. If the training task of the neural network model is determined to be a first target task for instructing VPU signal denoising processing, then the neural network model can be trained according to the first target task.

[0082] Optionally, when determining the training task, after the neural network model receives the input first frequency domain signal, it can receive the training instruction input by the user, or directly determine the training rule corresponding to the first frequency domain signal. According to the training rule or training instruction, the corresponding training task is selected from each training task. If the selected training task is a training task used to indicate VPU signal denoising processing, then the training task is determined as the first target task.

[0083] Optionally, in one scenario, the first target task can also be obtained by concatenating the character indicating VPU signal denoising processing with the first frequency domain signal in the neural network model. The neural network model is trained based on the generated first target task to train the neural network model to achieve the first target. Optionally, the first target can be that the neural network model can achieve VPU signal denoising processing.

[0084] Optionally, after the neural network model receives the input first frequency domain signal and determines that the training task corresponding to the first frequency domain signal is the first target task, the neural network model can perform the first target task on the first frequency domain signal, that is, perform noise reduction processing on the first frequency domain signal to obtain a prediction result, and iteratively train the neural network model based on the prediction result. This continues until the preset training iteration conditions are met, resulting in the neural network model after training.

[0085] Optionally, the preset training iteration conditions can be pre-set conditions required for the neural network model to terminate training. These conditions can be set as needed, and are not limited in this embodiment. For example, in a specific implementation, they can be set to the convergence of the total loss function of the neural network model, or the number of training epochs reaching a set number of epochs, or the training duration reaching a set duration, etc.

[0086] In this embodiment, when the training task of the neural network model is determined to be the first target task for instructing VPU signal denoising processing, the first target of the neural network model is trained according to the first target task and the first frequency domain signal until the pre-training iteration conditions are met, and the neural network model after the model training is completed is obtained. In this way, the smart wearable device can realize VPU signal denoising processing through the neural network model.

[0087] Furthermore, in a feasible embodiment, step b10, the step of training the neural network model for the first objective based on the first objective task and the first frequency domain signal, includes steps b11-b14.

[0088] Step b11: Based on the neural network model and the first target task, the first frequency domain signal is denoised to generate the second frequency domain signal;

[0089] Optionally, in the neural network model, the first frequency domain signal can be denoised according to the first objective task. For example, the U-Net model with an Encode-Decode structure can be used to denoise the first frequency domain signal to generate the second frequency domain signal.

[0090] Step b12 performs a Fourier transform on the second frequency domain signal to obtain a clean first VPU speech signal;

[0091] Optionally, after obtaining the second frequency domain signal, a Fourier transform can be performed on the second frequency domain signal to obtain the corresponding VPU signal, and this VPU signal can be used as the pure first VPU speech signal.

[0092] Step b13: Input the VPU speech signal corresponding to the first VPU mixed signal and the first VPU speech signal into a preset noise reduction processing loss function for calculation to obtain the first prediction loss;

[0093] Optionally, a pre-set VPU speech signal corresponding to the first VPU mixed signal can be determined. This VPU speech signal is a clean, noise-free VPU speech signal. The VPU speech signal corresponding to the first VPU mixed signal and the first VPU speech signal are input into a preset noise reduction processing loss function for calculation to obtain the first prediction loss.

[0094] Optionally, the preset noise reduction loss function can be a conventional loss function, such as mean squared error and mean squared logarithmic error.

[0095] Step b14: Update the neural network model based on the first prediction loss to train the neural network model for the first objective.

[0096] Optionally, the parameters of the neural network model can be updated and optimized in reverse based on the first prediction loss, and then the next round of iterative training can be carried out until the preset training termination condition is met, so as to achieve the first goal of training the neural network model.

[0097] In this embodiment, the first frequency domain signal is denoised based on the neural network model and the first target task to generate a second frequency domain signal. A Fourier transform is then performed to obtain a clean first VPU speech signal. The VPU speech signal corresponding to the mixed signal of the first VPU and the second VPU speech signal is input into the denoising loss function for calculation to obtain the first prediction loss. The neural network model is then updated based on the first prediction loss to train the first target of the neural network model. In this way, the smart wearable device can achieve VPU signal denoising through the neural network model.

[0098] Furthermore, in a feasible embodiment, step S20, which involves training a preset neural network model based on a first frequency domain signal, includes step b20.

[0099] Step b20: If the training task of the neural network model is determined to be a first target task for instructing VPU signal denoising processing and a second target task for instructing VPU signal speech activity detection processing, then the first target of the neural network model is trained according to the first target task and the first frequency domain signal, and the second target of the neural network model is trained according to the second target task and the first frequency domain signal, until the preset training iteration conditions are met, and the neural network model after model training is obtained.

[0100] In this embodiment, during the model training phase of the neural network model, the training task of the neural network model can be determined first. If the training task of the neural network model is determined to be a joint training task, and the joint training task includes a first objective task for instructing VPU signal denoising processing and a second objective task for instructing VPU signal speech activity detection processing, then the neural network model can be jointly trained according to the first objective task and the second objective task.

[0101] Optionally, when determining the training task, after the neural network model receives the input first frequency domain signal, it can receive the training instruction input by the user, or directly determine the training rule corresponding to the first frequency domain signal. According to the training rule or training instruction, the corresponding training task can be selected from each training task. If the selected training task is a joint training task, the neural network model can be trained according to the joint training task so that the neural network model can achieve multiple objectives.

[0102] Optionally, in one scenario, the first target task can be obtained by concatenating the characters indicating VPU signal denoising processing with the first frequency domain signal in the neural network model. The second target task can be obtained by concatenating the characters indicating VPU signal speech activity detection processing with the first frequency domain signal.

[0103] In each round of iterative training, the neural network model is trained based on the generated first objective task to train the neural network model for the first objective. Then, it is trained based on the generated second objective task to train the neural network model for the second objective.

[0104] Optionally, the first objective can be that the neural network model can perform VPU signal noise reduction processing. The second objective can be that the neural network model can perform VPU signal speech activity detection processing. Optionally, VPU signal speech activity detection involves detecting whether there is a VPU signal generated by user speech in the VPU signal. That is, detecting whether there is a frequency domain signal corresponding to the VPU speech signal in the first frequency domain signal corresponding to the VPU mixed signal. If it exists, the detection result of the VPU signal speech activity detection processing is determined to be passed, that is, the model output for the second objective task is passed. If it does not exist, the detection result of the VPU signal speech activity detection processing is determined to be failed, that is, the model output for the second objective task is failed.

[0105] Optionally, after the neural network model receives the input first frequency domain signal and determines that the training task corresponding to the first frequency domain signal is a joint training task, the neural network model can perform a joint training task on the first frequency domain signal.

[0106] Optionally, the first and second objective tasks in the joint training task can be executed in parallel within the neural network model. Alternatively, the second objective task can be executed first, followed by the first objective task. No further restrictions are imposed.

[0107] Optionally, in the neural network model, a second objective task can be performed on the first frequency domain signal to detect whether a frequency domain signal corresponding to the VPU speech signal exists in the first frequency domain signal, thus obtaining a first prediction result. If the first prediction result indicates that a frequency domain signal corresponding to the VPU speech signal exists in the first frequency domain signal, the neural network model performs the first objective task on the first frequency domain signal, i.e., performs noise reduction processing on the first frequency domain signal, thus obtaining a second prediction result. The neural network model is then iteratively trained based on the first and second prediction results until the preset training iteration conditions are met, resulting in the neural network model after training.

[0108] Optionally, during iterative training, a first prediction loss corresponding to the first target task can be calculated based on the first prediction result. A second prediction loss corresponding to the second target task can be calculated based on the second prediction result. The first and second prediction losses are then weighted and summed to obtain the total loss function value. The model parameters of the neural network model are optimized based on the total loss function value to train the neural network model for the first and second targets.

[0109] In this embodiment, when the training task of the neural network model is a first target task and a second target task, the first target of the neural network model is trained based on the first target task and the first frequency domain signal, and the second target of the neural network model is trained based on the second target task and the first frequency domain signal, until the preset training iteration conditions are met, and the neural network model after the model training is completed is obtained. This ensures the effective training of the neural network model and facilitates subsequent smart wearable devices to effectively identify whether the VPU signal is a valid VPU speech signal through the neural network model, and to perform noise reduction processing on the VPU signal through the neural network model.

[0110] Furthermore, in a feasible embodiment, step b20, the step of training the neural network model for the second objective based on the second objective task and the first frequency domain signal, includes steps b21-b23.

[0111] Step b21: Based on the neural network model and the second target task, perform speech activity detection on the first frequency domain signal to obtain the speech activity detection result;

[0112] Optionally, in the neural network model, speech activity detection can be performed on the first frequency domain signal according to the second objective task to obtain the speech activity detection result, and the speech activity detection result can be output as the second prediction result.

[0113] Optionally, the speech activity detection result may include the presence of a frequency domain signal corresponding to the VPU speech signal in the first frequency domain signal corresponding to the VPU mixed signal. It may also include the absence of a frequency domain signal corresponding to the VPU speech signal in the first frequency domain signal corresponding to the VPU mixed signal.

[0114] Step b22: Input the speech activity detection result and the speech activity label corresponding to the first VPU mixed signal into the preset speech activity detection loss function for calculation to obtain the second prediction loss;

[0115] Optionally, a voice activity tag corresponding to the first VPU mixed signal at each time step can be pre-set. Optionally, the voice activity tag can include whether a VPU voice signal exists in the first VPU mixed signal at that time step, and whether a VPU voice signal does not exist in the first VPU mixed signal at that time step.

[0116] Optionally, the speech activity label of the first VPU mixed signal corresponding to the speech activity detection result can be determined, and the label and the speech activity detection result can be input into a preset speech activity detection loss function for calculation to obtain the second prediction loss.

[0117] Optionally, the preset speech activity detection loss function can be a conventional loss function, such as mean squared error and mean squared logarithmic error.

[0118] Step b23: Update the neural network model according to the second prediction loss to train the neural network model for the second objective.

[0119] Optionally, the parameters of the neural network model can be updated and optimized in reverse based on the second prediction loss, and then the next round of iterative training can be carried out until the preset training termination condition is met, so as to achieve the second objective of training the neural network model.

[0120] Optionally, in one scenario, the first prediction loss and the second prediction loss can be weighted and summed to obtain the total loss function value. The parameters of the neural network model can then be updated and optimized based on the total loss function value. Then, the next round of iterative training can be carried out until the preset training termination condition is met, so as to achieve the first and second objectives of training the neural network model.

[0121] In this embodiment, the speech activity detection result of the first frequency domain signal is performed based on the neural network model and the second target task. The speech activity detection result and the speech activity label are input into the speech acquisition detection loss function for calculation to obtain the second prediction loss. The neural network model is then updated based on the second prediction loss to train the second target of the neural network model. This enables the subsequent smart wearable device to effectively identify whether the VPU signal is a valid VPU speech signal based on the neural network model.

[0122] Based on the first or second embodiment of this application, a third embodiment of this application is proposed. In this third embodiment, content that is the same as or similar to the above embodiments can be referred to the above description and will not be repeated hereafter. Based on this, step S30, the step of performing VPU signal denoising processing according to the neural network model after model training, includes step c10.

[0123] Step c10: If the actual VPU signal is acquired, the actual VPU signal is converted into a third frequency domain signal and input into the neural network model after model training is completed, so that the third frequency domain signal can be denoised according to the neural network model after training to generate a fourth frequency domain signal, and the fourth frequency domain signal is converted into a clean VPU speech signal corresponding to the actual VPU signal.

[0124] Optionally, in the practical application stage of the neural network model, the VPU signal can be acquired in real time by the VPU sensor in the smart wearable device to obtain the actual VPU signal. Optionally, the actual VPU signal may include at least one of the actually acquired VPU speech signal and VPU noise signal.

[0125] Optionally, the actual VPU signal can be Fourier transformed into a third frequency domain signal and input into a neural network model for noise reduction processing. This generates a noise-reduced fourth frequency domain signal, which is then Fourier transformed to obtain a clean VPU speech signal corresponding to the actual VPU signal. This VPU speech signal is then output, thereby enabling the smart wearable device to perform noise reduction processing on the VPU signal through a neural network model.

[0126] In this embodiment, by performing VPU signal noise reduction processing on the actual VPU signal collected by the smart wearable device according to the neural network model in the actual application stage, a pure VPU voice signal is obtained, thereby avoiding the interference of noise signal on the VPU signal.

[0127] Furthermore, to aid in understanding the VPU-based noise reduction process in this embodiment, an example is provided below.

[0128] For example, such as Figure 3 As shown, after the smart wearable device collects the VPU signal through its own VPU sensor, it performs Fourier transform processing on the collected VPU signal to obtain a frequency domain signal (such as the first or third frequency domain signal in the above embodiment). The first or third frequency domain signal is then input into a neural network model, and a new frequency domain signal (such as the second or fourth frequency domain signal in the above embodiment) is output. This new frequency domain signal is then subjected to Fourier transform processing to obtain an enhanced signal (i.e., the clean VPU speech signal in the above embodiment). In other words, the neural network model achieves noise reduction processing of the VPU signal, thus suppressing non-stationary noise in the VPU signal.

[0129] Furthermore, embodiments of this application provide a noise reduction processing device based on a VPU, referring to... Figure 4 The VPU-based noise reduction processing device includes:

[0130] The conversion module A10 is used to convert and merge the preset microphone voice signal and microphone noise signal according to the preset conversion rules between the microphone signal and the target VPU signal to obtain the VPU mixed signal.

[0131] The model training module A20 is used to convert the VPU mixed signal into a first frequency domain signal and train a preset neural network model based on the first frequency domain signal.

[0132] The noise reduction module A30 is used to perform VPU signal noise reduction processing based on the neural network model after model training.

[0133] The VPU-based noise reduction processing device provided in this application, employing the VPU-based noise reduction processing method described in the above embodiments, can solve the technical problem of how to perform noise reduction processing on VPU signals in smart wearable devices. Compared with the prior art, the beneficial effects of the VPU-based noise reduction processing device provided in this application are the same as those of the VPU-based noise reduction processing method provided in the above embodiments, and other technical features in the VPU-based noise reduction processing device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0134] This application provides a smart wearable device, which includes: a VPU; at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the VPU-based noise reduction processing method in Embodiment 1 above.

[0135] The following is for reference. Figure 5 The figure illustrates a structural schematic diagram suitable for implementing the embodiments of this application of a smart wearable device. The smart wearable device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The device shown in the figure is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0136] The smart wearable device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for device operation. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows smart wearable devices to communicate wirelessly or wiredly with other devices to exchange data. While the figures show smart wearable devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0137] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0138] The smart wearable device provided in this application employs the VPU-based noise reduction processing method described in the above embodiments, which solves the technical problem of how to perform noise reduction processing on the VPU signal in the smart wearable device. Compared with the prior art, the beneficial effects of the smart wearable device provided in this application are the same as those of the VPU-based noise reduction processing method provided in the above embodiments, and other technical features of the smart wearable device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0139] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0140] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0141] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the VPU-based noise reduction processing method in the above embodiments.

[0142] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0143] The aforementioned computer-readable storage medium may be included in the smart wearable device; or it may exist independently and not assembled into the smart wearable device.

[0144] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a smart wearable device, enable the smart wearable device to perform the steps in the aforementioned VPU-based noise reduction processing method.

[0145] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0147] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0148] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described VPU-based noise reduction processing method, thereby solving the technical problem of how to perform noise reduction processing on VPU signals in smart wearable devices. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the VPU-based noise reduction processing method provided in the above embodiments, and will not be repeated here.

[0149] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the VPU-based noise reduction processing method described above.

[0150] The computer program product provided in this application can solve the technical problem of how to perform noise reduction processing on VPU signals in smart wearable devices. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the VPU-based noise reduction processing method provided in the above embodiments, and will not be repeated here.

[0151] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A noise reduction processing method based on VPU, characterized in that, The VPU-based noise reduction processing method includes the following steps: Based on the preset conversion rules between the microphone signal and the target VPU signal, the preset microphone voice signal and microphone noise signal are converted and merged to obtain the VPU mixed signal; The VPU mixed signal is converted into a first frequency domain signal, and a preset neural network model is trained based on the first frequency domain signal. VPU signal denoising is performed based on the neural network model after training.

2. The VPU-based noise reduction method as described in claim 1, characterized in that, The step of converting and merging a preset microphone voice signal and a microphone noise signal to obtain a VPU mixed signal according to a preset conversion rule between the microphone signal and the target VPU signal includes: When the conversion rule includes a first transfer function corresponding to the microphone voice signal, the microphone voice signal is converted into a VPU voice signal according to the first transfer function; When the conversion rule includes the second transfer function corresponding to the microphone noise signal, the microphone noise signal is converted into a VPU noise signal according to the second transfer function; The VPU speech signal and the VPU noise signal are mixed according to a preset signal-to-noise ratio to obtain a VPU mixed signal.

3. The VPU-based noise reduction method as described in claim 1, characterized in that, The step of training a preset neural network model based on the first frequency domain signal includes: If the training task of the neural network model is determined to be a first target task for instructing VPU signal denoising processing, then the neural network model is trained according to the first target task and the first frequency domain signal until the preset training iteration conditions are met, and the neural network model after training is obtained.

4. The VPU-based noise reduction method as described in claim 3, characterized in that, The first objective step of training the neural network model based on the first objective task and the first frequency domain signal includes: Based on the neural network model and the first target task, the first frequency domain signal is denoised to generate a second frequency domain signal; Perform a Fourier transform on the second frequency domain signal to obtain a clean first VPU speech signal; The VPU speech signal corresponding to the first VPU mixed signal and the first VPU speech signal are input into a preset noise reduction processing loss function for calculation to obtain the first prediction loss; The neural network model is updated based on the first prediction loss to train the neural network model for the first objective.

5. The VPU-based noise reduction processing method as described in claim 1, characterized in that, The step of training a preset neural network model based on the first frequency domain signal further includes: If the training task of the neural network model is determined to be a first target task for instructing VPU signal noise reduction processing and a second target task for instructing VPU signal speech activity detection processing, then the first target of the neural network model is trained according to the first target task and the first frequency domain signal, and the second target of the neural network model is trained according to the second target task and the first frequency domain signal, until the preset training iteration conditions are met, and the neural network model after model training is obtained.

6. The VPU-based noise reduction processing method as described in claim 1, characterized in that, The step of training the neural network model based on the second target task and the first frequency domain signal for the second target task includes: Based on the neural network model and the second target task, speech activity detection is performed on the first frequency domain signal to obtain speech activity detection results; The speech activity detection result and the speech activity label corresponding to the first VPU mixed signal are input into a preset speech activity detection loss function for calculation to obtain the second prediction loss; The neural network model is updated based on the second prediction loss to train the neural network model for the second objective.

7. The VPU-based noise reduction processing method according to any one of claims 1-6, characterized in that, The step of performing VPU signal denoising processing based on the neural network model after model training includes: If an actual VPU signal is acquired, the actual VPU signal is converted into a third frequency domain signal, and the third frequency domain signal is input into the neural network model after model training is completed, so that the third frequency domain signal is denoised according to the neural network model after training to generate a fourth frequency domain signal, and the fourth frequency domain signal is converted into a clean VPU speech signal corresponding to the actual VPU signal.

8. A noise reduction processing device based on VPU, characterized in that, The VPU-based noise reduction processing device includes: The conversion module is used to convert and merge the preset microphone voice signal and microphone noise signal according to the preset conversion rules between the microphone signal and the target VPU signal to obtain the VPU mixed signal. The model training module is used to convert the VPU mixed signal into a first frequency domain signal and train a preset neural network model based on the first frequency domain signal. The noise reduction module is used to perform VPU signal noise reduction processing based on the neural network model after model training.

9. A smart wearable device, characterized in that, The smart wearable device includes: a VPU, a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the VPU-based noise reduction processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the VPU-based noise reduction processing method as described in any one of claims 1 to 7.

Images