Voice signal processing method, voice signal processing device and storage medium

Abstract

The present application discloses a speech signal processing method, a speech signal processing device and a storage medium, and relates to the field of speech signal processing technology. The speech signal processing is disclosed, including: obtaining an ambient audio signal of the current environment; performing frequency domain conversion and feature extraction processing on the ambient audio signal, and determining an ambient noise estimation value of the ambient audio signal after frequency domain conversion and feature extraction; when receiving a speech signal to be processed, denoising the speech signal to be processed based on the ambient noise estimation value and a deep neural network to obtain a denoised first speech signal. When the present application processes a speech signal, the ambient audio signal of the current environment is first extracted, and then the ambient noise estimation value corresponding to the ambient audio signal is extracted. Finally, based on the ambient noise estimation value and the deep neural network, speech denoising and enhancement processing is performed on the user's speech signal to provide a clearer speech output result and improve the speech denoising effect.

Description

Technical Field

[0001] The present application relates to the field of speech signal processing technology, and in particular to a speech signal processing method, a speech signal processing device and a storage medium. Background Art

[0002] In the related noise processing methods, complex environmental noise is usually processed by spectral subtraction, Wiener filtering and adaptive filtering. However, these processing methods rely on accurate estimation of noise and have limitations when processing non-stationary noise or noise with a frequency close to that of speech signals, which makes the noise estimation inaccurate, easily introduces artifacts or distortion, and leads to poor denoising effect.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a speech signal processing method, a speech signal processing device and a storage medium, aiming to solve the technical problem of poor denoising effect in related noise processing technologies.

[0005] To achieve the above object, the present application proposes a speech signal processing method, the method comprising:

[0006] Get the ambient audio signal of the current environment;

[0007] Performing frequency domain conversion and feature extraction processing on the ambient audio signal, and determining an ambient noise estimation value of the ambient audio signal after the frequency domain conversion and feature extraction;

[0008] When a speech signal to be processed is received, denoising is performed on the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain a denoised first speech signal.

[0009] In one embodiment, when receiving the speech signal to be processed, the step of performing denoising on the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain the denoised first speech signal includes:

[0010] When receiving the speech signal to be processed, converting the speech signal to be processed into a frequency domain signal;

[0011] The frequency domain signal and the environmental noise estimation value are input into the deep neural network, and the frequency domain signal is denoised based on the deep neural network to obtain the first speech signal.

[0012] In one embodiment, when receiving the speech signal to be processed, after the step of denoising the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain the denoised first speech signal, the method further includes:

[0013] Determining a target gain value corresponding to the first speech signal based on the first speech signal and the environmental noise estimation value;

[0014] Based on the target gain value, the first speech signal is gain adjusted by an automatic gain control algorithm to obtain a second speech signal.

[0015] In one embodiment, the step of determining a target gain value corresponding to the first speech signal based on the first speech signal and the environmental noise estimation value comprises:

[0016] Based on the voice activity detection module and the environmental noise estimation value, signal differentiation is performed on the first voice signal to obtain the voice signal to be amplified and other voice signals;

[0017] Determine the signal strength of the speech signal to be amplified and the other speech signals according to the envelope detection module, and determine the environmental noisiness based on the signal strength;

[0018] The target gain value is generated based on the environmental noisiness.

[0019] In one embodiment, the step of performing frequency domain conversion and feature extraction processing on the ambient audio signal and determining the ambient noise estimation value of the ambient audio signal after the frequency domain conversion and feature extraction comprises:

[0020] Performing short-time Fourier transform processing on the ambient audio signal to obtain a preset frequency domain signal;

[0021] Performing feature extraction processing on the preset frequency domain signal based on a convolutional neural network, and generating an estimated value of the noise level and noise type of the ambient audio signal according to the extracted features;

[0022] The estimated values ​​of the noise level and the noise type are set as the environmental noise estimated values.

[0023] In one embodiment, when receiving the speech signal to be processed, after the step of denoising the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain the denoised first speech signal, the method further includes:

[0024] Determine the user's hearing profile and audio preference model;

[0025] generating audio adjustment parameters based on the hearing characteristic curve and the audio preference model;

[0026] The first voice signal is audio-adjusted according to the audio adjustment parameter to obtain a third voice signal.

[0027] In one embodiment, the step of determining the user's hearing characteristic curve and audio preference model includes:

[0028] Determining the audio playback history and feedback data of the user, and training a pre-stored learning model based on the audio playback history, and iteratively optimizing the trained model based on the feedback data to obtain the audio preference model;

[0029] Acquire hearing test data of the user, and determine the hearing characteristic curve according to the hearing test data.

[0030] In one embodiment, when receiving the speech signal to be processed, after the step of denoising the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain the denoised first speech signal, the method further includes:

[0031] Determining a deviation value between the first speech signal and a predicted audio signal generated by the deep neural network;

[0032] The deep neural network is iteratively optimized based on the deviation value.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a speech signal processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the speech signal processing method described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the speech signal processing method described above are implemented.

[0035] One or more technical solutions proposed in this application have at least the following technical effects:

[0036] When performing noise reduction processing on the speech signal to be processed, the current ambient audio signal is first obtained, and the ambient noise estimation value is calculated based on the ambient audio signal, so that when the deep neural network processes the speech signal, the speech and noise in the speech signal can be effectively separated in combination with the ambient noise estimation value, thereby enhancing the speech output effect and improving the denoising effect of the speech signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A schematic diagram of a first embodiment of the speech signal processing method provided by the present application;

[0040] Figure 2 A flowchart of a second embodiment of the speech signal processing method provided by the present application;

[0041] Figure 3 A flowchart of a third embodiment of the speech signal processing method provided by the present application;

[0042] Figure 4 Schematic diagram of the device structure of the hardware operating environment involved in the voice signal processing method in the embodiment of the present application.

[0043] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0045] The main solution of the embodiment of the present application is: obtaining the ambient audio signal of the current environment;

[0046] Performing frequency domain conversion and feature extraction processing on the ambient audio signal, and determining an ambient noise estimation value of the ambient audio signal after the frequency domain conversion and feature extraction;

[0047] When a speech signal to be processed is received, denoising is performed on the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain a denoised first speech signal.

[0048] In this embodiment, for the convenience of description, the following description is made by taking the speech signal processing device as the execution subject.

[0049] Traditional speech enhancement and noise processing technologies include spectral subtraction, Wiener filtering and adaptive filtering. Spectral subtraction estimates the power spectrum of the noise and subtracts it from the noisy speech to improve the clarity of the speech signal. Wiener filtering is based on a statistical model and enhances the speech signal by minimizing the mean square error. Adaptive filtering uses the least mean square (LMS) algorithm to suppress noise by adjusting the filter coefficients in real time.

[0050] However, spectral subtraction easily produces musical noise, that is, it introduces unnatural audio artifacts during the noise reduction process. Wiener filtering requires accurate modeling of noise and speech signals, and robust estimation of model parameters is more difficult. Adaptive filtering has poor adaptability to changes in environmental noise and requires prior knowledge.

[0051] Based on this, current noise processing technology has limitations when dealing with non-stationary noise or noise with a frequency close to that of speech signals. It is easy to introduce artifacts or distortion, resulting in poor denoising effect.

[0052] The present application provides a solution. When performing noise reduction processing on a speech signal to be processed, the current ambient audio signal is first obtained, and an ambient noise estimation value is calculated based on the ambient audio signal, so that when a deep neural network processes the speech signal, the speech and noise in the speech signal can be effectively separated in combination with the ambient noise estimation value, thereby enhancing the speech output effect and improving the denoising effect of the speech signal.

[0053] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a voice signal processing device, etc. The following takes a voice signal processing device as an example to illustrate this embodiment and the following embodiments.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The present application provides a method for processing a speech signal. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the speech signal processing method of the present application.

[0056] In this embodiment, the speech signal processing method includes steps S10 to S30:

[0057] Step S10, obtaining an ambient audio signal of the current environment;

[0058] It should be noted that the ambient audio signal refers to all sound signals existing in the current position, including but not limited to human voice, mechanical sound, natural sound, etc. These sound signals may interfere with the speech signal to be processed.

[0059] Therefore, in this embodiment, when the voice signal processing device is in operation, the ambient audio signal of the current environment can be obtained in real time, so as to analyze the current ambient noise based on the ambient audio signal, and then perform denoising on the voice signal to be processed based on the ambient noise to improve the denoising effect. Among them, the ambient audio signal can be collected by the microphone of the voice signal processing device, and the voice signal to be processed can be a voice signal input by the user through the recording device, a voice signal pre-stored in the voice signal processing device, such as recorded audio including meeting records and telephone communications, songs and radio programs. Optionally, the voice signal to be processed can also be a voice signal sent by the server. The specific source of the voice signal to be processed is not limited here. The voice signal processing device can obtain the current ambient audio signal based on a preset time interval, or it can obtain the ambient audio signal in real time, and it can also obtain the ambient audio signal when the decibel of the sound in the current environment is greater than a certain threshold.

[0060] Step S20, performing frequency domain conversion and feature extraction processing on the ambient audio signal, and determining an ambient noise estimation value of the ambient audio signal after the frequency domain conversion and feature extraction;

[0061] It should be noted that frequency domain conversion refers to converting a time domain signal into a frequency domain signal. The conversion method is usually to perform Fourier transform processing on the ambient audio signal, such as short-time Fourier transform (STFT), fast Fourier transform (FFT), etc. Feature extraction refers to extracting the spectrum characteristics of the converted frequency domain signal, so as to determine the ambient noise estimation value of the ambient audio signal based on the spectrum characteristics. The ambient noise estimation value includes the estimation value of the noise level and the noise type.

[0062] In this embodiment, when processing the ambient audio signal, the spectrum characteristics of the ambient noise can be analyzed in real time through deep learning models such as convolutional neural networks (CNN) or long short-term memory networks (LSTM), so as to obtain an estimated value of the noise level of the current environment and an estimated value of the noise type based on the spectrum characteristics. Among them, the estimated values ​​of the noise level and the noise type are the estimated values ​​of the ambient noise. The estimated value of the noise level can be used to determine the noisiness corresponding to different sound frequency bands in the current environment, and targeted noise optimization processing can be performed based on the noise type to improve the denoising effect.

[0063] As an optional implementation, after the ambient audio signal is collected by the microphone, the signal needs to be preprocessed. The preprocessing process includes performing short-time Fourier transform processing on the ambient audio signal, and then converting the time domain signal into a preset frequency domain signal, so as to extract the spectral features of the frequency domain signal through a convolutional neural network. Next, feature extraction processing is performed on the preset frequency domain signal based on the convolutional neural network, and based on the extracted features, an estimated value of the noise level and noise type of the ambient audio signal is generated, and finally the estimated value of the noise level and the noise type is set as the ambient noise estimate.

[0064] It should be noted that when performing signal conversion processing on the ambient audio signal, in order to reduce spectrum leakage, a window function such as a Hanning window can be applied before the short-time Fourier transform. When performing spectrum feature extraction processing, the convolutional neural network automatically learns and extracts the features of the frequency domain signal through the convolution layer, and then outputs the estimated value of the noise level and the estimated value of the noise type based on the extracted features.

[0065] It should be noted that when using a convolutional neural network or a long short-term memory network model, it is necessary to use a large amount of labeled environmental noise data to train it, and then learn the spectral characteristics of different noise types. After obtaining the corresponding spectral characteristics, the corresponding noise level and noise type estimation values ​​are generated based on the spectral characteristics to improve the denoising effect.

[0066] Step S30: When a speech signal to be processed is received, denoising is performed on the speech signal based on the environmental noise estimation value and the deep neural network to obtain a denoised first speech signal.

[0067] In this embodiment, when denoising the speech signal to be processed, the estimated value of the ambient noise can be used as a reference value for denoising, and input into a deep neural network together with the speech signal, and then processed based on the deep neural network (DNN). When the speech signal is input into the deep neural network for processing, it is also necessary to convert the speech signal into a frequency domain signal.

[0068] Based on this, when the voice signal to be processed is received, the voice signal to be processed is converted into a frequency domain signal. In this process, the voice signal can also be short-time Fourier transformed to convert the time domain signal into a frequency domain signal. The frequency domain signal and the environmental noise estimation value are then input into the deep neural network, and then the frequency domain signal is denoised based on the deep neural network to obtain the first voice signal. Among them, the stage of denoising the frequency domain signal based on the deep neural network belongs to the signal enhancement stage.

[0069] For example, on a street, a user is talking on the phone with someone. The smartphone obtains the ambient audio signal of the current call environment through the built-in microphone, and then performs short-time Fourier transform processing on the ambient audio signal to obtain the frequency domain signal of the ambient audio signal. Subsequently, the features of the frequency domain signal are extracted through a convolutional neural network to obtain the current ambient noise estimate, which includes 80 decibels of street noise, including car horns, children playing, and dogs barking, as well as the spectral characteristics of these noises. Based on this, during a user call, the received voice signal to be processed includes the user's call voice and the mixed sound of the current environmental noise. At this time, the mixed voice signal is first converted into a frequency domain signal through short-time Fourier transform, and then the frequency domain signal and the environmental noise estimation value are input into the deep neural network (DNN) model. The deep neural network model analyzes the frequency band of the environmental noise estimation value and the corresponding noise loudness. For example, the frequency band of the car horn sound is 100-200 Hz and the decibel is 80 decibels. At this time, the deep learning model eliminates 80 decibels of noise within the frequency band. Similarly, the deep learning model eliminates the noise of children playing and dog barking based on the same method, and then outputs the first voice signal after denoising. It can be understood that the above parameters are only used for explanation and are not a limitation of this application.

[0070] It should be noted that in the training phase, the DNN model establishes a mapping relationship between a large number of noisy speech and the corresponding clean speech signals by learning them. This mapping relationship can be spectral mapping or masking, and the specific method depends on the specific model architecture used. For example, the spectral mapping-based method directly learns the relationship between the amplitude spectrum of noisy speech and the amplitude spectrum of clean speech, while the masking-based method distinguishes between speech and noise components by learning a masking matrix. Based on this, in the enhancement phase, when the real-time collected noisy speech signal and the noise estimation value are input, the DNN model estimates the amplitude spectrum of the clean speech or applies the masking matrix based on the previously learned mapping relationship, so as to remove the noise component as much as possible and retain the pure speech signal. Finally, these frequency domain representations are converted back to time domain signals through waveform reconstruction technology and other technologies to obtain the enhanced speech signal, that is, the first speech signal after denoising.

[0071] Based on this, traditional spectral subtraction and Wiener filtering have limited effects when dealing with complex noise environments and are prone to introducing artifacts or distortion. The audio enhancement algorithm of this application uses a deep neural network for speech enhancement, which can more accurately separate speech and noise, provide clearer speech output, and improve the denoising effect of speech signals.

[0072] Optionally, after obtaining the first voice signal, the audio signal may be directly output, or the audio signal may be processed in a personalized manner and the audio signal after the personalized processing may be output.

[0073] In another optional implementation, the deep neural network model can also be iteratively optimized. Specifically, when receiving a speech signal to be processed, the deep neural network model usually generates a predicted audio signal, and then after denoising the speech signal to be processed mixed with noise and obtaining a first speech signal, determines the deviation value between the predicted audio signal and the first speech signal, and then iteratively optimizes the deep neural network model based on the deviation value.

[0074] Based on this, by calculating the deviation between the predicted audio signal and the first speech signal, the model can more accurately capture the characteristics of the actual speech signal, thereby reducing the prediction error. At the same time, iterative optimization based on the deviation value helps to adjust the model parameters, so that the model can better adapt to complex speech data and improve the accuracy of prediction. After multiple iterative optimizations, it is ensured that the model can maintain high performance in various noise environments and speech conditions, that is, it enhances its generalization ability on different data sets, making it more robust to sudden changes or different loudness in speech signals.

[0075] The present embodiment provides a speech signal processing method, which first obtains the current ambient audio signal, then extracts the frequency spectrum features corresponding to the ambient audio signal, and determines the corresponding ambient noise estimation value based on the features, then performs frequency domain conversion processing on the received speech signal, and processes the features after the frequency domain conversion and the ambient noise estimation value based on a deep neural network, so that the deep neural network can more accurately separate the speech and noise in the speech signal to be processed, thereby improving the denoising effect of the speech signal.

[0076] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2 After step S30, the speech signal processing method further includes steps S40 to S50:

[0077] Step S40, determining a target gain value corresponding to the first speech signal based on the first speech signal and the environmental noise estimation value;

[0078] In this embodiment, after the first voice signal is generated, the voice signal can be directly output, or the voice signal can be output after gain processing. Among them, in different environments, the volume of the first voice signal output after denoising the voice signal to be processed may be inconsistent, thereby affecting the user experience. For example, the frequency band coverage of different noises is different, resulting in different attenuation degrees of the corresponding frequency bands in the voice signal in the filter during the processing process, thereby affecting the output volume. Based on this, after obtaining the denoised first voice signal, the gain of the audio output can be dynamically adjusted through the automatic gain control (AGC) algorithm to ensure the consistency of the volume in different environments.

[0079] When processing the first voice signal, it is necessary to first determine the target gain value of the gain control algorithm. In the process of determining the target gain value based on the first voice signal and the estimated value of the ambient noise, the optimal gain value in the current environment can be calculated based on the WebRTC (Web Real-Time Communication) framework using voice activity detection (VAD) and envelope detection modules.

[0080] Specifically, based on the voice activity module and the ambient noise value, the first voice signal can be differentiated to obtain the voice signal to be gained and other voice signals, wherein the voice signal to be gained is related to the sound to be gained, for example, if the human voice needs to be gained, the voice signal to be gained is the human voice, and the remaining non-gained sounds are other sounds. Similarly, if animal sounds, mechanical sounds, or music sounds need to be gained, the corresponding sounds to be gained are animal sounds, mechanical sounds, or music sounds. For example, the sound that currently needs to be gained is a human voice. When processing based on the voice activity module, the noise level in the ambient noise estimation value can be used to determine which sounds in which frequency bands belong to non-gained sounds, and then the voice parts to be gained and other voice parts, i.e., noise parts, in the audio stream can be identified.

[0081] After obtaining the voice signal to be gained, the signal strength of the voice signal to be gained and the other voice signals is determined according to the envelope detection module of the WebRTC framework, and the environmental noise is determined based on the signal strength. In this process, the energy level of the voice signal to be gained and the other voice signals is estimated through the envelope detection module, and the energy level is characterized as the current environmental noise. It can be understood that when the environmental noise is positively correlated with the current target gain value, the higher the environmental noise, the greater the noise in the surrounding environment. At this time, the sound of the audio signal needs to be increased synchronously to avoid being interfered by the environmental noise and causing the sound to be unclear.

[0082] Finally, after the environmental noise is obtained, the target gain value is generated based on the environmental noise. There is a corresponding mapping relationship between the environmental noise and the target gain value. Based on the mapping relationship, after the environmental noise is obtained, the specific gain value can be determined. The mapping relationship can be a functional relationship, a numerical matching relationship, an algorithm conversion, etc.

[0083] Step S50: Based on the target gain value, the first speech signal is gain adjusted according to an automatic gain control algorithm to obtain a gain speech signal.

[0084] In this embodiment, after obtaining the target gain value, the automatic gain control algorithm can perform gain processing on the first voice signal based on the gain value. For example, the gain value is to increase the volume by 20 decibels in all frequency bands. At this time, the overall volume of the first voice signal is increased by 20 decibels based on the automatic gain control algorithm.

[0085] This embodiment provides a speech signal processing method. After obtaining a first speech signal, gain processing is performed on the first speech signal through an automatic gain control algorithm to ensure consistency of volume in different environments. During the processing, the automatic gain control algorithm can optimize the speech signal based on the WebRTC framework, and process the first speech signal and the environmental noise estimation value by adding a voice activity detection and envelope detection module to obtain a gain value that meets the current gain requirement, thereby being able to adjust the gain more quickly and accurately, thereby improving the output effect of the speech after denoising processing.

[0086] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated later. Figure 3 After step S30, the speech signal processing method further includes steps S60 to S80:

[0087] Step S60, determining the hearing characteristic curve and audio preference model of the user;

[0088] After denoising the voice signal to be processed and obtaining the first voice signal, the first voice signal can be directly output. When audio output is performed based on the first voice signal, it can also be adjusted based on the user's personalized needs, so as to output a voice signal that meets the user's actual needs, thereby improving the user's auditory experience and the audio output effect.

[0089] Specifically, in this process, the user's hearing characteristics and preference data can be analyzed to automatically adjust audio settings, such as equalizer parameters, volume gain, etc., to provide a personalized auditory experience. Among them, the hearing characteristic curve includes the user's hearing characteristics in different frequency bands, namely low, medium and high audio frequency bands. The audio preference model is a model built based on the user's actual audio preferences.

[0090] Therefore, in this embodiment, it is necessary to first determine the user's hearing characteristic curve and the audio preference model that characterizes the user's audio preference, wherein the user's hearing characteristics and preference data can be collected through questionnaires or user feedback, and the audio preference model can be determined through the user's audio playback history, feedback data, etc., and the hearing characteristic curve can be determined through the collected hearing test data.

[0091] As an optional implementation method for determining the hearing characteristic curve and the audio preference model, the user's audio playback history and feedback data can be determined, a pre-stored learning model can be trained based on the audio playback history, and the trained model can be iteratively optimized based on the feedback data to obtain the audio preference model. In this process, the audio playback history data is used as training data to train the model to obtain an initial model. At the same time, the training data can be updated through the feedback information input by the user, and then the initial model is iteratively optimized based on the updated training data until the performance evaluation of the model meets the requirements, and finally the model that meets the performance requirements is used as the audio preference model. In this process, the user's audio playback history and feedback data can be obtained first, and then a deep learning model such as a convolutional neural network, a recurrent neural network (RNN, Recurrent Neural Network), etc. can be used to learn the user's audio preferences to obtain an audio preference model.

[0092] Furthermore, the hearing characteristic curve can be determined based on the user's hearing test data. In the process of determining the hearing characteristic curve, after obtaining the hearing test data, the user's hearing sensitivity in different frequency bands such as low frequency, medium frequency and high frequency, as well as the user's hearing characteristics are extracted based on the machine learning model, and finally, based on the hearing characteristics and hearing sensitivity, the user's hearing characteristic curves corresponding to different frequency bands are generated.

[0093] By obtaining the user's hearing characteristic curve and audio preference model, the processing algorithm can dynamically adjust the audio settings based on the user's real-time feedback and environmental changes, providing an auditory experience that better meets user needs, that is, conforms to the user's hearing characteristics and audio preferences, thereby improving the output effect.

[0094] Step S70, generating audio adjustment parameters based on the hearing characteristic curve and the audio preference model;

[0095] In this embodiment, the hearing characteristic curve includes the hearing sensitivity of the user in different frequency bands. Based on this, after the hearing characteristic curve is input into the audio preference model, the model can calculate the gain curve of the hearing characteristic curve at each frequency point. The gain value in the gain curve is used to compensate for the hearing loss of the user at a specific frequency, as well as the filter parameters for parameter adjustment of the filter. The gain curve and the filter are audio adjustment parameters. Based on the audio adjustment parameters, the audio of different frequency bands in the first voice signal can be subjected to corresponding gain processing. Therefore, after obtaining the audio adjustment parameters, the first voice signal is adjusted based on the filter and the automatic gain control algorithm of the filter.

[0096] Step S80: Perform audio adjustment on the first voice signal according to the audio adjustment parameter to obtain a third voice signal.

[0097] In this embodiment, after obtaining the audio adjustment parameters, the first voice signal is adjusted and processed based on the audio adjustment parameters through a filter and a gain control algorithm of the filter. For example, the first voice signal is filtered based on an adaptive filter, and at the same time, the gain of a certain frequency band or all frequency bands is adjusted through an automatic gain control algorithm. Finally, a personalized third voice signal for output is obtained, thereby improving the output effect of the voice signal after noise reduction.

[0098] This embodiment provides a speech signal processing method, which generates the hearing characteristic curves corresponding to different frequency bands through the user's hearing sensitivity and hearing characteristics, and inputs the hearing characteristic curves as hearing characteristic curves into an audio preference model to make the output parameters meet the user's actual hearing needs. Finally, the first speech signal is adjusted based on the audio adjustment parameters to ensure that the output personalized audio signal matches the user's hearing characteristics and preferences, thereby improving the audio output effect.

[0099] The present application provides a speech signal processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the speech signal processing method in the above-mentioned first embodiment.

[0100] Reference below Figure 4, which shows a schematic diagram of the structure of a speech signal processing device suitable for implementing the embodiments of the present application. The speech signal processing device in the embodiments of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The voice signal processing device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0101] like Figure 4 As shown, the speech signal processing device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the speech signal processing device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the speech signal processing device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a speech signal processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided alternatively.

[0102] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0103] The speech signal processing device provided by the present application adopts the speech signal processing method in the above embodiment, which can solve the technical problem of poor denoising effect in the relevant noise processing technology. Compared with the prior art, the beneficial effects of the speech signal processing device provided by the present application are the same as the beneficial effects of the speech signal processing method provided by the above embodiment, and the other technical features in the speech signal processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0104] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0105] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0106] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the speech signal processing method in the above-mentioned embodiment.

[0107] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0108] The computer-readable storage medium may be included in the speech signal processing device; or may exist independently without being assembled into the speech signal processing device.

[0109] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the speech signal processing device, the speech signal processing device:

[0110] Get the ambient audio signal of the current environment;

[0111] Performing frequency domain conversion and feature extraction processing on the ambient audio signal, and determining an ambient noise estimation value of the ambient audio signal after the frequency domain conversion and feature extraction;

[0112] When a speech signal to be processed is received, denoising is performed on the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain a denoised first speech signal.

[0113] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0116] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned speech signal processing method, and can solve the technical problem of poor denoising effect in related noise processing technologies. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the speech signal processing method provided in the above-mentioned embodiment, and will not be described in detail here.

[0117] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A speech signal processing method, characterized in that: The method comprises: Get the ambient audio signal of the current environment; Performing short-time Fourier transform processing on the ambient audio signal to obtain a preset frequency domain signal; Performing feature extraction processing on the preset frequency domain signal based on a convolutional neural network, and generating an estimated value of the noise level and noise type of the ambient audio signal according to the extracted features; Setting the estimated values ​​of the noise level and the noise type as estimated values ​​of environmental noise; When receiving a speech signal to be processed, denoising the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain a denoised first speech signal; Based on the voice activity detection module and the environmental noise estimation value, signal differentiation is performed on the first voice signal to obtain a voice signal to be gained and other voice signals, wherein the voice activity detection module determines the voice part and the noise part to be gained of the first voice signal according to the noise level in the environmental noise estimation value; Determine the signal strength of the speech signal to be amplified and the other speech signals according to the envelope detection module, and determine the environmental noisiness based on the signal strength; generating a target gain value based on the environmental noisiness; Based on the target gain value, the first speech signal is gain adjusted by an automatic gain control algorithm to obtain a second speech signal.

2. The method according to claim 1, characterized in that When receiving the speech signal to be processed, the step of performing denoising on the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain the denoised first speech signal includes: When receiving the speech signal to be processed, converting the speech signal to be processed into a frequency domain signal; The frequency domain signal and the environmental noise estimation value are input into the deep neural network, and the frequency domain signal is denoised based on the deep neural network to obtain the first speech signal.

3. The method according to claim 1, characterized in that After the step of performing denoising on the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain a denoised first speech signal when the speech signal to be processed is received, the method further includes: Determine the user's hearing profile and audio preference model; generating audio adjustment parameters based on the hearing characteristic curve and the audio preference model; The first voice signal is audio-adjusted according to the audio adjustment parameter to obtain a third voice signal.

4. The method according to claim 3, characterized in that The step of determining the user's hearing characteristic curve and audio preference model comprises: Determining the audio playback history and feedback data of the user, and training a pre-stored learning model based on the audio playback history, and iteratively optimizing the trained model based on the feedback data to obtain the audio preference model; Acquire hearing test data of the user, and determine the hearing characteristic curve according to the hearing test data.

5. The method according to claim 1, characterized in that After the step of performing denoising on the speech signal to be processed based on the environmental noise estimation value and the deep neural network to obtain a denoised first speech signal when the speech signal to be processed is received, the method further includes: Determining a deviation value between the first speech signal and a predicted audio signal generated by the deep neural network; The deep neural network is iteratively optimized based on the deviation value.

6. A speech signal processing device, characterized in that: The speech signal processing device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the speech signal processing method according to any one of claims 1 to 5.

7. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the speech signal processing method according to any one of claims 1 to 5 are implemented.

Images