Speech enhancement model training method, speech enhancement method, equipment, medium and product

Abstract

The invention discloses a speech enhancement model training method, a speech enhancement method, equipment, a medium and a product, and relates to the technical field of machine learning, and the method comprises the steps: obtaining a training data set and a to-be-trained speech enhancement model, the training data set comprises at least one piece of input data and label data corresponding to each piece of input data, at least one piece of label data in the label data is a mixed signal mixed with voice and noise; and training the speech enhancement model by taking the input data as the input of the speech enhancement model and taking the tag data as the tag of the speech enhancement model to obtain a trained speech enhancement model. According to the invention, the suppression amount of the speech enhancement model to the speech is reduced, and the speech definition and quality are improved.

Description

Technical Field

[0001] The present application relates to the field of machine learning technology, and in particular to a speech enhancement model training method, a speech enhancement method, a device, a medium and a product. Background Art

[0002] Speech Enhancement (SE) refers to the technology of recovering useful speech signals from disturbed speech signals to improve the quality and clarity of speech signals. With the rapid development of communication technology and smart devices, people's demand for clear and high-quality speech is increasing. However, in practical applications, speech signals are often affected by various noises and interferences, which reduces the speech quality and recognition. In order to solve this problem, speech enhancement plays a vital role in the field of digital signal processing.

[0003] Generally speaking, a speech enhancement model can be used for speech enhancement. Therefore, the training of the speech enhancement model is the key to improving the quality of speech noise reduction. The traditional speech enhancement model training method first mixes the clean speech data in the clean speech database and the noise data in the noise database through data simulation to obtain a noisy speech database. Among them, for any noisy speech in the noisy speech database, the noisy speech corresponds to a clean speech in the clean speech database. Then, the speech enhancement model is trained with the noisy speech database as input and the clean speech as the label, and the speech enhancement model is optimized based on the loss value. The above process is iterated until the given training end condition is reached.

[0004] However, the conventional speech enhancement model training method always uses clean speech as the training label, that is, the training goal of the speech enhancement model is always to make the output speech as close to the clean speech as possible. During this training process, the model may mistakenly remove some subtle features of the speech or low-energy speech parts as noise, resulting in excessive suppression of the speech, which further leads to a decrease in the clarity and quality of the speech.

[0005] Therefore, how to reduce the amount of speech suppression by the speech enhancement model to improve the clarity and quality of speech is a technical problem that needs to be solved urgently in this technical field. Summary of the invention

[0006] The main purpose of this application is to provide a speech enhancement model training method, speech enhancement method, device, medium and product, aiming to solve the technical problem of how to reduce the amount of speech suppression by the speech enhancement model to improve the clarity and quality of the speech.

[0007] To achieve the above object, the present application provides a speech enhancement model training method, the speech enhancement model training method comprising the following steps:

[0008] Acquire a training data set and a speech enhancement model to be trained, wherein the training data set includes at least one input data and label data corresponding to each input data, and at least one label data in each label data is a mixed signal of speech and noise;

[0009] The speech enhancement model is trained using each of the input data as the input of the speech enhancement model and each of the label data as the label of the speech enhancement model to obtain the trained speech enhancement model.

[0010] In one embodiment, before the step of obtaining a training data set and a speech enhancement model to be trained, the method further includes:

[0011] Acquire an original data set, wherein the original data set includes a speech signal and a noise signal;

[0012] Mixing the speech signal with the noise signal at at least one first signal-to-noise ratio to obtain input data;

[0013] For each piece of the input data, determining a second signal-to-noise ratio based on the first signal-to-noise ratio of the input data, mixing the speech signal and the noise signal with the second signal-to-noise ratio to obtain label data corresponding to the input data, wherein the second signal-to-noise ratio is greater than or equal to the first signal-to-noise ratio;

[0014] The input data and the label data are combined to obtain a training data set.

[0015] In one embodiment, the step of determining the second signal-to-noise ratio based on the first signal-to-noise ratio of the input data comprises:

[0016] Obtain a mapping relationship table, and search for a second signal-to-noise ratio corresponding to the first signal-to-noise ratio of the input data based on the mapping relationship table; or,

[0017] The first signal-to-noise ratio corresponding to the input data is input into a preset mapping function to obtain a second signal-to-noise ratio.

[0018] In one embodiment, before the step of obtaining the mapping relationship table, the method further includes:

[0019] Inputting at least one preset test data into a preset conventional speech enhancement model respectively to obtain an output result corresponding to each preset test data, wherein the preset conventional speech enhancement model is the speech enhancement model pre-trained with a speech signal as a label;

[0020] Obtaining an input signal-to-noise ratio interval to which the signal-to-noise ratio of each of the preset test data belongs;

[0021] For each of the input signal-to-noise ratio intervals, calculate the average value of the output results corresponding to all preset test data belonging to the input signal-to-noise ratio interval, determine the label signal-to-noise ratio interval based on the average value, and configure a corresponding relationship between the input signal-to-noise ratio interval and the label signal-to-noise ratio interval, wherein the lower limit value of the label signal-to-noise ratio interval is greater than or equal to the average value;

[0022] The corresponding relationships between the input signal-to-noise ratio intervals and the label signal-to-noise ratio intervals are combined to obtain a mapping relationship table.

[0023] In one embodiment, the original data set further includes a room impulse response. After the step of obtaining the original data set, the method further includes:

[0024] Adding the room impulse response to the speech signal to obtain a first speech signal, and adding the room impulse response to the noise signal to obtain a first noise signal;

[0025] The first speech signal is used as a new speech signal, the first noise signal is used as a new noise signal, and based on the new speech signal and the new noise signal, the step of mixing the speech signal and the noise signal at at least one first signal-to-noise ratio to obtain input data is performed.

[0026] In one embodiment, after the step of determining the second signal-to-noise ratio based on the first signal-to-noise ratio of the input data, the method further comprises:

[0027] Obtaining a training target for the speech enhancement model;

[0028] If the training objective indicates dereverberation, early reverberation in the room impulse response is added to the speech signal to obtain a second speech signal, the second speech signal is used as a new speech signal, and based on the new speech signal, the step of mixing the speech signal with the noise signal at the second signal-to-noise ratio to obtain label data corresponding to the input data is performed.

[0029] In addition, to achieve the above purpose, the present application also provides a speech enhancement method, which comprises the following steps:

[0030] Acquire an original signal to be enhanced and a target speech enhancement model, and input the original signal into the target speech enhancement model to obtain a speech enhancement result;

[0031] The target speech enhancement model is a speech enhancement model trained by the speech enhancement model training method as described above.

[0032] In addition, to achieve the above-mentioned purpose, the present application also provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the speech enhancement model training method and / or the speech enhancement method as described above.

[0033] In addition, to achieve the above-mentioned purpose, the present application also provides a readable storage medium, which is a computer-readable storage medium, and a computer program is stored on the computer-readable storage medium. The computer program is executed by a processor to implement the steps of the speech enhancement model training method and / or the speech enhancement method as described above.

[0034] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the above-mentioned speech enhancement model training method and / or speech enhancement method.

[0035] One or more technical solutions proposed in this application have at least the following technical effects:

[0036] Obtain a training data set and a speech enhancement model to be trained, wherein the training data set includes at least one input data and label data corresponding to each input data, and at least one label data in each label data is a mixed signal mixed with speech and noise; train the speech enhancement model with each input data as the input of the speech enhancement model and each label data as the label of the speech enhancement model to obtain the trained speech enhancement model. In this way, the embodiment of the present application uses at least one mixed signal of mixed speech and noise as the training label of the speech enhancement model to train the speech enhancement model, instead of always using clean speech as the training label of the speech enhancement model, so that the training goal of the speech enhancement model can retain some noise on the basis of being as close to clean speech as possible, so as to reduce the situation in which the model mistakenly removes some subtle features of the speech or low-energy speech parts as noise during the training process, thereby reducing the amount of speech suppression by the speech enhancement model and improving the clarity and quality of the speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 This is a flow chart of the first embodiment of the speech enhancement model training method of the present application;

[0040] Figure 2 A schematic diagram of a mapping function curve involved in an embodiment of a speech enhancement model training method of the present application;

[0041] Figure 3 A schematic diagram of a room impulse response involved in an embodiment of a speech enhancement model training method of the present application;

[0042] Figure 4 This is a brief flowchart of a specific embodiment of the speech enhancement model training method of the present application;

[0043] Figure 5 This is a schematic diagram of the device structure of the speech enhancement model training device of the present application;

[0044] Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the speech enhancement model training method device in the embodiment of the present application.

[0045] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0047] With the popularity of various smart devices, sound has gradually become one of the main ways for humans to obtain information. However, most of the sounds that humans receive every day contain various noises. For example, in remote suburbs, the average ambient noise is 30 decibels; in a more noisy environment, the noise level can reach 77 decibels. Therefore, voice interaction tasks in real scenarios face severe challenges.

[0048] Speech enhancement refers to the technology of recovering useful speech signals from disturbed speech signals. Traditional signal processing-based speech enhancement methods regard noise reduction as a simple signal processing problem to establish mathematical and statistical models, and often require artificial addition of prior information. However, in real complex scenarios, the performance of traditional enhancement methods will be greatly attenuated due to the large difference between prior assumptions and real scenarios. In recent years, with the development of deep learning technology, deep neural networks have begun to be widely used in speech enhancement. By using noisy speech-clean speech data pairs for large-scale supervised training, neural networks can learn the rules of information related to clean speech. With the continuous exploration of researchers, many advanced speech enhancement models based on time domain and time-frequency domain have emerged, such as DCCRN (Deep Complex Convolution Recurrent Network), DPCRN (Dual-Path Convolution Recurrent Network), Conv-TasNet (Convolutional Time-domain audio separation Network), etc.

[0049] In the above research, no matter how much noise is in the audio, the prediction target is clean audio, which is often not the optimal target for enhancement models, especially lightweight models. When the audio signal-to-noise ratio is high, the model can get clean audio by removing very little noise; but when the audio signal-to-noise ratio is low, the total amount of noise that needs to be removed may exceed the capabilities of the lightweight model, resulting in the model being unable to converge to the optimal state during training, ultimately resulting in over-suppression of the enhancement result or excessive noise residue. Speech enhancement models often have the problem of over-suppression of speech under conditions of low parameter count and low computing resources.

[0050] Based on this, the main solution of the present application is: obtain a training data set and a speech enhancement model to be trained, wherein the training data set includes at least one input data and label data corresponding to each input data, and there is at least one label data in each label data that is a mixed signal of speech and noise; train the speech enhancement model using each input data as the input of the speech enhancement model and each label data as the label of the speech enhancement model to obtain the trained speech enhancement model.

[0051] The present application trains the speech enhancement model by using at least one mixed signal of mixed speech and noise as the training label of the speech enhancement model, instead of always using clean speech as the training label of the speech enhancement model. This allows the training goal of the speech enhancement model to be as close to clean speech as possible while retaining some noise, so as to reduce the situation in which the model mistakenly removes some subtle features of the speech or low-energy speech parts as noise during the training process, thereby reducing the amount of speech suppression by the speech enhancement model and improving speech clarity and quality.

[0052] It should be noted that the execution subject of each embodiment of the speech enhancement model training method of the present application can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, such as headphones, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, AR helmets, VR helmets, smart glasses, etc. The embodiments of the speech enhancement model training method of the present application do not make specific restrictions on this.

[0053] Based on this, this application proposes a speech enhancement model training method of the first embodiment, please refer to Figure 1 As shown, the speech enhancement model training method includes the following steps S10 to S20:

[0054] Step S10, obtaining a training data set and a speech enhancement model to be trained, wherein the training data set includes at least one input data and label data corresponding to each input data, and at least one label data in each label data is a mixed signal of speech and noise;

[0055] The speech enhancement model may be a pre-built model for speech enhancement. The speech enhancement model may be a deep learning model, such as a DCCRN model, a DPCRN model, a Conv-TasNet model, etc., which is not specifically limited in this embodiment.

[0056] It should be noted that, in this embodiment, each piece of input data and each piece of label data in the training data set can specifically be audio data.

[0057] The training data set includes at least one input data and label data corresponding to each input data. It can be understood that in order to improve the training effect of the model, multiple input data and label data corresponding to each input data can be prepared in advance as the training data set.

[0058] For each piece of input data, the label data corresponding to the input data may specifically be the expected output corresponding to the input data. In layman's terms, after the input data is input into the speech enhancement model, the speech enhancement model is expected to output ideal data.

[0059] Each piece of input data may be a mixed signal of speech and noise. It should be noted that when there are multiple pieces of input data, one or more pieces of input data may be a signal that only includes speech or a signal that only includes noise, but at least one piece of input data should be a mixed signal that mixes speech and noise.

[0060] Furthermore, input data that only includes speech or only includes noise is recorded as pure input data, and input data that is a mixture of speech and noise is recorded as mixed input data. The total number of pure input data is less than the total number of mixed input data, so as to improve the training effect of the model and enable the model to have a certain processing capability for pure speech signals and pure noise signals, thereby further improving the robustness of the model.

[0061] Furthermore, the first signal-to-noise ratio of the input data is less than or equal to the second signal-to-noise ratio of the label data corresponding to the input data, so as to ensure the noise removal effect of the speech enhancement model.

[0062] It should be noted that, in this embodiment, the voice signal (referred to as voice) may specifically refer to a sound signal emitted by humans, which may usually be manifested as voice, music, etc., and the noise signal (referred to as noise) refers to a signal that does not contain useful information and is an undesirable signal, which may usually be manifested as environmental noise, thermal noise, electromagnetic interference noise, etc.

[0063] Step S20, training the speech enhancement model with each of the input data as the input of the speech enhancement model and each of the label data as the label of the speech enhancement model to obtain the trained speech enhancement model.

[0064] After obtaining the training data set, each input data in the training data set is used as the input of the speech enhancement model, and each label data is used as the label of the speech enhancement model. It is easy to understand that for each input data, the label data corresponding to the input data is the label in the model training process.

[0065] The training end condition can be pre-set. During the training process of the speech enhancement model, if the training end condition is met, the training of the speech enhancement model is terminated to obtain a trained speech enhancement model. If the training end condition is not met, the speech enhancement model is continuously iterated and optimized until the training end condition is met. The training end condition can be a condition set in advance, such as reaching a predetermined number of iterations, the loss function value is lower than a predetermined threshold, the computing resources are exhausted, the time limit is reached, the accuracy is greater than a predetermined threshold, etc. This embodiment does not impose specific restrictions on this.

[0066] It should be noted that the trained speech enhancement model can specifically be the speech enhancement model obtained when the training end conditions are met, that is, it can be understood that the model parameters of the trained speech enhancement model are the model parameters obtained when the training is finished; or the trained speech enhancement model can be the speech enhancement model with the best model performance ability in the iterative process. Specifically, the model parameters obtained by training in each round of iterative training can be recorded, and then a set of target model parameters with the best model performance ability can be selected, and the model parameters of the speech enhancement model are set to the target model parameters to obtain the trained speech enhancement model.

[0067] Specifically, known model evaluation methods can be used to evaluate the performance of the model, such as STOI (Short-Time Objective Intelligibility) evaluation, PESQ (Perceptual Evaluation of Speech Quality) evaluation, etc. This implementation does not impose specific restrictions on this and will not describe the specific evaluation process in detail.

[0068] Furthermore, rules for evaluating the optimal model performance capability can be set in advance, such as the maximum value among the model performance capabilities being the optimal model performance capability, and the optimal model performance capability among the model performance capabilities can be determined based on the preset rules to select the model parameters corresponding to the optimal model performance capability as the target model parameters.

[0069] After obtaining the trained speech enhancement model, the trained speech enhancement model can also be deployed to other devices so that other devices can perform speech enhancement processing based on the trained speech enhancement model.

[0070] This embodiment obtains a training data set and a speech enhancement model to be trained, wherein the training data set includes at least one input data and label data corresponding to each input data, and at least one label data in each label data is a mixed signal mixed with speech and noise; the speech enhancement model is trained with each input data as the input of the speech enhancement model and each label data as the label of the speech enhancement model to obtain the trained speech enhancement model. In this way, this embodiment trains the speech enhancement model with at least one mixed signal of mixed speech and noise as the training label of the speech enhancement model, instead of always using clean speech as the training label of the speech enhancement model, so that the training goal of the speech enhancement model can retain some noise on the basis of being as close to clean speech as possible, so as to reduce the situation in which the model mistakenly removes some subtle features of speech or low-energy speech parts as noise during the training process, thereby reducing the amount of speech suppression by the speech enhancement model and improving the clarity and quality of speech.

[0071] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description and will not be described in detail later. On this basis, before the step of obtaining the training data set and the speech enhancement model to be trained, the method further includes:

[0072] Step A10, obtaining an original data set, wherein the original data set includes a speech signal and a noise signal;

[0073] The original data set may be a data set prepared in advance by relevant personnel. It should be noted that the original data set includes a speech signal and a noise signal. Specifically, the original data set may include one or more speech signals and one or more noise signals.

[0074] Furthermore, a speech signal and a noise signal may be randomly selected from the original data set each time, or may be selected from the original data set based on a predetermined rule, and no specific limitation is made here.

[0075] Step A20, mixing the speech signal and the noise signal with at least one first signal-to-noise ratio to obtain input data;

[0076] It should be noted that one speech signal and one noise signal can be selected from the original data set each time to execute step A20 and step A30 until the number of input data obtained meets the preset threshold. The preset threshold can be any positive value set in advance by relevant personnel, and this embodiment does not impose any specific restrictions on this.

[0077] The first signal-to-noise ratio may specifically be one or more randomly generated signal-to-noise ratios. For example, in a specific implementation, the first signal-to-noise ratio may specifically be a signal-to-noise ratio randomly selected from a signal-to-noise ratio interval of [-20, 20].

[0078] Step A30: for each piece of the input data, determine a second signal-to-noise ratio based on the first signal-to-noise ratio of the input data, and mix the speech signal and the noise signal with the second signal-to-noise ratio to obtain label data corresponding to the input data, wherein the second signal-to-noise ratio is greater than or equal to the first signal-to-noise ratio;

[0079] The input data and label data that have a corresponding relationship are obtained by mixing the same speech signal and the same noise signal at different signal-to-noise ratios. For example, a first signal-to-noise ratio A is randomly generated, and a second signal-to-noise ratio A is determined based on the first signal-to-noise ratio A. The speech signal A and the noise signal are mixed at the first signal-to-noise ratio A to obtain input data A. Then, the speech signal A and the noise signal A are mixed at the second signal-to-noise ratio A to obtain label data A. Label data A is the label data corresponding to input data A. This process is repeated until label data corresponding to all input data are obtained.

[0080] It should be noted that the upper limit of the signal-to-noise ratio can also be set. The upper limit represents the signal-to-noise ratio when the signal is all voice. If the signal-to-noise ratio is the upper limit, the voice signal can be used as input data or label data. Specifically, when the first signal-to-noise ratio is the upper limit, the voice signal is used as input data. When the second signal-to-noise ratio is the upper limit, the voice signal is used as label data. Similarly, the lower limit of the signal-to-noise ratio can also be set. The lower limit represents the signal-to-noise ratio when the signal is all noise. If the signal-to-noise ratio is the lower limit, the noise signal can be used as input data or label data. Specifically, when the first signal-to-noise ratio is the lower limit, the noise signal is used as input data. When the second signal-to-noise ratio is the lower limit, the noise signal is used as label data.

[0081] Furthermore, when the first signal-to-noise ratio is an upper limit value, the second signal-to-noise ratio may be equal to the first signal-to-noise ratio, and when the first signal-to-noise ratio is not an upper limit value, the second signal-to-noise ratio is greater than the first signal-to-noise ratio to ensure the noise reduction effect of the speech enhancement model.

[0082] Furthermore, the upper limit and lower limit can be defined in advance based on actual needs. They can be specific values ​​or user-defined strings. For example, the string "MAX" can be defined as the upper limit and the string "MIN" as the lower limit.

[0083] Step A40, combining each of the input data and each of the label data to obtain a training data set.

[0084] In this embodiment, the input data is obtained by mixing the speech signal and the noise signal with a first signal-to-noise ratio, and the label data is obtained by mixing the speech signal and the noise signal with a second signal-to-noise ratio greater than or equal to the first signal-to-noise ratio. In this way, it is easy to understand that the optimization goal of the speech enhancement model is to improve (at least not reduce) the signal-to-noise ratio of the signal, thereby ensuring that the speech enhancement model can reduce the residual noise and achieve a balance between noise suppression and speech retention.

[0085] In a possible implementation, the step of determining the second signal-to-noise ratio based on the first signal-to-noise ratio of the input data includes:

[0086] Step B10, obtaining a mapping relationship table, and searching for a second signal-to-noise ratio corresponding to the first signal-to-noise ratio of the input data based on the mapping relationship table; or,

[0087] The mapping relationship table may specifically be a pre-set mapping table storing signal-to-noise ratio correspondence relationships.

[0088] Furthermore, the mapping relationship table may be a mapping table containing multiple one-to-one corresponding signal-to-noise ratios. However, considering the continuity of the signal-to-noise ratio and the wide value space, storing the corresponding relationship in units of signal-to-noise ratio requires a large amount of data to be stored, which results in the need for more storage space. Based on this, as another implementation, the mapping relationship table may be a mapping table containing multiple one-to-one corresponding signal-to-noise ratio intervals to reduce the amount of data stored. For example, in a specific implementation, the mapping relationship table may be a mapping table as shown in Table 1 below, where the input signal-to-noise ratio interval in Table 1 represents the interval to which the signal-to-noise ratio of the input data belongs, the label signal-to-noise ratio interval represents the interval to which the signal-to-noise ratio of the label data belongs, and "Clean Label" represents the upper limit of the signal-to-noise ratio.

[0089] Input signal-to-noise ratio range Tag signal-to-noise ratio range [-20，-15) [-8，-3) [-15，-10) [-3，2) [-10，-5) [3，8) [-5，0) [10，18) [0，20] Clean Label

[0090] Table 1

[0091] Further, when the mapping relationship table is a mapping table containing multiple one-to-one corresponding signal-to-noise ratio intervals, the target signal-to-noise ratio interval to which the signal-to-noise ratio of the input data belongs may be determined first, and the signal-to-noise ratio interval corresponding to the target signal-to-noise ratio interval in the mapping relationship table is searched (for the convenience of subsequent description, referred to as the selected signal-to-noise ratio interval), and a signal-to-noise ratio is selected from the selected signal-to-noise ratio interval as the second signal-to-noise ratio. For example, in a specific implementation, a signal-to-noise ratio may be randomly selected from the selected signal-to-noise ratio interval as the second signal-to-noise ratio.

[0092] Step B20: inputting the first signal-to-noise ratio corresponding to the input data into a preset mapping function to obtain a second signal-to-noise ratio.

[0093] The preset mapping function is a preset mapping function, and satisfies that the value after mapping is greater than or equal to the value before mapping. In a preferred embodiment, the preset mapping function is f(snr)=30-7e -(snr+20) , where snr represents a mapping variable, and in this embodiment, specifically, is the first signal-to-noise ratio of the input data. The mapping curve of the preset mapping function is as follows: Figure 2 As shown, Figure 2 The horizontal axis represents the first signal-to-noise ratio, and the vertical axis represents the second signal-to-noise ratio.

[0094] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first and second embodiments can be referred to the above description, and will not be described in detail later. On this basis, before the step of obtaining the mapping relationship table, the method further includes:

[0095] Step C10, inputting at least one preset test data into a preset conventional speech enhancement model respectively, and obtaining an output result corresponding to each of the preset test data, wherein the preset conventional speech enhancement model is the speech enhancement model pre-trained with the speech signal as a label;

[0096] The preset conventional model is a speech enhancement model that is pre-trained using speech signals as labels, that is, the preset conventional speech enhancement model is a speech enhancement model that is trained using a conventional model training method.

[0097] The preset test data may be one or more test data prepared in advance. Generally, in order to improve the validity and accuracy of the test results, multiple test data are prepared in advance. For example, in a specific embodiment, 1000 test data with different signal-to-noise ratio intervals are prepared in advance, that is, 1000 preset test data are prepared.

[0098] Each test data is input into the preset conventional speech enhancement model to obtain the output result corresponding to each test data. It should be noted that the output result here can specifically be the signal-to-noise ratio of the output data of the speech enhancement model. For example, after the preset test data A is input into the preset conventional speech enhancement model, the output of the preset conventional speech enhancement model is output data A, and the output result corresponding to the preset test data A is the signal-to-noise ratio of output data A.

[0099] Step C20, obtaining the input signal-to-noise ratio interval to which the signal-to-noise ratio of each preset test data belongs;

[0100] It should be noted that one or more input signal-to-noise ratio intervals may be divided in advance, each input signal-to-noise ratio interval is a non-empty set, and different input signal-to-noise ratio intervals may have no intersection with each other. For example, in a specific implementation, the input signal-to-noise ratio intervals divided in advance are [-20, -15), [-15, -10), [-10, -5), [-5, 0), [0, 5), [5, 10), [10, 20].

[0101] According to the input signal-to-noise ratio interval divided in advance, the input signal-to-noise ratio interval to which the signal-to-noise ratio of each preset prediction data belongs can be obtained. For example, assuming that the signal-to-noise ratio of a preset test data is 5, and the input signal-to-noise ratio interval is as shown in the above specific implementation, the input signal-to-noise ratio interval to which the preset test data belongs is [5, 10).

[0102] Step C30, for each of the input signal-to-noise ratio intervals, calculating the average value of the output results corresponding to all preset test data belonging to the input signal-to-noise ratio interval, determining the label signal-to-noise ratio interval based on the average value, and configuring a corresponding relationship between the input signal-to-noise ratio interval and the label signal-to-noise ratio interval, wherein the lower limit value of the label signal-to-noise ratio interval is greater than or equal to the average value;

[0103] For each input signal-to-noise ratio interval, calculate the average value of the output results corresponding to all preset test data belonging to this input signal-to-noise ratio interval. The input signal-to-noise ratio interval to which the preset test data belongs is also the input signal-to-noise ratio interval to which the preset test data belongs. For example, for a certain input signal-to-noise ratio interval 1, the predicted test data whose signal-to-noise ratio belongs to this input signal-to-noise ratio interval are [preset test data A, predicted test data B, preset test data C], and the corresponding output results are [output result A, output result B, output result C], respectively. Then the average value is (output result A+output result B+output result C) / 3.

[0104] The tag signal-to-noise ratio interval is determined based on the average value, and specifically, the lower limit of the tag signal-to-noise ratio interval is greater than or equal to the average value. For example, in a specific implementation, as shown in Table 2 below, the tag signal-to-noise ratio interval is an interval with a fluctuation of 1 decibel around the tag average value after an increase of 10 decibels.

[0105]

[0106] Table 2

[0107] Step C40: combining the corresponding relationships between the input signal-to-noise ratio intervals and the label signal-to-noise ratio intervals to obtain a mapping relationship table.

[0108] The mapping relationship table stores the corresponding relationship between the input signal-to-noise ratio interval and the label signal-to-noise ratio interval. For example, assuming that the input signal-to-noise ratio interval and the label signal-to-noise ratio interval are as shown in Table 2 above, the mapping relationship table is as shown in Table 3.

[0109]

[0110]

[0111] Table 3

[0112] In this embodiment, the preset test data is input into the preset conventional speech enhancement model to obtain the output result, and the label signal-to-noise ratio interval is determined based on the average value of the output results of each preset test data. In this way, the label signal-to-noise ratio interval is set based on the output result of the conventional training method, and the lower limit value of the label signal-to-noise ratio interval is greater than the average value of the output result, which ensures that the signal-to-noise ratio selected from the label signal-to-noise ratio interval subsequently is greater than the average value, that is, the speech enhancement model obtained by the subsequent training of the label signal-to-noise ratio interval has a better noise reduction effect.

[0113] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment, the second embodiment and the third embodiment can be referred to the above introduction, and will not be repeated in detail later. On this basis, the original data set also includes a room impulse response, and after the step of obtaining the original data set, the method further includes:

[0114] Step D10, adding the room impulse response to the speech signal to obtain a first speech signal, and adding the room impulse response to the noise signal to obtain a first noise signal;

[0115] Room Impulse Response (RIR) is a term used in acoustics to describe the interaction between sound waves and surfaces such as walls and furniture in a closed space (such as a room) when the sound propagates in the room. It reflects the process of multiple reflections, interference and attenuation of the sound in the room after the sound is emitted from the sound source. Specifically, the room impulse response can be a time domain function, which describes the change of the sound pressure received at any position in the room over time when the sound source emits a very short pulse sound.

[0116] It can be understood that the room impulse response may be prepared in advance. For example, relevant personnel may prepare the room impulse response based on a public room impulse response data set.

[0117] The room impulse response is added to the signal (including the speech signal and the noise signal) by performing convolution calculation on the room impulse response and the signal.

[0118] It should be noted that part or all of the speech signals and noise signals in the original data set may be selected to execute step D10, that is, part or all of the signals may be added with room impulse responses. The specific number of speech signals and noise signals to be added with room impulse responses may be set by relevant personnel based on actual needs, and this embodiment does not impose any specific restrictions on this.

[0119] Furthermore, similar to speech signals and noise signals, the original data set may include one or more groups of room impulse responses. For speech signals and noise signals to which room impulse responses are to be added, a group of room impulse responses can be selected from the original data set each time and added to a pair of speech signals and noise signals, and the speech signal is added with the first room impulse response in the group of room impulse responses (hereinafter referred to as the first one), and the noise signal is added with the second room impulse response in the group of room impulse responses (hereinafter referred to as the second one). For example, assuming that the original data set is [speech signal A, speech signal B, speech signal C, noise signal A, noise signal B, noise signal C, a first group of room impulse responses, a second group of room impulse responses], and room impulse responses are added to all speech signals and noise signals, then in a specific implementation, the first item in the first group of room impulse responses can be added to the speech signal A, and the second item in the first group of room impulse responses can be added to the noise signal A; the first item in the first group of room impulse responses can be added to the speech signal B, and the second item in the first group of room impulse responses can be added to the noise signal B; the first item in the second group of room impulse responses can be added to the speech signal C, and the second item in the second group of room impulse responses can be added to the noise signal C.

[0120] Reference Figure 3 As shown, each group of room impulse responses simulates the same scene (such as Figure 3 The office scene or the lecture hall scene shown in the figure), and the position of the microphone in the scene is fixed; but the sound source positions of the two scenes are different, and the two sound sources are used to simulate the speech sound source and the noise sound source respectively. Therefore, each set of room impulse responses is composed of two different room impulse responses, among which the first RIR.1 describes the sound source of the speech when it emits a very short pulse sound, at any position in the room ( Figure 3 The figure shows the change of the sound pressure received at the microphone position over time. The second RIR.2 describes the change of the sound pressure at any position in the room (at the microphone position) when the noise source emits a very short pulse sound. Figure 3 Shown in FIG. 1 is the variation of the sound pressure received at the microphone position over time.

[0121] Step D20, taking the first speech signal as a new speech signal, taking the first noise signal as a new noise signal, and performing the step of mixing the speech signal and the noise signal with at least one first signal-to-noise ratio to obtain input data based on the new speech signal and the new noise signal.

[0122] In this embodiment, the room impulse response is added to the speech signal and the noise signal, that is, the reverberation effect is added to the speech signal and the noise signal. It can be understood that the signals in the real environment usually have a reverberation effect. In this way, a training data set is subsequently constructed based on the signal with the reverberation effect added, so that the signals in the training data set are closer to the signals in the real environment, thereby further improving the training effect of the speech enhancement model.

[0123] In a possible implementation manner, after the step of determining the second signal-to-noise ratio based on the first signal-to-noise ratio of the input data, the method further includes:

[0124] Step E10, obtaining a training target of the speech enhancement model;

[0125] It should be noted that the training objectives of the speech enhancement model can be set in advance by relevant personnel based on actual needs, and specifically can be two objectives: dereverberation or non-dereverberation. Dedereverberation is used to indicate that the reverberation in the signal needs to be removed, and non-dereverberation is used to indicate that the reverberation in the signal does not need to be removed.

[0126] Step E20, if the training objective indicates dereverberation, the early reverberation in the room impulse response is added to the speech signal to obtain a second speech signal, the second speech signal is used as a new speech signal, and based on the new speech signal, the step of mixing the speech signal with the noise signal at the second signal-to-noise ratio to obtain label data corresponding to the input data is performed.

[0127] It should be noted that step E20 may be performed only for the speech signal to which the room impulse response is added.

[0128] If the training goal indicates reverberation, the early reverberation in the room impulse response is added to the speech signal. The early reverberation is the part before the index corresponding to the element with the largest value in the room impulse response. The early reverberation is the front part of the room impulse response very shortly after the direct sound (usually 10ms). There is no sense of reverberation in the auditory sense. Adding early reverberation to the speech signal can make the second speech signal have the same delay as the first speech signal and the first noise signal, and then make the input data obtained by subsequent mixing have the same delay as the label data, that is, align the label data and the input data in the time dimension, so as to facilitate the learning of the speech enhancement model.

[0129] It should be noted that the training target indicates reverberation. When constructing label data based on the second speech signal, if the input data is constructed by adding the first noise data of the room impulse response, the second speech signal and the first noise signal are mixed to obtain label data corresponding to the input data. For example, assuming that the original data set is [speech signal A, speech signal B, speech signal C, noise signal A, noise signal B, noise signal C, first group of room impulse response, second group of room impulse response], the first group of room impulse response is added to the speech signal A and the noise signal A to obtain the first speech signal A and the first noise signal A, the speech signal B and the noise signal B are not added with room impulse response, the first speech signal A and the first noise signal A are mixed to obtain input data A, and the speech signal B and the noise signal B are mixed to obtain input data B. If the training target indicates reverberation, the early reverberation of the first group of room impulse response is added to the speech signal A to obtain the second speech signal A, the second speech signal A and the first noise signal A are mixed to obtain label data A corresponding to the input data A, and the speech signal B and the noise signal B are mixed to obtain label data B corresponding to the input data B.

[0130] Furthermore, if the training objective indicates that reverberation is not to be removed, the first speech signal and the first noise signal are mixed to obtain label data to retain the reverberation effect of the signal. For example, assuming that the original data set is [speech signal A, speech signal B, speech signal C, noise signal A, noise signal B, noise signal C, first group of room impulse responses, second group of room impulse responses], the second group of room impulse responses are added to the speech signal C and the noise signal C to obtain the first speech signal C and the first noise signal C, and the first speech signal C and the second noise signal C are mixed to obtain input data C. If the training objective indicates that reverberation is not to be removed, the first speech signal C and the second noise signal C are mixed to obtain label data C corresponding to the input data C.

[0131] In this embodiment, when the training goal indicates dereverberation, the early reverberation in the room impulse response is added to the speech signal to obtain a second speech signal, and label data is constructed based on the second speech signal, thereby ensuring that the trained speech enhancement model has a dereverberation effect.

[0132] For example, to help understand the technical concept or technical principle of the speech enhancement model training method after combining this embodiment with the first embodiment, the second embodiment and the third embodiment, please refer to Figure 4 As shown, the training method of the speech enhancement model includes:

[0133] 1. Randomly select data. From the speech data of the original data set ( Figure 4 A clean human voice (i.e., speech signal) is randomly selected from the speech data set shown in , and the noise data ( Figure 4A noise signal is randomly selected from the noise data set shown in the figure, and the room impulse response data ( Figure 4 A set of room impulse responses rir_list is randomly selected from the room impulse response data set shown in FIG.

[0134] 2. Generate signal-to-noise ratio (including input signal-to-noise ratio and label signal-to-noise ratio). There are three signal-to-noise ratio generation methods:

[0135] Inter-partition mapping: After randomly generating the first signal-to-noise ratio mix_snr from the signal-to-noise ratio range [-20, 20], according to the interval where mix_snr is located, randomly sample a value from the corresponding interval (based on the interval determined in Table 1) as the second signal-to-noise ratio label_snr.

[0136] Statistical mapping: After randomly generating the input signal-to-noise ratio mix_snr from the signal-to-noise ratio interval [-20, 20], according to the interval where mix_snr is located, randomly sample a value from the corresponding interval (based on the interval determined in Table 3) as the second signal-to-noise ratio label_snr.

[0137] Function mapping: randomly generate a first signal-to-noise ratio mix_snr from the signal-to-noise ratio interval [-20, 20], and calculate the second signal-to-noise ratio through the mapping function f(snr)=30-7*exp(-(snr+20)).

[0138] Figure 4 Only the method of generating the signal-to-noise ratio based on the function mapping is shown.

[0139] 3. Add reverberation as needed, that is, add room impulse response as needed. First, add the second rir in the room impulse response rir_list to the noise noise to obtain the first noise signal reverb_noise. In the case where the model needs to have dereverberation capability, add the first rir in rir_list to clean to obtain the first speech signal reverb_clean, the first noise signal reverb_noise and the first speech signal reverb_clean are mixed at a certain first signal-to-noise ratio to obtain the input data mix; clean plus early reverberation obtains the second speech signal, the second speech signal and the first noise signal reverb_noise are mixed at a certain second signal-to-noise ratio to obtain the label data label.

[0140] In the case where the model does not need to have dereverberation capabilities ( Figure 4(not shown), the first speech signal reverb_clean and the first noise signal reverb_noise are mixed at a certain first signal-to-noise ratio to obtain input data mix, and the first speech signal reverb_clean and the first noise signal reverb_noise are mixed at a certain second signal-to-noise ratio to obtain label data label.

[0141] 4. Train the model, repeat steps 1 to 3 until the amount of input data and label data obtained meets certain requirements, and then use the input data as input and the label data as labels to train the speech enhancement model.

[0142] It should be noted that the above examples are only used to assist in understanding the present application and do not constitute a limitation on the speech enhancement model training method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0143] In addition, the present invention also provides a method for speech enhancement, which comprises the following steps:

[0144] Acquire an original signal to be enhanced and a target speech enhancement model, and input the original signal into the target speech enhancement model to obtain a speech enhancement result;

[0145] The target speech enhancement model is a speech enhancement model trained by the speech enhancement model training method described in any of the above embodiments.

[0146] The execution subject of each embodiment of the speech enhancement training method can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device that can realize the above functions, such as headphones, AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, AR helmets, VR helmets, smart glasses, etc. This embodiment does not make specific restrictions on this. Furthermore, the execution subjects of the speech enhancement method of the present application and the speech enhancement model training method may be the same or different, and no specific restrictions are made here.

[0147] It is easy to understand that after obtaining the original signal to be enhanced, the original signal is input into the target speech enhancement model to obtain a speech enhancement result, which can specifically be the output of the target speech enhancement model.

[0148] In addition, the present application also provides a speech enhancement model training device, referring to Figure 5 As shown, the speech enhancement model training device includes:

[0149] An acquisition module 10 is used to acquire a training data set and a speech enhancement model to be trained, wherein the training data set includes at least one input data and label data corresponding to each input data, and at least one label data in each label data is a mixed signal of speech and noise;

[0150] The training module 20 is used to train the speech enhancement model using each of the input data as the input of the speech enhancement model and each of the label data as the label of the speech enhancement model to obtain the trained speech enhancement model.

[0151] In one embodiment, the speech enhancement model training device further includes a data preparation module, and the data preparation module is used to:

[0152] Acquire an original data set, wherein the original data set includes a speech signal and a noise signal;

[0153] Mixing the speech signal with the noise signal at at least one first signal-to-noise ratio to obtain input data;

[0154] For each piece of the input data, determining a second signal-to-noise ratio based on the first signal-to-noise ratio of the input data, mixing the speech signal and the noise signal with the second signal-to-noise ratio to obtain label data corresponding to the input data, wherein the second signal-to-noise ratio is greater than or equal to the first signal-to-noise ratio;

[0155] The input data and the label data are combined to obtain a training data set.

[0156] In one embodiment, the data preparation module is further used to:

[0157] Obtain a mapping relationship table, and search for a second signal-to-noise ratio corresponding to the first signal-to-noise ratio of the input data based on the mapping relationship table; or,

[0158] The first signal-to-noise ratio corresponding to the input data is input into a preset mapping function to obtain a second signal-to-noise ratio.

[0159] In one embodiment, the data preparation module is further used to:

[0160] Inputting at least one preset test data into a preset conventional speech enhancement model respectively to obtain an output result corresponding to each preset test data, wherein the preset conventional speech enhancement model is the speech enhancement model pre-trained with a speech signal as a label;

[0161] Obtaining an input signal-to-noise ratio interval to which the signal-to-noise ratio of each of the preset test data belongs;

[0162] For each of the input signal-to-noise ratio intervals, calculate the average value of the output results corresponding to all preset test data belonging to the input signal-to-noise ratio interval, determine the label signal-to-noise ratio interval based on the average value, and configure a corresponding relationship between the input signal-to-noise ratio interval and the label signal-to-noise ratio interval, wherein the lower limit value of the label signal-to-noise ratio interval is greater than or equal to the average value;

[0163] The corresponding relationships between the input signal-to-noise ratio intervals and the label signal-to-noise ratio intervals are combined to obtain a mapping relationship table.

[0164] In one embodiment, the data preparation module is further used to:

[0165] Adding the room impulse response to the speech signal to obtain a first speech signal, and adding the room impulse response to the noise signal to obtain a first noise signal;

[0166] The first speech signal is used as a new speech signal, and the first noise signal is used as a new noise signal.

[0167] In one embodiment, the data preparation module is further used to:

[0168] Obtaining a training target for the speech enhancement model;

[0169] If the training objective indicates reverberation, the early reverberation in the room impulse response is added to the speech signal to obtain a second speech signal, and the second speech signal is used as a new speech signal.

[0170] In addition, an embodiment of the present application also proposes an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the speech enhancement model training method and / or the speech enhancement method as described above.

[0171] refer to Figure 6 , which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic devices in the embodiments of the present application may also include, but are not limited to, mobile terminals such as headphones, AR glasses, VR glasses, AR helmets, VR helmets, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0172] like Figure 6 As shown, the electronic device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows an electronic device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0173] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0174] The electronic device provided in the embodiment of the present application adopts the speech enhancement model training method in the above embodiment, which can solve the technical problem of how to reduce the amount of suppression of speech by the speech enhancement model to improve the clarity and quality of speech. Compared with the prior art, the beneficial effects of the electronic device provided in the present application are the same as the beneficial effects of the speech enhancement model training method provided in the above embodiment, and the other technical features in the electronic device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0175] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0176] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0177] In addition, to achieve the above-mentioned purpose, an embodiment of the present application also provides a readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the speech enhancement model training method in the above-mentioned embodiment.

[0178] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0179] The computer-readable storage medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0180] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by an electronic device, the electronic device: obtains a training data set and a speech enhancement model to be trained, wherein the training data set includes at least one input data and label data corresponding to each of the input data, and at least one label data in each of the label data is a mixed signal of speech and noise; trains the speech enhancement model with each of the input data as the input of the speech enhancement model and each of the label data as the label of the speech enhancement model to obtain the trained speech enhancement model.

[0181] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0182] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0183] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of a module does not, in some cases, constitute a limitation on the module itself.

[0184] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned speech enhancement model training method and / or speech enhancement method, and can solve the technical problem of how to reduce the amount of speech suppression by the speech enhancement model to improve the clarity and quality of speech. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the speech enhancement model training method provided in the above-mentioned embodiment, and will not be repeated here.

[0185] In addition, an embodiment of the present application also proposes a computer program product, including a computer program, which, when executed by a processor, implements the steps of the speech enhancement model training method and / or the speech enhancement method as described above.

[0186] The specific implementation of the computer program product of the present application is basically the same as the above-mentioned speech enhancement model training and / or speech enhancement method embodiments, and will not be repeated here.

[0187] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0188] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0189] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software sensor, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, including a number of instructions for a terminal device (which can be a mobile phone, computer, server or network device, etc.) to execute the methods described in each embodiment of the present application.

[0190] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A speech enhancement model training method, characterized in that: The speech enhancement model training method comprises the following steps: Acquire a training data set and a speech enhancement model to be trained, wherein the training data set includes at least one input data and label data corresponding to each input data, and at least one label data in each label data is a mixed signal of speech and noise; The speech enhancement model is trained using each of the input data as the input of the speech enhancement model and each of the label data as the label of the speech enhancement model to obtain the trained speech enhancement model.

2. The speech enhancement model training method according to claim 1, characterized in that: Before the step of obtaining the training data set and the speech enhancement model to be trained, the method further includes: Acquire an original data set, wherein the original data set includes a speech signal and a noise signal; Mixing the speech signal and the noise signal at at least one first signal-to-noise ratio to obtain input data; For each piece of the input data, determining a second signal-to-noise ratio based on the first signal-to-noise ratio of the input data, mixing the speech signal and the noise signal with the second signal-to-noise ratio to obtain label data corresponding to the input data, wherein the second signal-to-noise ratio is greater than or equal to the first signal-to-noise ratio; The input data and the label data are combined to obtain a training data set.

3. The speech enhancement model training method according to claim 2, characterized in that: The step of determining the second signal-to-noise ratio based on the first signal-to-noise ratio of the input data comprises: Obtain a mapping relationship table, and search for a second signal-to-noise ratio corresponding to the first signal-to-noise ratio of the input data based on the mapping relationship table; or, The first signal-to-noise ratio corresponding to the input data is input into a preset mapping function to obtain a second signal-to-noise ratio.

4. The speech enhancement model training method according to claim 3, characterized in that: Before the step of obtaining the mapping relationship table, the method further includes: Inputting at least one preset test data into a preset conventional speech enhancement model respectively to obtain an output result corresponding to each preset test data, wherein the preset conventional speech enhancement model is the speech enhancement model pre-trained with a speech signal as a label; Obtaining an input signal-to-noise ratio interval to which the signal-to-noise ratio of each of the preset test data belongs; For each of the input signal-to-noise ratio intervals, calculate the average value of the output results corresponding to all preset test data belonging to the input signal-to-noise ratio interval, determine the label signal-to-noise ratio interval based on the average value, and configure a corresponding relationship between the input signal-to-noise ratio interval and the label signal-to-noise ratio interval, wherein the lower limit value of the label signal-to-noise ratio interval is greater than or equal to the average value; The corresponding relationships between the input signal-to-noise ratio intervals and the label signal-to-noise ratio intervals are combined to obtain a mapping relationship table.

5. The speech enhancement model training method according to claim 2, characterized in that: The original data set also includes a room impulse response. After the step of obtaining the original data set, the method further includes: Adding the room impulse response to the speech signal to obtain a first speech signal, and adding the room impulse response to the noise signal to obtain a first noise signal; The first speech signal is used as a new speech signal, the first noise signal is used as a new noise signal, and based on the new speech signal and the new noise signal, the step of mixing the speech signal and the noise signal at at least one first signal-to-noise ratio to obtain input data is performed.

6. The speech enhancement model training method according to claim 5, characterized in that: After the step of determining a second signal-to-noise ratio based on the first signal-to-noise ratio of the input data, the method further comprises: Obtaining a training target for the speech enhancement model; If the training objective indicates dereverberation, early reverberation in the room impulse response is added to the speech signal to obtain a second speech signal, the second speech signal is used as a new speech signal, and based on the new speech signal, the step of mixing the speech signal with the noise signal at the second signal-to-noise ratio to obtain label data corresponding to the input data is performed.

7. A speech enhancement method, characterized in that: The speech enhancement method comprises the following steps: Acquire an original signal to be enhanced and a target speech enhancement model, and input the original signal into the target speech enhancement model to obtain a speech enhancement result; The target speech enhancement model is a speech enhancement model trained by the speech enhancement model training method according to any one of claims 1 to 6.

8. An electronic device, characterized in that: The electronic device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, which implements the steps of the method according to any one of claims 1 to 7 when executed by a processor.

Images