Model training method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a model training method and device, electronic equipment and a readable storage medium, and belongs to the field of artificial intelligence. The method comprises the following steps: inputting an audio signal sample into a first noise reduction model, carrying out noise reduction processing on the audio signal sample, and outputting a noise-reduced audio signal; determining a first loss value based on a first signal envelope of the audio signal sample and a second signal envelope of the noise-reduced audio signal, the first loss value being used to characterize a correlation between the first signal envelope and the second signal envelope; based on the Euclidean distance value between the first signal envelope and the second signal envelope, a second loss value is determined, and the second loss value is used for representing the difference between the first signal envelope and the second signal envelope; based on the first loss value and the second loss value, training a first noise reduction model to obtain a second noise reduction model; wherein the first signal envelope is at least used for representing the timbre characteristics of the audio signal samples, and the second signal envelope is at least used for representing the timbre characteristics of the noise reduction audio signals.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence, and specifically relates to a model training method, device, electronic device and readable storage medium. Background Art

[0002] With the rapid development of economic globalization and the Internet, machine translation technology can not only translate text, but also translate audio.

[0003] Usually, when translating audio, the electronic device can first perform noise reduction on the audio, and then translate the audio by replicating the user's timbre in the noise-reduced audio, thereby obtaining a more realistic translated audio. Since there may be a lot of noise in the audio, electronic devices usually use a general noise reduction model for noise reduction. If the electronic device believes that the noise reduction effect of the noise reduction model is not good, it will use a model training method to improve the noise reduction effect of the noise reduction model.

[0004] In related technologies, the noise reduction model is generally trained using basic training loss functions such as the error method, the cross entropy loss method, and the contrast loss method. However, no matter which training loss function is used, the trained model will treat part of the user's voice in the audio as noise for noise reduction, resulting in poor results of the trained model. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a model training method, device, electronic device and readable storage medium, which can improve the effect of the trained model.

[0006] In a first aspect, an embodiment of the present application provides a model training method, which includes: inputting an audio signal sample into a first noise reduction model, performing noise reduction processing on the audio signal sample, and outputting a noise-reduced audio signal; determining a first loss value based on a first signal envelope of the audio signal sample and a second signal envelope of the noise-reduced audio signal, the first loss value being used to characterize the correlation between the first signal envelope and the second signal envelope; determining a second loss value based on a Euclidean distance value between the first signal envelope and the second signal envelope, the second loss value being used to characterize the difference between the first signal envelope and the second signal envelope; training a first noise reduction model based on the first loss value and the second loss value to obtain a second noise reduction model; wherein the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal.

[0007] In a second aspect, an embodiment of the present application provides a model training device, which includes: a processing module and a training module; the processing module is used to input an audio signal sample into a first noise reduction model, perform noise reduction processing on the audio signal sample, and output a noise-reduced audio signal; the processing module is also used to determine a first loss value based on a first signal envelope of the audio signal sample and a second signal envelope of the noise-reduced audio signal, and the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; the processing module is also used to determine a second loss value based on a Euclidean distance value between the first signal envelope and the second signal envelope, and the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; the training module is used to train the first noise reduction model based on the first loss value determined by the above-mentioned processing module and the second loss value determined by the above-mentioned processing module to obtain a second noise reduction model; wherein the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal.

[0008] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0009] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0011] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0012] In the model training method provided in the embodiment of the present application, the electronic device inputs an audio signal sample into a first noise reduction model, performs noise reduction processing on the audio signal sample, and outputs a noise-reduced audio signal; based on the first signal envelope of the audio signal sample and the second signal envelope of the noise-reduced audio signal, a first loss value is determined, and the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; based on the Euclidean distance value between the first signal envelope and the second signal envelope, a second loss value is determined, and the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; based on the first loss value and the second loss value, the first noise reduction model is trained to obtain the second noise reduction model; wherein the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal. In the present scheme, since the element that has the greatest and most intuitive impact on the timbre is the signal envelope of the audio signal, the electronic device calculates the first loss value and the second loss value by combining the correlation and difference between the signal envelope of the audio signal sample and the signal envelope of the noise reduction audio signal, and trains the model part of the noise reduction model that identifies the signal envelope according to the first loss value and the second loss value, so that the trained noise reduction model can accurately identify the signal envelope of the audio signal during the process of denoising the audio signal, thereby protecting the audio signal part within the signal envelope that reflects the user's timbre, thereby improving the effect of the trained model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flowchart of a model training method provided in an embodiment of the present application;

[0014] Figure 2 is a schematic diagram of a signal envelope of an audio signal provided in an embodiment of the present application;

[0015] Figure 3 1 is a schematic diagram of a volume value of an audio signal for automatic gain adjustment provided in an embodiment of the present application;

[0016] Figure 4 This is one of the example schematic diagrams of a mobile phone interface provided in an embodiment of the present application;

[0017] Figure 5 This is one of the example schematic diagrams of a mobile phone interface provided in an embodiment of the present application;

[0018] Figure 6 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0019] Figure 7 It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0020] Figure 8It is a structural schematic diagram of a model training device provided in an embodiment of the present application;

[0021] Fig. 9 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0022] Fig.10 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0024] Some terms and nouns involved in the embodiments of the present application are explained below.

[0025] Timbre: Timbre refers to the sensory characteristics of sound. Timbre is one of the properties of sound. For example, the properties of sound can include loudness, pitch, or timbre. It reflects the unique quality of the sound produced by each object. The difference in each person's voice and the sound produced by various instruments is caused by different timbres.

[0026] Envelope: Envelope refers to the outline or contour characteristics of a speech signal, which describes the amplitude and frequency characteristics of a sound signal over time. Envelopes are often used to describe the spectral characteristics of noise, speech, or other sounds, usually including the formant and harmonic structure of a speech signal.

[0027] Automatic Gain Control: By controlling the gain, it can ensure that the output signal has a consistent amplitude at different input levels, aiming to automatically adjust the gain level of the signal.

[0028] Sound replication: A sound model with a similar sound style to the user is generated by learning and training a large amount of data through an artificial intelligence (AI) model.

[0029] Least Mean Square (LMS): Least Mean Square is an adaptive filtering algorithm based on gradient descent. Its main goal is to optimize the filter coefficients by minimizing the mean square of the error. In short, the purpose of the LMS algorithm is to minimize the error between the filter output and the desired output.

[0030] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0031] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0032] The logo in this application refers to text, symbols, images, etc. used to indicate information. Logos or other containers can be used as carriers to display information, including but not limited to text logos, image logos, symbol logos, etc.

[0033] It should be noted that the model training method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the model training method provided in the embodiments of the present application is illustrated by taking the electronic device as the execution subject to execute the model training method.

[0034] With the development of globalization, exchanges and cooperation between countries are becoming more and more frequent. Through translation, people can better understand the views, values ​​and ways of thinking of other cultures, and promote more harmonious relations between countries in the world. Machine translation technology can convert one natural language into another natural language. It is a branch of computational linguistics and one of the ultimate goals of artificial intelligence. It has important scientific research value.

[0035] At the same time, machine translation has important practical value. With the rapid development of economic globalization and the Internet, machine translation technology plays an increasingly important role in promoting political, economic, and cultural exchanges. Voice replication is to learn and train a large amount of data through AI models to generate a voice model similar to the user's timbre and pronunciation style. This technology can quickly "replicate" personalized voices, make up for the shortcomings of traditional speech synthesis technology in digitizing human voices, and generate voices with more realistic textures.

[0036] When we have meetings with foreign colleagues or communicate with foreigners, we may need to use translation applications due to language barriers. Usually, conventional translation applications can only convert speech into text, or read out text with a mechanical voice, or record the user's own voice and read out the translated text in combination with the user's voice.

[0037] However, in the process of electronic devices combining the translated text with the user's voice to broadcast, there are still the following problems:

[0038] 1. There may be noise in the process of recording the user's voice. To avoid noise in the audio synthesized by the model, the recorded audio needs to be denoised. Since in related technologies, a simple noise reduction method is usually selected to reduce the noise of the audio, it may affect the user's own voice.

[0039] 2. The volume of the translated report can only be set in advance. If the noise level in the current environment is too high, the preset volume may be too low, making it difficult for users to hear the report clearly. Therefore, it is necessary to dynamically adjust the volume of the report according to the actual environmental noise. For example, the volume requirements on a noisy road and a quiet meeting room are naturally different.

[0040] 3. The tone and emotion of the broadcast should be consistent with the speaker's tone to create a better experience of interpersonal communication.

[0041] In this regard, in the model training method provided in the embodiment of the present application, the electronic device inputs an audio signal sample into a first noise reduction model, performs noise reduction on the audio signal sample, and outputs a noise-reduced audio signal; based on the first signal envelope of the audio signal sample and the second signal envelope of the noise-reduced audio signal, a first loss value is determined, and the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; based on the Euclidean distance value between the first signal envelope and the second signal envelope, a second loss value is determined, and the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; based on the first loss value and the second loss value, the first noise reduction model is trained to obtain a second noise reduction model; wherein the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal. In the present scheme, since the element that has the greatest and most intuitive impact on the timbre is the signal envelope of the audio signal, the electronic device calculates the first loss value and the second loss value by combining the correlation and difference between the signal envelope of the audio signal sample and the signal envelope of the noise reduction audio signal, and trains the model part of the noise reduction model that identifies the signal envelope according to the first loss value and the second loss value, so that the trained noise reduction model can accurately identify the signal envelope of the audio signal during the process of denoising the audio signal, thereby protecting the audio signal part within the signal envelope that reflects the user's timbre, thereby improving the effect of the trained model.

[0042] The execution subject of the model training method provided in the embodiment of the present application may be a model training device. Exemplarily, the model training device may be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The model training method provided in the embodiment of the present application will be exemplarily described below using an electronic device as an example.

[0043] The present application embodiment provides a model training method. Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown, and the method can be applied to electronic devices. Figure 1 As shown, the model training method provided in the embodiment of the present application may include the following steps 201 to 204.

[0044] Step 201: The electronic device inputs an audio signal sample into a first noise reduction model, performs noise reduction processing on the audio signal sample, and outputs a noise-reduced audio signal.

[0045] In some embodiments of the present application, the audio signal sample includes a speech signal.

[0046] Exemplarily, the audio signal sample may be audio recorded by a user through an electronic device, or audio with human voice stored in the electronic device.

[0047] Exemplarily, the electronic device may record an audio signal of the user's voice input through a translation application or a recording application.

[0048] It is understandable that the above-mentioned translation application or recording application can be an application that comes with the electronic device system, or it can be a third-party application, such as "Smart Translator".

[0049] In some embodiments of the present application, the first noise reduction model is used to perform noise reduction processing on the noise part of the audio signal.

[0050] Exemplarily, the first denoising model may be a denoising model based on a deep neural network (DNN), a convolutional neural network (CNN), or a recurrent neural network (RNN).

[0051] In some embodiments of the present application, the first noise reduction model may be set in a cloud server of the electronic device, or may be set in an application installed in the electronic device.

[0052] In some embodiments of the present application, the electronic device performs noise reduction processing on the entire audio signal sample through a first noise reduction model, and outputs the above-mentioned noise-reduced audio signal, that is, the noise-reduced audio signal sample.

[0053] Step 202: The electronic device determines a first loss value based on a first signal envelope of the audio signal sample and a second signal envelope of the noise reduction audio signal.

[0054] In some embodiments of the present application, the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal.

[0055] In some embodiments of the present application, the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope.

[0056] In a possible embodiment, the electronic device may determine the first loss value according to the similarity between the first signal envelope and the second signal envelope.

[0057] In one example, the electronic device determines the first loss value by calculating a correlation coefficient between the first signal envelope and the second signal envelope.

[0058] Step 203: The electronic device determines a second loss value based on the Euclidean distance value between the first signal envelope and the second signal envelope.

[0059] In some embodiments of the present application, the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope.

[0060] In a possible embodiment, the electronic device may determine the second loss value according to a difference value between the first signal envelope and the second signal envelope.

[0061] In one example, the electronic device may determine the second loss value according to an average value of the Euclidean distance between the first signal envelope and the second signal envelope.

[0062] Step 204: The electronic device trains the first noise reduction model based on the first loss value and the second loss value to obtain a second noise reduction model.

[0063] In some embodiments of the present application, when the first signal envelope and the second signal envelope are different, the electronic device continuously adjusts the model parameters of the first noise reduction model based on the first loss value and the second loss value until the first signal envelope and the second signal envelope are the same, and the electronic device determines the noise reduction model after adjusting the model parameters as the second noise reduction model.

[0064] In one example, the electronic device may train the first noise reduction model based on the sum of the first loss value and the second loss value.

[0065] In some embodiments of the present application, the electronic device may use the first formula to obtain the sum of the first loss value and the second loss value.

[0066] Exemplarily, the first formula above is as follows:

[0067] loss=loss r +loss d (1)

[0068] Among them, loss r Represents the first loss value, loss d Represents the second loss value.

[0069] In this way, since the first loss value represents both the timbre similarity between the audio signal sample and the noise-reduced audio signal and the timbre difference between the audio signal sample and the noise-reduced audio signal, the timbre in the audio can be fully protected from being affected during the noise reduction process by training the denoising model with the first loss value.

[0070] It is understandable that when the first signal envelope and the second signal envelope are the same, it means that the timbre of the audio signal sample and the noise reduction audio signal is substantially the same. At this time, the current noise reduction model no longer needs to be trained.

[0071] In the model training method provided in the embodiment of the present application, the electronic device inputs an audio signal sample into a first noise reduction model, performs noise reduction processing on the audio signal sample, and outputs a noise-reduced audio signal; based on the first signal envelope of the audio signal sample and the second signal envelope of the noise-reduced audio signal, a first loss value is determined, and the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; based on the Euclidean distance value between the first signal envelope and the second signal envelope, a second loss value is determined, and the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; based on the first loss value and the second loss value, the first noise reduction model is trained to obtain the second noise reduction model; wherein the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal. In the present scheme, since the element that has the greatest and most intuitive impact on the timbre is the signal envelope of the audio signal, the electronic device calculates the first loss value and the second loss value by combining the correlation and difference between the signal envelope of the audio signal sample and the signal envelope of the noise reduction audio signal, and trains the model part of the noise reduction model that identifies the signal envelope according to the first loss value and the second loss value, so that the trained noise reduction model can accurately identify the signal envelope of the audio signal during the process of noise reduction of the audio signal, thereby protecting the audio signal part within the signal envelope that reflects the user's timbre, thereby improving the noise reduction effect of the electronic device on the audio.

[0072] Optionally, in some embodiments of the present application, the above step 202 can be specifically implemented by the following steps 202a and 202b.

[0073] Step 202a: The electronic device determines a correlation coefficient between the first signal envelope and the second signal envelope based on a covariance value of the first signal envelope and the second signal envelope.

[0074] In some embodiments of the present application, the above correlation coefficient is used to characterize the similarity between the first signal envelope and the second signal envelope.

[0075] In some embodiments of the present application, the magnitude of the above correlation coefficient is positively correlated with the similarity between the first signal envelope and the second signal envelope.

[0076] Exemplarily, the closer the above correlation coefficient is to 1, the more similar the first signal envelope and the second signal envelope are, that is, the more similar the timbre of the two audio signals are; conversely, the closer the above correlation coefficient is to 0, the less similar the first signal envelope and the second signal envelope are, that is, the less similar the timbre of the two audio signals are.

[0077] In some embodiments of the present application, the electronic device uses the second formula to calculate the correlation coefficient between the first signal envelope and the second signal envelope.

[0078] Exemplarily, the tenth formula above is as follows:

[0079]

[0080] Where r is the correlation coefficient, Cov(X, Y) is the covariance of X and Y, Var[X] is the variance of X, Var[Y] is the variance of Y, and X and Y are the first signal envelope h0(f) and the second signal envelope h1(f) respectively.

[0081] Step 202b: The electronic device determines the difference between the first value and the correlation coefficient as the first loss value.

[0082] In some embodiments of the present application, the first value may be predefined by the electronic device or may be set by the user.

[0083] In some embodiments of the present application, the first value is any value. Usually, the first value is 1.

[0084] Exemplarily, when the first value is 1, the electronic device calculates the first loss value using the third formula.

[0085] Exemplarily, the third formula above is as follows:

[0086] loss r =1-r (3)

[0087] In this way, if the signal envelopes of the two audio signals are exactly the same, the correlation coefficient between them will tend to 1, and the first loss value will tend to 0; if the signal envelopes of the two audio signals are completely different, the correlation coefficient between them will tend to 0, and the first loss value will tend to 1. The smaller the first loss value, the smaller the intensity of training the denoising model. Therefore, by continuously using the first loss value to train the denoising model, a denoising model for protecting the signal envelope characteristics in the audio signal can be obtained.

[0088] Optionally, in some embodiments of the present application, the above step 203 can be specifically implemented through the following steps 203a to 203c.

[0089] Step 203a: The electronic device calculates the Euclidean distance value between the first signal envelope and the second signal envelope corresponding to each frame of the audio signal.

[0090] In some embodiments of the present application, the audio signal samples include X frames of audio signals, and the noise reduction audio signal includes X frames of audio signals.

[0091] It can be understood that the above audio signal samples and the noise reduction audio signal can be divided into audio signals with the same number of frames, and each frame of the audio signal corresponds to a section of the signal envelope.

[0092] In some embodiments of the present application, the Euclidean metric is a commonly used distance definition, which refers to the real distance between two points in m-dimensional space, or the natural length of a vector, that is, the distance from the point to the origin. In other words, the calculation process of the above Euclidean distance is to calculate the absolute value of the distance between each coordinate point in the first signal envelope and the second signal envelope.

[0093] In some embodiments of the present application, the electronic device uses the fourth formula to calculate the Euclidean distance between the first signal envelope and the second signal envelope.

[0094] Exemplarily, the fourth formula above is as follows:

[0095] D=|h1(f)-h0(f)| (4)

[0096] Wherein, h1(f) represents the first signal envelope, and h0(f) represents the second signal envelope.

[0097] Step 203b: The electronic device determines a weighting coefficient for each Euclidean distance value according to the energy value corresponding to each frame of the audio signal.

[0098] In some embodiments of the present application, different energy values ​​correspond to different weighting coefficients.

[0099] It can be understood that the higher the weighting coefficient, the more suitable the energy value is for human ears to accept.

[0100] For example, suppose the energy of the samples below 4000 Hz, 4000-6000 Hz, and above 6000 Hz are S l , S m , S h ; Assuming S l The energy accounts for more than 0.7 of the total energy, the weighting coefficient is 1. m If the total energy accounts for more than 0.3, the weighting coefficient is 0.8, and in other cases the weighting coefficient is 0.5. The specific weighting coefficient R can be expressed by the fifth formula.

[0101] Specifically, the fifth formula above is as follows:

[0102]

[0103] Step 203c: The electronic device accumulates each weighted Euclidean distance value based on the weighting coefficient, and calculates an average value of the accumulated Euclidean distance values ​​to obtain a second loss value.

[0104] In some embodiments of the present application, the electronic device first weights the Euclidean distance value between the first signal envelope and the second signal envelope according to the energy value corresponding to each frame of the audio signal, and then accumulates the weighted Euclidean distance values ​​and calculates the average value to obtain the above-mentioned second loss value.

[0105] In some embodiments of the present application, the electronic device may use the fifth formula to calculate the second loss value.

[0106] Exemplarily, the sixth formula above is as follows:

[0107]

[0108] For example, the above To weight and accumulate each Euclidean distance value.

[0109] As human voices are mostly distributed below 4000 Hz, the human ear is more sensitive to high-frequency signals of 4000-6000 Hz, which sound harsh, while those above 6000 Hz are basically noise. Therefore, envelopes at different energy values ​​are emphasized by weighting. The sum of the Euclidean distances between signal envelopes multiplied by the weighting coefficient is used as the loss for training the denoising model, which can protect the signal envelope of the audio signal and thus retain more of the original timbre of the audio signal.

[0110] Optionally, in some embodiments of the present application, after the above-mentioned step 204 "the electronic device trains the first noise reduction model based on the first loss value and the second loss value to obtain the second noise reduction model", the model training method provided in the embodiments of the present application also includes the following steps 301 to 303.

[0111] Step 301: The electronic device inputs a first audio signal into a second noise reduction model, performs noise reduction processing on the first audio signal, and outputs a second audio signal.

[0112] In some embodiments of the present application, the first audio signal includes a speech signal.

[0113] Exemplarily, the first audio signal may be audio recorded by a user through an electronic device, or audio with human voice stored in the electronic device.

[0114] Exemplarily, the electronic device may record an audio signal of the user's voice input through a translation application or a recording application.

[0115] It is understandable that the above-mentioned translation application or recording application can be an application that comes with the electronic device system, or it can be a third-party application, such as "Smart Translator".

[0116] In some embodiments of the present application, the second denoising model is a denoising model trained by the first denoising model.

[0117] In some embodiments of the present application, the second noise reduction model is used to perform noise reduction processing on the noise part of the audio signal.

[0118] In some embodiments of the present application, the electronic device inputs the first audio signal into the above-mentioned first noise reduction model, identifies the signal envelope of the first audio signal, and then uses a noise reduction algorithm to perform noise reduction processing on other audio signal parts except the audio signal part within the signal envelope, for example, the median method, the standard deviation method, or the sliding average method, etc., to obtain a second audio signal after noise reduction, and then outputs the second audio signal to the first noise model.

[0119] In some embodiments of the present application, the signal envelope is at least used to characterize the timbre characteristics of the audio signal.

[0120] It can be understood that the above signal envelope is at least used to characterize the amplitude profile of the audio signal in the frequency spectrum.

[0121] Exemplarily, the above-mentioned contour feature information can be represented by the frequency spectrum, power and resonance peak of the audio.

[0122] For example, Figure 2 As shown, taking the sample audio as an example, curve 1 is the signal curve of the first audio signal, and curve 2 is the signal envelope curve of the first audio signal, wherein the horizontal axis is the sampling point of the first audio signal, such as a 10-second (s) audio, 100 sampling points can be set, with 0.01s as a sampling point. The vertical axis is used to represent the amplitude of the first audio signal at the sampling point.

[0123] Exemplarily, the first signal portion is a signal portion of the first audio signal excluding a signal portion within a signal envelope. Figure 2 As shown, the portion below curve 2 is the signal portion of the signal envelope content, and the portion above curve 2 is the above-mentioned first signal portion.

[0124] In some embodiments of the present application, in the above-mentioned second noise reduction model, the electronic device first uses the seventh formula to perform a convolution operation on the first audio signal, and then uses the eighth formula to perform a Hilbert transform calculation on the first audio signal after the convolution operation to obtain the signal envelope h(f) of the first audio signal.

[0125] Exemplarily, the seventh formula above is as follows:

[0126]

[0127] in, is a function expression of the first audio signal after the convolution operation.

[0128] Exemplarily, the eighth formula above is as follows:

[0129]

[0130] Where (-j2πft)dt represents the instantaneous phase.

[0131] For example, in a meeting scenario, the electronic device can receive input from a user clicking on the "smart translator" to start the recording program. The user inputs "Hello" through voice, which is the first audio signal mentioned above. The electronic device uses the first noise reduction model to identify the signal envelope of "Hello", performs noise reduction on the background noise, and then outputs the noise-reduced audio signal to the second noise reduction model.

[0132] In an example, the electronic device generates the second audio signal by performing noise reduction processing on audio signal parts other than the audio signal part within the signal envelope and then re-synthesizing the audio signal part with the audio signal part within the signal envelope.

[0133] In another example, the electronic device directly generates a new first audio signal, i.e., the above-mentioned second audio signal, from the original first audio signal by performing noise reduction processing on audio signal parts other than the audio signal part within the signal envelope, without re-synthesizing with the audio signal part within the signal envelope.

[0134] Step 302: The electronic device inputs the second audio signal into a speech translation model to identify speech emotion features in the second audio signal.

[0135] In some embodiments of the present application, the speech translation model is used to translate a language in an audio signal into another language.

[0136] Exemplarily, the above-mentioned speech translation model can be used to translate Chinese audio into English audio, or translate English audio into Chinese audio, or translate Chinese audio into French audio, etc.

[0137] Exemplarily, the electronic device may pre-train a speech translation model that can translate by replicating the user's timbre in an audio signal. That is, any sentence the user says can be synthesized into audio of any other text content through this model by replicating the user's timbre.

[0138] Exemplarily, the speech translation model may be a large language model (LLM), a multimodal model, or a transform model, etc.

[0139] In some embodiments of the present application, the above-mentioned speech emotion features are used to characterize the emotional expression of the user in the audio.

[0140] Exemplarily, the above-mentioned voice emotion characteristics may include but are not limited to happiness, anger, shock or sadness.

[0141] Optionally, in some embodiments of the present application, the electronic device trains the speech translation model in combination with the speech emotion features in the training set.

[0142] Exemplarily, the above-mentioned speech emotion features may correspond to corresponding speech emotion labels, so that the speech translation model may directly obtain the speech emotion features in the audio signal by identifying the speech emotion labels corresponding to the audio signal.

[0143] Step 303: The electronic device translates the second audio signal based on the speech emotion feature and outputs a translated audio signal.

[0144] In some embodiments of the present application, the electronic device identifies the speech emotion features in the second audio signal through a speech translation model, and after completing the translation of the second audio signal, generates and outputs a translated audio signal in combination with the identified implied emotion features.

[0145] It is understandable that people's emotions when speaking are very important. For example, the tone of voice in normal speech can easily lead to contradictions. Therefore, electronic devices add voice emotion tags for training, so that the voice translation model can recognize the user's emotions when speaking, such as happy, sad, angry, and add speech style emotion tags to the training data of voice reproduction, so that the translated synthetic audio can reproduce the user's timbre while containing the user's original emotional expression.

[0146] In this way, the electronic device uses a noise reduction model to reduce the noise of the audio while retaining the user's voice, and translates the noise-reduced audio through a speech translation model, thereby realizing the ability to synthesize audio using the user's own voice; at the same time, the speech emotion tags of the audio are identified through the speech translation model, thereby realizing the ability to synthesize audio using the user's own voice and emotions, thereby initially realizing the ability to broadcast the translated text with one's own voice and original emotions after translation.

[0147] Optionally, in some embodiments of the present application, after the above step 303, the model training method provided in the embodiments of the present application further includes the following steps 401 to 403.

[0148] Step 401: The electronic device obtains a first volume value of an ambient noise signal from a translated audio signal.

[0149] In some embodiments of the present application, the electronic device extracts an audio signal of the ambient noise signal from the translated audio signal, and obtains a noise volume value in the audio signal of the ambient noise signal, that is, the above-mentioned first volume value.

[0150] For example, it is first assumed that the environmental noise signal in the translated audio signal is a random stationary additive noise and is unrelated to the user's voice signal. The noisy speech power spectrum of each frequency point in the translated audio signal is smoothed. The short-term smoothing method is to use the ninth formula for processing.

[0151] Specifically, the ninth formula is as follows:

[0152] P(λ,k)=αP(λ-1,k)+(1-α)||Y(λ,k)| 2 (9)

[0153] Here, is the signal power spectrum of the kth frequency point and λ frame after smoothing, α is the smoothing factor, which is usually taken as: 0.7≤α≤0.9, and is the noisy speech power spectrum of the current kth frequency point and frame.

[0154] Next, the following tenth formula is used to perform nonlinear tracking on the minimum value of the noisy speech power at each frequency point in the translated audio signal.

[0155]

[0156] Among them, P min (λ, k) is the noise power spectrum estimated by tracking the minimum of the continuous spectrum, while β and γ are constants between 0 and 1 and serve as smoothing factors.

[0157] It is understandable that in an audio signal, there may be different background scenes, such as a meeting scene or a subway scene. Since noise is usually the audio signal with the smallest power in an audio signal, by performing nonlinear tracking on the minimum power value of noisy speech at each frequency point in different scenes, the corresponding noise audio signal can be extracted from a translated audio signal.

[0158] Then, the square root of the estimated noise power spectrum is taken and then an inverse Fourier transform is performed to obtain the amplitude of the time domain signal of the restored noise. Then, the eleventh formula is used to obtain the volume value of the ambient noise signal according to the amplitude of the time domain signal of the restored noise.

[0159] e0=220log 10 (A) (11)

[0160] Where A is the amplitude of the restored noise, and e0 is the calculated volume value of the ambient noise in decibels (dB).

[0161] Step 402: The electronic device determines a second volume value based on the first volume value and a preset volume value.

[0162] In some embodiments of the present application, the preset volume value may be user-defined or predefined by the electronic device.

[0163] Exemplarily, the preset volume value may be an optimal hearing volume value set to adapt to the optimal hearing sense of the human ear, or may be a volume value of a current player of the electronic device.

[0164] It is understandable that in order to ensure the balanced volume of the broadcast audio, electronic devices need to first set a basic optimal listening volume x0. According to the human ear's perception of sound and search data, usually 50 decibels at 3kHz is determined as the optimal listening volume.

[0165] In one example, the electronic device determines the sum of the first volume value and the preset volume value as the second volume value.

[0166] In another example, the electronic device weights the first volume value and the preset volume value respectively, and then determines the sum of the weighted first volume value and the preset volume value as the second volume value.

[0167] Step 403: The electronic device adjusts the volume value of the translated audio signal to a second volume value.

[0168] In some embodiments of the present application, the translated audio signal includes N frames of audio signal.

[0169] In some embodiments of the present application, the electronic device adjusts the volume value of each frame of the audio signal to the second volume value.

[0170] It can be understood that, since the volume value of each frame of the audio signal is the same, the amplitude of each frame of the audio signal is also the same.

[0171] In this way, since there may be ambient noise during audio recording, the volume should be dynamically adjusted according to the volume of the ambient noise during broadcasting. In other words, the volume requirements in a quiet environment and a noisy subway are different. Therefore, the audio broadcast volume can be adjusted according to the volume value of the ambient noise signal and the preset volume value, so that the electronic device can adaptively and dynamically adjust the playback volume value according to the noise volume of the real environment.

[0172] Optionally, in some embodiments of the present application, the above-mentioned translated audio signal includes N frames of audio signals, where N is a positive integer. After the above-mentioned step 403, the model training method provided in the embodiment of the present application also includes the following steps 501 to 504.

[0173] Step 501: The electronic device obtains an amplitude adjustment coefficient corresponding to an i-th frame audio signal among N frames of audio signals.

[0174] Among them, i∈[1,N], i is an integer.

[0175] In some embodiments of the present application, the electronic device may determine the amplitude adjustment coefficient corresponding to the i-th frame audio signal by analyzing the signal envelope of the i-th frame audio signal and an error value of a preset signal envelope.

[0176] In some embodiments of the present application, the amplitude adjustment coefficient is used to adjust the amplitude of each frame of the audio signal.

[0177] In some embodiments of the present application, a frame of audio signal corresponds to at least one amplitude adjustment coefficient.

[0178] Step 502: The electronic device adjusts the amplitude of the i-th frame audio signal based on the amplitude adjustment coefficient.

[0179] In some embodiments of the present application, the electronic device adds the amplitude adjustment coefficient to the amplitude of the i-th frame audio signal to obtain an adjusted amplitude of the i-th frame audio signal.

[0180] In some embodiments of the present application, the electronic device uses the twelfth formula to adjust the amplitude of the i-th frame audio signal in the translated audio.

[0181] Exemplarily, the twelfth formula above is as follows:

[0182] y(n,i)=x(n,i)G(i) (12)

[0183] Among them, y(n, i) is the amplitude of the nth sampling point of the i-th audio frame of the translated audio after adjustment, x(n, i) is the amplitude of the nth sampling point of the i-th audio frame of the translated audio, and G(i) is the above amplitude adjustment parameter.

[0184] Step 503: The electronic device performs volume conversion on the adjusted amplitude of the i-th frame audio signal to obtain a third volume value.

[0185] In some embodiments of the present application, the electronic device may take the square root of the amplitude of the i-th frame audio signal and then perform an inverse Fourier transform to restore the amplitude of the i-th frame audio signal, and then use the ninth formula in the above step 401 to perform volume conversion on the restored amplitude of the i-th frame audio signal to obtain a third volume value.

[0186] Step 504: The electronic device adjusts the volume value of the i-th frame audio signal from the second volume value to the third volume value.

[0187] It can be understood that the electronic device determines the third volume value corresponding to each frame of the audio signal in the translated audio signal through the above steps 501 to 503, and adjusts the volume value of each frame of the audio signal to the corresponding third volume value.

[0188] In this way, the electronic device adjusts the volume of the translated audio signal by partition through automatic gain, so that when the electronic device adjusts the playback volume of the translated audio signal, the intonation of the voice in the audio signal can be retained, avoiding the translated audio signal from being too harsh.

[0189] Optionally, in some embodiments of the present application, the above step 501 can be specifically implemented by the following steps 501a and 501b.

[0190] Step 501a: The electronic device obtains the signal envelope and envelope error value of the i-th frame audio signal.

[0191] In some embodiments of the present application, the envelope error value is a difference between a preset signal envelope and a signal envelope of the i-th frame audio signal.

[0192] Step 501b: The electronic device determines an amplitude adjustment coefficient according to the signal envelope and envelope error value of the i-th frame audio signal.

[0193] In some embodiments of the present application, the electronic device determines the amplitude adjustment coefficient of the i-th frame audio signal according to the signal envelope, envelope error value and step factor of the i-th frame audio signal.

[0194] In some embodiments of the present application, the preset signal envelope is the signal envelope of the i-th frame audio signal under an ideal state.

[0195] In some embodiments of the present application, the above step factor is determined based on the envelope error value and can be expressed as μ(i).

[0196] Exemplarily, the above step length can be calculated using the thirteenth formula.

[0197] Exemplarily, the above thirteenth formula is as follows:

[0198]

[0199] Wherein, i represents the i-th frame audio signal, and e(i) is the envelope error value of the i-th frame audio signal.

[0200] It should be noted that in the thirteenth formula, when the envelope error value approaches 0, the step size still changes very quickly, which will cause a large steady-state error of the noise algorithm. In view of the above problem, this embodiment improves the above formula and introduces control parameters α, β, and m.

[0201] Specifically, the above adjusted variable step length function, that is, the fourteenth formula is as follows:

[0202]

[0203] like Figure 3 As shown in the figure, when the values ​​of β and m are fixed, the larger the value of β, the smaller the initial convergence compensation of the algorithm and the slower the convergence. Therefore, after comprehensive experimental consideration, the variable step size of α = 3, β = 0.2, and m = 2 is usually selected for processing.

[0204] In some embodiments of the present application, the electronic device uses the fifteenth formula to iteratively calculate the amplitude adjustment coefficient corresponding to the i-th frame audio signal.

[0205] Exemplarily, the fifteenth formula above is as follows:

[0206] G(i)=G(i-1)+μ(i)B x (i) e(i) (15)

[0207] Among them, G(i) is the amplitude adjustment coefficient corresponding to the i-th frame, G(i-1) is the amplitude adjustment coefficient of the previous frame, and B x (i) is the power value of the signal envelope of the i-th frame audio signal, and e(i) is the envelope error value.

[0208] Thus, since the step factor in this embodiment is determined according to the envelope error value of each frame of the translated audio signal, a more accurate amplitude adjustment coefficient can be obtained when the amplitude adjustment coefficient of each frame of the audio signal is iteratively calculated.

[0209] The following is a specific example to illustrate the model training method provided in the embodiments of the present application.

[0210] The model training method may include the following steps A1 to A10.

[0211] Step A1: The electronic device runs a translation application, such as an intelligent translator.

[0212] For example, Figure 4 As shown, a “start recording” control 41 is displayed in the setting interface of the translation application.

[0213] Exemplarily, the above-mentioned "start recording" control is used to start the audio recording function.

[0214] Step 102: The electronic device receives input from a user clicking a “start recording” control.

[0215] Exemplarily, the user long presses the "Start Recording" control to arbitrarily input a voice audio of 10 seconds in length, hereinafter referred to as recorded audio.

[0216] It is understandable that the above recorded audio is used by the electronic device to reproduce the user's timbre information when translating. Since the timbre of people speaking in different emotions and states is quite different, for example, the timbre is different when in a meeting and chatting. Therefore, in this step, the user can be asked to record in advance in the tone of daily speaking to retain the user's timbre and ensure the relative uniformity of the timbre.

[0217] It should be noted that the above-mentioned recorded audio may be recorded in a scenario of extracting timbre or in a scenario of translation.

[0218] Step A3: The electronic device calculates the signal envelopes of the recorded audio and the denoised recorded audio through Hilbert transform, and calculates the correlation coefficient of the signal envelopes of the two as the loss value for loss training the denoising model.

[0219] It is understandable that since the recording environment in step A2 is likely to have a large amount of ambient noise, if the recorded audio being learned contains noise, the audio that is finally broadcast is likely to contain noise as well, which will seriously affect the listening experience. Therefore, it is necessary to perform noise reduction on the recording first. There are many factors that affect the timbre, and most of the elements in the spectrum will affect the timbre. Among these factors, the factor that has the greatest and most intuitive impact on the timbre is the envelope. Therefore, in order to protect the user's timbre from being affected during noise reduction, it is necessary to consider the similarity of the envelope of the audio before and after noise reduction while training the noise reduction model, that is, to obtain it by calculating the above-mentioned correlation coefficient, so that the trained noise reduction model can better retain the timbre.

[0220] Step A4, the electronic device calculates the Euclidean distance between the signal envelope of the recorded audio and the envelope of the recorded audio after noise reduction, and performs smooth weighting according to the human ear's perception of different frequency bands, that is, calculates the weighting coefficient for the signal energy in different frequency bands, and uses the average value of the weighted Euclidean distance as one of the loss values ​​for training the noise reduction model, and trains the noise reduction model together with the loss value in step A3, so as to retain the most realistic user timbre while suppressing the noise during recording.

[0221] It is understandable that, in order to further protect the envelope information, the electronic device may design the loss in step A4 based on step A3, that is, the third loss value mentioned above, to jointly protect the envelope.

[0222] At the same time, in order to unify the amplitude mean of the recorded audio and the recorded audio after noise reduction, assuming that the signal envelope amplitude mean of the recorded audio is E0, and the signal envelope amplitude mean of the recorded audio after noise reduction is E1, then E0-E1 is added to the entire signal envelope after noise reduction to make the overall amplitude mean the same to avoid affecting the subsequent calculation results.

[0223] Step A5: The electronic device performs noise reduction processing on the audio to be translated, that is, the first audio signal, through a noise reduction model.

[0224] It should be noted that the audio to be translated may be the recorded audio in step A2, or may be the audio re-recorded by the user.

[0225] Step A6: The electronic device sets the translation language in the translation application, and then inputs the above-mentioned noise-reduced audio to be translated. The electronic device performs speech recognition on the audio to be translated and then performs conventional text translation.

[0226] For example, Figure 5 As shown, when using the smart translator application, the user needs to first enter the language to be translated, such as "Chinese to English" in the text box 52 of the interface 51. After the application performs voice recognition on the input audio, it translates the audio according to the language to be translated to obtain the translated audio of the audio to be translated.

[0227] Step A7: The electronic device estimates the ambient noise volume value e0 of the translated audio based on the continuous spectrum minimum tracking noise estimation, that is, the first volume value mentioned above, which is used to adjust the playback volume value of the translated audio.

[0228] Step A8: The electronic device sets the optimal listening volume x0, that is, the above-mentioned preset playback volume value, to adjust the playback volume value of the translation audio.

[0229] Step A9, the volume of the sound broadcast by the electronic device may be uneven, and the volume needs to be balanced. At the same time, according to the ambient noise volume value e0 in step A6 and the optimal listening volume x0 in step A7, the playback volume value needs to be adaptively and dynamically adjusted to the target volume value x0+e0, that is, the above-mentioned second volume value.

[0230] Step A10: The electronic device combines the envelope of the translated audio, uses a variable step-size LMS algorithm improved by a softsign function, and adopts automatic gain control to adjust the actual playback volume value of the translated audio frame by frame.

[0231] In this way, this embodiment provides an intelligent translation method that can broadcast the translated text with one's own voice and original emotions after translation when meeting with foreign colleagues or communicating with foreigners, retaining the user's true timbre. Conventional translation applications can only convert speech into text, or mechanically broadcast text with voice, which has a cold sense of technology, while this embodiment can broadcast the translated text with one's own voice and original emotions after translation, which will be warmer and more like the experience of communication between people. At the same time, the method proposed in this embodiment can better protect the user's true timbre, and adaptively adjust the playback volume according to the current real external environment.

[0232] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0233] The model training method provided in the embodiment of the present application can be executed by an electronic device or a model training device. In the embodiment of the present application, the model training device executing the model training method is taken as an example to illustrate the model training device provided in the embodiment of the present application.

[0234] Figure 6 A possible structural diagram of the model training device involved in the embodiment of the present application is shown. Figure 6 As shown, the model training device 600 may include: a processing module 601 and a training module 602.

[0235] Among them, the above-mentioned processing module 601 is used to input the audio signal sample into the first noise reduction model, perform noise reduction processing on the audio signal sample, and output the noise-reduced audio signal; the above-mentioned processing module 601 is also used to determine a first loss value based on the first signal envelope of the audio signal sample and the second signal envelope of the noise-reduced audio signal, and the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; the above-mentioned processing module 601 is also used to determine a second loss value based on the Euclidean distance value between the first signal envelope and the second signal envelope, and the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; the above-mentioned training module 602 is used to train the first noise reduction model based on the first loss value determined by the above-mentioned processing module 601 and the second loss value determined by the above-mentioned processing module 601 to obtain the second noise reduction model; wherein the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal.

[0236] Optionally, in some embodiments of the present application, the processing module 601 is specifically used to:

[0237] Determine a correlation coefficient between the first signal envelope and the second signal envelope based on a covariance value between the first signal envelope and the second signal envelope, where the correlation coefficient is used to characterize the correlation between the first signal envelope and the second signal envelope;

[0238] The difference between the first value and the correlation coefficient is determined as the first loss value.

[0239] Optionally, in some embodiments of the present application, the audio signal sample includes X frames of audio signals, and the noise reduction audio signal includes X frames of audio signals;

[0240] The processing module 601 is specifically used for:

[0241] Calculating the Euclidean distance between the first signal envelope and the second signal envelope corresponding to each frame of the audio signal;

[0242] Determine a weighting coefficient for each Euclidean distance value according to an energy value corresponding to each frame of the audio signal;

[0243] Based on the weighting coefficient, each weighted Euclidean distance value is accumulated, and the average value of the accumulated Euclidean distance values ​​is calculated to obtain a second loss value.

[0244] Optionally, in some embodiments of the present application, combined with Figure 6 ,like Figure 7 As shown, the above-mentioned device 600 also includes: a recognition module 603; the above-mentioned processing module 601 is also used for the above-mentioned training module 602 to train the first denoising model based on the first loss value and the second loss value, and after obtaining the second denoising model, the first audio signal is input into the second denoising model, the first audio signal is denoised, and the second audio signal is output; the above-mentioned recognition module 603 is used to input the second audio signal processed by the above-mentioned processing module 601 into the speech translation model to identify the speech emotion features in the second audio signal; the above-mentioned processing module 601 is also used to translate the second audio signal based on the speech emotion features identified by the above-mentioned recognition module 603, and output the translated audio signal.

[0245] Optionally, in some embodiments of the present application, combined with Figure 7 ,like Figure 8 As shown, the above-mentioned device 600 also includes: an acquisition module 604 and an adjustment module 605; the above-mentioned acquisition module 604 is used for the above-mentioned processing module 601 to translate the second audio signal based on the speech emotion feature, and after outputting the translated audio signal, obtain the first volume value of the environmental noise signal from the translated audio signal; the above-mentioned processing module 601 is also used to determine the second volume value based on the first volume value and the preset volume value obtained by the above-mentioned acquisition module 604; the above-mentioned adjustment module 605 is used to adjust the volume value of the translated audio signal processed by the above-mentioned processing module 601 to the second volume value determined by the above-mentioned processing module 601.

[0246] Optionally, in some embodiments of the present application, the translated audio signal includes N frames of audio signals, where N is a positive integer; the above-mentioned acquisition module 604 is also used to obtain the amplitude adjustment coefficient corresponding to the i-th frame audio signal in the N frames of audio signals after the above-mentioned adjustment module 605 adjusts the volume value of the translated audio signal to the second volume value, i∈[1,N], where i is an integer; the above-mentioned adjustment module 605 is also used to adjust the amplitude of the i-th frame audio signal based on the amplitude adjustment coefficient obtained by the above-mentioned acquisition module 604; the above-mentioned processing module 601 is also used to perform volume conversion on the amplitude of the i-th frame audio signal adjusted by the above-mentioned adjustment module 605 to obtain a third volume value; the above-mentioned adjustment module 605 is also used to adjust the volume value of the i-th frame audio signal from the second volume value to the third volume value.

[0247] Optionally, in some embodiments of the present application, the acquisition module 604 is specifically used to:

[0248] Acquire a signal envelope and an envelope error value of an i-th frame audio signal, where the envelope error value is a difference between a preset signal envelope and a signal envelope of the i-th frame audio signal;

[0249] An amplitude adjustment coefficient is determined according to a signal envelope and an envelope error value of an i-th frame audio signal.

[0250] In the model training device provided in the embodiment of the present application, the model training device inputs the audio signal sample into the first noise reduction model, performs noise reduction processing on the audio signal sample, and outputs the noise-reduced audio signal; based on the first signal envelope of the audio signal sample and the second signal envelope of the noise-reduced audio signal, a first loss value is determined, and the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; based on the Euclidean distance value between the first signal envelope and the second signal envelope, a second loss value is determined, and the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; based on the first loss value and the second loss value, the first noise reduction model is trained to obtain the second noise reduction model; wherein the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal. In the present scheme, since the signal envelope of the audio signal is the element that has the greatest and most intuitive impact on the timbre, the model training device calculates the first loss value and the second loss value by combining the correlation and difference between the signal envelope of the audio signal sample and the signal envelope of the noise reduction audio signal, and trains the model part of the signal envelope in the noise reduction model according to the first loss value and the second loss value, so that the trained noise reduction model can accurately identify the signal envelope of the audio signal during the process of denoising the audio signal, thereby protecting the audio signal part within the signal envelope that reflects the user's timbre, thereby improving the effect of the trained model.

[0251] The model training device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtualreality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personalcomputer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personalcomputer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0252] The model training device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0253] The model training device provided in the embodiment of the present application can implement each process implemented in the embodiment of the model training method and achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0254] Alternatively, if Fig. 9 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores programs or instructions that can be executed on the processor 801, and when the program or instructions are executed by the processor 801, the various steps of the above-mentioned model training method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0255] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0256] Fig.10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0257] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0258] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0259] The processor 110 is used to input the audio signal sample into the first noise reduction model, perform noise reduction on the audio signal sample, and output the noise-reduced audio signal; the processor 110 is also used to determine a first loss value based on the first signal envelope of the audio signal sample and the second signal envelope of the noise-reduced audio signal, and the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; the processor 110 is also used to determine a second loss value based on the Euclidean distance value between the first signal envelope and the second signal envelope, and the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; the processor 110 is used to train the first noise reduction model based on the first loss value and the second loss value to obtain the second noise reduction model; the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal.

[0260] Optionally, in some embodiments of the present application, the processor 110 is specifically configured to:

[0261] Determine a correlation coefficient between the first signal envelope and the second signal envelope based on a covariance value between the first signal envelope and the second signal envelope, where the correlation coefficient is used to characterize the correlation between the first signal envelope and the second signal envelope;

[0262] The difference between the first value and the correlation coefficient is determined as the first loss value.

[0263] Optionally, in some embodiments of the present application, the audio signal sample includes X frames of audio signals, and the noise reduction audio signal includes X frames of audio signals;

[0264] The processor 110 is specifically configured to:

[0265] Calculating the Euclidean distance between the first signal envelope and the second signal envelope corresponding to each frame of the audio signal;

[0266] Determine a weighting coefficient for each Euclidean distance value according to an energy value corresponding to each frame of the audio signal;

[0267] Based on the weighting coefficient, each weighted Euclidean distance value is accumulated, and the average value of the accumulated Euclidean distance values ​​is calculated to obtain a second loss value.

[0268] Optionally, in some embodiments of the present application, the processor 110 is further used to train a first noise reduction model based on the first loss value and the second loss value, and after obtaining the second noise reduction model, input the first audio signal into the second noise reduction model, perform noise reduction processing on the first audio signal, and output a second audio signal; the processor 110 is further used to input the second audio signal into a speech translation model to identify speech emotion features in the second audio signal; the processor 110 is further used to translate the second audio signal based on the speech emotion features, and output a translated audio signal.

[0269] Optionally, in some embodiments of the present application, the processor 110 is further used to translate the second audio signal based on the speech emotion feature, and after outputting the translated audio signal, obtain a first volume value of the ambient noise signal from the translated audio signal; the processor 110 is further used to determine the second volume value based on the first volume value and a preset volume value; the processor 110 is used to adjust the volume value of the translated audio signal to the second volume value.

[0270] Optionally, in some embodiments of the present application, the translated audio signal includes N frames of audio signals, where N is a positive integer; the processor 110 is further used to adjust the volume value of the translated audio signal to the second volume value, and then obtain an amplitude adjustment coefficient corresponding to the i-th frame audio signal in the N frames of audio signals, where i∈[1,N], and i is an integer; the processor 110 is further used to adjust the amplitude of the i-th frame audio signal based on the amplitude adjustment coefficient; the processor 110 is further used to perform volume conversion on the adjusted amplitude of the i-th frame audio signal to obtain a third volume value; the processor 110 is further used to adjust the volume value of the i-th frame audio signal from the second volume value to the third volume value.

[0271] Optionally, in some embodiments of the present application, the processor 110 is specifically configured to:

[0272] Acquire a signal envelope and an envelope error value of an i-th frame audio signal, where the envelope error value is a difference between a preset signal envelope and a signal envelope of the i-th frame audio signal;

[0273] An amplitude adjustment coefficient is determined according to a signal envelope and an envelope error value of an i-th frame audio signal.

[0274] In the electronic device provided in the embodiment of the present application, the electronic device inputs an audio signal sample into a first noise reduction model, performs noise reduction processing on the audio signal sample, and outputs a noise-reduced audio signal; determines a first loss value based on a first signal envelope of the audio signal sample and a second signal envelope of the noise-reduced audio signal, and the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; determines a second loss value based on a Euclidean distance value between the first signal envelope and the second signal envelope, and the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; trains a first noise reduction model based on the first loss value and the second loss value to obtain a second noise reduction model; wherein the first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise-reduced audio signal. In the present scheme, since the element that has the greatest and most intuitive impact on the timbre is the signal envelope of the audio signal, the electronic device calculates the first loss value and the second loss value by combining the correlation and difference between the signal envelope of the audio signal sample and the signal envelope of the noise reduction audio signal, and trains the model part of the noise reduction model that identifies the signal envelope according to the first loss value and the second loss value, so that the trained noise reduction model can accurately identify the signal envelope of the audio signal during the process of denoising the audio signal, thereby protecting the audio signal part within the signal envelope that reflects the user's timbre, thereby improving the effect of the trained model.

[0275] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0276] The memory 109 can be used to store application programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0277] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0278] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned model training method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0279] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0280] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0281] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0282] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned model training method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0283] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0284] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of an application program plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer application product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0285] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Inputting the audio signal sample into the first noise reduction model, performing noise reduction processing on the audio signal sample, and outputting the noise-reduced audio signal; Determining a first loss value based on a first signal envelope of the audio signal sample and a second signal envelope of the noise-reduced audio signal, wherein the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; Determine a second loss value based on a Euclidean distance value between the first signal envelope and the second signal envelope, wherein the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; Based on the first loss value and the second loss value, training the first denoising model to obtain a second denoising model; The first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise reduction audio signal.

2. The method according to claim 1, characterized in that The determining of a first loss value based on a first signal envelope of the audio signal sample and a second signal envelope of the noise reduction audio signal comprises: Determine a correlation coefficient between the first signal envelope and the second signal envelope based on a covariance value between the first signal envelope and the second signal envelope, wherein the correlation coefficient is used to characterize the correlation between the first signal envelope and the second signal envelope; The difference between the first value and the correlation coefficient is determined as the first loss value.

3. The method according to claim 1, characterized in that The audio signal samples include X frames of audio signals, and the noise reduction audio signal includes X frames of audio signals; The determining a second loss value based on the Euclidean distance value between the first signal envelope and the second signal envelope comprises: Calculating the Euclidean distance between the first signal envelope and the second signal envelope corresponding to each frame of the audio signal; Determining a weighting coefficient of each of the Euclidean distance values ​​according to the energy value corresponding to each frame of the audio signal; Based on the weighting coefficient, each weighted Euclidean distance value is accumulated, and an average value of the accumulated Euclidean distance values ​​is calculated to obtain a second loss value.

4. The method according to claim 1, characterized in that: After training the first denoising model based on the first loss value and the second loss value to obtain the second denoising model, the method further includes: Inputting the first audio signal into the second noise reduction model, performing noise reduction processing on the first audio signal, and outputting a second audio signal; Inputting the second audio signal into a speech translation model to identify speech emotion features in the second audio signal; Based on the speech emotion feature, the second audio signal is translated and a translated audio signal is output.

5. The method according to claim 4, characterized in that After translating the second audio signal based on the speech emotion feature and outputting the translated audio signal, the method further includes: Acquire a first volume value of an ambient noise signal from the translated audio signal; Determining a second volume value based on the first volume value and a preset volume value; The volume value of the translated audio signal is adjusted to the second volume value.

6. The method according to claim 5, characterized in that The translated audio signal includes N frames of audio signals, where N is a positive integer; After adjusting the volume value of the translated audio signal to the second volume value, the method further includes: Obtaining an amplitude adjustment coefficient corresponding to an i-th frame of audio signal in the N frames of audio signal, i∈[1,N], where i is an integer; Based on the amplitude adjustment coefficient, adjusting the amplitude of the i-th frame audio signal; Performing volume conversion on the adjusted amplitude of the i-th frame audio signal to obtain a third volume value; The volume value of the i-th frame audio signal is adjusted from the second volume value to the third volume value.

7. The method according to claim 6, characterized in that The obtaining the amplitude adjustment coefficient corresponding to the i-th frame audio signal in the N frames of audio signals includes: Acquire a signal envelope and an envelope error value of the i-th frame audio signal, wherein the envelope error value is a difference between a preset signal envelope and the signal envelope of the i-th frame audio signal; The amplitude adjustment coefficient is determined according to the signal envelope of the i-th frame audio signal and the envelope error value.

8. A model training device, characterized in that: The model training device comprises: a processing module and a training module; The processing module is used to input the audio signal sample into the first noise reduction model, perform noise reduction processing on the audio signal sample, and output the noise-reduced audio signal; The processing module is further used to determine a first loss value based on a first signal envelope of the audio signal sample and a second signal envelope of the noise reduction audio signal, wherein the first loss value is used to characterize the correlation between the first signal envelope and the second signal envelope; The processing module is further used to determine a second loss value based on a Euclidean distance value between the first signal envelope and the second signal envelope, wherein the second loss value is used to characterize the difference between the first signal envelope and the second signal envelope; The training module is used to train the first denoising model based on the first loss value determined by the processing module and the second loss value determined by the processing module to obtain a second denoising model; The first signal envelope is at least used to characterize the timbre characteristics of the audio signal sample, and the second signal envelope is at least used to characterize the timbre characteristics of the noise reduction audio signal.

9. The device according to claim 8, characterized in that The processing module is specifically used for: Determine a correlation coefficient between the first signal envelope and the second signal envelope based on a covariance value between the first signal envelope and the second signal envelope, wherein the correlation coefficient is used to characterize the correlation between the first signal envelope and the second signal envelope; The difference between the first value and the correlation coefficient is determined as the first loss value.

10. The device according to claim 8, characterized in that The audio signal samples include X frames of audio signals, and the noise reduction audio signal includes X frames of audio signals; The processing module is specifically used for: Calculating the Euclidean distance between the first signal envelope and the second signal envelope corresponding to each frame of the audio signal; Determining a weighting coefficient of each of the Euclidean distance values ​​according to the energy value corresponding to each frame of the audio signal; Based on the weighting coefficient, each weighted Euclidean distance value is accumulated, and an average value of the accumulated Euclidean distance values ​​is calculated to obtain a second loss value.

11. The device according to claim 8, characterized in that The device further comprises: an identification module; The processing module is further configured to: train the first denoising model based on the first loss value and the second loss value, obtain the second denoising model, input the first audio signal into the second denoising model, perform denoising on the first audio signal, and output a second audio signal; The recognition module is used to input the second audio signal processed by the processing module into a speech translation model to recognize speech emotion features in the second audio signal; The processing module is further configured to translate the second audio signal based on the speech emotion feature identified by the recognition module, and output a translated audio signal.

12. The device according to claim 11, characterized in that The device also includes: an acquisition module and an adjustment module; The acquisition module is used for the processing module to translate the second audio signal based on the speech emotion feature, and after outputting the translated audio signal, obtain the first volume value of the environmental noise signal from the translated audio signal; The processing module is further configured to determine a second volume value based on the first volume value and the preset volume value acquired by the acquisition module; The adjustment module is used to adjust the volume value of the translated audio signal processed by the processing module to the second volume value determined by the processing module.

13. The device according to claim 12, characterized in that The translated audio signal includes N frames of audio signals, where N is a positive integer; The acquisition module is further configured to acquire an amplitude adjustment coefficient corresponding to the i-th frame audio signal in the N frames of audio signals after the adjustment module adjusts the volume value of the translated audio signal to the second volume value, where i∈[1,N], i is an integer; The adjustment module is further configured to adjust the amplitude of the i-th frame audio signal based on the amplitude adjustment coefficient acquired by the acquisition module; The processing module is further used to perform volume conversion on the amplitude of the i-th frame audio signal adjusted by the adjustment module to obtain a third volume value; The adjustment module is further configured to adjust the volume value of the i-th frame audio signal from the second volume value to the third volume value.

14. The device according to claim 13, characterized in that The acquisition module is specifically used for: Acquire a signal envelope and an envelope error value of the i-th frame audio signal, wherein the envelope error value is a difference between a preset signal envelope and the signal envelope of the i-th frame audio signal; The amplitude adjustment coefficient is determined according to the signal envelope of the i-th frame audio signal and the envelope error value.

15. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the model training method as described in any one of claims 1 to 7 are implemented.

16. A readable storage medium, characterized in that: The readable storage medium stores programs or instructions, and when the programs or instructions are executed by the processor, the steps of the model training method as described in any one of claims 1 to 7 are implemented.

17. A computer program product, wherein the computer program product is stored in a storage medium and is executed by at least one processor to implement the steps of the model training method as described in any one of claims 1 to 7.

Images