Audio noise reduction model training method, audio noise reduction method and device

Abstract

The present disclosure provides a training method for an audio noise reduction model, a method and device for audio noise reduction, and relates to the fields of computer technology, in particular to artificial intelligence, speech technology, and deep learning technology. The training method for the audio noise reduction model includes: inputting the spectrum characteristics of the noise-containing frequency into the noise token model to obtain the noise characteristics; inputting the spectrum characteristics and the noise characteristics into the noise reduction model to obtain the noise-reduced frequency; inputting the noise-reduced frequency into the generative adversarial network to obtain the audio prediction value and the audio true value; using the loss function, according to the noise-reduced frequency, the noise-free frequency, the audio prediction value, the audio true value and the audio preset value, adjusting the parameters of the noise token model, the noise reduction model and the generative adversarial network; when the noise token model, the noise reduction model and the generative adversarial network are adjusted to converge, a trained audio noise reduction model is obtained. According to the solution of the present disclosure, the noise interference in the audio can be reduced, the audio quality can be enhanced, and the audio intelligibility can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to the fields of artificial intelligence, voice technology, and deep learning technology. Background Art

[0002] In real-time audio and video communications, various noises are inevitable, such as loud noises, keyboard tapping, and clamor. To suppress these noises, audio noise reduction (or audio enhancement) is necessary. When an audio signal is disrupted or even drowned out by various background noises, audio noise reduction (or audio enhancement) is used to extract the useful audio signal (or pure audio signal) from the noisy audio signal as much as possible, suppressing or reducing the noise interference. This noise is often present not only at the far-end source but also at the near-end receiver. Summary of the Invention

[0003] The present disclosure provides a training method for an audio noise reduction model, and a method and apparatus for audio noise reduction.

[0004] According to one aspect of the present disclosure, a method for training an audio noise reduction model is provided, comprising:

[0005] Input the spectrum features containing noise frequency into the noise token model to obtain noise features;

[0006] Input the spectrum characteristics and noise characteristics into the noise reduction model to obtain the noise reduction frequency;

[0007] Input the noise-reduced audio into the generative adversarial network to obtain the audio prediction value and the audio true value;

[0008] Using the loss function, adjust the parameters of the noise token model, the noise reduction model, and the generative adversarial network according to the noise-reduced audio, the noise-free frequency corresponding to the noisy frequency, the audio prediction value, the audio ground truth value, and the audio preset value; and

[0009] After adjusting the noise token model, denoising model and generative adversarial network to converge, a trained audio denoising model is obtained.

[0010] According to another aspect of the present disclosure, a method for audio noise reduction is provided, comprising:

[0011] Using a pre-trained audio noise reduction model to process the target noisy audio at the audio transmitter; and

[0012] The noise reduction enhanced audio obtained after processing by the pre-trained audio noise reduction model is sent to the audio receiving end; wherein the pre-trained audio noise reduction model is obtained using the training method of the audio noise reduction model of any embodiment of the present disclosure.

[0013] According to another aspect of the present disclosure, a training device for an audio noise reduction model is provided, comprising:

[0014] A feature module is used to input the spectrum features containing noise frequency into the noise token model to obtain noise features;

[0015] A noise reduction module is used to input the spectrum characteristics and noise characteristics into the noise reduction model to obtain the noise-reduced frequency spectrum;

[0016] A calculation module is used to input the noise reduction audio into the generative adversarial network to obtain the audio prediction value and the audio true value;

[0017] An adjustment module, configured to adjust parameters of the noise token model, the noise reduction model, and the generative adversarial network, respectively, based on the noise-reduced audio, the noise-free audio corresponding to the noisy audio, the audio prediction value, the audio true value, and the audio preset value, using a loss function; and

[0018] The training module is used to obtain a trained audio denoising model when the noise token model, the denoising model and the generative adversarial network are adjusted to converge.

[0019] According to another aspect of the present disclosure, there is provided an apparatus for audio noise reduction, comprising:

[0020] a processing module for processing the target noisy audio at the audio sending end using a pre-trained audio noise reduction model; and

[0021] A sending module is used to send the noise reduction enhanced audio obtained after processing by the pre-trained audio noise reduction model to the audio receiving end; wherein, the pre-trained audio noise reduction model is obtained using the training method of the audio noise reduction model of any embodiment of the present disclosure, or is obtained using the training device of the audio noise reduction model of any embodiment of the present disclosure.

[0022] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0023] at least one processor; and

[0024] a memory communicatively connected to at least one processor; wherein,

[0025] The memory stores instructions that can be executed by at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method in any embodiment of the present disclosure.

[0026] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the audio noise reduction model training method and / or audio noise reduction method in any embodiment of the present disclosure.

[0027] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the audio noise reduction model training method and / or audio noise reduction method in any embodiment of the present disclosure.

[0028] According to the solution disclosed in the present invention, noise interference in audio can be reduced, audio quality can be enhanced, and audio intelligibility can be improved.

[0029] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0031] Figure 1 is a schematic diagram of a training method for an audio noise reduction model according to an embodiment of the present disclosure;

[0032] Figure 2 2 is a schematic diagram of an application of a training method for an audio noise reduction model according to an embodiment of the present disclosure;

[0033] Figure 3 2 is a schematic diagram of an application of a training method for an audio noise reduction model according to an embodiment of the present disclosure;

[0034] Figure 4 is a structural diagram of an audio noise reduction model according to an embodiment of the present disclosure;

[0035] Figure 5 is a structural diagram of a noise token model according to an embodiment of the present disclosure;

[0036] Figure 6 is a structural diagram of a noise reduction model according to an embodiment of the present disclosure;

[0037] Figure 7 is a schematic structural diagram of a generator according to an embodiment of the present disclosure;

[0038] Figure 8 is a schematic structural diagram of a discriminator according to an embodiment of the present disclosure;

[0039] Figure 9 is a schematic diagram of the structure of a generative adversarial network according to an embodiment of the present disclosure;

[0040] Figure 10 is a schematic diagram of the structure of a generative adversarial network according to an embodiment of the present disclosure;

[0041] Figure 11is a schematic diagram of the structure of a generative adversarial network according to an embodiment of the present disclosure;

[0042] Figure 12 is a schematic diagram of a method for audio noise reduction according to an embodiment of the present disclosure;

[0043] Figure 13 is a schematic diagram of an application of the audio noise reduction method according to an embodiment of the present disclosure;

[0044] Figure 14 is a schematic diagram of a training device for an audio noise reduction model according to an embodiment of the present disclosure;

[0045] Figure 15 is a schematic diagram of an apparatus for audio noise reduction according to an embodiment of the present disclosure;

[0046] Figure 16 It is a block diagram of an electronic device used to implement the audio noise reduction method and / or the audio noise reduction model training method of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0047] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0048] The embodiment of the present disclosure provides a method for training an audio noise reduction model, such as Figure 1 As shown, it is a flow chart of the data processing method of this embodiment, which may include the following steps:

[0049] Step S101: Input the spectrum characteristics of the noise-containing frequency into the noise token model to obtain the noise characteristics.

[0050] Step S102: Inputting the spectrum characteristics and noise characteristics into the noise reduction model to obtain the noise-reduced frequency spectrum.

[0051] Step S103: Input the noise reduction audio into the generative adversarial network to obtain an audio prediction value and an audio true value.

[0052] Step S104: Using the loss function, adjust the parameters of the noise token model, the noise reduction model and the generative adversarial network according to the noise reduction audio, the noise-free audio corresponding to the noisy audio, the audio prediction value, the audio true value and the audio preset value.

[0053] Step S105: When the noise token model, the noise reduction model, and the generative adversarial network are adjusted to converge, a trained audio noise reduction model is obtained.

[0054] It should be noted that noisy audio can be understood as speech or audio containing background noise. Background noise can include noise, clamor, keyboard tapping, vehicle horns, etc. in the environment. Voice can include voice sent by a user through a smart terminal (for example, a mobile phone call, an Internet call, or a video call). Audio can include music played by the terminal, videos, or interactive voice generated by the smart terminal. Noisy audio can be understood as one frame of audio in a continuous audio stream. The length and frame shift of each frame of audio can be selected and adjusted as needed.

[0055] A spectral feature can be understood as any feature related to the spectrum of an audio signal or speech signal. The spectral features of the noise-containing frequency spectrum can be extracted from the audio using any method known in the art, such as Fourier transform, short-time Fourier transform, or spectral feature extraction models.

[0056] The network structure of the noise token model can be selected and adjusted as needed, without specific limitations here. As long as the noise characteristics of the noise can be extracted from the noisy frequency spectrum, the noise characteristics can be represented in the form of vectors. The noise characteristics can be any feature that characterizes the noise, and the dimensionality of the noise characteristics can be selected and adjusted as needed.

[0057] The specific noise reduction model employed can be any model known in the art that can achieve audio noise reduction, and is not specifically limited herein. The network structure of the noise reduction model can be selected and adjusted as needed, and is not specifically limited herein. The noise reduction audio output by the noise reduction model can be the noise reduction audio itself, or the spectral characteristics of the noise reduction audio.

[0058] The generator and discriminator in the generative adversarial network can be composed of neural networks or functions, which is not specifically limited in this embodiment.

[0059] The audio prediction value can be the predicted value calculated by the discriminator based on the output of the generator. The audio true value can be calculated based on the noise-reduced audio using a preset function in the generative adversarial network. The preset function can exist independently of the generator and discriminator. The audio preset value can be a pre-set value. The noise-free audio corresponding to the noisy audio can be understood as the actual speech spoken by the speaker at the audio transmitter, or the audio of the audio or video actually played by the audio transmitter.

[0060] Adjusting the parameters of the noise token model, denoising model, and generative adversarial network separately can be understood as adjusting the parameters of all three models in a single parameter adjustment process based on the loss function. It can also be understood as adjusting the parameters of only one or more of the models in a single parameter adjustment process based on the loss function, rather than adjusting the parameters of all models.

[0061] Loss functions can be selected and adjusted as needed. There can be one or more loss functions. For example, the Noise Token Model, the Denoising Model, and the Generative Adversarial Network each have their own loss functions, while the Noise Token Model, the Denoising Model, and the Generative Adversarial Network also share a common loss function. The loss values from these loss functions are used to comprehensively adjust the parameters of the Noise Token Model, the Denoising Model, and the Generative Adversarial Network.

[0062] According to the solution disclosed in the present invention, the trained audio noise reduction model can reduce noise interference in audio, enhance audio quality, and improve audio intelligibility. Since the solution disclosed in the present invention jointly utilizes the noise token model, the noise reduction model and the generative adversarial network to construct the audio noise reduction model, the trained model can process different types of noise, will not limit the upper limit of audio noise, and can process non-stationary noise. The model training method disclosed in the present invention uses the loss function to adjust the parameters of the noise token model, the noise reduction model and the generative adversarial network respectively, realizing the overall consideration and analysis from the transmission link of the audio signal, global modeling optimization, and improving the experience of the listener at the audio receiving end. The audio noise reduction model trained by the solution disclosed in the present invention can be applied to audio signal transmission in audio and video conferencing, entertainment interactive live broadcasts, and online education products. Since the generative adversarial neural network is designed in the trained audio noise reduction model, the generative adversarial neural network can be used to automatically learn and discover the characteristics of audio and noise, thereby enhancing the effect of model training. At the same time, adding generative adversarial networks to model training can effectively avoid the computational complexity and non-differentiability shortcomings of traditional criteria (such as speech quality perception evaluation criteria, virtual speech quality target audience criteria, short-time objective intelligibility criteria, and coherent speech intelligibility index criteria).

[0063] According to an embodiment of the present disclosure, Figure 2 This is a schematic diagram of a distributed cluster processing scenario according to an embodiment of the present disclosure. The distributed cluster system is an example of a cluster system. It exemplifies how the distributed cluster system can be used to train an audio noise reduction model. The present disclosure is not limited to model training on a single machine or multiple machines. Distributed processing can further improve the accuracy of model training. Figure 2As shown, the distributed cluster system includes multiple nodes (such as server cluster 201, server 202, server cluster 203, server 204, server 205, and server 205 can also be connected to electronic devices, such as mobile phone 2051 and desktop computer 2052), and the training tasks of one or more audio noise reduction models can be performed together between multiple nodes, and between multiple nodes and connected electronic devices. Optionally, multiple nodes in the distributed cluster system can adopt a data-parallel audio noise reduction model training method, and multiple nodes can perform audio noise reduction model training tasks based on the same training method to better train the model; if multiple nodes in the distributed cluster system adopt a model-parallel model training method, multiple nodes can perform model training tasks based on different training methods to better train the model. Optionally, after each round of model training is completed, data exchange (such as data synchronization) can be performed between multiple nodes.

[0064] According to an embodiment of the present disclosure, Figure 3 This is an application scenario of the audio noise reduction model trained by the audio noise reduction model training method disclosed in the present invention. The audio transmitting end is the real environment where the speaker is located. The audio receiving end is the real environment where the listener is located. The audio transmitting end includes the voice audio (Hello) spoken by the speaker and the far-end noise in the real environment where the speaker is located (the far-end is the audio transmitting end). The audio receiving end includes the enhanced audio received by the listener and the near-end noise in the real environment where the listener is located (the near-end is the audio receiving end). The enhanced audio is obtained by the audio noise reduction model trained by the audio noise reduction model training method disclosed in the present invention by processing the noisy audio (the voice audio spoken by the speaker + the far-end noise) emitted by the audio transmitting end.

[0065] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, and may further include the steps of: before inputting the spectral features of the noise-containing frequency into the noise token model to obtain the noise features, further including:

[0066] Construct a noisy audio file based on the background noise audio and the noise-free audio file at the audio sending end.

[0067] The short time Fourier transformation (STFT) is used to obtain the spectrum characteristics of the noisy frequency.

[0068] It should be noted that noise-free audio can be understood as the actual spoken voice of the speaker or the audio in the audio or video played by the terminal. Background noise audio can include environmental noise, clamor, keyboard tapping, vehicle horns, and so on. The constructed noise-containing audio can be understood as continuous audio or as a single frame of continuous audio. The length and frame shift of each audio frame can be selected and adjusted as needed.

[0069] According to the disclosed solution, the Short-Time Fourier Transform (STFT) can be used to obtain more accurate spectral characteristics of noisy audio. By constructing noisy audio from background noise audio and noise-free audio, the noisy audio can be made more consistent with the speaker's actual audio transmitter environment in real-world usage. This allows for highly effective audio noise reduction models trained using noisy audio as training samples.

[0070] In one example, the noisy frequency is first divided into frames using a Hanning window, with each frame being W milliseconds (eg, W=64 milliseconds) and a frame shift of H milliseconds (eg, H=8 milliseconds). Spectral features are then obtained by performing a short-time Fourier transform on each frame.

[0071] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, wherein step S102: inputting the spectral features of the noise-containing frequency into the noise token model to obtain the noise features may include:

[0072] The spectrum feature containing the noise frequency is input into the first network of the noise token model to extract the first feature.

[0073] The first feature is operated on the high-dimensional noise matrix of the noise token model to obtain a noise feature, wherein the high-dimensional noise matrix is constructed by high-dimensional features of different types of noise.

[0074] It should be noted that the first feature, noise feature, and high-dimensional features of different types of noise in the embodiments of the present disclosure can all be represented in the form of vectors. In other words, the first feature can be understood as a first vector, the noise feature can be understood as a noise embedding, and the high-dimensional features of different types of noise can be understood as high-dimensional vectors of different types of noise.

[0075] The first feature is operated on the high-dimensional noise matrix of the noise token model, which can be understood as multiplying or adding or subtracting the first feature and the high-dimensional noise matrix.

[0076] Since the high-dimensional noise matrix is constructed from the high-dimensional features of different types of noise, it may include the high-dimensional features of the noise in the noisy frequency. If so, the noise features of the noisy frequency can be accurately extracted through the high-dimensional noise matrix. If not, the high-dimensional noise matrix can also accurately obtain the noise features representing the noisy frequency, because the high-order features of different types of noise in the high-dimensional noise matrix can be combined to represent the noise features of the noisy frequency. By combining the high-dimensional features of known noise, the noise features close to the unknown noise can be represented.

[0077] According to the disclosed solution, the noise token model is designed into the trained audio noise reduction model, effectively modeling the ambient noise. Furthermore, by introducing a high-dimensional noise matrix constructed from the high-dimensional features of different types of noise, this high-dimensional noise matrix can be used as a vector template to obtain noise characteristics of known or unknown noise. By operating the first feature with the high-dimensional noise matrix of the noise token model, the spectral characteristics of the noisy frequency can be mapped to the corresponding subspace, thereby achieving more accurate noise modeling and noise characteristics.

[0078] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, such as Figure 4 As shown in the figure, the first network is composed of N1 two-dimensional convolutional layers (Conv2D), long short-term memory networks (LSTM, Long Short-Term Memory) and multi-head attention layers (Multi-head-attention) connected in sequence, where N1 is a positive integer. Figure 4 In the example shown, N1=6.

[0079] The multi-head attention layer multiplies the output of the long short-term memory network with the high-dimensional noise matrix to obtain a set of coefficients (e.g. Figure 4 0.20, 0.1, 0.7 in the high-dimensional noise matrix). This coefficient can represent the probability that it is a high-dimensional feature of a certain noise type in the high-dimensional noise matrix. Then, this set of coefficients is used to compare with the high-dimensional feature of a certain noise type in the high-dimensional noise matrix (for example, Figure 4 The noise features are finally obtained by weighting the Class A noise high-dimensional features, Class B noise high-dimensional features, and Class C noise high-dimensional features.

[0080] According to the solution disclosed in the present invention, the noise token model constructed includes a two-dimensional convolutional layer, a long short-term memory network, a multi-head attention layer and a high-dimensional noise matrix. Therefore, the noise token model can more accurately extract the required noise characteristics of the noise frequency.

[0081] In a specific example, the calculation formula of the multi-head attention layer is:

[0082]

[0083] The algorithm consists of three variables: Q, K, and V. Q is the output vector of the long short-term memory network layer, and K and V are high-dimensional noise matrices. By multiplying Q and K, a set of correlation coefficients is found, which is then weighted with V to characterize the noise characteristics of the noise-containing frequency.

[0084] In an embodiment of the present disclosure, the input of the noise token model is the spectral characteristics of each frame of audio, and the output is the noise characteristics of the frame, which is then input into the noise reduction model to assist in training the audio noise reduction model's ability to perceive environmental noise, and then perform targeted noise reduction on the audio.

[0085] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, wherein the process of constructing the high-dimensional noise matrix includes:

[0086] Different types of noise are input into the preset noise recognition network respectively.

[0087] High-dimensional features of different types of noise are extracted from the underlying network layers of the preset noise recognition network.

[0088] According to the high-dimensional characteristics of different types of noise, a high-dimensional noise matrix is constructed.

[0089] It should be noted that the preset noise recognition network can adopt any network structure in the prior art as long as it can realize noise recognition.

[0090] The high-dimensional features of different types of noise can be represented by vectors. That is, the high-dimensional features of different types of noise can be understood as high-dimensional vectors of different types of noise. In other words, a high-dimensional noise matrix can be constructed based on the high-dimensional vectors of different types of noise.

[0091] According to the solution disclosed in the present invention, since the features are extracted from the underlying network layer of the preset noise recognition network, the high-dimensional features of different types of noise obtained can more accurately represent the characteristics of the noise, thereby making the constructed high-dimensional noise matrix more effective in applications.

[0092] In one example, the preset noise recognition network can be composed of a 5-layer TDNN (time delay neural network) and / or LSTM. By inputting different types of noise for classification and recognition training, a noise recognition network is obtained. Then, each type of noise is input into the noise recognition network, and a vector is extracted from the penultimate or second layer of the network structure as a high-dimensional vector representing the type of noise, thereby obtaining a vector template corresponding to each type of noise. Among them, the noise used to train the noise recognition network includes open source noise data (for example, Aurora2, HuCorpus, etc.), airports, restaurants, streets, stations, cars, booths, room reverberation and other types of noise.

[0093] In one example, 16 categories of noise can be identified, and 256-dimensional vectors can be extracted from the penultimate or second-layer network structure of the noise identification network. Then, a high-dimensional noise matrix can be constructed using the 256-dimensional vectors of the 16 categories of noise.

[0094] In one embodiment, the training method of the audio noise reduction model provided by the embodiments of the present disclosure includes steps S101 to S105, wherein step S102: inputting spectral features and noise features into the noise reduction model to obtain the noise reduction frequency may include:

[0095] The spectral features and noise features are concatenated and then fed into the encoder of the denoising model. The spectral features are obtained by short-time Fourier transform (STFT) of the noise-containing frequency spectrum.

[0096] The decoder of the noise reduction model is used to reconstruct the audio output of the encoder.

[0097] The reconstructed audio is subjected to an inverse short-time Fourier transform (Inverse STFT) to obtain the noise-reduced audio.

[0098] According to the solution disclosed in the present invention, audio noise reduction processing can be effectively implemented by using an encoder, a decoder and Fourier transform.

[0099] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, such as Figure 5 As shown, the denoising model includes an encoder, a long short-term memory network and a decoder.

[0100] The encoder consists of N2 two-dimensional convolutional layers (Conv2D), normalization layers (layer norm) and linear rectification units (ReLU, Linear rectification function) connected in sequence, where N2 is a positive integer.

[0101] The number of long short-term memory networks is N3, where N3 is a positive integer.

[0102] The decoder includes a real decoder and an imaginary decoder connected in parallel. The real decoder and the imaginary decoder are both composed of N4 two-dimensional convolutional layers, where N4 is a positive integer.

[0103] The linear rectifier unit of the encoder is connected to the input of the first long short-term memory network, and the output of the last long short-term memory network is connected to the real decoder and the imaginary decoder respectively. The encoder is also directly connected (skip connection) to the real decoder and the imaginary decoder.

[0104] According to the solution disclosed in the present invention, since the designed noise reduction model includes an encoder, a long short-term memory network and a decoder, and the encoder is directly connected to the real decoder and the imaginary decoder, it is ensured that the noise reduction model can better implement audio noise reduction processing, so that the output noise reduction frequency can be better utilized by the generative adversarial network, thereby ensuring the training effect of the overall audio noise reduction model.

[0105] In one example, the encoder includes five Conv2D layers, layerNorm, and ReLU. The real decoder includes five Conv2D layers, and the imaginary decoder includes five Conv2D layers. The denoising model includes an encoder, two LSTM layers, an imaginary decoder, and a real decoder.

[0106] The direct connection between the encoder and the real decoder can be understood as the output of the encoder's first convolutional layer (Conv2D) being fed into the real decoder's fifth convolutional layer (Conv2D), the second convolutional layer being fed into the real decoder's fourth convolutional layer, and so on. The direct connection between the encoder and the imaginary decoder follows a similar approach and will not be further explained here.

[0107] The decoder uses a Complex Ideal Ratio Mask (CIRM) network to reconstruct the clean speech signal, predicting its real and imaginary parts. After obtaining these parts, it applies an inverse short-time Fourier transform (STFT) to produce the denoised audio. This approach achieves better noise reduction results.

[0108] In one embodiment, the training method of the audio noise reduction model provided by the embodiments of the present disclosure includes steps S101 to S105, wherein step S103: inputting the noise reduction audio into the generative adversarial network to obtain the audio prediction value and the audio true value, including:

[0109] Step S1031: Input the noise reduction audio and the background noise audio of the audio receiving end into the generator of the generative adversarial network to obtain enhanced audio.

[0110] Step S1032: Input the enhanced audio and the noise reduction audio into the discriminator of the generative adversarial network to obtain an audio prediction value.

[0111] Step S1033: using a preset function, based on the enhanced audio and the noise-reduced audio, to obtain an audio true value.

[0112] It should be noted that the noise reduction audio input into the generator of the generative adversarial network can be understood as the spectral characteristics of the noise reduction audio, and the background noise audio input into the audio receiving end of the generator of the generative adversarial network can be understood as the spectral characteristics of the background noise audio.

[0113] According to the solution disclosed in the present invention, noise not only exists at the audio transmitting end (speaker's side), but also exists at the audio receiving end (listener's side). Most noise reduction technologies only consider the situation at one end, without considering the overall situation together for analysis and optimization to improve the quality and intelligibility of the audio. The solution disclosed in the present invention combines the background noise audio of the audio transmitting end and the audio receiving end, and performs a unified modeling of the noise-containing frequency throughout the process. On the one hand, it reduces background noise interference and improves audio (speech) quality, and on the other hand, it improves the intelligibility of speech. The solution disclosed in the present invention inputs the background noise audio of the audio receiving end into the generator, which can enable the generator to perceive the environmental noise in advance, which is conducive to the generated audio adapting to the surrounding environment and integrating better with the surrounding environment.

[0114] In one example, the preset function may adopt a complementary cumulative distribution function (Q function).

[0115] In one embodiment, the training method of the audio noise reduction model provided by the embodiments of the present disclosure includes steps S101 to S105, wherein step S1032 of step S103: inputting the enhanced audio and the noise reduction audio into the discriminator of the generative adversarial network to obtain the audio prediction value may include:

[0116] The enhanced audio and the noise-reduced audio are input into the first discriminator of the generative adversarial network to obtain the audio quality prediction value.

[0117] The enhanced audio and the noise-reduced audio are fed into the second discriminator of the generative adversarial network to obtain the audio intelligibility prediction value.

[0118] Using preset functions, the true audio value is obtained based on the enhanced audio and the noise-reduced audio, including:

[0119] A first preset function is used to obtain a true value of audio quality based on the enhanced audio and the noise-reduced audio.

[0120] A second preset function is used to obtain a true value of audio intelligibility based on the enhanced audio and the noise-reduced audio.

[0121] It should be noted that the first discriminator is used to determine the intelligibility of the enhanced audio based on the enhanced audio output by the generator. The second discriminator is used to determine the quality of the enhanced audio based on the enhanced audio output by the generator. The first preset function and the second preset function can adopt complementary cumulative distribution functions.

[0122] In the embodiments of the present disclosure, the criteria commonly used to evaluate the quality and intelligibility of speech include perceptual evaluation of speech quality (PESQ, with a score range of -0.5 to 4.5), virtual speech quality objective listener (ViSQOL, with a score range of 1 to 5), short-time objective intelligibility (STOI, with a score range of 0 to 1), and coherence and speech intelligibility index (CSII, with a score range of 0 to +∞). Since these criteria are relatively complex and non-differentiable, it is not easy to directly optimize the audio. Therefore, the present disclosure scheme designs a generative adversarial network in the audio denoising model, using a generator to generate the desired audio, and then using a first discriminator and a second discriminator to determine whether it meets the expectations or how far it is from the expectations. The generative adversarial network is then optimized to achieve the predetermined goal, so that the generative adversarial network has the ability to distinguish and generate. The disclosed solution imparts the traditional ability to measure the quality and intelligibility of speech signals to the first and second discriminant models, enabling the audio noise reduction model to discern the quality and intelligibility of audio, thereby effectively improving the listener experience.

[0123] In one example, if Figure 6 As shown, the general model structure of the audio noise reduction model to be trained is shown. The audio noise reduction model to be trained includes a noise token model, a noise reduction model and a generative adversarial network. The audio transmitter constructs a noisy frequency based on the noise-free frequency (the speaker's voice or the audio and video played by the terminal) and the background noise. The spectral features of the noisy frequency are input into the noise token model and the noise reduction model. The noise token model inputs the noise features obtained based on the noisy frequency into the noise reduction model. The noise reduction model inputs the noise reduction frequency obtained based on the noise features and the spectral features of the noisy frequency into the generator of the generative adversarial network. The generator inputs the enhanced audio obtained into the first discriminator and the second discriminator of the generative adversarial network, and at the same time inputs the noise reduction frequency output by the noise reduction model into the first discriminator and the second discriminator. Using the loss value obtained by the first loss function based on the noise-free frequency and the noise-reduced frequency, the loss value obtained by the second loss function based on the first discriminator and the first preset function, and the loss value obtained by the third loss function based on the second discriminator and the second preset function, a fourth loss function is constructed and the fourth loss value is calculated. Based on the first loss value, the second loss value, the third loss value and the fourth loss value, the parameters of the noise token model, the noise reduction model and the generative adversarial network (generator, first discriminator and second discriminator) are adjusted respectively.

[0124] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, such as Figure 7 As shown in the figure, the generator is composed of N5 second networks, a fully connected layer (FC), an exponential activation layer, and an energy normalization layer, where N5 is a positive integer. The second network is composed of a one-dimensional convolutional layer (Conv1D), a normalization layer (layer Norm), and a linear rectifier unit (ReLU) connected in sequence. The input of the second network is the spectrum of the noise-reduced frequency after short-time Fourier transform (STFT).

[0125] According to the solution disclosed in the present invention, the generator includes a convolution layer, a normalization layer, a linear rectification unit, a fully connected layer, an exponential activation layer and an energy normalization layer. Therefore, the enhanced audio quality output by the constructed generator is higher.

[0126] In one example, the generator structure consists of {{Conv1D+layer Norm+ReLU}*6+FC+ExponentialActivation+Energy Normalization}. The input is the spectrum of the spliced noise reduction audio and the spectrum of the background noise audio at the audio receiving end. After convolution and fully connected layers, the audio spectrum is adjusted by the exponential activation function of the exponential activation layer, the energy of the adjusted spectrum is regularized, and then the enhanced audio is obtained through inverse short-time Fourier transform (inverseSTFT).

[0127] The exponential activation function is as follows:

[0128] α=exp(4*tanh(u))

[0129] Where u is the output of the FC layer. α is the amplification factor, preferably between 0.02 and 55. This factor is then multiplied by the denoised spectrum. If it is greater than 1, the spectrum energy is enhanced, and vice versa.

[0130] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, wherein the generative adversarial network includes a first discriminator and a second discriminator arranged in parallel, the first discriminator is used to calculate the audio quality prediction value, and the second discriminator is used to calculate the audio intelligibility prediction value.

[0131] like Figure 8As shown in the figure, the structure of the first discriminator or the second discriminator is shown. The first discriminator and the second discriminator are both composed of N6 third networks, a global average pooling layer (GAPooling), a first fully connected layer, a leaky linear rectification unit (LReLU, Leaky Linear Rectification function) and a second fully connected layer connected in sequence, where N6 is a positive integer. The third network is composed of a two-dimensional convolutional layer (Conv2D), a normalization layer (layerNorm) and a linear rectification unit (ReLU) connected in sequence.

[0132] According to the scheme disclosed in the present invention, the structure of the discriminator (first discriminator and second discriminator) includes a two-dimensional convolution layer, a normalization layer, a linear rectification unit, a global average pooling layer, a first fully connected layer, a leaky linear rectification unit and a second fully connected layer, so the discriminator can achieve more accurate audio prediction.

[0133] In one example, the first and second discriminators have the same structure, consisting of {Concat+{Conv2D+layerNorm+ReLU}*5+GAPooling+FC+LReLU+FC+sigmod}. The inputs are the denoised audio and the enhanced audio output by the generator, and the outputs are predicted scores corresponding to the PESQ, ViSQOL, STOI, and CSII criteria, all in two dimensions. The audio ground truth is the accurate calculated value obtained by directly applying these criteria using a preset function (the true value calculated according to traditional criteria). This allows the discriminator's predictions to approach the target ground truth through training, enabling the discriminator to evaluate audio quality and intelligibility.

[0134] It should be noted that Concat and sigmod are functions used to perform operations on the input and output of the discriminator (the first discriminator and the second discriminator).

[0135] In one example, if Figure 9 As shown, the generative adversarial network includes: a generator, a first discriminator and a second discriminator. The noise reduction audio output by the noise reduction model is input into the generator, and the generator inputs the obtained enhanced audio into the first discriminator and the second discriminator of the generative adversarial network. At the same time, the noise reduction audio output by the noise reduction model is input into the first discriminator and the second discriminator.

[0136] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, wherein step S104: using a loss function, according to the noise reduction frequency, the noise-free frequency corresponding to the noisy frequency, the audio prediction value, the audio true value and the audio preset value, respectively adjusting the parameters of the noise token model, the noise reduction model and the generative adversarial network, may include:

[0137] A first loss value is calculated using the first loss function according to the noise-reduced frequency and the noise-free frequency corresponding to the noisy frequency.

[0138] A second loss value is calculated using a second loss function based on the audio quality prediction value and the audio quality true value, and / or the audio quality prediction value and the audio quality preset value.

[0139] A third loss value is calculated using a third loss function according to the audio intelligibility prediction value and the audio intelligibility true value, and / or the audio intelligibility prediction value and the audio intelligibility preset value.

[0140] A fourth loss value is calculated based on the first loss value, the second loss value, and the third loss value using the fourth loss function.

[0141] According to the first loss value, the second loss value, the third loss value and the fourth loss value, the parameters of the noise token model, the denoising model and the generative adversarial network are adjusted respectively.

[0142] It should be noted that the first loss function, the second loss function, the third loss function and the fourth loss function can be selected and adjusted as needed and are not specifically limited here.

[0143] According to the scheme disclosed in the present invention, the parameters of the noise token model, the noise reduction model and the generative adversarial network can be adjusted individually through the first loss function, the second loss function and the third loss function. The fourth loss function can be used to globally adjust the parameters of the noise token model, the noise reduction model and the generative adversarial network based on the first loss function, the second loss function and the third loss function, thereby realizing the overall parameter tuning of the audio noise reduction model and completing the model training faster.

[0144] In one example, the first loss function may use a scale-invariant signal-to-noise ratio (SI-SNR) as the loss function. The training objective of the first loss function is to maximize the scale-invariant signal-to-noise ratio. The calculation formula is as follows:

[0145]

[0146] in, represents the noise-reduced frequency, s represents the noise-free frequency, ||s|| 2 =<s,s> Represents the signal power. Here, the reconstructed signal is zero-mean normalized to ensure scale invariance.

[0147] In one example, the second loss function and the third loss function may use mean square error (MSE) as the loss function.

[0148] In one example, the loss calculation for overall model training consists of three parts. One part is the noise reduction model for the audio transmitter, which aims to improve the purity of the audio signal and reduce the impact of noise from the perspective of signal-to-noise ratio. The other two parts are the generative adversarial network for the audio receiver, which aims to give the first and second discriminators the traditional ability to measure the quality and intelligibility of speech signals, and enable the generator to generate the desired audio towards the established goal. This is achieved through deep learning, and the losses introduced are:

[0149] L=L int +α*L qua +β*L sisnr

[0150] Among them, α and β represent hyperparameters, which are empirical values (for example, α is 0.6 and β is 0.05). L represents the fourth loss function, L int is the first loss function, L qua is the second loss function, L sisnr is the third loss function.

[0151] In another embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, wherein step S104: using a loss function, according to the noise reduction frequency, the noise-free frequency corresponding to the noisy frequency, the audio prediction value, the audio true value and the audio preset value, respectively adjusting the parameters of the noise token model, the noise reduction model and the generative adversarial network, may include:

[0152] A fifth loss value is calculated using the fifth loss function according to the noise-reduced frequency and the noise-free frequency corresponding to the noisy frequency.

[0153] The sixth loss value is calculated using the sixth loss function according to the audio prediction value, the audio true value and the audio preset value.

[0154] A seventh loss value is calculated based on the fifth loss value and the sixth loss value using the seventh loss function.

[0155] According to the fifth loss value, the sixth loss value and the seventh loss value, the parameters of the noise token model, the denoising model and the generative adversarial network are adjusted respectively.

[0156] According to the scheme disclosed in the present invention, the fifth loss function and the sixth loss function can be used to adjust the parameters of the noise token model, the noise reduction model and the generative adversarial network individually. The seventh loss function can be used to globally adjust the parameters of the noise token model, the noise reduction model and the generative adversarial network based on the fifth loss function and the sixth loss function, thereby realizing the overall parameter tuning of the audio noise reduction model and completing the model training faster.

[0157] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, wherein step S104: using a loss function, according to the noise reduction frequency, the noise-free frequency corresponding to the noisy frequency, the audio prediction value, the audio true value and the audio preset value, respectively adjusting the parameters of the noise token model, the noise reduction model and the generative adversarial network, may include:

[0158] Using the loss function, the parameters of the noise token model, the denoising model, and the discriminator of the generative adversarial network are adjusted for the i-th time according to the denoised frequency, the noise-free frequency corresponding to the noisy frequency, the audio prediction value, the audio true value, and the audio preset value, where i is a positive integer.

[0159] Based on the parameter-adjusted noise token model, noise reduction model and generative adversarial network, the i+1th noise reduction frequency, audio prediction value and audio true value are obtained according to the spectral characteristics of the noisy frequency.

[0160] Using the loss function, the parameters of the noise token model, denoising model and generator of the generative adversarial network after parameter adjustment are adjusted for the i+1th time according to the noise-free frequency corresponding to the noisy frequency, the audio preset value, the i+1th noise-reduced frequency, the audio prediction value and the audio true value.

[0161] According to the solution disclosed in the present invention, the generator and the discriminator are trained alternately, that is, in the iterative process, the loss is calculated according to the sixth loss function, and then the discriminator is updated in reverse according to the stochastic gradient descent criterion (such as Figure 10 As shown), the generator is not updated for the time being, the remote denoising model and the noise token model are updated, and then the discriminator parameters are fixed. In the next iteration, after calculating the discriminator prediction score, the loss is calculated between the corresponding audio preset value, and then the discriminator is fixed and the generator is updated in reverse (as shown). Figure 11 As shown), denoising model, noise token model, and so on. The loss in the discriminator is calculated as the difference between the audio prediction value and the audio true value calculated by the traditional criterion, while the generator is the difference between the audio prediction value and the audio preset value (for example, the audio preset value is 1, the maximum value of sigmod).

[0162] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, wherein step S105: obtaining a trained audio noise reduction model when adjusting the noise token model, the noise reduction model, and the generative adversarial network to converge, may include:

[0163] When the noise token model, denoising model and generative adversarial network are adjusted to converge, a trained audio denoising model is obtained based on the generator of the noise token model, denoising model and generative adversarial network.

[0164] The embodiment of the present disclosure provides a method for audio noise reduction, such as Figure 12 As shown, it is a flowchart of the audio noise reduction method of this embodiment, which may include the following steps:

[0165] Step S1201: Use the pre-trained audio noise reduction model to process the target noisy audio at the audio sending end.

[0166] Step S1202: Send the noise reduction enhanced audio obtained after processing by the pre-trained audio noise reduction model to the audio receiving end. The pre-trained audio noise reduction model is obtained by using the audio noise reduction model training method of any embodiment of the present disclosure.

[0167] According to the solution of the present disclosure, an audio noise reduction model is obtained by using the training method of the audio noise reduction model of any embodiment of the present disclosure, which can effectively eliminate noise interference at the audio sending end and the audio receiving end, enhance audio quality, improve audio intelligibility, and ensure the listener experience at the audio receiving end.

[0168] According to an embodiment of the present disclosure, Figure 13 This is an application scenario of the audio noise reduction method disclosed in the present invention. The audio transmitting end is the real environment where the speaker is located. The audio receiving end is the real environment where the listener is located. The audio transmitting end includes the voice audio (Hello) spoken by the speaker and the far-end noise in the real environment where the speaker is located (the far-end is the audio transmitting end). The audio receiving end includes the enhanced audio received by the listener and the near-end noise in the real environment where the listener is located (the near-end is the audio receiving end). The enhanced audio is obtained by the audio noise reduction model trained by the training method of the audio noise reduction model disclosed in the present invention by processing the noisy audio (the voice audio spoken by the speaker + the far-end noise) emitted by the audio transmitting end.

[0169] In one embodiment, the audio noise reduction model training method provided by the embodiments of the present disclosure includes steps S1201 and S1202, wherein step S1201: using the pre-trained audio noise reduction model to process the target noisy audio at the audio sending end may include:

[0170] The spectrum features of the target noisy frequency at the audio transmitter are input into the noise token model of the pre-trained audio noise reduction model to obtain the noise features of the target noisy frequency.

[0171] The noise features of the target noisy frequency and the spectrum features of the target noisy frequency are input into the denoising model of the pre-trained audio denoising model to obtain the target noise-reduced frequency.

[0172] The target noise reduction audio is input into the generator of the generative adversarial network of the pre-trained audio noise reduction model to obtain the noise reduction enhanced audio.

[0173] The embodiment of the present disclosure provides a training device for an audio noise reduction model, such as Figure 14 As shown in FIG, it is a structural block diagram of the training device of the audio noise reduction model of this embodiment, and the device may include:

[0174] The feature module 140 is configured to input the spectrum feature containing the noise frequency into the noise token model to obtain the noise feature.

[0175] The noise reduction module 141 is configured to input the spectrum characteristics and the noise characteristics into the noise reduction model to obtain the noise-reduced frequency spectrum.

[0176] The calculation module 142 is used to input the noise reduction audio into the generative adversarial network to obtain the audio prediction value and the audio true value.

[0177] The adjustment module 143 is configured to adjust the parameters of the noise token model, the noise reduction model, and the generative adversarial network respectively according to the noise reduction audio, the noise-free audio corresponding to the noisy audio, the audio prediction value, the audio true value, and the audio preset value using the loss function.

[0178] The training module 144 is used to obtain a trained audio noise reduction model when the noise token model, the noise reduction model and the generative adversarial network are adjusted to converge.

[0179] In one embodiment, the training device for the audio noise reduction model further includes:

[0180] The construction module is used to construct a noisy audio according to the background noise audio and the noise-free audio of the audio sending end.

[0181] The acquisition module is used to obtain the spectrum characteristics of the noise frequency using short-time Fourier transform.

[0182] In one embodiment, the feature module 140 includes:

[0183] The extraction submodule is used to input the spectrum feature containing the noise frequency into the first network of the noise token model to extract the first feature.

[0184] The operator module is configured to operate the first feature with a high-dimensional noise matrix of the noise token model to obtain a noise feature, wherein the high-dimensional noise matrix is constructed from high-dimensional features of different types of noise.

[0185] In one embodiment, the noise reduction module 141 includes:

[0186] The first input submodule is used to input the spectrum features and noise features into the encoder of the noise reduction model, wherein the spectrum features are obtained by short-time Fourier transform of the noisy frequency spectrum.

[0187] The reconstruction submodule is used to reconstruct the audio output of the encoder using the decoder of the noise reduction model.

[0188] The computing submodule is used to perform inverse short-time Fourier transform on the reconstructed audio to obtain the noise-reduced audio.

[0189] In one embodiment, the calculation module 142 includes:

[0190] The second input submodule is used to input the noise reduction audio and the background noise audio of the audio receiving end into the generator of the generative adversarial network to obtain enhanced audio.

[0191] The third input submodule is used to input the enhanced audio and the noise reduction audio into the discriminator of the generative adversarial network to obtain an audio prediction value.

[0192] The truth value submodule is used to obtain the audio truth value based on the enhanced audio and the noise-reduced audio using a preset function.

[0193] In one embodiment, the third input submodule is configured to input the enhanced audio and the noise reduction audio into the first discriminator of the generative adversarial network to obtain an audio quality prediction value. The enhanced audio and the noise reduction audio are input into the second discriminator of the generative adversarial network to obtain an audio intelligibility prediction value. And

[0194] The truth value submodule is configured to obtain a truth value of audio quality based on the enhanced audio and the noise-reduced audio using a first preset function and a truth value of audio intelligibility based on the enhanced audio and the noise-reduced audio using a second preset function.

[0195] In one embodiment, the adjustment module 143 includes:

[0196] The first calculation submodule is configured to calculate a first loss value based on the noise-reduced frequency and the noise-free frequency corresponding to the noisy frequency by using a first loss function.

[0197] The second calculation submodule is used to calculate a second loss value based on the audio quality prediction value and the audio quality true value, and / or the audio quality prediction value and the audio quality preset value using a second loss function.

[0198] The third calculation submodule is used to calculate a third loss value based on the audio intelligibility prediction value and the audio intelligibility true value, and / or the audio intelligibility prediction value and the audio intelligibility preset value using a third loss function.

[0199] The fourth calculation submodule is used to calculate a fourth loss value according to the first loss value, the second loss value and the third loss value by using a fourth loss function.

[0200] The first adjustment submodule is used to adjust the parameters of the noise token model, the noise reduction model and the generative adversarial network according to the first loss value, the second loss value, the third loss value and the fourth loss value.

[0201] In one embodiment, the adjustment module 143 includes:

[0202] The fifth calculation submodule is used to calculate a fifth loss value according to the noise-reduced frequency and the noise-free frequency corresponding to the noisy frequency by using a fifth loss function.

[0203] The sixth calculation submodule is used to calculate a sixth loss value based on the audio prediction value, the audio true value and the audio preset value using a sixth loss function.

[0204] The seventh calculation submodule is used to calculate a seventh loss value according to the fifth loss value and the sixth loss value by using a seventh loss function.

[0205] The second adjustment submodule is used to adjust the parameters of the noise token model, the noise reduction model and the generative adversarial network according to the fifth loss value, the sixth loss value and the seventh loss value.

[0206] In one embodiment, the adjustment module 143 includes:

[0207] The third adjustment submodule is used to use the loss function to adjust the parameters of the noise token model, the noise reduction model and the discriminator of the generative adversarial network for the i-th time according to the noise-reduced frequency, the noise-free frequency corresponding to the noisy frequency, the audio prediction value, the audio true value and the audio preset value, where i is a positive integer.

[0208] The acquisition submodule is used to obtain the i+1th noise reduction frequency, audio prediction value and audio true value based on the spectral characteristics of the noisy frequency based on the parameter-adjusted noise token model, noise reduction model and generative adversarial network.

[0209] The fourth adjustment submodule is used to use the loss function to adjust the parameters of the noise token model, the noise reduction model and the generator of the generative adversarial network for the i+1th time according to the noise-free frequency corresponding to the noisy frequency, the audio preset value, the i+1th noise reduction frequency, the audio prediction value and the audio true value.

[0210] In one embodiment, the training module 144 includes:

[0211] The training submodule is used to obtain a trained audio noise reduction model based on the noise token model, the noise reduction model and the generator of the generative adversarial network when the noise token model, the noise reduction model and the generative adversarial network are adjusted to converge.

[0212] In one embodiment, the training method of the audio noise reduction model provided by the embodiment of the present disclosure includes steps S101 to S105, wherein the first network is composed of N1 two-dimensional convolutional layers, long short-term memory networks and multi-head attention layers connected in sequence, wherein N1 is a positive integer.

[0213] In one embodiment, the process of constructing a high-dimensional noise matrix includes:

[0214] Different types of noise are input into the preset noise recognition network respectively.

[0215] High-dimensional features of different types of noise are extracted from the underlying network layers of the preset noise recognition network.

[0216] According to the high-dimensional characteristics of different types of noise, a high-dimensional noise matrix is constructed.

[0217] In one embodiment, the denoising model includes an encoder, a long short-term memory network, and a decoder.

[0218] The encoder consists of N2 two-dimensional convolutional layers, normalization layers and linear rectification units connected in sequence, where N2 is a positive integer.

[0219] The number of long short-term memory networks is N3, where N3 is a positive integer.

[0220] The decoder includes a real decoder and an imaginary decoder connected in parallel. Both the real decoder and the imaginary decoder are composed of N4 two-dimensional convolutional layers, where N4 is a positive integer.

[0221] The linear rectifier unit of the encoder is connected to the input of the first long short-term memory network, and the output of the last long short-term memory network is connected to the real decoder and the imaginary decoder respectively. The first two-dimensional convolutional layer of the encoder is also connected to the real decoder and the imaginary decoder.

[0222] In one embodiment, the generator is composed of N5 second networks, fully connected layers, exponential activation layers and energy normalization layers connected in sequence, where N5 is a positive integer, and the second network is composed of a one-dimensional convolutional layer, a normalization layer and a linear rectification unit connected in sequence.

[0223] In one embodiment, the generative adversarial network includes a first discriminator and a second discriminator arranged in parallel, the first discriminator is used to calculate the audio quality prediction value, and the second discriminator is used to calculate the audio intelligibility prediction value.

[0224] Both the first discriminator and the second discriminator are composed of N6 third networks, a global average pooling layer, a first fully connected layer, a leaky linear rectifier unit and a second fully connected layer connected in sequence, where N6 is a positive integer, and the third network is composed of a two-dimensional convolutional layer, a normalization layer and a linear rectifier unit connected in sequence.

[0225] The embodiment of the present disclosure provides an audio noise reduction device, such as Figure 15 As shown in FIG, which is a structural block diagram of the audio noise reduction device of this embodiment, the device may include:

[0226] The processing module 150 is used to process the target noisy audio at the audio sending end using the pre-trained audio noise reduction model.

[0227] The sending module 151 is configured to send the noise-reduced and enhanced audio obtained after processing by the pre-trained audio noise reduction model to an audio receiving end. The pre-trained audio noise reduction model is obtained using the audio noise reduction model training method of any embodiment of the present disclosure, or using the audio noise reduction model training device of any embodiment of the present disclosure.

[0228] In one embodiment, the processing module 150 includes:

[0229] The spectrum feature input submodule is used to input the spectrum features of the target noisy frequency at the audio sending end into the noise token model of the pre-trained audio noise reduction model to obtain the noise features of the target noisy frequency.

[0230] The target denoising submodule is used to input the noise characteristics and the spectral characteristics of the target noisy frequency into the denoising model of the pre-trained audio denoising model to obtain the target denoised frequency.

[0231] The denoising enhancement submodule is used to input the target denoised audio into the generator of the generative adversarial network of the pre-trained audio denoising model to obtain denoised enhanced audio.

[0232] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0233] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0234] Figure 16A schematic block diagram of an example electronic device 1600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0235] like Figure 16 As shown, device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded from a storage unit 1608 into a random access memory (RAM) 1603. Various programs and data required for the operation of device 1600 can also be stored in RAM 1603. Computing unit 1601, ROM 1602, and RAM 1603 are connected to each other via a bus 1604. An input / output (I / O) interface 1605 is also connected to bus 1604.

[0236] Various components in device 1600 are connected to I / O interface 1605, including an input unit 1606, such as a keyboard and mouse; an output unit 1607, such as various types of displays and speakers; a storage unit 1608, such as a magnetic disk and optical disk; and a communication unit 1609, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1609 allows device 1600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0237] The computing unit 1601 can be various general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1601 performs the various methods and processes described above, such as the audio noise reduction model training method and / or the audio noise reduction method. For example, in some embodiments, the audio noise reduction model training method and / or the audio noise reduction method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 1608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1600 via the ROM 1602 and / or the communication unit 1609. When the computer program is loaded into the RAM 1603 and executed by the computing unit 1601, one or more steps of the audio noise reduction model training method and / or the audio noise reduction method described above can be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to execute the audio noise reduction model training method and / or the audio noise reduction method in any other appropriate manner (for example, by means of firmware).

[0238] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system comprising at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0239] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0240] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0241] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0242] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0243] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0244] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0245] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for training an audio noise reduction model, comprising: Input the spectrum features containing noise frequency into the noise token model to obtain noise features; The noise token model is used to extract noise features of the noise from the noisy frequency spectrum; Inputting the frequency spectrum characteristics and the noise characteristics into a noise reduction model to obtain a noise reduction frequency spectrum; Inputting the noise-reduced audio into a generative adversarial network to obtain an audio prediction value and an audio true value; Using a loss function, adjust the parameters of the noise token model, the noise reduction model, and the generative adversarial network according to the noise-reduced audio, the noise-free audio corresponding to the noisy audio, the audio prediction value, the audio true value, and the audio preset value; as well as When the noise token model, the noise reduction model, and the generative adversarial network are adjusted to converge, a trained audio noise reduction model is obtained.

2. The method according to claim 1, before inputting the spectrum characteristics of the noise-containing frequency into the noise token model to obtain the noise characteristics, further comprising: Construct a noisy audio frequency based on the background noise audio frequency and the noise-free audio frequency of the audio sending end; The frequency spectrum characteristics of the noisy frequency are obtained by short-time Fourier transform.

3. The method according to claim 1, wherein The process of inputting the spectrum characteristics of the noise-containing frequency into the noise token model to obtain the noise characteristics includes: Inputting the spectrum feature containing the noise frequency into the first network of the noise token model to extract the first feature; The first feature is operated on the high-dimensional noise matrix of the noise token model to obtain a noise feature; wherein the high-dimensional noise matrix is constructed by high-dimensional features of different types of noise.

4. The method according to claim 1, wherein The step of inputting the spectrum feature and the noise feature into a noise reduction model to obtain a noise reduction frequency comprises: Inputting the spectral features and the noise features into an encoder of a noise reduction model; wherein the spectral features are obtained by short-time Fourier transforming the noisy frequency; Reconstructing the audio output of the encoder using the decoder of the noise reduction model; The reconstructed audio is subjected to inverse short-time Fourier transform to obtain the noise-reduced audio.

5. The method according to claim 1, wherein Inputting the noise reduction audio into a generative adversarial network to obtain an audio prediction value and an audio true value includes: Inputting the noise reduction audio and the background noise audio of the audio receiving end into a generator of a generative adversarial network to obtain enhanced audio; Inputting the enhanced audio and the noise-reduced audio into a discriminator of the generative adversarial network to obtain an audio prediction value; A preset function is used to obtain an audio true value based on the enhanced audio and the noise-reduced audio.

6. The method according to claim 5, wherein: Inputting the enhanced audio and the noise reduction audio into the discriminator of the generative adversarial network to obtain an audio prediction value includes: Inputting the enhanced audio and the noise reduction audio into a first discriminator of the generative adversarial network to obtain an audio quality prediction value; Inputting the enhanced audio and the noise-reduced audio into a second discriminator of the generative adversarial network to obtain an audio intelligibility prediction value; and The obtaining of an audio true value based on the enhanced audio and the noise-reduced audio by using a preset function includes: Obtaining a true audio quality value based on the enhanced audio and the noise-reduced audio using a first preset function; A second preset function is used to obtain a true value of audio intelligibility based on the enhanced audio and the noise-reduced audio.

7. The method according to claim 6, wherein: The loss function is used to adjust the parameters of the noise token model, the noise reduction model, and the generative adversarial network according to the noise reduction frequency, the noise-free frequency corresponding to the noisy frequency, the audio prediction value, the audio true value, and the audio preset value, including: Calculating a first loss value based on the noise-reduced frequency and the noise-free frequency corresponding to the noisy frequency using a first loss function; Calculating a second loss value according to the audio quality prediction value and the audio quality true value, and / or the audio quality prediction value and the audio quality preset value, using a second loss function; Calculating a third loss value using a third loss function based on the audio intelligibility prediction value and the audio intelligibility true value, and / or the audio intelligibility prediction value and the audio intelligibility preset value; Calculating a fourth loss value based on the first loss value, the second loss value, and the third loss value using a fourth loss function; According to the first loss value, the second loss value, the third loss value and the fourth loss value, the parameters of the noise token model, the denoising model and the generative adversarial network are adjusted respectively.

8. The method according to claim 1, wherein The loss function is used to adjust the parameters of the noise token model, the noise reduction model, and the generative adversarial network according to the noise reduction frequency, the noise-free frequency corresponding to the noisy frequency, the audio prediction value, the audio true value, and the audio preset value, including: Calculating a fifth loss value based on the noise-reduced frequency and the noise-free frequency corresponding to the noisy frequency using a fifth loss function; Calculating a sixth loss value according to the audio prediction value, the audio true value, and the audio preset value using a sixth loss function; Calculating a seventh loss value according to the fifth loss value and the sixth loss value using a seventh loss function; According to the fifth loss value, the sixth loss value and the seventh loss value, the parameters of the noise token model, the denoising model and the generative adversarial network are adjusted respectively.

9. The method according to claim 1, wherein The loss function is used to adjust the parameters of the noise token model, the noise reduction model, and the generative adversarial network according to the noise reduction frequency, the noise-free frequency corresponding to the noisy frequency, the audio prediction value, the audio true value, and the audio preset value, including: Using a loss function, adjust the parameters of the noise token model, the noise reduction model, and the discriminator of the generative adversarial network for an i-th time according to the denoised audio, the noise-free audio corresponding to the noisy audio, the audio prediction value, the audio true value, and the audio preset value, where i is a positive integer; Based on the noise token model, noise reduction model and generative adversarial network after parameter adjustment, according to the spectral characteristics of the noisy frequency, the i+1th noise reduction frequency, the audio prediction value and the audio true value are obtained; Utilizing the loss function, the parameters of the noise token model, the noise reduction model, and the generator of the generative adversarial network after parameter adjustment are adjusted for the i+1th time according to the noise-free frequency corresponding to the noisy frequency, the audio preset value, the i+1th noise reduction frequency, the audio prediction value, and the audio true value.

10. The method according to claim 5, wherein The method further comprises: obtaining a trained audio noise reduction model when the noise token model, the noise reduction model, and the generative adversarial network are all adjusted to converge. When the noise token model, the noise reduction model and the generative adversarial network are adjusted to converge, a trained audio noise reduction model is obtained based on the noise token model, the noise reduction model and the generator of the generative adversarial network.

11. The method according to claim 3, wherein: The first network is composed of N1 two-dimensional convolutional layers, long short-term memory networks and multi-head attention layers connected in sequence, where N1 is a positive integer.

12. The method according to claim 3 or 11, wherein: The construction process of the high-dimensional noise matrix includes: Inputting the different types of noise into a preset noise recognition network respectively; Extracting high-dimensional features of the different types of noise from the bottom network layer of the preset noise recognition network; The high-dimensional noise matrix is constructed according to the high-dimensional features of the different types of noise.

13. The method according to claim 4, wherein: The denoising model includes the encoder, the long short-term memory network and the decoder; The encoder is composed of N2 two-dimensional convolutional layers, normalization layers and linear rectification units connected in sequence, where N2 is a positive integer; The number of the long short-term memory networks is N3, where N3 is a positive integer; The decoder includes a real part decoder and an imaginary part decoder arranged in parallel, and each of the real part decoder and the imaginary part decoder is composed of N4 two-dimensional convolutional layers, where N4 is a positive integer; The linear rectification unit of the encoder is connected to the input end of the first long short-term memory network, and the output end of the last long short-term memory network is connected to the real part decoder and the imaginary part decoder respectively. The encoder is also connected to the real part decoder and the imaginary part decoder.

14. The method according to claim 5, wherein The generator is composed of N5 second networks, a fully connected layer, an exponential activation layer and an energy normalization layer connected in sequence, where N5 is a positive integer, and the second network is composed of a one-dimensional convolutional layer, a normalization layer and a linear rectification unit connected in sequence.

15. The method according to claim 5 or 6, wherein: The generative adversarial network includes a first discriminator and a second discriminator arranged in parallel, the first discriminator is used to calculate an audio quality prediction value, and the second discriminator is used to calculate an audio intelligibility prediction value; Both the first discriminator and the second discriminator are composed of N6 third networks, a global average pooling layer, a first fully connected layer, a leaky linear rectification unit and a second fully connected layer connected in sequence, where N6 is a positive integer, and the third network is composed of a two-dimensional convolutional layer, a normalization layer and a linear rectification unit connected in sequence.

16. A method for audio noise reduction, comprising: Use the pre-trained audio noise reduction model to process the target noisy audio at the audio transmitter; as well as The noise reduction enhanced audio obtained after processing by the pre-trained audio noise reduction model is sent to an audio receiving end; wherein the pre-trained audio noise reduction model is obtained by the method according to any one of claims 1 to 15.

17. The method according to claim 16, wherein The method of using a pre-trained audio noise reduction model to process the target noisy audio at the audio transmitter includes: Inputting the spectral features of the target noisy frequency at the audio transmitting end into the noise token model of the pre-trained audio noise reduction model to obtain the noise features of the target noisy frequency; Inputting the noise characteristics of the target noisy frequency and the spectrum characteristics of the target noisy frequency into the noise reduction model of the pre-trained audio noise reduction model to obtain the target noise-reduced frequency; The target noise reduction audio is input into the generator of the generative adversarial network of the pre-trained audio noise reduction model to obtain noise reduction enhanced audio.

18. A training device for an audio noise reduction model, comprising: A feature module is used to input the spectrum features containing noise frequency into the noise token model to obtain noise features; The noise token model is used to extract noise features of the noise from the noisy frequency spectrum; A noise reduction module, configured to input the spectrum characteristics and the noise characteristics into a noise reduction model to obtain a noise-reduced frequency spectrum; A calculation module, configured to input the noise reduction audio into a generative adversarial network to obtain an audio prediction value and an audio true value; an adjustment module, configured to adjust parameters of the noise token model, the noise reduction model, and the generative adversarial network, respectively, using a loss function according to the noise-reduced audio, the noise-free audio corresponding to the noisy audio, the audio prediction value, the audio true value, and the audio preset value; as well as A training module is used to obtain a trained audio noise reduction model when the noise token model, the noise reduction model and the generative adversarial network are adjusted to converge.

19. The apparatus according to claim 18, further comprising: A construction module, configured to construct a noisy audio according to the background noise audio and the noise-free audio of the audio sending end; The acquisition module is used to obtain the spectrum characteristics of the noisy frequency by using short-time Fourier transform.

20. The apparatus according to claim 18, wherein The feature module includes: an extraction submodule, configured to input the spectrum feature containing the noise frequency into the first network of the noise token model to extract the first feature; An operation submodule is used to operate the first feature with the high-dimensional noise matrix of the noise token model to obtain a noise feature; wherein the high-dimensional noise matrix is constructed by high-dimensional features of different types of noise.

21. The apparatus according to claim 18, wherein The noise reduction module includes: A first input submodule, configured to input the spectral features and the noise features into an encoder of a noise reduction model; wherein the spectral features are obtained by short-time Fourier transforming the noisy frequency spectrum; A reconstruction submodule, configured to reconstruct the audio output of the encoder using the decoder of the noise reduction model; The computing submodule is used to perform inverse short-time Fourier transform on the reconstructed audio to obtain the noise-reduced audio.

22. The apparatus according to claim 18, wherein The calculation module includes: A second input submodule is configured to input the noise reduction audio and background noise of the audio receiving end into a generator of a generative adversarial network to obtain enhanced audio; A third input submodule, configured to input the enhanced audio and the noise-reduced audio into a discriminator of the generative adversarial network to obtain an audio prediction value; The truth value submodule is configured to obtain an audio truth value based on the enhanced audio and the noise reduction audio using a preset function.

23. The device according to claim 22, wherein The third input submodule is configured to input the enhanced audio and the noise reduction audio into the first discriminator of the generative adversarial network to obtain an audio quality prediction value; Inputting the enhanced audio and the noise-reduced audio into a second discriminator of the generative adversarial network to obtain an audio intelligibility prediction value; as well as The truth value submodule is configured to obtain a true value of audio quality based on the enhanced audio and the noise-reduced audio using a first preset function; A second preset function is used to obtain a true value of audio intelligibility based on the enhanced audio and the noise-reduced audio.

24. The device according to claim 23, wherein The adjustment module includes: a first calculation submodule, configured to calculate a first loss value based on the noise-reduced frequency and the noise-free frequency corresponding to the noisy frequency using a first loss function; a second calculation submodule, configured to calculate a second loss value based on the audio quality prediction value and the audio quality true value, and / or the audio quality prediction value and the audio quality preset value, using a second loss function; a third calculation submodule, configured to calculate a third loss value based on the audio intelligibility prediction value and the audio intelligibility true value, and / or the audio intelligibility prediction value and the audio intelligibility preset value, using a third loss function; a fourth calculation submodule, configured to calculate a fourth loss value according to the first loss value, the second loss value, and the third loss value by using a fourth loss function; The first adjustment submodule is used to adjust the parameters of the noise token model, the denoising model and the generative adversarial network according to the first loss value, the second loss value, the third loss value and the fourth loss value.

25. The apparatus according to claim 18, wherein The adjustment module includes: a fifth calculation submodule, configured to calculate a fifth loss value according to the noise-reduced frequency and the noise-free frequency corresponding to the noisy frequency by using a fifth loss function; a sixth calculation submodule, configured to calculate a sixth loss value according to the audio prediction value, the audio true value, and the audio preset value by using a sixth loss function; a seventh calculation submodule, configured to calculate a seventh loss value according to the fifth loss value and the sixth loss value by using a seventh loss function; The second adjustment submodule is used to adjust the parameters of the noise token model, the noise reduction model and the generative adversarial network according to the fifth loss value, the sixth loss value and the seventh loss value.

26. The apparatus according to claim 18, wherein The adjustment module includes: a third adjustment submodule, configured to adjust the parameters of the noise token model, the noise reduction model, and the discriminator of the generative adversarial network for an i-th time, respectively, using a loss function according to the denoised audio, the noise-free audio corresponding to the noisy audio, the audio prediction value, the audio true value, and the audio preset value, where i is a positive integer; An acquisition submodule is configured to obtain the (i+1)th noise-reduced audio, the audio prediction value, and the audio true value based on the spectral characteristics of the noisy audio based on the noise token model, the noise reduction model, and the generative adversarial network after parameter adjustment; The fourth adjustment submodule is used to use the loss function to adjust the parameters of the noise token model, noise reduction model and generator of the generative adversarial network for the i+1th time according to the noise-free frequency corresponding to the noisy frequency, the audio preset value, the i+1th noise reduction frequency, the audio prediction value and the audio true value.

27. The apparatus according to claim 22, wherein the training module comprises: A training submodule is used to obtain a trained audio noise reduction model based on the noise token model, the noise reduction model and the generator of the generative adversarial network when the noise token model, the noise reduction model and the generative adversarial network are adjusted to converge.

28. An audio noise reduction device, comprising: A processing module, configured to process the target noisy audio at the audio transmitter using a pre-trained audio noise reduction model; as well as A sending module, used to send the noise reduction and enhanced audio obtained after processing by the pre-trained audio noise reduction model to an audio receiving end; wherein, the pre-trained audio noise reduction model is obtained by the method described in any one of claims 1 to 15, or by the device described in any one of claims 18 to 27.

29. The apparatus according to claim 28, wherein The processing module includes: A spectrum feature input submodule, configured to input the spectrum feature of the target noisy frequency at the audio transmitting end into the noise token model of the pre-trained audio noise reduction model to obtain the noise feature of the target noisy frequency; a target noise reduction submodule, configured to input the noise characteristics of the target noisy frequency and the spectral characteristics of the target noisy frequency into the noise reduction model of the pre-trained audio noise reduction model to obtain the target noise-reduced frequency; The noise reduction enhancement submodule is used to input the target noise reduction audio into the generator of the generative adversarial network of the pre-trained audio noise reduction model to obtain noise reduction enhanced audio.

30. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 17.

31. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 17.

32. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 17.

Images