Audio signal processing method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to an audio signal processing method, device, electronic equipment and storage medium, the method comprising: obtaining a to-be-repaired audio signal; the to-be-repaired audio signal is a distorted audio signal containing an interference signal; processing the to-be-repaired audio signal according to a pre-trained noise reduction processing model to obtain a noise-reduced audio signal; the noise reduction processing model is trained based on a first audio sample containing an interference signal; inputting the to-be-repaired audio signal and the noise-reduced audio signal into a pre-trained audio repair model to obtain a repaired target audio signal; the audio repair model is trained based on a second audio sample constructed based on a sample distorted audio signal and a sample noise-reduced audio signal corresponding to the sample distorted audio signal. By using the method, the accuracy and quality of audio repair are improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of audio technology, and in particular, to an audio signal processing method and device, an electronic device, and a storage medium. BACKGROUND

[0002] With the development of audio technology, an audio restoration technology appears, which can realize noise reduction and repair of problem audio.

[0003] The current audio restoration method selects a problem audio for a problem audio that may exist with noise, reverberation or clipping, and performs audio restoration training on a deep learning model based on the problem audio and the normal audio corresponding to the problem audio, to obtain a trained deep learning model. Based on the trained deep learning model, any problem audio is processed for audio restoration to obtain a restored repaired audio.

[0004] However, in the current audio restoration method, since the problem audio usually has more than one problem, the output repaired audio still has unclear pronunciation, some pronunciation is still missing, and the audio repair quality is poor, according to the centralized learning of various problems in the problem audio by the deep learning model. SUMMARY

[0005] The present disclosure provides an audio signal processing method, device, electronic device and storage medium to at least solve the problem of low audio repair accuracy and poor quality in related technologies. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of an embodiment of the present disclosure, an audio signal processing method is provided, comprising:

[0007] obtaining a to-be-repaired audio signal; the to-be-repaired audio signal is a distorted audio signal containing an interference signal;

[0008] processing the to-be-repaired audio signal according to a pre-trained noise reduction processing model to obtain a noise-reduced audio signal; the noise reduction processing model is trained based on a first audio sample containing an interference signal; the noise-reduced audio signal is a distorted audio signal in which the interference signal is eliminated;

[0009] inputting the to-be-repaired audio signal and the noise-reduced audio signal into a pre-trained audio repair model to obtain a repaired target audio signal; the audio repair model is trained based on a second audio sample constructed based on a sample distorted audio signal and a sample noise-reduced audio signal corresponding to the sample distorted audio signal.

[0010] In an exemplary embodiment, the obtaining a to-be-repaired audio signal comprises:

[0011] Acquire the first audio signal to be repaired; the first audio signal is a distorted audio signal containing interference signals;

[0012] Perform a short-time Fourier transform on the first audio signal to obtain the first converted audio signal in the time-frequency domain;

[0013] The first converted audio signal is subjected to Mel frequency cepstral conversion processing to obtain the audio signal to be repaired.

[0014] In one exemplary embodiment, processing the audio signal to be repaired according to a pre-trained noise reduction processing model to obtain a noise-reduced audio signal includes:

[0015] The audio signal to be repaired, containing interference signals, is input into a pre-trained noise reduction processing model; the interference signals include noise signals and reverberation signals.

[0016] The noise signal and reverberation signal in the audio signal to be repaired are processed by a pre-trained noise reduction processing model to obtain a noise-reduced frequency signal that eliminates the noise signal and the reverberation signal.

[0017] In an exemplary embodiment, after inputting the audio signal to be repaired and the noise-reduced audio signal into a pre-trained audio repair model to obtain the repaired target audio signal, the method includes:

[0018] The target audio signal is input to an audio encoder, which converts the target audio signal on the Mel frequency cepstral to obtain a repaired audio signal; the repaired audio signal is the time-domain audio signal after the repaired audio signal is repaired.

[0019] In one exemplary embodiment, the training process of the noise reduction model includes:

[0020] A first audio sample is acquired, the first audio sample containing a first sample audio signal and the original audio signal corresponding to the first sample audio signal; the first sample audio signal contains interference signals.

[0021] The first sample audio signal is input into the noise reduction processing model, and the noise reduction processing model processes the first sample audio signal to obtain the first processed audio signal.

[0022] Loss calculation is performed based on the first processed audio signal and the original audio signal to obtain the loss result corresponding to the first processed audio signal;

[0023] Determine whether the loss result meets the preset loss condition, and determine that the noise reduction processing model training is complete when the loss result meets the preset loss condition; the first processed audio signal output by the trained noise reduction processing model is the noise reduction frequency signal.

[0024] In one exemplary embodiment, the training process of the audio restoration model includes:

[0025] A second audio sample is obtained, which includes a second sample audio signal, a sample noise-reduced frequency signal corresponding to the second sample audio signal, and the original audio signal corresponding to the second sample audio signal; the second sample audio signal is a sample distorted audio signal containing interference signals.

[0026] The second sample audio signal and the sample noise-reduced frequency signal are input into the audio restoration model. The second sample audio signal and the sample noise-reduced frequency signal are processed by the audio restoration model to obtain the second processed audio signal.

[0027] Loss calculation is performed based on the original audio signal corresponding to the second processed audio signal and the second sample audio signal to obtain the loss result corresponding to the second processed audio signal;

[0028] Determine whether the loss result meets the preset loss condition, and determine that the audio restoration model training is complete when the loss result meets the preset loss condition.

[0029] In one exemplary embodiment, obtaining the second audio sample includes:

[0030] Acquire a second audio signal and the original audio signal corresponding to the second audio signal; the second audio signal is a sample-distorted audio signal containing interference signals;

[0031] Short-time Fourier transforms are performed on the second audio signal and the original audio signal corresponding to the second audio signal, respectively, to obtain the second sample audio signal in the time-frequency domain and the original audio signal in the time-frequency domain;

[0032] The second sample audio signal in the time-frequency domain is processed according to the pre-trained noise reduction processing model to obtain the sample noise-reduced frequency signal corresponding to the second sample audio signal;

[0033] The second audio sample is constructed based on the second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced audio signal in the time-frequency domain.

[0034] In one exemplary embodiment, before the construction to obtain the second audio sample, the method further includes:

[0035] The second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced frequency signal are respectively subjected to spectrum transformation to obtain the second sample audio signal in the Mel frequency cepstral, the original audio signal in the Mel frequency cepstral, and the sample noise-reduced frequency signal in the time-frequency domain.

[0036] The construction yields a second audio sample, including:

[0037] The second audio sample is constructed based on the second sample audio signal on the Mel frequency cepstral, the original audio signal on the Mel frequency cepstral, and the sample noise-reduced audio signal on the Mel frequency cepstral.

[0038] According to a second aspect of the present disclosure, an audio signal processing apparatus is provided, comprising:

[0039] The acquisition unit is configured to acquire the audio signal to be repaired; the audio signal to be repaired is a distorted audio signal containing interference signals;

[0040] An interference removal unit is configured to process the audio signal to be repaired according to a pre-trained noise reduction processing model to obtain a noise-reduced audio signal; the noise reduction processing model is trained based on a first audio sample containing interference signals; the noise-reduced audio signal is a distorted audio signal that eliminates the interference signals.

[0041] The repair unit is configured to input the audio signal to be repaired and the noise-reduced frequency signal into a pre-trained audio repair model to obtain the repaired target audio signal; the audio repair model is trained based on a second audio sample constructed from the sample distorted audio signal and the sample noise-reduced frequency signal corresponding to the sample distorted audio signal.

[0042] In one exemplary embodiment, the acquisition unit includes:

[0043] The acquisition subunit is configured to acquire a first audio signal to be repaired; the first audio signal is a distorted audio signal containing interference signals.

[0044] The first conversion subunit is configured to perform a short-time Fourier transform on the first audio signal to obtain the first converted audio signal in the time-frequency domain;

[0045] The second conversion subunit is configured to perform Mel frequency cepstral conversion processing on the first converted audio signal to obtain the audio signal to be repaired.

[0046] In an exemplary embodiment, the interference cancellation unit includes:

[0047] The input subunit is configured to perform the input of the audio signal to be repaired, which contains interference signals, into a pre-trained noise reduction processing model; the interference signals include noise signals and reverberation signals.

[0048] The interference elimination subunit is configured to process the noise signal and reverberation signal in the audio signal to be repaired through a pre-trained noise reduction processing model to obtain a noise-reduced frequency signal that eliminates the noise signal and the reverberation signal.

[0049] In one exemplary embodiment, the apparatus further includes:

[0050] The restoration unit is configured to input the target audio signal to an audio encoder, and convert the target audio signal on the Mel frequency cepstral spectrum through the audio encoder to obtain a repaired audio signal; the repaired audio signal is the time-domain audio signal after the repaired audio signal is restored.

[0051] In one exemplary embodiment, the apparatus further includes:

[0052] The first sample acquisition unit is configured to acquire a first audio sample, the first audio sample including a first sample audio signal and the original audio signal corresponding to the first sample audio signal; the first sample audio signal includes an interference signal.

[0053] The first sample processing unit is configured to input the first sample audio signal into a noise reduction processing model, and process the first sample audio signal through the noise reduction processing model to obtain a first processed audio signal.

[0054] The first loss calculation unit is configured to perform loss calculation based on the first processed audio signal and the original audio signal to obtain the loss result corresponding to the first processed audio signal.

[0055] The first discrimination unit is configured to judge whether the loss result meets the preset loss condition, and determine that the noise reduction processing model training is completed when the loss result meets the preset loss condition; the first processed audio signal output by the trained noise reduction processing model is the noise reduction frequency signal.

[0056] In one exemplary embodiment, the apparatus further includes:

[0057] The second sample acquisition unit is configured to acquire a second audio sample, the second audio sample including a second sample audio signal, a sample noise-reduced audio signal corresponding to the second sample audio signal, and the original audio signal corresponding to the second sample audio signal; the second sample audio signal is a sample distorted audio signal containing interference signals.

[0058] The second sample processing unit is configured to input the second sample audio signal and the sample noise-reduced frequency signal into the audio restoration model, and process the second sample audio signal and the sample noise-reduced frequency signal through the audio restoration model to obtain a second processed audio signal;

[0059] The second loss calculation unit is configured to perform loss calculation based on the original audio signal corresponding to the second processed audio signal and the second sample audio signal, and obtain the loss result corresponding to the second processed audio signal.

[0060] The second discrimination unit is configured to judge whether the loss result meets the preset loss condition, and determine that the audio restoration model training is complete when the loss result meets the preset loss condition.

[0061] In one exemplary embodiment, the second sample acquisition unit includes:

[0062] The second acquisition subunit is configured to acquire a second audio signal and the original audio signal corresponding to the second audio signal; the second audio signal is a sample-distorted audio signal containing interference signals.

[0063] The third conversion subunit is configured to perform short-time Fourier transform on the second audio signal and the original audio signal corresponding to the second audio signal, respectively, to obtain the second sample audio signal in the time-frequency domain and the original audio signal in the time-frequency domain.

[0064] The processing subunit is configured to perform processing on the second sample audio signal in the time-frequency domain according to the pre-trained noise reduction processing model to obtain the sample noise-reduced frequency signal corresponding to the second sample audio signal;

[0065] The construction subunit is configured to perform the operation of constructing a second audio sample based on the second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced audio signal in the time-frequency domain.

[0066] In one exemplary embodiment, the apparatus further includes:

[0067] The fourth conversion subunit is configured to perform spectral conversion on the second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced frequency signal, respectively, to obtain the second sample audio signal in the Mel frequency cepstral, the original audio signal in the Mel frequency cepstral, and the sample noise-reduced frequency signal in the time-frequency domain.

[0068] The construction subunit is configured to construct a second audio sample based on the second sample audio signal on the Mel frequency cepstral, the original audio signal on the Mel frequency cepstral, and the sample noise-reduced audio signal on the Mel frequency cepstral.

[0069] According to a third aspect of the present disclosure, an electronic device is provided, comprising:

[0070] processor;

[0071] Memory used to store the processor's executable instructions;

[0072] The processor is configured to execute the instructions to implement the audio signal processing method as described in any one of the first aspects above.

[0073] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform an audio signal processing method as described in any one of the first aspects above.

[0074] According to a fifth aspect of the present disclosure, a computer program product is provided, wherein when the instructions are executed by a processor of an electronic device, the electronic device is enabled to perform the audio signal processing method described in any one of the first aspects.

[0075] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0076] This method uses a pre-trained noise reduction model to process the audio signal to be repaired. Based on this, the pre-trained audio restoration model is used to process both the noise-reduced audio signal and the audio signal to be repaired, resulting in a target audio signal that is free of interference and audio distortion after audio restoration. This method effectively repairs the problems of interference and audio distortion in the audio signal to be repaired, improving the accuracy and quality of audio restoration.

[0077] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0078] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0079] Figure 1 This is a flowchart illustrating an audio signal processing method according to an exemplary embodiment.

[0080] Figure 2 This is a flowchart illustrating a method for acquiring an audio signal to be repaired, according to an exemplary embodiment.

[0081] Figure 3 This is a flowchart illustrating a training method for a noise reduction processing model according to an exemplary embodiment.

[0082] Figure 4 This is a schematic diagram illustrating a noise reduction processing model training process according to an exemplary embodiment.

[0083] Figure 5 This is a flowchart illustrating a training method for an audio restoration model according to an exemplary embodiment.

[0084] Figure 6 This is a flowchart illustrating a method for constructing a second audio sample according to an exemplary embodiment.

[0085] Figure 7 This is a schematic diagram illustrating an audio restoration model training process according to an exemplary embodiment.

[0086] Figure 8 This is a flowchart illustrating another method for constructing a second audio sample according to an exemplary embodiment.

[0087] Figure 9 This is a block diagram illustrating an audio signal processing apparatus according to an exemplary embodiment.

[0088] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment.

[0089] Figure 11 This is a block diagram illustrating another electronic device according to an exemplary embodiment. Detailed Implementation

[0090] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0091] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0092] It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0093] Figure 1 This is a flowchart illustrating an audio signal processing method according to an exemplary embodiment, such as... Figure 1 As shown in the example, this disclosure uses the audio signal processing method applied to a terminal. Optionally, the method can also be applied to a server, or to a system where the terminal and server interact. This disclosure does not limit the application of this method. The specific method includes the following steps:

[0094] In step S110, the audio signal to be repaired is acquired.

[0095] The audio signal to be repaired is a distorted audio signal containing interference signals. Specifically, this distorted audio signal is one with issues such as packet loss, voids, or clipping.

[0096] In practice, the audio signal to be repaired often suffers from multiple types of audio signal loss, such as noise, reverberation, and other interference signals, as well as voids and clipping. Therefore, it is necessary to repair the audio signal to address these various types of audio signal loss. Consequently, a computer acquires the audio signal to be repaired and performs audio repair processing.

[0097] In step S120, the audio signal to be repaired is processed according to the pre-trained noise reduction processing model to obtain the noise-reduced audio signal.

[0098] The noise reduction model is trained based on a first audio sample containing interference signals. The noise-reduced audio signal is the distorted audio signal with interference signals removed.

[0099] In implementation, a noise reduction processing model is pre-stored in the terminal. This model has been trained based on a first audio sample containing interference signals. Therefore, when audio restoration of the audio signal to be restored is required, the audio signal to be restored is first input into the noise reduction processing model. Interference removal processing is performed only on the audio signal to be restored, resulting in a noise-reduced audio signal. This noise-reduced audio signal eliminates interference, but still has audio loss issues such as missing audio data (e.g., audio packet loss, holes, clipping, etc.).

[0100] Optionally, the noise reduction model may be, but is not limited to, a deep neural network model (DNN), which will not be described further in this disclosure.

[0101] In step S130, the audio signal to be repaired and the noise-reduced audio signal are input into a pre-trained audio repair model to obtain the repaired target audio signal.

[0102] The audio restoration model is trained based on a second audio sample constructed from two signals: a sample distorted audio signal and a sample noise-reduced audio signal corresponding to the distorted audio signal.

[0103] In practice, the terminal inputs both the audio signal to be repaired and the noise-reduced frequency signal into a pre-trained audio restoration model. The model processes these two signals to obtain the restored target audio signal. Specifically, the two signals serve as a reference to prevent the model from losing the original audio characteristics during simultaneous interference removal and restoration processes, thus avoiding issues such as unclear pronunciation and voiceless consonant removal after restoration. The target audio signal is the final audio signal obtained after noise removal and audio loss correction of the original audio signal.

[0104] Optionally, the audio restoration model may be, but is not limited to, a deep neural network model (DNN), which will not be described further in this disclosure.

[0105] In the above audio signal processing method, a pre-trained noise reduction processing model is used to process the audio signal to be repaired. Based on this, the noise-reduced audio signal and the audio signal to be repaired are processed by a pre-trained audio repair model to obtain a target audio signal without interference signals and without audio distortion after audio repair. This realizes the repair of interference signals and audio signal distortion in the audio signal to be repaired, and improves the accuracy and quality of audio repair.

[0106] In one exemplary embodiment, when processing the audio signal to be repaired, the directly acquired audio signal in the time domain can be frequency domain transformed to the time-frequency domain or even the Mel-frequency cepstral spectra before undergoing audio processing based on a deep learning model. This improves the accuracy of audio signal processing. Specifically, for example... Figure 2 As shown, step S110 can be implemented through the following steps:

[0107] In step S211, the first audio signal to be repaired is acquired.

[0108] The first audio signal is a distorted audio signal containing interference signals.

[0109] In practice, for the first audio signal to be repaired, since there is more than one audio problem in the first audio signal, the terminal acquires the first audio signal to be repaired.

[0110] Optionally, the first audio signal may contain interference signals, which may be classified into various types such as noise signals and reverberation signals. In this embodiment of the present disclosure, the type and number of interference signals present in the first audio signal to be repaired are not limited.

[0111] Optionally, the first audio signal may also be a distorted audio signal, which may, but is not limited to, having problems such as audio packet loss, voids, and clipping. This disclosure does not limit the types and number of audio distortions.

[0112] In step S212, a short-time Fourier transform is performed on the first audio signal to obtain the first converted audio signal in the time-frequency domain.

[0113] In practice, for the first audio signal in the time domain, the terminal performs a Short-Time Fourier Transform (STFT) on the first audio signal, converting the original first audio signal to be repaired in the time domain to the time-frequency domain, obtaining a first converted audio signal in the time-frequency domain. This first converted audio signal still contains interference signals and has audio distortion problems.

[0114] In step S213, the first converted audio signal is subjected to Mel frequency cepstral conversion processing to obtain the audio signal to be repaired.

[0115] In practice, the terminal performs Mel-frequency cepstrum conversion on the first converted audio signal, transforming the first converted audio signal in the time-frequency domain spectrum to the Mel-frequency cepstrum, thus obtaining the audio signal to be repaired in the Mel-frequency cepstrum. This audio signal to be repaired is the audio signal that can be input into the deep learning model (i.e., the noise reduction model and the audio restoration model) for processing.

[0116] In this embodiment, the first audio signal to be repaired is processed by converting it from the time domain to the time-frequency domain, and then to the Mel frequency cepstral spectrum to obtain the audio signal to be repaired on the Mel frequency cepstral spectrum. This allows for subsequent processing of the audio signal to be repaired on the Mel frequency cepstral spectrum. Through the conversion of the audio signal, the discriminative power of the deep learning model is increased, thereby improving the model's discriminative accuracy.

[0117] In one embodiment, step S120, which processes the audio signal to be repaired according to a pre-trained noise reduction processing model to obtain a noise-reduced audio signal, includes:

[0118] Step S120A: Input the audio signal to be repaired, which contains interference signals, into the pre-trained noise reduction processing model.

[0119] The interference signal includes noise and reverberation signals.

[0120] During implementation, the original audio signal is easily affected by the surrounding environment and recording equipment, resulting in interference signals in the recorded original audio signal. These interference signals may originate from noise in the recording environment or from reverberation interference caused by sound leakage from the recording equipment. In other words, the original audio signal contains reverberation signals. Therefore, when repairing the audio signal to be repaired, it is necessary to eliminate these interference signals. The terminal stores a pre-trained noise reduction processing model. When repairing the audio signal to be repaired, the terminal inputs the audio signal containing interference signals to the pre-trained noise reduction processing model for processing.

[0121] Step S120B: The noise signal and reverberation signal in the audio signal to be repaired are processed by the pre-trained noise reduction processing model to obtain the noise-reduced frequency signal corresponding to the audio signal to be repaired.

[0122] In practice, the terminal processes the noise and reverberation signals in the audio signal to be repaired through a pre-trained noise reduction processing model, thereby eliminating the noise and reverberation signals contained in the audio signal to be repaired and obtaining a noise-reduced audio signal.

[0123] In this embodiment, the audio signal to be repaired is preprocessed to obtain a noise-reduced frequency signal with interference signals eliminated. The obtained noise-reduced frequency signal is then cross-referenced with the audio signal to be repaired to achieve dual repair of interference elimination and distortion repair of the audio signal to be repaired.

[0124] In an exemplary embodiment, in step 130: after inputting the audio signal to be repaired and the noise-reduced audio signal into a pre-trained audio repair model to obtain the repaired target audio signal, the method further includes:

[0125] In step S131, the target audio signal is input to the audio encoder, and the target audio signal on the Mel frequency cepstral is converted by the audio encoder to obtain the repaired audio signal.

[0126] Among them, the repaired audio signal is the audio signal in the time domain after the audio signal to be repaired is repaired.

[0127] In implementation, the target audio signal output by the audio restoration model is an audio signal on the Mel-frequency cepstral spectrum. Having undergone restoration processing by the model, the target audio signal has had interference signals eliminated and audio distortion repaired. Then, based on this restored target audio signal, the terminal inputs it into an audio encoder. The encoder converts the Mel-frequency cepstral target audio signal directly to the time domain, obtaining the restored audio signal in the time domain. This restored audio signal in the time domain is the playable audio signal.

[0128] Specifically, the target audio signal on the Mel frequency cepstral is represented by MelS... 48k (n, m) represents the formula for converting the target audio signal on the Mel frequency cepstral, as shown below:

[0129] s(t) = Vocoder(MelS) 48k (n, m))

[0130] Where s(t) represents the converted, repaired audio signal in the time domain. Vocoder represents the audio encoder conversion process.

[0131] Optionally, the converted and restored audio signal is a playable audio signal in the time domain. Therefore, playing the restored audio signal can verify the accuracy and quality of the restoration of the audio signal to be restored.

[0132] In this embodiment, the target audio signal in the Mel frequency domain spectrum obtained after restoration is converted by an audio encoder to obtain the restored audio signal in the time domain after conversion, thereby realizing the restoration of the target audio signal obtained after interference elimination processing and audio restoration processing.

[0133] In one exemplary embodiment, the noise reduction processing model used in the audio signal processing process needs to be pre-trained, such as... Figure 3 As shown, the training process of this noise reduction model includes:

[0134] In step S302, the first audio sample is obtained.

[0135] The first audio sample includes a first sample audio signal and the original audio signal corresponding to the first sample audio signal; the first sample audio signal only has one type of audio loss problem, namely, the first sample audio signal contains interference signals such as noise signals and reverberation signals, but the first sample audio signal is not a distorted audio signal.

[0136] In implementation, when training the noise reduction model, the terminal first acquires a first audio sample. Specifically, the process of acquiring the first audio sample includes: the terminal first acquires a first sample audio signal of length T and the corresponding original audio signal. The original audio signal x1 and the first sample audio signal y1 containing interference signals are x1(t) and y1(t) in the time domain, respectively. Here, t represents time, and 0 < t ≤ T. Then, the terminal performs a short-time Fourier transform on the first sample audio signal in the time domain and the corresponding original audio signal to obtain the first sample audio signal in the time-frequency domain (using STFT(y1)). 48k (t) represents the original audio signal in the time-frequency domain (using STFT(x1)). 48k (t) represents this. Specifically, it can be expressed by the following formula:

[0137] X1 48k (n, k) = STFT(x1) 48k (t)) (1)

[0138] Y1 48k (n, k) = STFT(y1) 48k (t)) (2)

[0139] Where n is the frame sequence, 0 < n ≤ N (N is the total number of frames), and k is the center frequency sequence, 0 < k ≤ K. 48 (K 48 (Total number of bandwidths).

[0140] Then, for the first sample audio signal (STFT(y1) in the time-frequency domain) 48k (t) and the original audio signal in the time-frequency domain (STFT(x1) 48k (t)), respectively, to obtain the amplitude information of each frame of audio signal. Specifically, the formula for calculating the amplitude of the audio signal is as follows:

[0141] MagX1 48k (n, k) = abs(X1) 48k (n, k)) (3)

[0142] MagY1 48k (n, k) = abs(Y1) 48k (n, k)) (4)

[0143] Among them, MagX1 48k (n, k) represents the amplitude information (or amplitude feature) of the original audio signal in the time-frequency domain, abs(X1) 48k (n, k) represents taking the absolute value of the original audio signal. MagY1 48k(n, k) represents the amplitude information of the first sample audio signal in the time-frequency domain, abs(Y1) 48k (n, k)) represents taking the absolute value of the first sample audio signal.

[0144] Finally, the terminal constructs the first audio sample based on the amplitude information of the first sample audio signal and the amplitude information of the original audio signal corresponding to the first sample audio signal.

[0145] In step S304, the first sample audio signal is input into the noise reduction processing model, and the first sample audio signal is processed by the noise reduction processing model to obtain the first processed audio signal.

[0146] In implementation, the terminal inputs the first sample audio signal from the first audio sample into the noise reduction processing model. The noise reduction processing model then performs interference removal processing on the first sample audio signal to obtain the first processed audio signal. Specifically, Figure 4 This is a schematic diagram of the training process for a noise reduction model, such as... Figure 4 As shown, y1 48 (t) represents the first sample audio signal containing interference at 48kHz in the time domain obtained after short-time Fourier transform. After short-time Fourier transform, the first sample audio signal in the time-frequency domain is obtained, denoted as y1. 48 (n, k) represents the frequency domain. The amplitude information MagY1 of the first sample audio signal at 48kHz in this time-frequency domain is extracted. 48k (n, k), the amplitude information of the first sample audio signal is input into the noise reduction processing model (DNN1 model). The noise reduction processing model performs interference removal processing on the amplitude information of the first sample audio signal to obtain the audio signal output by the model, and uses the amplitude information of the original audio signal MagX1. 48 (n, k) is used as the target of deep learning. Then, loss is calculated on the audio signal output by the model, and the model is iterated based on the result of the loss calculation to complete the training of the noise reduction model.

[0147] In step S306, loss calculation is performed based on the first processed audio signal and the original audio signal to obtain the loss result corresponding to the first processed audio signal.

[0148] In implementation, the terminal performs loss calculation based on the first processed audio signal and the original audio signal to obtain the loss result corresponding to the first processed audio signal. Specifically, the loss calculation can be amplitude spectrum distance calculation, that is, the terminal can calculate the amplitude spectrum distance between the first processed audio signal output by the noise reduction processing model and the original audio signal, and use the calculated amplitude spectrum distance as the loss result corresponding to the first processed audio signal.

[0149] In step S308, it is determined whether the loss result meets the preset loss condition. The training of the noise reduction processing model is completed when the loss result meets the preset loss condition.

[0150] The first processed audio signal output by the trained noise reduction model is the noise-reduced audio signal.

[0151] In implementation, the terminal determines whether the loss result corresponding to the current first processed audio signal meets the preset loss conditions. Specifically, if the loss result meets the preset loss conditions, the terminal determines that the noise reduction processing model has been trained successfully. If the loss result does not meet the preset loss conditions, the terminal executes steps S302 to S304 as described above until the loss result meets the preset loss conditions. The processes of steps S302 to S304 will not be described in detail here.

[0152] Optionally, the preset loss condition can be less than or equal to a preset first loss threshold. Therefore, if the calculated loss result corresponding to the first processed audio signal is less than or equal to the preset first loss threshold, it indicates that the loss result meets the preset loss condition. Conversely, if the loss result corresponding to the first processed audio signal is greater than the preset first loss threshold, it indicates that the loss result does not meet the preset loss condition.

[0153] In this embodiment, the noise reduction processing model is trained using the first audio sample, thereby obtaining the trained noise reduction processing model, which is then used to process the problem of interference signals in the audio signal to be repaired.

[0154] In one exemplary embodiment, the audio restoration model used in the audio signal processing needs to be pre-trained, such as... Figure 5 As shown, the training process of this audio restoration model includes:

[0155] In step S502, a second audio sample is obtained.

[0156] The second audio sample includes a second sample audio signal, a sample noise-reduced frequency signal corresponding to the second sample audio signal, and the original audio signal corresponding to the second sample audio signal.

[0157] The second sample audio signal is a sample-distorted audio signal containing interference signals.

[0158] In implementation, the terminal acquires a second audio sample. Specifically, the terminal can acquire a second sample audio signal containing interference signals with a resampling rate of 48kHz and a length of T, a sample noise-reduced frequency signal corresponding to the second sample audio signal, and the original audio signal corresponding to the second sample audio signal. The sample noise-reduced frequency signal is the noise-reduced frequency signal obtained after interference removal processing of the second sample audio signal, and the original audio signal is the audio signal corresponding to the second sample audio signal without any audio loss. Then, the terminal preprocesses the second sample audio signal, the sample noise-reduced frequency signal, and the original audio signal, and constructs the second audio sample based on the preprocessed audio signals.

[0159] In step S504, the second sample audio signal and the sample noise-reduced frequency signal are input into the audio restoration model. The second sample audio signal and the sample noise-reduced frequency signal are processed by the audio restoration model to obtain the second processed audio signal.

[0160] In implementation, the terminal inputs the second sample audio signal and the sample noise-reduced frequency signal from the second audio sample into the audio restoration model (DNN2 model). The audio restoration model processes the second sample audio signal and the sample noise-reduced frequency signal, and outputs a second processed audio signal. This second processed audio signal is the restored audio signal on the Mel frequency cepstral.

[0161] In step S506, loss calculation is performed based on the original audio signals corresponding to the second processed audio signal and the second sample audio signal to obtain the loss result corresponding to the second processed audio signal.

[0162] The second processed audio signal is used to perform loss calculation with the original audio signal, which is the training target, to determine the loss result of the model in this training.

[0163] In implementation, the terminal performs loss calculations based on the original audio signal corresponding to the second processed audio signal and the second sample audio signal to obtain the loss result corresponding to the second processed audio signal. Optionally, the terminal can calculate the amplitude spectrum distance between the second processed audio signal and the original audio signal, and use this amplitude spectrum distance as the loss result corresponding to the second processed audio signal.

[0164] In step S508, it is determined whether the loss result meets the preset loss condition. When the loss result meets the preset loss condition, the training of the audio restoration model is determined to be complete.

[0165] In implementation, the terminal stores preset loss conditions. Then, for each loss result corresponding to the second processed audio signal output by the audio restoration model, the terminal determines whether the loss result meets the preset loss conditions. The audio restoration model training is considered complete when the loss result meets the preset loss conditions. Specifically, the preset loss condition is that the loss result is less than or equal to a preset second loss threshold. If the loss result corresponding to the second processed audio signal is less than or equal to the preset second loss threshold, the loss result is determined to meet the preset loss conditions. The terminal then determines that the audio restoration model training is complete. If the loss result corresponding to the second processed audio signal is greater than the preset second loss threshold, the loss result is determined not to meet the preset loss conditions. The terminal then continues to execute steps S502 to S506 until the loss result meets the preset loss conditions, at which point the audio restoration model training is considered complete.

[0166] In this embodiment, the audio restoration model is trained using a second audio sample to obtain a trained audio restoration model, which is then used to address issues such as interference signals and audio signal distortion in the audio signal to be restored.

[0167] In one exemplary embodiment, during the acquisition of the second audio samples, the terminal preprocesses the directly acquired second audio signal in the time domain, the noise-reduced frequency signal corresponding to the second audio signal, and the original audio signal corresponding to the second audio signal to obtain preprocessed audio signals. Then, the second audio samples are constructed based on the preprocessed audio signals. The specific preprocessing process is as follows: Figure 6 As shown, obtaining the second audio sample in step S502 specifically includes the following steps:

[0168] In step S602, the second audio signal and the original audio signal corresponding to the second audio signal are acquired.

[0169] The second audio signal is a sample-distorted audio signal containing interference signals.

[0170] In implementation, the terminal acquires the second audio signal in the time domain and the corresponding original audio signal. The second audio signal in the time-frequency domain is represented by y2. 48k (t) represents the original audio signal corresponding to the second audio signal, which is the audio signal that does not contain any audio loss issues.

[0171] In step S604, short-time Fourier transforms are performed on the second audio signal and the original audio signal corresponding to the second audio signal to obtain the second sample audio signal in the time-frequency domain and the original audio signal in the time-frequency domain.

[0172] In implementation, the terminal performs short-time Fourier transforms on the second audio signal in the time domain and the corresponding original audio signal, respectively, to obtain the second sample audio signal in the time-frequency domain and the original audio signal in the time-frequency domain. Specifically, as follows... Figure 7 As shown, through STFT processing, y2 48k (t) is converted into the second sample audio signal y2 in the time-frequency domain. 48k (n, k). The original audio signal in the time domain x2 48k (t) is converted into the original audio signal x2 in the time-frequency domain. 48k (n, k). Specifically, for the original audio signal in the time-frequency domain, the amplitude spectrum information of the original audio signal in the time-frequency domain is determined. This amplitude spectrum information of the original audio signal is represented by MagX2. 48 (n, k),

[0173] In step S606, the second sample audio signal in the time-frequency domain is processed according to the pre-trained noise reduction processing model to obtain the sample noise-reduced frequency signal corresponding to the second sample audio signal.

[0174] In implementation, the terminal performs interference removal processing on the second sample audio signal in the time domain according to a pre-trained noise reduction processing model, obtaining the sample noise-reduced audio signal corresponding to the second sample audio signal. For example... Figure 7 As shown, the second sample audio signal is backed up. Then, one copy of the second sample audio signal is input into a pre-trained denoising model (DNN1 model). After processing by the denoising model (DNN1 model), the sample denoised frequency signal corresponding to the second sample audio signal is obtained. This sample denoised frequency signal is the amplitude spectrum information in the time-frequency domain, and is represented by MagY2. 48k (n, k) represents the expression.

[0175] In step S608, a second audio sample is constructed based on the second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced audio signal in the time-frequency domain.

[0176] In practice, the terminal constructs the second audio sample based on the second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced audio signal in the time-frequency domain.

[0177] In this embodiment, the second audio signal in the time domain, the noise-reduced frequency signal corresponding to the second audio signal, and the original audio signal corresponding to the second audio signal are preprocessed to obtain each preprocessed audio signal. Based on the preprocessed second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced frequency signal in the time-frequency domain, a second audio sample is constructed to train the audio restoration model based on the second audio sample.

[0178] In one exemplary embodiment, such as Figure 8 As shown, in step S608, before constructing the second audio sample, the method further includes:

[0179] In step S802, the second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced frequency signal are subjected to spectral conversion to obtain the second sample audio signal in the Mel frequency cepstral, the original audio signal in the Mel frequency cepstral, and the sample noise-reduced frequency signal in the time-frequency domain.

[0180] In practice, the terminal performs spectrum conversion on the second sample audio signal, the original audio signal, and the sample noise-reduced frequency signal in the time-frequency domain, respectively, and converts each audio signal to the Mel frequency cepstral to obtain the second sample audio signal on the Mel frequency cepstral, the original audio signal on the Mel frequency cepstral, and the sample noise-reduced frequency signal in the time-frequency domain.

[0181] Then, the processing procedure in step S608 is updated to the following steps:

[0182] In step S804, the second audio sample is constructed based on the second sample audio signal on the Mel frequency cepstral, the original audio signal on the Mel frequency cepstral, and the sample noise-reduced audio signal on the Mel frequency cepstral.

[0183] In practice, the terminal constructs a second audio sample based on the second sample audio signal on the Mel frequency cepstral, the original audio signal on the Mel frequency cepstral, and the sample noise-reduced audio signal on the Mel frequency cepstral, thereby enabling the audio restoration model to be trained based on the second audio sample.

[0184] In this embodiment, by converting the second sample audio signal, the original audio signal, and the sample noise-reduced frequency signal in the time-frequency domain, a second audio sample is obtained, which is constructed from the second sample audio signal, the original audio signal, and the sample noise-reduced frequency signal on the Mel frequency cepstral. This allows the model to be trained using the second audio sample on the Mel frequency cepstral, preserving the speech information contained in the second audio sample and improving the accuracy of the audio restoration model.

[0185] It should be understood that, although Figures 1-3 ,Figures 5 to 6 , Figure 8 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 1-3 , Figures 5 to 6 , Figure 8 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0186] It is understood that the same / similar parts between the various embodiments of the methods described above in this specification can be referred to each other. Each embodiment focuses on the differences from other embodiments, and relevant parts can be referred to the description of other method embodiments.

[0187] Figure 9 This is a block diagram illustrating an audio signal processing apparatus according to an exemplary embodiment. (Refer to...) Figure 9 The device includes an acquisition unit 902, an interference elimination unit 904, and a repair unit 906.

[0188] The acquisition unit 902 is configured to acquire the audio signal to be repaired; the audio signal to be repaired is a distorted audio signal containing interference signals;

[0189] The interference removal unit 904 is configured to process the audio signal to be repaired according to a pre-trained noise reduction processing model to obtain a noise-reduced audio signal; the noise reduction processing model is trained based on a first audio sample containing the interference signal.

[0190] Repair unit 906 is configured to input the audio signal to be repaired and the noise-reduced frequency signal into a pre-trained audio repair model to obtain the repaired target audio signal; the audio repair model is trained based on a second audio sample constructed from the sample distorted audio signal and the sample noise-reduced frequency signal corresponding to the sample distorted audio signal.

[0191] In one exemplary embodiment, the acquisition unit 902 includes:

[0192] The acquisition subunit is configured to acquire a first audio signal to be repaired; the first audio signal is a distorted audio signal containing interference signals.

[0193] The first conversion subunit is configured to perform a short-time Fourier transform on the first audio signal to obtain the first converted audio signal in the time-frequency domain;

[0194] The second conversion subunit is configured to perform Mel frequency cepstral conversion processing on the first converted audio signal to obtain the audio signal to be repaired.

[0195] In one exemplary embodiment, the interference cancellation unit 904 includes:

[0196] The input subunit is configured to perform the input of the audio signal to be repaired, which contains interference signals, into a pre-trained noise reduction processing model; the interference signals include noise signals and reverberation signals.

[0197] The interference elimination subunit is configured to process the noise signal and reverberation signal in the audio signal to be repaired through a pre-trained noise reduction processing model to obtain a noise-reduced frequency signal that eliminates the noise signal and the reverberation signal.

[0198] In one exemplary embodiment, the device further includes:

[0199] The restoration unit is configured to input the target audio signal into the audio encoder, and convert the target audio signal on the Mel frequency cepstral through the audio encoder to obtain the restored audio signal; the restored audio signal is the audio signal in the time domain after the audio signal to be restored is restored.

[0200] In one exemplary embodiment, the device further includes:

[0201] The first sample acquisition unit is configured to acquire a first audio sample, the first audio sample containing a first sample audio signal and the original audio signal corresponding to the first sample audio signal; the first sample audio signal contains interference signals.

[0202] The first sample processing unit is configured to input the first sample audio signal into the noise reduction processing model, process the first sample audio signal through the noise reduction processing model, and obtain the first processed audio signal.

[0203] The first loss calculation unit is configured to perform loss calculation based on the first processed audio signal and the original audio signal to obtain the loss result corresponding to the first processed audio signal.

[0204] The first discrimination unit is configured to judge whether the loss result meets the preset loss conditions, and determine that the noise reduction processing model training is complete when the loss result meets the preset loss conditions; the first processed audio signal output by the trained noise reduction processing model is the noise reduction frequency signal.

[0205] In one exemplary embodiment, the device further includes:

[0206] The second sample acquisition unit is configured to acquire a second audio sample, which includes a second sample audio signal, a sample noise-reduced audio signal corresponding to the second sample audio signal, and the original audio signal corresponding to the second sample audio signal; the second sample audio signal is a sample distorted audio signal containing interference signals.

[0207] The second sample processing unit is configured to input the second sample audio signal and the sample noise-reduced frequency signal into the audio restoration model, and process the second sample audio signal and the sample noise-reduced frequency signal through the audio restoration model to obtain the second processed audio signal;

[0208] The second loss calculation unit is configured to perform loss calculation based on the original audio signal corresponding to the second processed audio signal and the second sample audio signal, and obtain the loss result corresponding to the second processed audio signal.

[0209] The second discrimination unit is configured to judge whether the loss result meets the preset loss conditions, and determine that the audio restoration model training is complete when the loss result meets the preset loss conditions.

[0210] In one exemplary embodiment, the second sample acquisition unit includes:

[0211] The second acquisition subunit is configured to acquire a second audio signal and the original audio signal corresponding to the second audio signal; the second audio signal is a sample-distorted audio signal containing interference signals.

[0212] The third conversion subunit is configured to perform short-time Fourier transform on the second audio signal and the original audio signal corresponding to the second audio signal, respectively, to obtain the second sample audio signal in the time-frequency domain and the original audio signal in the time-frequency domain.

[0213] The processing subunit is configured to process the second sample audio signal in the time-frequency domain according to a pre-trained noise reduction processing model to obtain the sample noise-reduced frequency signal corresponding to the second sample audio signal.

[0214] The construction subunit is configured to perform the operation of constructing a second audio sample based on the second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced audio signal in the time-frequency domain.

[0215] In one exemplary embodiment, the device further includes:

[0216] The fourth conversion subunit is configured to perform spectral conversion on the second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced frequency signal, respectively, to obtain the second sample audio signal in the Mel frequency cepstral, the original audio signal in the Mel frequency cepstral, and the sample noise-reduced frequency signal in the time-frequency domain.

[0217] The construction subunit is configured to perform the operation of constructing a second audio sample based on the second sample audio signal on the Mel frequency cepstral, the original audio signal on the Mel frequency cepstral, and the sample noise-reduced audio signal on the Mel frequency cepstral.

[0218] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0219] Figure 10 This is a block diagram illustrating an electronic device 1000 for an audio signal processing method according to an exemplary embodiment. For example, the electronic device 1000 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0220] Reference Figure 10 The electronic device 1000 may include one or more of the following components: processing component 1002, memory 1004, power supply component 1006, multimedia component 1008, audio component 1010, input / output (I / O) interface 1012, sensor component 1014, and communication component 1016.

[0221] Processing component 1002 typically controls the overall operation of electronic device 1000, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1002 may include one or more processors 1020 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1002 may include one or more modules to facilitate interaction between processing component 1002 and other components. For example, processing component 1002 may include a multimedia module to facilitate interaction between multimedia component 1008 and processing component 1002.

[0222] Memory 1004 is configured to store various types of data to support the operation of electronic device 1000. Examples of such data include instructions for any application or method operating on electronic device 1000, contact data, phonebook data, messages, pictures, videos, etc. Memory 1004 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, optical disk, or graphene storage.

[0223] Power supply component 1006 provides power to various components of electronic device 1000. Power supply component 1006 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1000.

[0224] Multimedia component 1008 includes a screen that provides an output interface between the electronic device 1000 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1008 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1000 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0225] Audio component 1010 is configured to output and / or input audio signals. For example, audio component 1010 includes a microphone (MIC) configured to receive external audio signals when electronic device 1000 is in an operating mode, such as call mode, recording mode, and audio recognition mode. The received audio signals may be further stored in memory 1004 or transmitted via communication component 1016. In some embodiments, audio component 1010 also includes a speaker for outputting audio signals.

[0226] I / O interface 1012 provides an interface between processing component 1002 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0227] Sensor assembly 1014 includes one or more sensors for providing state assessments of various aspects of electronic device 1000. For example, sensor assembly 1014 can detect the on / off state of electronic device 1000, the relative positioning of components such as the display and keypad of electronic device 1000, changes in position of electronic device 1000 or its components, the presence or absence of user contact with electronic device 1000, orientation or acceleration / deceleration of device 1000, and temperature changes of electronic device 1000. Sensor assembly 1014 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1014 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1014 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0228] Communication component 1016 is configured to facilitate wired or wireless communication between electronic device 1000 and other devices. Electronic device 1000 can access wireless networks based on communication standards, such as WiFi, carrier networks (such as 2G, 3G, 4G, or 5G), or combinations thereof. In one exemplary embodiment, communication component 1016 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1016 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0229] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0230] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 1004 including instructions, which can be executed by a processor 1020 of an electronic device 1000 to perform the above-described method. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0231] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by the processor 1020 of the electronic device 1000 to perform the above method.

[0232] Figure 11 This is a block diagram illustrating an electronic device 1100 for audio signal processing according to an exemplary embodiment. For example, the electronic device 1100 may be a server. (Refer to...) Figure 11 The electronic device 1100 includes a processing component 1120, which further includes one or more processors, and memory resources represented by memory 1122 for storing instructions, such as application programs, that can be executed by the processing component 1120. The application programs stored in memory 1122 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1120 is configured to execute instructions to perform the methods described above.

[0233] Electronic device 1100 may further include: a power supply component 1124 configured to perform power management of electronic device 1100, a wired or wireless network interface 1126 configured to connect electronic device 1100 to a network, and an input / output (I / O) interface 1128. Electronic device 1100 may operate on an operating system stored in memory 1122, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, or similar.

[0234] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory S22 including instructions, which can be executed by a processor of an electronic device S00 to perform the above-described method. The storage medium can be a computer-readable storage medium, for example, a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0235] In an exemplary embodiment, a computer program product is also provided, the computer program product including instructions that can be executed by the processor of the electronic device 1100 to perform the above-described method.

[0236] It should be noted that the above-mentioned apparatus, electronic equipment, computer-readable storage medium, computer program product, etc., may also include other implementation methods according to the description of the method embodiments. For specific implementation methods, please refer to the description of the relevant method embodiments, which will not be elaborated here.

[0237] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0238] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio signal processing method, characterized in that, include: Acquire the audio signal to be repaired; the audio signal to be repaired is a distorted audio signal containing interference signals; The audio signal to be repaired is processed according to a pre-trained noise reduction processing model to obtain a noise-reduced audio signal; the noise reduction processing model is trained based on a first audio sample containing interference signals; the noise-reduced audio signal is a distorted audio signal that eliminates the interference signals. The audio signal to be repaired and the noise-reduced frequency signal are input into a pre-trained audio repair model to obtain the repaired target audio signal; the audio repair model is trained based on a second audio sample constructed from the sample distorted audio signal and the sample noise-reduced frequency signal corresponding to the sample distorted audio signal.

2. The audio signal processing method according to claim 1, characterized in that, The acquisition of the audio signal to be repaired includes: Acquire the first audio signal to be repaired; the first audio signal is a distorted audio signal containing interference signals; Perform a short-time Fourier transform on the first audio signal to obtain the first converted audio signal in the time-frequency domain; The first converted audio signal is subjected to Mel frequency cepstral conversion processing to obtain the audio signal to be repaired.

3. The audio signal processing method according to claim 1, characterized in that, The step of processing the audio signal to be repaired according to a pre-trained noise reduction processing model to obtain a noise-reduced audio signal includes: The audio signal to be repaired, containing interference signals, is input into a pre-trained noise reduction processing model; the interference signals include noise signals and reverberation signals. The noise signal and reverberation signal in the audio signal to be repaired are processed by a pre-trained noise reduction processing model to obtain a noise-reduced frequency signal that eliminates the noise signal and the reverberation signal.

4. The audio signal processing method according to claim 1, characterized in that, After inputting the audio signal to be repaired and the noise-reduced audio signal into a pre-trained audio repair model to obtain the repaired target audio signal, the method further includes: The target audio signal is input to an audio encoder, which converts the target audio signal on the Mel frequency cepstral to obtain a repaired audio signal; the repaired audio signal is the time-domain audio signal after the repaired audio signal is repaired.

5. The audio signal processing method according to claim 1, characterized in that, The training process of the noise reduction model includes: A first audio sample is acquired, the first audio sample containing a first sample audio signal and the original audio signal corresponding to the first sample audio signal; the first sample audio signal contains interference signals. The first sample audio signal is input into the noise reduction processing model, and the noise reduction processing model processes the first sample audio signal to obtain the first processed audio signal. Loss calculation is performed based on the first processed audio signal and the original audio signal to obtain the loss result corresponding to the first processed audio signal; Determine whether the loss result meets the preset loss condition, and determine that the noise reduction processing model training is complete when the loss result meets the preset loss condition; the first processed audio signal output by the trained noise reduction processing model is the noise reduction frequency signal.

6. The audio signal processing method according to claim 1 or 2, characterized in that, The training process of the audio restoration model includes: A second audio sample is obtained, which includes a second sample audio signal, a sample noise-reduced frequency signal corresponding to the second sample audio signal, and the original audio signal corresponding to the second sample audio signal; the second sample audio signal is a sample distorted audio signal containing interference signals. The second sample audio signal and the sample noise-reduced frequency signal are input into the audio restoration model. The second sample audio signal and the sample noise-reduced frequency signal are processed by the audio restoration model to obtain the second processed audio signal. Loss calculation is performed based on the original audio signal corresponding to the second processed audio signal and the second sample audio signal to obtain the loss result corresponding to the second processed audio signal; Determine whether the loss result meets the preset loss condition, and determine that the audio restoration model training is complete when the loss result meets the preset loss condition.

7. The audio signal processing method according to claim 6, characterized in that, The acquisition of the second audio sample includes: Acquire a second audio signal and the original audio signal corresponding to the second audio signal; the second audio signal is a sample-distorted audio signal containing interference signals; Short-time Fourier transforms are performed on the second audio signal and the original audio signal corresponding to the second audio signal, respectively, to obtain the second sample audio signal in the time-frequency domain and the original audio signal in the time-frequency domain; The second sample audio signal in the time-frequency domain is processed according to the pre-trained noise reduction processing model to obtain the sample noise-reduced frequency signal corresponding to the second sample audio signal; The second audio sample is constructed based on the second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced audio signal in the time-frequency domain.

8. The audio signal processing method according to claim 7, characterized in that, Before constructing the second audio sample, the method further includes: The second sample audio signal in the time-frequency domain, the original audio signal in the time-frequency domain, and the sample noise-reduced frequency signal are respectively subjected to spectrum transformation to obtain the second sample audio signal in the Mel frequency cepstral, the original audio signal in the Mel frequency cepstral, and the sample noise-reduced frequency signal in the time-frequency domain. The construction yields a second audio sample, including: The second audio sample is constructed based on the second sample audio signal on the Mel frequency cepstral, the original audio signal on the Mel frequency cepstral, and the sample noise-reduced audio signal on the Mel frequency cepstral.

9. An audio signal processing device, characterized in that, include: The acquisition unit is configured to acquire the audio signal to be repaired; the audio signal to be repaired is a distorted audio signal containing interference signals; An interference removal unit is configured to process the audio signal to be repaired according to a pre-trained noise reduction processing model to obtain a noise-reduced audio signal; the noise reduction processing model is trained based on a first audio sample containing interference signals; the noise-reduced audio signal is a distorted audio signal that eliminates the interference signals. The repair unit is configured to input the audio signal to be repaired and the noise-reduced frequency signal into a pre-trained audio repair model to obtain the repaired target audio signal; the audio repair model is trained based on a second audio sample constructed from the sample distorted audio signal and the sample noise-reduced frequency signal corresponding to the sample distorted audio signal.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the audio signal processing method as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the audio signal processing method as described in any one of claims 1 to 7.

Images