Audio processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an audio processing method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining a to-be-processed audio signal, and carrying out the frequency domain transformation of the to-be-processed audio signal, and obtaining frequency domain information; performing feature compression processing on the frequency domain information to obtain first intermediate information, and inputting the first intermediate information into the trained deep learning model to obtain predicted feature information; and performing feature fusion processing on the frequency domain information and the prediction feature information to obtain audio correction information, and performing time domain transformation processing based on the audio correction information and the frequency domain information to obtain a target audio signal. According to the scheme, frequency domain information processing based on the deep learning model is realized, the method is suitable for audio signal processing in different complex acoustic scenes, the high-frequency damage of the audio signal is effectively compensated, and the high-frequency quality of the recovered audio signal is guaranteed.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio technology, and in particular, to an audio processing method, apparatus, device, storage medium, and program product. Background Art

[0002] With the rapid development of Internet technology, network bandwidth and transmission speed have been significantly improved, providing a reliable network environment for real-time calls. As an efficient and convenient way of communication, real-time calls meet people's needs for instant communication in different scenarios. Among them, in a one-to-one real-time call scenario, due to the influence of the sound reception conditions, hardware status and the processing algorithm in the data sending end, the sound signal received by the user at the data receiving end will have a high-frequency missing problem, which affects the clarity of the sound and also causes the information to not be transmitted in place.

[0003] In related technologies, based on the correlation between low-frequency signals and high-frequency signals, the high-frequency part is restored through the low-frequency information in the signal. For example, the audio is divided into several sub-bands by sub-band decomposition, and then the high-frequency sub-bands are analyzed to obtain high-frequency parameters. During reconstruction, the low-frequency information is copied to the high frequency, and the high-frequency information is adjusted according to the high-frequency analysis parameters to obtain a full-band audio signal. However, this type of method is only suitable for scenarios where the signal has a relatively fixed effective bandwidth, and cannot be adapted to audio signal processing in complex acoustic scenarios. The restored audio signal is prone to high-frequency noise or noise. Summary of the invention

[0004] The embodiments of the present application provide an audio processing method, apparatus, device, storage medium and program product, which solve the problem that the relevant technology is only suitable for scenarios where the signal has a relatively fixed effective bandwidth, cannot adapt to audio signal processing in complex acoustic scenarios, and is prone to high-frequency noise or noise in the relevant processing results. It realizes frequency domain information processing based on a deep learning model, is suitable for audio signal processing in different complex acoustic scenarios, effectively compensates for the high-frequency damage of the audio signal, and ensures the high-frequency quality of the restored audio signal.

[0005] In a first aspect, an embodiment of the present application provides an audio processing method, the method comprising:

[0006] Acquire an audio signal to be processed, and perform frequency domain transformation on the audio signal to be processed to obtain frequency domain information;

[0007] Performing feature compression processing on the frequency domain information to obtain first intermediate information, and inputting the first intermediate information into a trained deep learning model to obtain predicted feature information;

[0008] The frequency domain information and the predicted feature information are subjected to feature fusion processing to obtain audio correction information, and a time domain transformation processing is performed based on the audio correction information and the frequency domain information to obtain a target audio signal.

[0009] In a second aspect, an embodiment of the present application further provides an audio processing device, the device comprising:

[0010] A frequency domain transformation module is configured to obtain an audio signal to be processed, and perform frequency domain transformation processing on the audio signal to be processed to obtain frequency domain information;

[0011] A feature prediction module is configured to perform feature compression processing on the frequency domain information to obtain first intermediate information, and input the first intermediate information into a trained deep learning model to obtain predicted feature information;

[0012] The time domain transformation module is configured to perform feature fusion processing on the frequency domain information and the predicted feature information to obtain audio correction information, and perform time domain transformation processing based on the audio correction information and the frequency domain information to obtain a target audio signal.

[0013] In a third aspect, an embodiment of the present application further provides an audio processing device, the device comprising:

[0014] one or more processors;

[0015] a storage device configured to store one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio processing method described in the embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the audio processing method described in the embodiment of the present application when executed by a computer processor.

[0018] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the audio processing method described in the embodiment of the present application.

[0019] In an embodiment of the present application, by acquiring an audio signal to be processed, frequency domain transformation processing is performed on the audio signal to be processed to obtain frequency domain information; feature compression processing is performed on the frequency domain information to obtain first intermediate information, and the first intermediate information is input into a trained deep learning model to obtain predicted feature information; feature fusion processing is performed on the frequency domain information and the predicted feature information to obtain audio correction information, and time domain transformation processing is performed based on the audio correction information and the frequency domain information to obtain the target audio signal. In the above scheme, by performing feature compression processing on the frequency domain information, the spectral features of the frequency domain information can be further compressed, reducing the computational pressure of the deep learning model. The deep learning model can effectively predict the high-frequency missing information of the audio signal to determine the predicted feature information. By performing feature fusion processing on the frequency domain information and the predicted feature information, the model prediction information can be more accurately concentrated on the high-frequency part to avoid damage to the low-frequency effective signal, thereby adapting to audio signal processing in complex acoustic scenes, effectively compensating for the high-frequency damage of the audio signal, and ensuring the high-frequency quality of the restored audio signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of an audio processing method provided in an embodiment of the present application;

[0021] Figure 2 A structural diagram of a feature input module provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the structure of a deep learning model provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the structure of a first convolutional layer provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of a second convolutional layer provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a feature output module provided in an embodiment of the present application;

[0026] Figure 7 A flowchart of an audio processing method including a process of obtaining frequency domain information provided in an embodiment of the present application;

[0027] Figure 8 A flowchart of an audio processing method including a process of obtaining a target audio signal provided in an embodiment of the present application;

[0028] Fig. 9 A flowchart of an audio processing method including a process of determining a cutoff frequency of frequency domain information provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of a process of performing audio processing using an audio processing method provided in an embodiment of the present application;

[0030] Fig.11 A structural block diagram of an audio processing device provided in an embodiment of the present application;

[0031] Fig.12 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only parts related to the embodiments of the present application are shown in the accompanying drawings, rather than all structures.

[0033] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0034] The audio processing method provided in the embodiment of the present application can sequentially perform feature compression processing, feature prediction of a deep learning model and feature fusion processing on the frequency domain information obtained by converting the audio signal to be processed, and obtain audio correction information, which is beneficial to repairing the high-frequency damage of the audio signal. Relevant application scenarios may include: audio calls, audio live broadcasts, etc. The several application scenarios listed above are only exemplary and explanatory. In actual applications, the audio processing method can also be used in audio processing in other scenarios, and the embodiment of the present application does not limit this. The audio processing method, device, equipment, storage medium and program product provided in the embodiment of the present application are intended to solve the problem that the relevant technology is only suitable for scenarios where the signal has a relatively fixed effective bandwidth, and cannot be adapted to audio signal processing in complex acoustic scenarios, and high-frequency noise or noise is prone to appear in the relevant processing results.

[0035] In the audio processing method provided in the embodiment of the present application, the executor of each step can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices, and can also be servers and other devices, which are not limited in the embodiment of the present application.

[0036] Figure 1 A flowchart of an audio processing method provided in an embodiment of the present application. In an audio transmission scenario, after the data sending end samples the audio signal to be transmitted through the audio acquisition device of the device, it will perform near-end algorithm processing on the sampled signal, and then encode it through the audio encoder, and transmit the encoded result to the data receiving end through the physical channel. After the data receiving end receives the encoded result, it will decode the encoded result through the audio decoder to obtain an audio signal, and apply the audio processing method provided in an embodiment of the present application to expand the bandwidth of the audio signal to obtain a target audio signal to compensate for the missing high-frequency part, and then play the target audio signal. The executor of the audio processing method can be a data receiving end, such as Figure 1 As shown, the audio processing method comprises the following steps:

[0037] Step S101: Acquire an audio signal to be processed, and perform frequency domain transformation on the audio signal to be processed to obtain frequency domain information.

[0038] Among them, the audio signal to be processed can be obtained after the data receiving end performs a decoding operation through an audio decoder. Due to the influence of the sound receiving conditions, the hardware status of the device and the audio processing algorithm of the data sending end, the audio signal to be processed is prone to high-frequency missing problems, resulting in poor sound quality. In one embodiment, since the audio data received by the data receiving end may contain both silent frames and non-silent frames, and the silent frames themselves do not contain useful language information, the audio processing of high-frequency recovery does not increase the quality or comprehensibility of the audio content, but consumes additional computing resources. Therefore, the audio signal to be processed can be first subjected to silent frame detection. If the audio signal to be processed belongs to a silent frame, there is no need to perform subsequent audio processing steps. If the audio signal to be processed belongs to a non-silent frame, the subsequent audio processing steps are continued. Thereby, unnecessary computing resources are avoided, audio quality is improved, and audio processing efficiency is guaranteed. In one embodiment, the frequency domain transform processing is used to convert the audio signal to be processed from the original time domain signal into frequency domain information, which can be specifically a method such as a short-time Fourier transform, a discrete cosine transform or a wavelet transform, which is not limited in this application. The frequency domain information may include amplitude signals and phase information.

[0039] Step S102: perform feature compression processing on the frequency domain information to obtain first intermediate information, and input the first intermediate information into the trained deep learning model to obtain predicted feature information.

[0040] Among them, feature compression processing is used to perform rough feature extraction on frequency domain information, and compress the corresponding dimension to be suitable for the input requirements of the deep learning model processing, and obtain the corresponding first intermediate information as the input information of the subsequent deep learning model. Specifically, it can be processed using a local linear embedding algorithm, a singular value decomposition algorithm, or a constructed neural network module, etc., which is not limited in this application. In one embodiment, feature compression processing is performed on frequency domain information to obtain the first intermediate information. The specific implementation process can be: input the frequency domain information into the set feature input module to obtain the first intermediate information. Among them, Figure 2 A structural diagram of a feature input module provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the feature input module includes a pre-filled layer, a deep separable convolution layer and an activation function layer connected in sequence. Since the subsequent deep learning model has a minimum input length requirement, it is necessary to fill the frequency domain information to meet the input requirements of the subsequent deep learning model. The pre-filled layer is used to receive the frequency domain information and perform data filling processing on the frequency domain information. The deep separable convolution layer can be divided into two stages: deep convolution and point-by-point convolution. In the deep convolution stage, the features of each channel can be independently extracted for spatial features to capture the local spatial information unique to different channels. Then, in the point-by-point convolution stage, these independently extracted channel features can be fused to generate new feature representations to achieve the purpose of feature extraction. The activation function layer can perform a nonlinear transformation on the output result of the deep separable convolution layer to obtain the first intermediate information, and output the first intermediate information. Thus, the frequency domain information can be further compressed to reduce the computational pressure of the deep learning model. In addition, the first intermediate information is input into the trained deep learning model to obtain predicted feature information, which includes high-frequency predicted amplitude information and predicted phase information.

[0041] In one embodiment, Figure 3 A schematic diagram of the structure of a deep learning model provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the deep learning model is a self-encoding structure, which specifically includes an encoder module, a timing modeling layer and a decoder module connected in sequence. The encoder module is jump-connected to the decoder module. The encoder module includes multiple encoders ( Figure 3 ), the encoder includes a first residual feature calculation module and a first convolution layer connected in sequence, and the first residual feature calculation module includes a plurality of first residual feature calculation units ( Figure 3), the first residual feature calculation unit includes a second convolution layer, a hole convolution layer and a third convolution layer connected in sequence, and the encoder module includes a plurality of decoders ( Figure 3 ), the decoder includes a second residual feature calculation module and a transposed convolution layer connected in sequence, and the second residual feature calculation module includes a plurality of second residual feature calculation units ( Figure 3 (not shown in the figure). Among them, the processing mode of the deep learning module is serial processing, and there are several skip connections for direct information interaction between codecs at the same level. The timing modeling layer between the encoder module and the decoder module can be used to capture timing information and purify features. Optionally, the timing modeling layer can be a gated recurrent unit layer. The first residual feature calculation module in the encoder is used to improve the receptive field of feature extraction, and the first convolutional layer is used for downsampling the feature processing results of the first residual feature calculation module. Specifically, the second convolutional layer and the third convolutional layer in the first residual feature calculation unit in the first residual feature calculation module can be standard convolutional layers, and the spatial convolutional layer can be used to increase the receptive field of the convolution kernel while keeping the resolution of the feature information unchanged. The relevant parameters of the decoder module are set layer by layer in correspondence with the encoder module, and the sampling coefficient of each encoder or decoder module is controlled by the step size parameter of the convolutional layer therein.

[0042] In one embodiment, in order to effectively reduce the amount of model calculation, the first convolution layer can be replaced by a depth-separable convolution process instead of a standard convolution process. Figure 4 A schematic diagram of the structure of a first convolutional layer provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the first convolution layer includes a first point convolution layer, a depth convolution layer and a second point convolution layer connected in sequence. The first point convolution layer is used to initially reduce the number of channels of the input feature, the depth convolution layer is used to perform a convolution operation on each channel of the input using an independent convolution kernel, and the second point convolution layer is used to combine information from different channels and output feature processing results. In addition, the second and third convolution layers can also be replaced by compression processing with random rearrangement of channels instead of standard convolution processing. Figure 5 A schematic diagram of the structure of a second convolutional layer provided in an embodiment of the present application is shown in FIG. Figure 5As shown, the second convolution layer includes a first channel splitting layer, a first channel convolution layer, a first channel merging layer and a first channel rearrangement layer connected in sequence, and the first channel splitting layer is connected to the first channel merging layer in a jump step. Among them, the first channel splitting layer can be used for feature channel splitting and grouping, and the first channel convolution layer can be used to calculate the convolution of some feature channels after the feature channel splitting and grouping, for example, the convolution of half of the feature channels, and the first channel merging layer can be used to merge the processing results of the first channel splitting layer and the processing results of the first channel convolution layer. The first channel rearrangement layer is used to disrupt the feature channels to ensure that all feature channels will be calculated in the overall model. Correspondingly, the third convolution layer includes a second channel splitting layer, a second channel convolution layer, a second channel merging layer and a second channel rearrangement layer connected in sequence. The second channel splitting layer is connected to the second channel merging layer in a jump step. Therefore, by performing structural replacement and structured pruning on the deep learning model, the amount of calculation and the amount of parameters are significantly reduced, so as to achieve the purpose of compressing the deep learning model, so that it can run stably in the low computing power scenario of the mobile terminal.

[0043] In one embodiment, the deep learning model can be trained using damaged audio samples containing original audio samples and high-frequency information removed, and the damaged audio samples can be obtained by inputting the original audio samples into a low-pass filter or other methods to remove the high-frequency information. In addition, a generative adversarial network training architecture can be adopted, wherein the discriminator is composed of a multi-scale frequency domain discriminator and a multi-period time domain discriminator. Generative adversarial loss function, time domain loss function, multi-scale spectral loss function, phase loss function and compatibility loss function can be used in training, wherein the generative adversarial loss function includes a generator loss function, a discriminator loss function and a discriminator feature loss function. The phase loss function is used to constrain the phase prediction result, and the compatibility loss function is used to alleviate the compatibility problem of amplitude phase. The specific formula of the relevant loss function is as follows:

[0044] Generator loss function: L G =E[Relu(1-D(G(x)))],

[0045] Discriminator loss function: L D =E[Relu(1+D(G(x)))]+E[Relu(1-D(y))],

[0046] Discriminator feature loss function: L Df =E[|D feat (y)-D feat (G(x))|1],

[0047] Time domain loss function: L T =|yG(x)|1,

[0048] Multi-scale amplitude spectrum loss function:

[0049]

[0050] Phase loss function:

[0051]

[0052] Compatibility loss function: L c =|STFT(G(x))-STFT(ISTFT(G(x)))|1,

[0053] The total loss function formula is:

[0054]

[0055] Where x represents the signal before processing, y represents the training reference signal, G(x) represents the signal after processing, e(*) represents the averaging function, Relu(*) is the activation function, D(*) represents the discriminator function, and D feat (*) indicates the discriminator feature output, Mag(*) indicates the amplitude spectrum, Phas(*) indicates the phase spectrum, STFT(*) indicates the short-time Fourier transform, ISTFT(*) indicates the inverse short-time Fourier transform, subscript n indicates the number of short-time Fourier transform points, which can be 16, 512, 2048, etc. according to empirical values, |*|1 indicates the 1-norm, and |*|2 indicates the 2-norm. Optionally, a, b, c, d, and e can be 1.0, 100, 0.5, 0.1, and 0.5, respectively, according to empirical values.

[0056] Therefore, the training process of the deep learning model can reasonably take into account both amplitude information and phase information, effectively avoiding the problems caused by the compatibility of amplitude and phase, and making the processed audio signal more natural.

[0057] Step S103: Perform feature fusion processing on the frequency domain information and the predicted feature information to obtain audio correction information, and perform time domain transformation processing based on the audio correction information and the frequency domain information to obtain a target audio signal.

[0058] Among them, the feature fusion processing is used to fuse the frequency domain information and the predicted feature information, blur the cutoff frequency band information of the predicted feature information, and prevent the output from affecting the existing low-frequency information. Specifically, it can be processed using a weighted average algorithm or a constructed neural network module, etc., which is not limited in this application. In one embodiment, the frequency domain information and the predicted feature information are subjected to feature fusion processing to obtain audio correction information. The specific implementation process can be: input the frequency domain information and the predicted feature information into the set feature output module to obtain the audio correction information. Among them, Figure 6 A structural diagram of a feature output module provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the feature output module includes a matrix addition layer, a fourth convolution layer and a fifth convolution layer connected to the matrix addition layer, a first matrix multiplication layer connected to the fourth convolution layer and the fifth convolution layer, a sixth convolution layer connected to the first matrix multiplication layer, and a second matrix multiplication layer connected to the sixth convolution layer. The matrix addition layer is used to receive frequency domain information and predicted feature information, and the second matrix multiplication layer is used to receive the output information and predicted feature information of the sixth convolution layer, and output audio correction information. Thus, the predicted feature information output by the deep learning module can be optimized so that the predicted feature information is more accurately concentrated on the high-frequency part to avoid damage to the low-frequency effective signal. In one embodiment, the audio correction information may include amplitude correction information and phase correction information, and the time domain transformation processing is used to fuse the audio correction information and the frequency domain information, and then convert the frequency domain information into a time domain signal. The specific fusion processing may be direct superposition, weighted averaging, etc., and the time domain transformation processing may be using methods such as short-time Fourier inverse transform, inverse discrete cosine transform or inverse wavelet transform, which are not limited in this application.

[0059] In the above, by acquiring the audio signal to be processed, the audio signal to be processed is subjected to frequency domain transformation processing to obtain frequency domain information; the frequency domain information is subjected to feature compression processing to obtain first intermediate information, and the first intermediate information is input into the trained deep learning model to obtain predicted feature information; the frequency domain information and the predicted feature information are subjected to feature fusion processing to obtain audio correction information, and the target audio signal is obtained by performing time domain transformation processing based on the audio correction information and the frequency domain information. In the above scheme, by performing feature compression processing on the frequency domain information, the spectral features of the frequency domain information can be further compressed, reducing the computational pressure of the deep learning model. The deep learning model can effectively predict the high-frequency missing information of the audio signal to determine the predicted feature information. By performing feature fusion processing on the frequency domain information and the predicted feature information, the model prediction information can be more accurately concentrated on the high-frequency part to avoid damage to the low-frequency effective signal, thereby adapting to the audio signal processing in complex acoustic scenes, effectively compensating for the high-frequency damage of the audio signal, and ensuring the high-frequency quality of the restored audio signal.

[0060] Figure 7 A flowchart of an audio processing method including a process of obtaining frequency domain information is provided in an embodiment of the present application. Figure 7 As shown, the audio processing method comprises the following steps:

[0061] Step S201, obtain the audio signal to be processed, perform short-time Fourier transform based on a short analysis window on the audio signal to be processed to obtain a frequency domain signal, map the phase information of the frequency domain signal within a first numerical range to obtain target phase information, and splice the amplitude information of the frequency domain signal and the target phase information to obtain frequency domain information.

[0062] Among them, the formula of short-time Fourier transform is as follows:

[0063]

[0064] Wherein, x[k] represents the time series of the audio signal to be processed, w[*] represents the window function. For example, the Hanning window is used in this embodiment. X[f,n] ​​represents the frequency domain signal, and f and n are the indexes of the frequency and time frames, respectively. N represents the number of Fourier transform points. In order to obtain a better phase estimation effect, a short analysis window is used in this embodiment, for example, N=16. Optionally, the first numerical range can be in the range of [0,1],

[0065] Step S202: perform feature compression processing on the frequency domain information to obtain first intermediate information, and input the first intermediate information into the trained deep learning model to obtain predicted feature information.

[0066] Step S203: Perform feature fusion processing on the frequency domain information and the predicted feature information to obtain audio correction information, and perform time domain transformation processing based on the audio correction information and the frequency domain information to obtain a target audio signal.

[0067] As mentioned above, by using a short analysis window and a small frame shift in the short-time Fourier transform, the phase fluctuation problem existing when using a long analysis window can be avoided and the phase winding phenomenon can be alleviated, and the continuity of the amplitude phase of the previous and next frames can be improved. By mapping the phase information to the first numerical range, the metric standard of the phase information can be unified, which is convenient for subsequent analysis and comparison; by splicing the amplitude information and the target phase information, the frequency domain characteristics of the signal can be fully represented, which is convenient for subsequent feature extraction and other operations.

[0068] Figure 8 The present invention provides a flowchart of an audio processing method including a process of processing a target audio signal. Figure 8 As shown, the audio processing method comprises the following steps:

[0069] Step S301, obtain the audio signal to be processed, perform short-time Fourier transform based on a short analysis window on the audio signal to be processed to obtain a frequency domain signal, map the phase information of the frequency domain signal in a first numerical range to obtain target phase information, and splice the amplitude information of the frequency domain signal and the target phase information to obtain frequency domain information.

[0070] Step S302: Perform feature compression processing on the frequency domain information to obtain first intermediate information, and input the first intermediate information into the trained deep learning model to obtain predicted feature information.

[0071] Step S303, perform feature fusion processing on the frequency domain information and the predicted feature information to obtain audio correction information, split the audio correction information to obtain amplitude correction information and phase correction information, map the phase correction information to a second numerical range to obtain target phase correction information, superimpose the amplitude correction information, the target phase correction information and the frequency domain information to obtain target frequency domain information, and perform short-time Fourier inverse transform based on a short analysis window on the target frequency domain information to obtain a target audio signal.

[0072] Among them, the formula for short-time inverse Fourier transform is as follows:

[0073]

[0074] Wherein, x[k] represents the time series of the target audio signal, w[*] represents the window function, for example, the Hanning window is used in this embodiment, X[f,n] ​​represents the target frequency domain signal, f and n are the indexes of the frequency and time frame respectively. N represents the number of Fourier transform points, in order to obtain a better phase estimation effect, a short analysis window is used in this embodiment, for example, N=16.

[0075] As described above, by splitting the audio correction information to obtain amplitude correction information and phase correction information, and mapping the phase correction information to a second numerical range to obtain the target phase correction information, it can correspond to the mapping operation and splicing operation in the frequency domain transformation processing, and effectively restore the different frequency domain characteristic information of the audio correction information; by superimposing the amplitude correction information, the target phase correction information and the frequency domain information to obtain the target frequency domain information, the frequency domain information can be effectively repaired at high frequencies to ensure the audio quality of the restored target audio signal.

[0076] Fig. 9 The present invention provides a flowchart of an audio processing method including a process of determining a cutoff frequency of frequency domain information. Fig. 9 As shown, the audio processing method comprises the following steps:

[0077] Step S401: Acquire an audio signal to be processed, and perform frequency domain transformation on the audio signal to be processed to obtain frequency domain information.

[0078] Step S402: Perform attenuation detection of frequency band energy on the frequency domain information to determine the cutoff frequency.

[0079] Among them, since the energy of the frequency domain signal will decay rapidly at the cutoff frequency, correspondingly, the slope of the frequency curve at the frequency point corresponding to the cutoff frequency is less than zero, and the absolute value reaches the maximum, and the energy will be significantly attenuated in a frequency band after the cutoff frequency. Therefore, the cutoff frequency can be determined by performing attenuation detection of the frequency band energy on the frequency domain information. The specific implementation process is as follows: First, determine the retrieval starting point and preset bandwidth length on the frequency curve corresponding to the frequency domain information. For example, for a 16kHz audio signal, its effective frequency range is 0-8kHz, and the corresponding retrieval starting point can be set within 0-4kHz according to the actual accuracy requirements. The preset bandwidth length can be set to 200Hz, 500Hz, etc., so as to obtain a first reference frequency band for calculating the slope. Based on the two endpoints of the first reference frequency band, the first energy attenuation value can be calculated, and the first energy attenuation value can be used to characterize the energy attenuation degree of the first reference frequency band. The first energy attenuation value is compared with the set first energy threshold. If the first energy attenuation value is less than or equal to the first energy threshold, it can be regarded that the energy attenuation degree of the first reference frequency band is small, and the possibility of it being the cutoff frequency is low. The search starting point can be right-shifted according to the preset step value, and the second reference frequency band is obtained in combination with the preset bandwidth length. The second energy attenuation value is calculated based on the two endpoints of the second reference frequency band, and the second energy attenuation value is compared with the first energy threshold, and the subsequent judgment process is re-executed. If the first energy attenuation value is greater than the first energy threshold, in order to prevent the two endpoints of the first reference frequency band from being extreme points, resulting in misjudgment of the cutoff frequency, the two endpoints of the first reference frequency band can be used as midpoints to determine the corresponding two narrow bands, and the difference in the average energy of the two narrow bands can be calculated respectively, and the difference is compared with the first energy threshold. If the difference is less than or equal to the first energy threshold, the search starting point is right-shifted according to the preset step value, and the subsequent judgment process is re-executed. If the difference is greater than the first energy threshold, the right endpoint of the first reference frequency band is shifted right according to the preset step value, and the third energy attenuation value is calculated based on the two endpoints of the first reference frequency band. The third energy attenuation value is compared with the second energy threshold. If the third energy attenuation value is greater than or equal to the second energy threshold, the right endpoint is continuously shifted right according to the step value, and the new energy attenuation value is re-determined to be compared with the second energy threshold until the step limit is reached. If the third energy attenuation value is less than the second energy threshold or the right shift of the right endpoint has reached the step limit, the in-band energy roll-off slope of the first reference frequency band is calculated. The in-band energy roll-off slope is compared with the recorded existing maximum slope. If the in-band energy roll-off slope exceeds the existing maximum slope, the in-band energy roll-off slope is updated to the new maximum slope. After recording the left endpoint corresponding to the first reference frequency band, the left endpoint is shifted right according to the preset step value, and a new reference frequency band is obtained in combination with the preset bandwidth length, and the subsequent judgment process is continued until the band edge of the frequency domain information is reached.If the energy roll-off slope in the band does not exceed the existing maximum slope, the left endpoint is directly shifted right according to the preset step value, and a new reference frequency band is obtained in combination with the preset bandwidth length, and the subsequent judgment process is continued until the frequency band edge of the frequency domain information is reached. Thus, the frequency of the left endpoint corresponding to the maximum slope finally obtained is the cutoff frequency.

[0080] Step S403: When the cutoff frequency is less than a preset frequency threshold, feature compression processing is performed on the frequency domain information to obtain first intermediate information, and the first intermediate information is input into the trained deep learning model to obtain predicted feature information.

[0081] Among them, if the cutoff frequency is greater than or equal to the preset frequency threshold, it can be regarded as that the high-frequency information of the audio signal to be processed is relatively complete, and there is no need to continue to execute subsequent steps. If the cutoff frequency is less than the preset frequency threshold, it can be regarded as that the high-frequency information of the audio signal to be processed is missing, and the subsequent steps are continued to perform high-frequency repair. For example, for a 16kHz audio signal, its effective frequency range is 8kHz, and the preset frequency threshold can be set to 6kHz. If the cutoff frequency of the audio signal to be processed is 4kHz, since 4kHz is less than 6kHz, it can be regarded that the audio signal to be processed is in a high-frequency missing state and needs to perform high-frequency repair. If the cutoff frequency of the audio signal to be processed is 7kHz, since 7kHz is greater than 6kHz, it can be regarded as that the high-frequency information of the audio signal to be processed is relatively complete, and there is no need to perform high-frequency repair. Of course, the specific preset frequency threshold can be adaptively set according to the effective frequency range and detection accuracy requirements of the actual audio signal, and this application is not limited here.

[0082] Step S404: perform feature fusion processing on the frequency domain information and the predicted feature information to obtain audio correction information, and perform time domain transformation processing based on the audio correction information and the frequency domain information to obtain a target audio signal.

[0083] As described above, by performing attenuation detection of the frequency band energy on the frequency domain information to determine the cutoff frequency, it is possible to effectively determine whether the audio signal to be processed is in a high-frequency missing state, and determine whether to continue to perform subsequent high-frequency repair steps based on the comparison result between the cutoff frequency and the preset frequency threshold. There is no need to repair the audio signal to be processed with relatively complete high frequencies, thereby improving the audio processing efficiency.

[0084] In one embodiment, Fig.10 A schematic diagram of a process of performing audio processing using an audio processing method provided in an embodiment of the present application is shown in FIG. Fig.10As shown, the silent frame detection module 101 performs silent detection on the input audio signal to be processed. If it is detected that the audio signal to be processed belongs to a non-silent frame, the audio signal to be processed is passed to the pre-processing module 102. The pre-processing module 102 performs frequency domain transformation on the audio signal to be processed to obtain frequency domain information, and passes the frequency domain information to the effective bandwidth detection module 103 and the feature output module 106. The effective bandwidth detection module 103 performs attenuation detection of the frequency band energy on the frequency domain information to determine the cutoff frequency. If the cutoff frequency is less than the preset frequency threshold, the frequency domain information is passed to the feature input module 104. The feature input module 106 04 performs feature compression processing on the frequency domain information to obtain the first intermediate information, and passes the first intermediate information to the deep learning model 105. The deep learning model 105 processes the first intermediate information and outputs predicted feature information, and passes the predicted feature information to the feature output module 106. The feature output module 106 performs feature fusion processing on the received frequency domain information and the predicted feature information to obtain audio correction information, and passes the audio correction information to the post-processing module 107. The post-processing module 107 performs time domain transformation processing based on the audio correction information and the frequency domain information to obtain the target audio signal, and outputs the target audio signal.

[0085] As described above, by applying the audio processing method provided in the embodiment of the present application to perform real-time processing of the audio signal in the call scenario, adaptive bandwidth expansion can be achieved to compensate for the damage to the high frequency of the signal caused by the physical environment, so that the audio signal has an objectively higher effective bandwidth and is subjectively clearer.

[0086] Fig.11 This is a structural block diagram of an audio processing device provided in an embodiment of the present application. The device is configured to execute the audio processing method provided in the above embodiment and has the corresponding functional modules and beneficial effects of the execution method. Fig.11 As shown, the device comprises:

[0087] The frequency domain transformation module 201 is configured to obtain the audio signal to be processed, and perform frequency domain transformation on the audio signal to be processed to obtain frequency domain information;

[0088] The feature prediction module 202 is configured to perform feature compression processing on the frequency domain information to obtain first intermediate information, and input the first intermediate information into the trained deep learning model to obtain predicted feature information;

[0089] The time domain transformation module 203 is configured to perform feature fusion processing on the frequency domain information and the predicted feature information to obtain audio correction information, and perform time domain transformation processing based on the audio correction information and the frequency domain information to obtain a target audio signal.

[0090] In the above, by acquiring the audio signal to be processed, the audio signal to be processed is subjected to frequency domain transformation processing to obtain frequency domain information; the frequency domain information is subjected to feature compression processing to obtain first intermediate information, and the first intermediate information is input into the trained deep learning model to obtain predicted feature information; the frequency domain information and the predicted feature information are subjected to feature fusion processing to obtain audio correction information, and the target audio signal is obtained by performing time domain transformation processing based on the audio correction information and the frequency domain information. In the above scheme, by performing feature compression processing on the frequency domain information, the spectral features of the frequency domain information can be further compressed, reducing the computational pressure of the deep learning model. The deep learning model can effectively predict the high-frequency missing information of the audio signal to determine the predicted feature information. By performing feature fusion processing on the frequency domain information and the predicted feature information, the model prediction information can be more accurately concentrated on the high-frequency part to avoid damage to the low-frequency effective signal, thereby adapting to the audio signal processing in complex acoustic scenes, effectively compensating for the high-frequency damage of the audio signal, and ensuring the high-frequency quality of the restored audio signal.

[0091] In one possible embodiment, the deep learning model includes an encoder module, a timing modeling layer and a decoder module connected in sequence, the encoder module is jump-connected to the decoder module, the encoder module includes a plurality of encoders connected in sequence, the encoder includes a first residual feature calculation module and a first convolutional layer connected in sequence, the first residual feature calculation module includes a plurality of first residual feature calculation units connected in sequence, the first residual feature calculation unit includes a second convolutional layer, a hole convolutional layer and a third convolutional layer connected in sequence, the encoder module includes a plurality of decoders connected in sequence, the decoder includes a second residual feature calculation module and a transposed convolutional layer connected in sequence, and the second residual feature calculation module includes a plurality of second residual feature calculation units connected in sequence.

[0092] In a possible embodiment, the first convolution layer includes a first point convolution layer, a depth convolution layer, and a second point convolution layer connected in sequence, the second convolution layer includes a first channel splitting layer, a first channel convolution layer, a first channel merging layer, and a first channel rearrangement layer connected in sequence, the first channel splitting layer is jump-connected to the first channel merging layer, the third convolution layer includes a second channel splitting layer, a second channel convolution layer, a second channel merging layer, and a second channel rearrangement layer connected in sequence, and the second channel splitting layer is jump-connected to the second channel merging layer.

[0093] In a possible embodiment, the frequency domain transformation module 201 is further configured as follows:

[0094] Performing short-time Fourier transform based on a short analysis window on the audio signal to be processed to obtain a frequency domain signal;

[0095] Mapping the phase information of the frequency domain signal to a first value range to obtain target phase information;

[0096] The frequency domain information is obtained by concatenating the amplitude information of the frequency domain signal and the target phase information.

[0097] In a possible embodiment, the time domain transformation module 203 is further configured as follows:

[0098] Splitting the audio correction information into amplitude correction information and phase correction information;

[0099] Mapping the phase correction information to a second value range to obtain target phase correction information;

[0100] The amplitude correction information, the target phase correction information and the frequency domain information are superimposed to obtain the target frequency domain information;

[0101] The target frequency domain information is subjected to short-time inverse Fourier transform based on a short analysis window to obtain the target audio signal.

[0102] In a possible embodiment, the feature prediction module 202 is further configured to:

[0103] The frequency domain information is input into a set feature input module to obtain the first intermediate information. The feature input module includes a pre-filled layer, a depth-separable convolutional layer and an activation function layer connected in sequence. The pre-filled layer is used to receive the frequency domain information, and the activation function layer is used to output the first intermediate information.

[0104] In a possible embodiment, the time domain transformation module 203 is further configured as follows:

[0105] The frequency domain information and the predicted feature information are input into a set feature output module to obtain the audio correction information. The feature output module includes a matrix addition layer, a fourth convolution layer and a fifth convolution layer connected to the matrix addition layer, a first matrix multiplication layer connected to the fourth convolution layer and the fifth convolution layer, a sixth convolution layer connected to the first matrix multiplication layer, and a second matrix multiplication layer connected to the sixth convolution layer. The matrix addition layer is used to receive the frequency domain information and the predicted feature information, the second matrix multiplication layer is used to receive the output information of the sixth convolution layer and the predicted feature information, and output the audio correction information.

[0106] In a possible embodiment, a cutoff frequency determination module is further included, configured as follows:

[0107] Performing attenuation detection of frequency band energy on frequency domain information to determine the cutoff frequency;

[0108] Accordingly, the feature prediction module 202 is further configured as follows:

[0109] When the cutoff frequency is less than the preset frequency threshold, feature compression processing is performed on the frequency domain information to obtain first intermediate information.

[0110] Fig.12A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the device includes a processor 301, a memory 302, an input device 303 and an output device 304; the number of processors 301 in the device can be one or more. Fig.12 A processor 301 is taken as an example; the processor 301, memory 302, input device 303 and output device 304 in the device can be connected by a bus or other means. Fig.12 The example of connection via bus is taken. The memory 302, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the audio processing method in the embodiment of the present application. The processor 301 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 302, that is, implements the above-mentioned audio processing method. The input device 303 can be configured to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 304 may include a display device such as a display screen.

[0111] An embodiment of the present application also provides a non-volatile storage medium containing computer executable instructions, which, when executed by a computer processor, is configured to execute an audio processing method described in the above embodiment, which includes: obtaining an audio signal to be processed, performing frequency domain transform processing on the audio signal to be processed to obtain frequency domain information; performing feature compression processing on the frequency domain information to obtain first intermediate information, and inputting the first intermediate information into a trained deep learning model to obtain predicted feature information; performing feature fusion processing on the frequency domain information and the predicted feature information to obtain audio correction information, and performing time domain transform processing based on the audio correction information and the frequency domain information to obtain a target audio signal.

[0112] It is worth noting that in the embodiment of the above-mentioned audio processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the protection scope of the embodiments of the present application.

[0113] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is configured to enable the computer device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the audio processing method described in the embodiment of the present application. The program product may be implemented in any combination of one or more readable media.

Claims

1. An audio processing method, characterized in that: include: Acquire an audio signal to be processed, and perform frequency domain transformation on the audio signal to be processed to obtain frequency domain information; Performing feature compression processing on the frequency domain information to obtain first intermediate information, and inputting the first intermediate information into a trained deep learning model to obtain predicted feature information; The frequency domain information and the predicted feature information are subjected to feature fusion processing to obtain audio correction information, and a time domain transformation processing is performed based on the audio correction information and the frequency domain information to obtain a target audio signal.

2. The audio processing method according to claim 1, characterized in that: The deep learning model includes an encoder module, a timing modeling layer and a decoder module connected in sequence, the encoder module is jump-connected to the decoder module, the encoder module includes a plurality of encoders connected in sequence, the encoder includes a first residual feature calculation module and a first convolutional layer connected in sequence, the first residual feature calculation module includes a plurality of first residual feature calculation units connected in sequence, the first residual feature calculation unit includes a second convolutional layer, a hole convolutional layer and a third convolutional layer connected in sequence, the encoder module includes a plurality of decoders connected in sequence, the decoder includes a second residual feature calculation module and a transposed convolutional layer connected in sequence, and the second residual feature calculation module includes a plurality of second residual feature calculation units connected in sequence.

3. The audio processing method according to claim 2, characterized in that: The first convolution layer includes a first point convolution layer, a depth convolution layer, and a second point convolution layer connected in sequence; the second convolution layer includes a first channel splitting layer, a first channel convolution layer, a first channel merging layer, and a first channel rearrangement layer connected in sequence; the first channel splitting layer is jump-connected to the first channel merging layer; the third convolution layer includes a second channel splitting layer, a second channel convolution layer, a second channel merging layer, and a second channel rearrangement layer connected in sequence; the second channel splitting layer is jump-connected to the second channel merging layer.

4. The audio processing method according to claim 1, characterized in that: The performing frequency domain transformation processing on the audio signal to be processed to obtain frequency domain information includes: Performing a short-time Fourier transform based on a short analysis window on the audio signal to be processed to obtain a frequency domain signal; Mapping the phase information of the frequency domain signal into a first value range to obtain target phase information; The amplitude information of the frequency domain signal and the target phase information are concatenated to obtain frequency domain information.

5. The audio processing method according to claim 4, characterized in that: The step of performing time domain transformation processing based on the audio correction information and the frequency domain information to obtain a target audio signal includes: Splitting the audio correction information into amplitude correction information and phase correction information; Mapping the phase correction information to a second value range to obtain target phase correction information; Superimposing the amplitude correction information, the target phase correction information and the frequency domain information to obtain target frequency domain information; The target frequency domain information is subjected to a short-time inverse Fourier transform based on a short analysis window to obtain a target audio signal.

6. The audio processing method according to claim 1, characterized in that: The step of performing feature compression processing on the frequency domain information to obtain first intermediate information includes: The frequency domain information is input into a set feature input module to obtain the first intermediate information, wherein the feature input module includes a pre-filled layer, a depth-separable convolutional layer and an activation function layer connected in sequence, the pre-filled layer is used to receive the frequency domain information, and the activation function layer is used to output the first intermediate information.

7. The audio processing method according to claim 1, characterized in that: The step of performing feature fusion processing on the frequency domain information and the predicted feature information to obtain the audio correction information includes: The frequency domain information and the predicted feature information are input into a set feature output module to obtain audio correction information, wherein the feature output module includes a matrix addition layer, a fourth convolution layer and a fifth convolution layer connected to the matrix addition layer, a first matrix multiplication layer connected to the fourth convolution layer and the fifth convolution layer, a sixth convolution layer connected to the first matrix multiplication layer, and a second matrix multiplication layer connected to the sixth convolution layer, the matrix addition layer is used to receive the frequency domain information and the predicted feature information, the second matrix multiplication layer is used to receive the output information of the sixth convolution layer and the predicted feature information, and output the audio correction information.

8. The audio processing method according to claim 1, characterized in that: Before performing feature compression processing on the frequency domain information to obtain the first intermediate information, the method further includes: Performing attenuation detection of frequency band energy on the frequency domain information to determine a cutoff frequency; Correspondingly, performing feature compression processing on the frequency domain information to obtain first intermediate information includes: When the cut-off frequency is less than a preset frequency threshold, feature compression processing is performed on the frequency domain information to obtain first intermediate information.

9. An audio processing device, characterized in that: include: A frequency domain transformation module is configured to obtain an audio signal to be processed, and perform frequency domain transformation processing on the audio signal to be processed to obtain frequency domain information; A feature prediction module is configured to perform feature compression processing on the frequency domain information to obtain first intermediate information, and input the first intermediate information into a trained deep learning model to obtain predicted feature information; The time domain transformation module is configured to perform feature fusion processing on the frequency domain information and the predicted feature information to obtain audio correction information, and perform time domain transformation processing based on the audio correction information and the frequency domain information to obtain a target audio signal.

10. An audio processing device, the device comprising: one or more processors; A storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the audio processing method described in any one of claims 1-8.

11. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the audio processing method according to any one of claims 1 to 8 when executed by a computer processor.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the audio processing method according to any one of claims 1 to 8 is implemented.

Images