Audio acquisition real-time repair method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose an audio collection real-time repair method and device, electronic equipment and a storage medium. In the method, real-time audio data generated by audio collection is obtained; the real-time audio data is input into a sound quality repair model pre-trained, the input of the sound quality repair model is a PQMF subband, the sound quality repair model is trained based on a generative adversarial network, the generative adversarial network includes a generator and a discriminator, the generator includes an encoding module and a decoding module, the encoding module includes a causal deep separable convolution module and a grouped LSTM module, the decoding module includes a mask path and a mapping path both using a causal deep separable transposed convolution, and the output complex spectrum of the mask path and the mapping path is feature fused with the PQMF subband; and the output of the sound quality repair model corresponding to the real-time audio data is taken as real-time repair audio. The present scheme can shorten the time required for audio repair to ideal quality, and meet the speed requirement of audio processing in real-time application scenarios.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of audio repair, and in particular to an audio collection real-time repair method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In application scenarios such as video conferencing, online communication audio and video communication, real-time audio collection and transmission will be performed. In the real-time collection and transmission process, the microphone collection and the 3A signal processing after collection will cause audio quality damage, and the packet loss in the process of audio coding and transmission for application scenarios can also cause further damage to the voice. In particular, in the case of high reverberation, low signal-to-noise ratio and low signal-to-reverberation ratio, all manufacturers will add noise reduction and echo cancellation based on NN. Due to the frequency aliasing of voice and noise and reverberation in the high reverberation case, the noise reduction and echo cancellation model is prone to overestimation after 3A processing, resulting in serious voice damage. In addition, the long-distance pickup microphone also causes serious damage to high frequencies. To improve the audio quality in the real-time audio collection and transmission scenario, the above multiple damages need to be repaired to ensure the clarity, restoration and intelligibility of the final playback voice. However, the existing model structure for repairing the above multiple damages is complex, and it takes a long time to repair the audio to the ideal quality, which cannot meet the speed requirement of audio processing in real-time application scenarios. SUMMARY

[0003] The present application provides an audio collection real-time repair method, device, electronic equipment and storage medium to solve the technical problem that the existing technology takes a long time to repair the audio to the ideal quality, which cannot meet the speed requirement of audio processing in real-time application scenarios.

[0004] In a first aspect, the embodiments of the present application provide an audio collection real-time repair method, comprising:

[0005] obtaining real-time audio data generated by audio collection;

[0006] inputting the real-time audio data into a pre-trained audio quality repair model, the input of the audio quality repair model being a PQMF subband, the audio quality repair model being trained based on a generative adversarial network, the generative adversarial network comprising a generator and a discriminator, the generator comprising an encoding module and a decoding module, the encoding module comprising a causal deep separable convolution module and a grouped LSTM module, the decoding module comprising a mask path and a mapping path both using a causal deep separable transposed convolution, and the output complex spectrum of the mask path and the mapping path being feature fused with the PQMF subband;

[0007] outputting the output of the audio quality repair model corresponding to the real-time audio data as real-time repaired audio.

[0008] In a second aspect, the embodiments of the present application provide an audio collection real-time repair device, comprising:

[0009] an audio data acquisition unit, configured to acquire real-time audio data generated by audio collection;

[0010] an audio data processing unit, configured to input the real-time audio data into a pre-trained sound quality repair model, wherein an input of the sound quality repair model is a PQMF subband, and the sound quality repair model is obtained based on a generative adversarial network, the generative adversarial network comprises a generator and a discriminator, the generator comprises an encoding module and a decoding module, the encoding module comprises a causal deep separable convolution module and a grouped LSTM module, the decoding module comprises a mask path and a mapping path both using a causal deep separable transposed convolution, and an output complex spectrum of the mask path and the mapping path is subjected to feature fusion with the PQMF subband;

[0011] a repair data output unit, configured to output, as real-time repair audio, an output of the sound quality repair model corresponding to the real-time audio data.

[0012] In a third aspect, the embodiments of the present application provide an electronic device, comprising:

[0013] one or more processors;

[0014] a memory, configured to store one or more computer programs;

[0015] When the one or more computer programs are executed by the one or more processors, the electronic device implements the audio collection real-time repair method according to the first aspect.

[0016] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the audio collection real-time repair method according to the first aspect.

[0017] In the aforementioned real-time audio acquisition and restoration method, device, electronic device, and storage medium, real-time audio data generated during audio acquisition is acquired. This real-time audio data is then input into a pre-trained audio quality restoration model. The input to the audio quality restoration model is a PQMF molecular band. The audio quality restoration model is trained based on a generative adversarial network (GAN), which includes a generator and a discriminator. The generator includes an encoding module and a decoding module. The encoding module consists of a causal depthwise separable convolutional module and a grouped LSTM module. The decoding module includes a mask path and a mapping path, both using causal depthwise separable transposed convolutions. The complex spectra of the outputs of the mask path and the mapping path are fused with the features of the PQMF molecular band. The output of the audio quality restoration model corresponding to the real-time audio data is used as the real-time restored audio. Addressing the audio quality issues after processing caused by real-time performance, high reverberation, low signal-to-noise ratio, low signal-to-back ratio, and long-distance pickup in real-time audio transmission scenarios, a dual-path GAN model is adopted. The input data is processed using PQMF molecular bands, thereby reducing the computational load of the model and shortening the time required to restore the audio to the ideal quality, meeting the speed requirements of audio processing in real-time application scenarios. Attached Figure Description

[0018] Figure 1 A flowchart illustrating the real-time audio acquisition and repair method provided in this embodiment of the application.

[0019] Figure 2 This is a schematic diagram illustrating the working principle of the audio restoration model provided in the embodiments of this application.

[0020] Figure 3 A schematic diagram of the specific structure of the generator for the audio restoration model provided in this application embodiment.

[0021] Figure 4 This is a schematic diagram of the spectrum of lossy audio.

[0022] Figure 5 The audio repair method provided in the embodiments of this application is for Figure 4 The image shows a spectrum diagram of the audio after restoration.

[0023] Figure 6 This is a schematic diagram of the structure of the real-time audio acquisition and repair device provided in the embodiments of this application.

[0024] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] The application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are intended to explain the application, but not to limit the application. In addition, it should be noted that only the parts related to the application are shown in the drawings for ease of description, rather than all the structures.

[0026] It should be noted that the present application does not exhaust all optional embodiments, and those skilled in the art should be able to think of any combination of technical features as long as the technical features do not contradict each other, which can constitute an optional embodiment.

[0027] At present, the quality of sound caused by microphone collection, 3A signal processing (acoustic echo cancellation (AEC), background noise suppression (ANS), automatic gain control (AGC), the three audio algorithms in combination) in the real-time audio collection transmission scene, the coding and packet loss of conference software can also cause further damage to the voice, especially in the case of high reverberation, low signal-to-noise ratio and low signal-to-reverberation ratio, all manufacturers will add noise reduction and echo cancellation based on NN (neural network). Due to the frequency aliasing of voice and noise, reverberation in the case of high reverberation, the noise reduction and echo cancellation model is prone to overestimation after 3A processing, resulting in serious voice damage. In addition, the long-distance pickup microphone also causes serious damage to high frequencies. The existing model structure for repairing the above multiple damages is complex, and it takes a long time to repair the audio to the ideal quality, which cannot meet the speed requirements of audio processing in real-time application scenarios.

[0028] The embodiments of the present application aim at the quality of sound after audio processing caused by real-time, high reverberation, low signal-to-noise ratio, low signal-to-reverberation ratio, long-distance pickup and the like in the real-time audio transmission scene. A dual-path generative adversarial network model is adopted, and PQMF molecular band processing is used for data input, thereby reducing the computational load of the model, shortening the time required for repairing the audio to the ideal quality, and meeting the speed requirements of audio processing in real-time application scenarios.

[0029] The audio collection real-time repair method in the embodiments of the present application can be applied to various types of node devices for real-time audio collection and transmission, such as microphones, terminal devices, network servers, etc. For example, the audio collection real-time repair method is implemented in a conference terminal to repair the quality of sound and transmit the audio to the conference software; for another example, the audio quality repair model integrated in the conference software repairs the real-time audio data after the conference software receives the real-time audio data transmitted through the network, and plays it to the user. No matter which link node device is used for processing, the processing speed can be effectively improved while ensuring the repair quality through the embodiments of the present application.

[0030] The embodiments of the present application will be described in detail below.

[0031] Step S110: Obtain real-time audio data generated by audio collection.

[0032] The real-time audio data generated by audio collection is completed by the microphone, and subsequent repair can be completed by any node device in a scene requiring real-time audio collection and transmission, such as the microphone directly obtaining the real-time audio data generated by itself, at which time at least 3A processing has been completed; for example, the terminal device obtains the real-time audio data generated by the microphone for audio collection, or the real-time audio data sent by another terminal device, at which time there may be a loss caused by transmission in addition to 3A processing; for example, the network server obtains the real-time audio data sent by the terminal device, at which time there may also be a loss caused by transmission in addition to 3A processing.

[0033] Step S120: Input the real-time audio data into the pre-trained sound quality repair model, the input of the sound quality repair model is the PQMF subband, the sound quality repair model is trained based on a generative adversarial network, the generative adversarial network includes a generator and a discriminator, the generator includes an encoding module and a decoding module, the encoding module includes a causal deep separable convolution module and a grouped LSTM module, the decoding module includes a mask path and a mapping path both using a causal deep separable transposed convolution, and the output complex spectrum of the mask path and the mapping path is feature fused with the PQMF subband.

[0034] As can be found from the damage analysis of the real-time audio data in the foregoing, repairing the lossy audio mainly includes two compensations, the first compensation is that due to the limitation of the 3A algorithm on the sampling rate, the sampling rate of the processed audio is only 16khz, and the speech needs to be super-resolved to compensate the high frequency part; the second compensation is that due to the damage to the low frequency characteristics of the speaker's voice mentioned in the previous section. In addition to the above two compensations, there may also be a microphone collection high frequency damage caused by a long distance pickup scene, and an encoding and packet loss damage in the network transmission process of the conference software.

[0035] For the reasons and specific types of damage, the sound quality repair model of the embodiments of the present application is based on a generative adversarial network in a large framework, but the model and training of the generator and the discriminator are specially designed. The sound quality repair model mainly includes three modules of the generator, the discriminator, and the loss function, wherein the generator mainly fits and compensates the damaged audio to generate a repaired model; the discriminator is mainly responsible for judging the difference between the generated audio and the lossless audio, thereby guiding the reverse learning of the generator. The loss function mainly includes a signal-based feature loss and a discriminator judgment loss. The overall design idea of the above processing process is as shown in Figure 2 .

[0036] Specific to the generator, in the embodiments of the present application, the mapping and masking dual-acting model that conforms to the actual use scene is designed according to the analysis of the real-time call scene and the characteristics of the voice to be repaired, and the model has the functions of compensation and masking; the damage of signal processing can be repaired, and the unprocessed noise or reverberation can be suppressed again. The generator is divided into an encoding module and a decoding module, wherein the encoding module is composed of causal deep separable convolution and grouped LSTM, and the use of causal deep separable convolution and grouped LSTM can greatly reduce the computational amount; in addition, the decoding module and the encoding module are symmetrical, and causal deep separable transposed convolution is used, but the decoding module is a double-path model, one of which is a mask path based on complex spectrum, and the other is a mapping path based on complex spectrum; wherein the mask path is used to input the complex spectrum.

[0037] In Figure 3 In the generator shown, the input of the model is processed by using the molecule band of the PQMF filter, the residual error output by the PQMF sub-band and the mask path is multiplied, so as to obtain the processing result after noise reduction, and then the processing result after noise reduction and the output complex spectrum of the mapping path (i.e. the processing result after repair) are spliced in the channel dimension, the spliced input is fused in the form of upsampling, and the fused result is subjected to inverse Fourier transform to obtain a relatively ideal sound output. The specific structure of the entire generator is shown in Figure 3 , wherein Causal_G_Conv represents causal deep separable convolution, Causal_G_transconv represents causal deep separable transposed convolution, and GLSTM represents grouped LSTM.

[0038] The generator of the embodiments of the present application can reduce the frequency dimension on the frame, indirectly reduce the downsampling multiple, and reduce the computational amount caused by downsampling; the output of the model is designed as a learnable PQMF filter for the synthesis of the voice, and the learnable parameter PQMF has better robustness than the fixed filter, preventing the boundary between sub-bands.

[0039] In the discriminator proposed in the embodiments of the present application, there are a multi-resolution time domain discriminator and an STFT discriminator with different Fourier transforms, and these discriminators jointly guide the learning of the generator to improve the granularity of voice learning. The experimental results show that the audio generated based on the present scheme has a more natural and full listening experience.

[0040] The loss function of the sound quality repair model proposed in the embodiments of the present application includes an error distance loss function based on signal characteristics and a discrimination loss function of the discriminator, and these loss functions jointly guide the update of the model parameters.

[0041] The adversarial loss provided by the discriminator to the generator:

[0042]

[0043] wherein k represents the number of discriminators, T represents the length of the discriminator output, Dk,t(x) represents the output of the target audio through the discriminator, and Dk,t(G(x) represents the discriminator output of the generated audio.

[0044] The feature loss provided by the discriminator to the generator is:

[0045]

[0046] wherein k represents the number of discriminators, and L represents the feature-map output of each layer.

[0047] The error distance loss function is the error distance loss function of the mel spectrum, and the loss can be specifically represented as:

[0048]

[0049] wherein Sts(x) represents the mel spectrum output of the t frame of the target audio, and Sst(G(x)) represents the mel spectrum output of the t frame of the generated audio.

[0050] The specific training sample of the sound quality restoration model can be a sample collected manually or a sample provided by a public data set. In the embodiment of the present application, the sound quality restoration model is designed, and the generator therein is used for the general process of model training and use in the macro processing process of sound quality restoration. Therefore, it will not be described in detail.

[0051] Step S130: taking the output of the sound quality restoration model corresponding to the real-time audio data as the real-time restored audio.

[0052] In a specific use of real-time audio, the audio data obtained after audio restoration by the sound quality restoration model provided in the embodiment of the present application can be used for subsequent processing until it is played to the user in a certain terminal device, and in a complete collection and transmission process, it can be processed in multiple node devices to restore the possible loss of each link and ensure the quality of the final output audio. Specifically, it can be compared with Figure 4 and Figure 5 wherein Figure 4 is a lossy audio, Figure 5 is the audio restored based on the sound quality restoration model in the embodiment of the present application. It can be found that the embodiment of the present application has good compensation effect on low frequency and high frequency, making the voice more natural and full, so as to realize fast and high-quality audio restoration as a whole and meet the sound quality restoration demand in the real-time audio transmission scene.

[0053] The real-time audio data generated by audio collection is acquired, and the real-time audio data is input into a sound quality restoration model obtained by pre-training. The input of the sound quality restoration model is a PQMF subband. The sound quality restoration model is obtained by training a generative adversarial network. The generative adversarial network includes a generator and a discriminator. The generator includes an encoding module and a decoding module. The encoding module includes a causal deep separable convolution module and a grouped LSTM module. The decoding module includes a mask path and a mapping path, both of which use a causal deep separable transposed convolution. The output complex spectrum of the mask path and the mapping path is feature fused with the PQMF subband. The output of the sound quality restoration model corresponding to the real-time audio data is used as real-time restoration audio. In the real-time audio transmission scene, the sound quality of the processed audio is affected by real-time performance, high reverberation, low signal-to-noise ratio, low signal-to-reverberation ratio, and long-distance pickup. A double-path generative adversarial network model is used to process the data input by using a PQMF subband, thereby reducing the computational amount of the model, shortening the time required to restore the audio to the ideal quality, and meeting the speed requirement of audio processing in real-time application scenarios.

[0054] Figure 6 A structural schematic diagram of an audio collection restoration device provided by the embodiment of the application is provided. Referring to Figure 6 The audio collection restoration device includes an audio data acquisition unit 310, an audio data processing unit 320, and a restoration data output unit 330.

[0055] The audio data acquisition unit 310 is configured to acquire real-time audio data generated by audio collection. The audio data processing unit 320 is configured to input the real-time audio data into a sound quality restoration model obtained by pre-training. The input of the sound quality restoration model is a PQMF subband. The sound quality restoration model is obtained by training a generative adversarial network. The generative adversarial network includes a generator and a discriminator. The generator includes an encoding module and a decoding module. The encoding module includes a causal deep separable convolution module and a grouped LSTM module. The decoding module includes a mask path and a mapping path, both of which use a causal deep separable transposed convolution. The output complex spectrum of the mask path and the mapping path is feature fused with the PQMF subband. The restoration data output unit 330 is configured to use the output of the sound quality restoration model corresponding to the real-time audio data as real-time restoration audio.

[0056] On the basis of the above embodiment, the output of the generator is the output of a learnable PWMF filter.

[0057] On the basis of the above embodiment, the discriminator includes a multi-resolution time domain discriminator and an STFT discriminator of different Fourier transforms.

[0058] On the basis of the above-mentioned embodiments, the discriminator comprises a multi-resolution time domain discriminator and an STFT discriminator of different Fourier transforms.

[0059] On the basis of the above-mentioned embodiments, the loss function of the sound quality restoration model comprises an error distance loss function based on signal features and a discrimination loss function of the discriminator.

[0060] On the basis of the above-mentioned embodiments, the error distance loss function is a mel spectrum error distance loss function.

[0061] The audio acquisition real-time restoration device provided by the embodiments of the present application is included in an electronic device and can be used to execute any audio acquisition real-time restoration method provided in the above-mentioned embodiments, and has the corresponding functions and beneficial effects.

[0062] It is worth noting that in the above-mentioned embodiments of the audio acquisition real-time restoration device, each unit and module included is only divided according to the function logic, but is not limited to the above-mentioned division, as long as the corresponding functions can be implemented; in addition, the specific name of each functional unit is only for easy mutual distinction, and does not limit the protection scope of the present application.

[0063] Figure 7 A structural schematic diagram of an electronic device provided by the embodiments of the present application is shown in FIG. 1. As shown in the figure, the electronic device comprises a processor 410 and a memory 420, and can further comprise an input device 430, an output device 440 and a communication device 450; the number of processors 410 in the electronic device can be one or more, and one processor 410 is taken as an example in the figure; the processor 410, the memory 420, the input device 430, the output device 440 and the communication device 450 in the electronic device can be connected through a bus or other means, and the connection through the bus is taken as an example in the figure. Figure 7 Figure 7 Figure 7

[0064] The memory 420 is a kind of computer readable storage medium, which can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the audio acquisition real-time restoration method in the embodiments of the present application. The processor 410 executes the software programs, instructions and modules stored in the memory 420, thereby performing various functional applications and data processing of the electronic device, i.e. implementing the above-mentioned audio acquisition real-time restoration method.

[0065] ​​​The memory 420 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs, etc. required by at least one function. The data storage area can store data created by the use of the electronic device, etc. In addition, the memory 420 can include a high-speed random access memory, and can also include a nonvolatile memory such as at least one disk memory device, a flash memory device, or other nonvolatile solid state memory device. In some examples, the memory 420 can further include a memory remotely located with respect to the processor 410, which can be connected to the electronic device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0066] The input device 430 can be used to receive input digital or character information, and to generate key signal input related to user settings and function controls of the electronic device. The output device 440 can include a display device such as a display screen, etc.

[0067] The embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, is configured to perform the related operations in the audio acquisition real-time repair method provided in any of the embodiments of the present application, and has the corresponding functions and beneficial effects.

[0068] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products.

[0069] Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer-usable program code embodied in the medium. The present application is described with reference to flowcharts and / or block diagrams illustrating the architecture, the devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one flow or a plurality of flows and / or blocks Figure 1apparatuses that carry out the specified functions of one or more blocks or a combination of blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the Figure 1 flowchart or flowchart and / or a block diagram Figure 1 flowchart or flowchart and / or a block diagram Figure 1 flowchart or flowchart and / or a block diagram Figure 1 flowchart or flowchart and / or a block diagram

[0070] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. The memory can include non-persistent memory, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash memory, among others, in a computer-readable medium. The memory is an example of computer-readable media.

[0071] Computer-readable media includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD), or other optical storage, magnetic cassette, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media, such as modulated data signals and carriers.

[0072] It is also to be noted that the terms "comprising", "including", and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0073] Note that the above merely describes preferred embodiments of the present application and the applied technical principles. Those skilled in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, reconfigurations, and substitutions can be made without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the appended claims.

Claims

1. An audio acquisition real-time repair method, characterized in that, The method comprises: obtaining real-time audio data generated by audio acquisition; inputting the real-time audio data into a pre-trained audio quality restoration model, wherein an input of the audio quality restoration model is a PQMF subband, the audio quality restoration model is trained based on a generative adversarial network, the generative adversarial network comprises a generator and a discriminator, the generator comprises an encoding module and a decoding module, the encoding module comprises a causal deep separable convolution module and a grouped LSTM module, the decoding module comprises a mask path and a mapping path both using a causal deep separable transposed convolution, and output complex spectra of the mask path and the mapping path are feature fused with the PQMF subband; outputting, by the audio quality restoration model, real-time restoration audio corresponding to the real-time audio data.

2. The audio acquisition real-time repair method of claim 1, wherein, An output of the generator is an output of a learnable PWMF filter.

3. The audio acquisition real-time repair method of claim 1, wherein, The discriminator comprises a multi-resolution time domain discriminator and an STFT discriminator of different Fourier transforms.

4. The audio acquisition real-time repair method of claim 1, wherein, A loss function of the audio quality restoration model comprises an error distance loss function based on a signal feature and a discrimination loss function of the discriminator.

5. The audio acquisition real-time repair method of claim 4, wherein, The error distance loss function is an error distance loss function of a mel spectrum.

6. Audio acquisition real-time repair apparatus, characterized in that, The method comprises: an audio data acquisition unit configured to obtain real-time audio data generated by audio acquisition; an audio data processing unit configured to input the real-time audio data into a pre-trained audio quality restoration model, wherein an input of the audio quality restoration model is a PQMF subband, the audio quality restoration model is trained based on a generative adversarial network, the generative adversarial network comprises a generator and a discriminator, the generator comprises an encoding module and a decoding module, the encoding module comprises a causal deep separable convolution module and a grouped LSTM module, the decoding module comprises a mask path and a mapping path both using a causal deep separable transposed convolution, and output complex spectra of the mask path and the mapping path are feature fused with the PQMF subband; a restoration data output unit configured to output, by the audio quality restoration model, real-time restoration audio corresponding to the real-time audio data.

7. The audio acquisition real-time repair apparatus of claim 6, wherein, An output of the generator is an output of a learnable PWMF filter.

8. The audio acquisition real-time repair apparatus of claim 6, wherein, The discriminator comprises a multi-resolution time domain discriminator and an STFT discriminator of different Fourier transforms.

9. An electronic device, characterized by The method comprises: one or more processors; a memory configured to store one or more computer programs; when the one or more computer programs are executed by the one or more processors, the electronic device implements the audio acquisition real-time restoration method according to any one of claims 1-5.

10. A computer readable storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the audio acquisition real-time restoration method according to any one of claims 1-5.

Images