Audio acquisition real-time restoration method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an audio acquisition real-time restoration method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring real-time audio data generated by audio acquisition; the real-time audio data is input into a tone quality restoration model obtained through pre-training, the input of the tone quality restoration model is a PQMF molecular band, the tone quality restoration model is obtained through training based on a generative adversarial network, the generative adversarial network comprises a generator and a discriminator, the generator comprises an encoding module and a decoding module, and the discriminator comprises an encoding module and a decoding module; the coding module comprises a causal depth separable convolution module and a grouping LSTM module, the decoding module comprises a mask path and a mapping path which both use causal depth separable transpose convolution, and feature fusion is carried out on output complex spectrums of the mask path and the mapping path and a PQMF molecular band; and taking the output of the sound quality restoration model corresponding to the real-time audio data as real-time restoration audio. According to the scheme, the time for repairing the audio to ideal quality can be shortened, and the speed requirement of audio processing in a real-time application scene is met.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio repair technology, and in particular to a method, device, electronic device and storage medium for real-time repair of audio acquisition. Background Art

[0002] In application scenarios such as video conferencing, online communication, audio and video communication, real-time audio acquisition and transmission are performed. In the process of real-time acquisition and transmission, microphone acquisition and 3A signal processing after acquisition will cause sound quality damage. Packet loss in the process of audio encoding, decoding and transmission in the application scenario may also cause further damage to the voice. Especially for high reverberation, low signal-to-noise ratio and low signal-to-echo ratio, all manufacturers will add NN-based noise reduction and echo cancellation. Due to the frequency aliasing of voice, noise and reverberation in the case of high reverberation, after 3A processing, the noise reduction and echo cancellation models are prone to over-estimation, resulting in serious voice damage. In addition, the long-distance pickup microphone also causes serious damage to the high frequency. In order to improve the audio quality in the audio real-time acquisition and transmission scenario, it is necessary to repair the above multiple damages to ensure the clarity, restoration and intelligibility of the final played voice. However, the existing model structure for repairing the above multiple damages is complex, and it takes a long time to repair the audio to the ideal quality, which cannot meet the speed requirements of audio processing in real-time application scenarios. Summary of the invention

[0003] The present invention provides a real-time repair method, device, electronic device and storage medium for audio acquisition to solve the technical problem in the prior art that it takes a long time to repair audio to an ideal quality and cannot meet the speed requirements of audio processing in real-time application scenarios.

[0004] In a first aspect, an embodiment of the present application provides a method for real-time repair of audio acquisition, comprising:

[0005] Acquire real-time audio data generated by audio collection;

[0006] Input the real-time audio data into the pre-trained sound quality restoration model. The input of the sound quality restoration model is the PQMF molecular band. The sound quality restoration model is trained based on a generative adversarial network. The generative adversarial network includes a generator and a discriminator. The generator includes an encoding module and a decoding module. The encoding module includes a causal deep separable convolution module and a grouped LSTM module. The decoding module includes a mask path and a mapping path that both use causal deep separable transposed convolution. The output complex spectra of the mask path and the mapping path are feature fused with the PQMF molecular band.

[0007] The output of the sound quality restoration model corresponding to the real-time audio data is used as the real-time restoration audio.

[0008] In a second aspect, an embodiment of the present application provides an audio acquisition real-time repair device, comprising:

[0009] An audio data acquisition unit, used to acquire real-time audio data generated by audio acquisition;

[0010] An audio data processing unit, used for inputting real-time audio data into a pre-trained sound quality restoration model, the input of the sound quality restoration model is a PQMF molecular band, the sound quality restoration model is trained based on a generative adversarial network, the generative adversarial network includes a generator and a discriminator, the generator includes an encoding module and a decoding module, the encoding module includes a causal deep separable convolution module and a grouped LSTM module, the decoding module includes a mask path and a mapping path both of which use causal deep separable transposed convolution, and the output complex spectra of the mask path and the mapping path are feature fused with the PQMF molecular band;

[0011] The repair data output unit is used to use the output of the sound quality repair model corresponding to the real-time audio data as real-time repair audio.

[0012] In a third aspect, an embodiment of the present application provides an electronic device, the electronic device comprising:

[0013] one or more processors;

[0014] a memory for storing one or more computer programs;

[0015] When one or more computer programs are executed by one or more processors, the electronic device implements the audio acquisition real-time repair method of the first aspect.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the audio acquisition real-time repair method as described in the first aspect.

[0017] In the above-mentioned audio acquisition real-time repair method, device, electronic device and storage medium, real-time audio data generated by audio acquisition is obtained; the real-time audio data is input into the pre-trained sound quality repair model, the input of the sound quality repair model is the PQMF molecular band, the sound quality repair model is obtained based on the generative adversarial network training, the generative adversarial network includes a generator and a discriminator, the generator includes an encoding module and a decoding module, the encoding module includes a causal depth separable convolution module and a grouped LSTM module, the decoding module includes a mask path and a mapping path both using causal depth separable transposed convolution, the output complex spectrum of the mask path and the mapping path is feature fused with the PQMF molecular band; the output of the sound quality repair model corresponding to the real-time audio data is used as the real-time repaired audio. In view of the sound quality problems after audio processing caused by real-time, high reverberation, low signal-to-noise ratio, low signal-to-return ratio, long-distance pickup, etc. in real-time audio transmission scenarios, a dual-path generative adversarial network model is adopted, and the data input is processed by PQMF molecular bands, thereby reducing the calculation amount of the model, shortening the time required for audio repair to the ideal quality, and meeting the speed requirements of audio processing in real-time application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A method flow chart of the real-time repair method for audio acquisition provided in an embodiment of the present application.

[0019] Figure 2 Schematic diagram of the working principle of the audio repair model provided in the embodiment of the present application.

[0020] Figure 3 A schematic diagram of the specific structure of the generator of the audio repair model provided in an embodiment of the present application.

[0021] Figure 4 Schematic diagram of the spectrum of lossy audio.

[0022] Figure 5 The audio repair method provided in the embodiment of the present application is Figure 4 Schematic diagram of the frequency spectrum of the audio after restoration.

[0023] Figure 6 A schematic diagram of the structure of an audio acquisition and real-time repair device provided in an embodiment of the present application.

[0024] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0026] It should be noted that the present application specification does not enumerate all optional implementation modes. After reading the present application specification, those skilled in the art should be able to understand that as long as the technical features do not contradict each other, any combination of the technical features can constitute an optional implementation mode.

[0027] At present, microphone acquisition and 3A signal processing (the collective name of three audio algorithms, acoustic echo cancellation (AEC), background noise suppression (ANS), and automatic gain control (AGC)) in real-time audio acquisition and transmission scenarios all cause sound quality damage. The encoding and decoding and packet loss of conference software can also cause further damage to the voice, especially in the case of high reverberation, low signal-to-noise ratio and low signal-to-echo ratio. At present, various manufacturers will add NN (neural network)-based noise reduction and echo cancellation. Due to the frequency aliasing of voice, noise and reverberation in the case of high reverberation, after 3A processing, the noise reduction and echo cancellation models are prone to over-estimation, resulting in serious voice damage. In addition, the long-distance pickup microphone also causes serious damage to the high frequency. The existing model structure for repairing the above multiple damages is complex, and it takes a long time to repair the audio to the ideal quality, which cannot meet the speed requirements of audio processing in real-time application scenarios.

[0028] The embodiment of the present application adopts a dual-path generative adversarial network model to address the sound quality problems after audio processing caused by real-time, high reverberation, low signal-to-noise ratio, low signal-to-echo ratio, and long-distance sound pickup in real-time audio transmission scenarios. PQMF molecular band processing is used for data input to reduce the computational complexity of the model, shorten the time required to repair the audio to the ideal quality, and meet the speed requirements of audio processing in real-time application scenarios.

[0029] The audio acquisition real-time repair method in the embodiment of the present application can be applied to various types of node devices for real-time audio acquisition and transmission, such as microphones, terminal devices, network servers, etc. For example, the audio acquisition real-time repair method is implemented in the conference terminal, and the sound quality is repaired and given to the conference software for audio transmission; for another example, it is integrated into the conference software. After the conference software receives the real-time audio data transmitted through the network, the sound quality repair model integrated in the conference software repairs the real-time audio data and plays it to the user. No matter which node device is used for processing, the embodiment of the present application can effectively improve the processing speed while ensuring the repair quality.

[0030] The following is a detailed description of the various embodiments of the present invention.

[0031] Step S110: Acquire real-time audio data generated by audio collection.

[0032] The microphone is used to collect audio and generate real-time audio data. Subsequent repair can be completed by any node device in the scenario where real-time audio collection and transmission are required. For example, the microphone directly obtains the real-time audio data generated by itself, and at least 3A processing has been completed at this time. For example, the terminal device obtains the real-time audio data generated by the microphone for audio collection, or the real-time audio data sent by another terminal device. In this case, there may be losses caused by transmission in addition to 3A processing. For another example, the network server obtains the real-time audio data sent by the terminal device. In this case, there may be losses caused by transmission in addition to 3A processing.

[0033] Step S120: input the real-time audio data into the pre-trained sound quality restoration model, the input of the sound quality restoration model is the PQMF molecular band, the sound quality restoration model is trained based on a generative adversarial network, the generative adversarial network includes a generator and a discriminator, the generator includes an encoding module and a decoding module, the encoding module includes a causal deep separable convolution module and a grouped LSTM module, the decoding module includes a mask path and a mapping path both of which use causal deep separable transposed convolution, and the output complex spectra of the mask path and the mapping path are feature fused with the PQMF molecular band.

[0034] As we can see from the damage analysis of real-time audio data in the previous article, there are two main types of compensation for repairing lossy audio. The first type of compensation is due to the sampling rate limitation of the 3A algorithm, which makes the processed audio sampling rate only 16khz, and the voice needs to be super-resolved to compensate for the high-frequency part; the second type is due to the damage to the low-frequency characteristics of the speaker's voice caused by the 3A algorithm mentioned in the previous section. In addition to the above two types of compensation, there may also be high-frequency damage to microphone collection caused by long-distance sound pickup scenarios, as well as coding and packet loss damage during network transmission of conference software.

[0035] In view of the causes and specific types of damage, the sound quality restoration model of the embodiment of the present application is based on a generative adversarial network in a large framework, but the models and training of the generator and discriminator are specially designed. The sound quality restoration model mainly includes three modules: generator, discriminator, and loss function. The generator mainly fits and compensates the lossy audio to obtain a repaired model; the discriminator is mainly responsible for judging the difference between the generated audio and the lossless audio, thereby guiding the reverse learning of the generator. The loss function mainly includes the feature loss based on the signal and the discriminant loss of the discriminator. The overall design idea of ​​the above processing process is as follows. Figure 2 shown.

[0036] Specifically for the generator, in the embodiment of the present application, according to the real-time call scenario and the characteristics of the voice to be repaired, a mapping and masking dual-action model that meets the actual use scenario is designed. The model has both compensation and masking functions; it can repair the damage of signal processing, and it can also perform secondary suppression on the noise or reverberation that has not been processed cleanly. The generator is divided into an encoding module and a decoding module, wherein the encoding module is based on causal deep separable convolution and grouped LSTM. The use of causal deep separable convolution and grouped LSTM can greatly reduce the amount of calculation; in addition, the decoding module is symmetrical with the encoding module, using causal deep separable transposed convolution, but the decoding module is a dual-path model, in which one is a mask path based on the complex spectrum, and the other is a mapping path based on the complex spectrum; wherein the mask path is used to input the complex spectrum.

[0037] exist Figure 3 In the generator shown in the figure, the model input is processed by the molecular band of the PQMF filter, and the residual output of the PQMF subband and the mask path is multiplied to obtain the denoised processing result. Then, the denoised processing result and the output complex spectrum of the mapping path (i.e., the repaired processing result) are spliced ​​in the channel dimension. The spliced ​​input is fused by upsampling the two spectra. The fused result is subjected to inverse Fourier transform to obtain a relatively ideal sound output. The specific structure of the entire generator is shown in the figure. Figure 3 As shown, Causal_G_Conv represents causal depthwise separable convolution, Causal_G_transconv represents causal depthwise separable transposed convolution, and GLSTM represents grouped LSTM.

[0038] The generator of the embodiment of the present application can reduce the frequency dimension on the frame, indirectly reduce the downsampling multiple, and reduce the amount of calculation caused by downsampling; the output of the model is designed as a learnable PQMF filter for speech synthesis. The PQMF with learnable parameters has better robustness than the fixed filter and prevents the boundaries between sub-bands.

[0039] The discriminators proposed in the embodiments of the present application include a multi-resolution time domain discriminator and an STFT discriminator with different Fourier transforms. These discriminators jointly guide the learning of the generator and improve the granularity of speech learning. Experimental results show that the audio generated based on this solution sounds more natural and full.

[0040] The loss function of the sound quality restoration model proposed in the embodiment of the present application includes an error distance loss function based on signal features and a discriminant loss function of the discriminator. These loss functions jointly guide the updating of model parameters.

[0041] The adversarial loss provided by the discriminator to the generator is:

[0042]

[0043] Where k represents the number of discriminators, T represents the discriminator output length, Dk,t(x) represents the output of the target audio after the discriminator, and Dk,t(G(x) represents the discriminator output of the generated audio.

[0044] The feature loss provided by the discriminator to the generator is:

[0045]

[0046] Among them, k represents the number of discriminators, and L represents the feature-map output of each layer.

[0047] The error distance loss function is the error distance loss function of the Mel spectrum, which can be specifically expressed as:

[0048]

[0049] Among them, Sts(x) represents the Mel spectrum output of the target audio frame t, and Sst(G(x)) represents the Mel spectrum output of the tth frame of the generated audio.

[0050] The specific training samples of the sound quality restoration model can be samples collected manually or samples provided by a public data set. On the basis of the sound quality restoration model designed in the embodiment of the present application, training is performed through samples, and the macro-processing process of using the generator therein for sound quality restoration is the general process of model training and use, which will not be elaborated in depth here.

[0051] Step S130: using the output of the sound quality restoration model corresponding to the real-time audio data as the real-time restoration audio.

[0052] In a specific scenario using real-time audio, the audio data obtained after audio repair by the sound quality repair model provided in the embodiment of this application can be used for subsequent processing until it is played to the user in a terminal device. In addition, in a complete acquisition and transmission process, it can be processed in multiple node devices to repair possible losses in each link to ensure the final output audio quality. Figure 4 and Figure 5 ,in Figure 4 is lossy audio, Figure 5 This is the audio repaired based on the sound quality restoration model in the embodiment of the present application. It can be found that the embodiment of the present application has a good compensation effect on both low and high frequencies, making the voice more natural and full, thereby achieving fast and high-quality audio restoration as a whole, and meeting the sound quality restoration needs in real-time audio transmission scenarios.

[0053] In the above, real-time audio data generated by audio acquisition is obtained; the real-time audio data is input into the sound quality restoration model obtained by pre-training, the input of the sound quality restoration model is the PQMF molecular band, the sound quality restoration model is obtained based on the generative adversarial network training, the generative adversarial network includes a generator and a discriminator, the generator includes an encoding module and a decoding module, the encoding module includes a causal deep separable convolution module and a grouped LSTM module, the decoding module includes a mask path and a mapping path both using causal deep separable transposed convolution, the output complex spectrum of the mask path and the mapping path is feature fused with the PQMF molecular band; the output of the sound quality restoration model corresponding to the real-time audio data is used as the real-time restoration audio. In view of the sound quality problems after audio processing caused by real-time, high reverberation, low signal-to-noise ratio, low signal-to-return ratio, and long-distance pickup in real-time audio transmission scenarios, a dual-path generative adversarial network model is adopted, and the data input is processed by PQMF molecular bands, thereby reducing the calculation amount of the model, shortening the time required to restore the audio to the ideal quality, and meeting the speed requirements of audio processing in real-time application scenarios.

[0054] Figure 6 This is a schematic diagram of the structure of the audio acquisition and repair device provided in the embodiment of the present application. Figure 6 The audio acquisition and repair device includes an audio data acquisition unit 310, an audio data processing unit 320 and a repair data output unit 330.

[0055] Among them, the audio data acquisition unit 310 is used to acquire real-time audio data generated by audio collection; the audio data processing unit 320 is used to input the real-time audio data into the pre-trained sound quality restoration model, the input of the sound quality restoration model is the PQMF molecular band, the sound quality restoration model is trained based on the generative adversarial network, the generative adversarial network includes a generator and a discriminator, the generator includes an encoding module and a decoding module, the encoding module includes a causal deep separable convolution module and a grouped LSTM module, the decoding module includes a mask path and a mapping path both using causal deep separable transposed convolution, the output complex spectra of the mask path and the mapping path are feature fused with the PQMF molecular band; the repair data output unit 330 is used to use the output of the sound quality restoration model corresponding to the real-time audio data as real-time repaired audio.

[0056] Based on the above embodiment, the output of the generator is the output of a learnable PWMF filter.

[0057] On the basis of the above embodiment, the discriminator includes a multi-resolution time domain discriminator and a STFT discriminator with different Fourier transforms.

[0058] On the basis of the above embodiment, the discriminator includes a multi-resolution time domain discriminator and a STFT discriminator with different Fourier transforms.

[0059] On the basis of the above embodiment, the loss function of the sound quality restoration model includes an error distance loss function based on signal features and a discriminant loss function of the discriminator.

[0060] Based on the above embodiment, the error distance loss function is an error distance loss function of the Mel spectrum.

[0061] The audio acquisition real-time repair device provided in the embodiment of the present application is included in an electronic device and can be used to execute any audio acquisition real-time repair method provided in the above embodiments, and has corresponding functions and beneficial effects.

[0062] It is worth noting that in the embodiment of the above-mentioned audio acquisition and real-time restoration device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0063] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 7 As shown, the electronic device includes a processor 410 and a memory 420, and may also include an input device 430, an output device 440, and a communication device 450; the number of processors 410 in the electronic device may be one or more, Figure 7 A processor 410 is taken as an example; the processor 410, the memory 420, the input device 430, the output device 440 and the communication device 450 in the electronic device can be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0064] The memory 420, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the audio acquisition real-time repair method in the embodiment of the present application. The processor 410 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 420, that is, realizing the above-mentioned audio acquisition real-time repair method.

[0065] The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include a memory remotely arranged relative to the processor 410, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0066] The input device 430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 440 may include a display device such as a display screen.

[0067] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is used to perform relevant operations in the audio acquisition real-time repair method provided in any embodiment of the present application, and has corresponding functions and beneficial effects.

[0068] Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0069] Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0070] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0071] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0072] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0073] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A real-time repair method for audio acquisition, characterized in that: include: Acquire real-time audio data generated by audio collection; Input the real-time audio data into a pre-trained sound quality restoration model, wherein the input of the sound quality restoration model is a PQMF molecular band, and the sound quality restoration model is obtained based on generative adversarial network training, wherein the generative adversarial network includes a generator and a discriminator, wherein the generator includes an encoding module and a decoding module, wherein the encoding module includes a causal depth-separable convolution module and a grouped LSTM module, and the decoding module includes a mask path and a mapping path both of which use causal depth-separable transposed convolution, and the output complex spectra of the mask path and the mapping path are feature fused with the PQMF molecular band; The output of the sound quality restoration model corresponding to the real-time audio data is used as real-time restoration audio.

2. The audio acquisition real-time repair method according to claim 1, characterized in that: The output of the generator is the output of the learnable PWMF filter.

3. The audio acquisition real-time repair method according to claim 1, characterized in that: The discriminator includes a multi-resolution time domain discriminator and a STFT discriminator with different Fourier transforms.

4. The audio acquisition real-time repair method according to claim 1, characterized in that: The loss function of the sound quality restoration model includes the error distance loss function based on signal features and the discriminant loss function of the discriminator.

5. The audio acquisition real-time repair method according to claim 4, characterized in that: The error distance loss function is an error distance loss function of a Mel spectrum.

6. The audio acquisition real-time repair device is characterized by: include: An audio data acquisition unit, used to acquire real-time audio data generated by audio acquisition; An audio data processing unit, used for inputting the real-time audio data into a pre-trained sound quality restoration model, wherein the input of the sound quality restoration model is a PQMF molecular band, and the sound quality restoration model is obtained by training based on a generative adversarial network, wherein the generative adversarial network includes a generator and a discriminator, wherein the generator includes an encoding module and a decoding module, wherein the encoding module includes a causal depth-separable convolution module and a grouped LSTM module, and wherein the decoding module includes a mask path and a mapping path both of which use causal depth-separable transposed convolution, and wherein the output complex spectra of the mask path and the mapping path are feature-fused with the PQMF molecular band; The repair data output unit is used to use the output of the sound quality repair model corresponding to the real-time audio data as real-time repair audio.

7. The audio acquisition real-time repair device according to claim 6, characterized in that: The output of the generator is the output of the learnable PWMF filter.

8. The audio acquisition real-time repair device according to claim 6, characterized in that: The discriminator includes a multi-resolution time domain discriminator and a STFT discriminator with different Fourier transforms.

9. An electronic device, characterized in that include: one or more processors; a memory for storing one or more computer programs; When the one or more computer programs are executed by the one or more processors, the electronic device implements the audio acquisition real-time repair method as described in any one of claims 1-5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the real-time audio acquisition repair method as described in any one of claims 1 to 5 is implemented.

Images