Audio data processing method and apparatus, electronic device, storage medium, and program product

Abstract

Embodiments of the present disclosure provide an audio data processing method and device, electronic equipment, storage medium and program product. The method comprises: obtaining first audio data, determining an audio processing model corresponding to the first audio data; inputting the first audio data into a harmonic generation module to generate first harmonic features for the first audio data, obtaining first audio features; inputting the first audio features into a harmonic processing module to generate second harmonic features for the first audio features and perform filtering processing, obtaining second audio features; inputting the second audio features into an audio reconstruction module to reconstruct the second audio features, obtaining second audio data. The technical solution provided by the embodiments of the present disclosure realizes harmonic generation, harmonic processing and audio reconstruction of the audio data to be processed by the audio processing model, so as to generate high-quality audio data with less artifacts.

Description

Technical Field

[0001] This disclosure relates to the field of computer processing technology, and in particular to an audio data processing method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In the field of digital audio processing, artifacts refer to unnatural distortions or interference introduced into the signal during acquisition, encoding, transmission, or processing. These artifacts severely affect audio fidelity and user experience.

[0003] The methods for suppressing artifacts in audio data in related technologies are difficult to effectively eliminate spectral aliasing and frequency blurring in sound wave data, resulting in poor audio data processing quality. Summary of the Invention

[0004] This disclosure provides an audio data processing method, apparatus, electronic device, storage medium, and program product to achieve the effect of generating, processing, and reconstructing harmonics in the audio data to be processed through an audio processing model, thereby generating high-quality audio data with fewer artifacts.

[0005] In a first aspect, embodiments of this disclosure provide an audio data processing method, the method comprising:

[0006] Acquire first audio data and determine the audio processing model corresponding to the first audio data; wherein, the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module;

[0007] The first audio data is input into the harmonic generation module to generate a first harmonic feature for the first audio data, thereby obtaining the first audio feature;

[0008] The first audio feature is input into the harmonic processing module to generate a second harmonic feature based on the first audio feature and perform filtering processing to obtain the second audio feature;

[0009] The second audio feature is input into the audio reconstruction module to reconstruct the second audio feature and obtain the second audio data.

[0010] Secondly, embodiments of this disclosure also provide an audio data processing apparatus, the apparatus comprising:

[0011] An audio data acquisition module is used to acquire first audio data and determine an audio processing model corresponding to the first audio data; wherein, the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module;

[0012] An audio harmonic generation module is used to input the first audio data into the harmonic generation module to generate a first harmonic feature for the first audio data, thereby obtaining the first audio feature;

[0013] An audio harmonic processing module is used to input the first audio feature into the harmonic processing module to generate a second harmonic feature for the first audio feature and perform filtering processing to obtain the second audio feature;

[0014] An audio data generation module is used to input the second audio feature into the audio reconstruction module to reconstruct the second audio feature and obtain the second audio data.

[0015] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0016] One or more processors;

[0017] Storage device for storing one or more programs.

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio data processing method as described in any of the embodiments of this disclosure.

[0019] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the audio data processing method as described in any of the embodiments of this disclosure.

[0020] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the audio data processing method as described in any of the embodiments of this disclosure.

[0021] The technical solution of this disclosure, by acquiring first audio data and determining the audio processing model corresponding to the first audio data, provides a data foundation and processing flow framework for subsequent audio data processing. Since the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module, it achieves the effect of applying a three-stage architecture of harmonic generation-harmonic processing-audio reconstruction to process the first audio data, laying a foundation for improving the quality of audio data from an overall architectural perspective. Furthermore, the first audio data is input into the harmonic generation module to generate first harmonic features based on the first audio data, obtaining first audio features. This achieves the effect of feature extraction and feature transformation of audio data by generating first harmonic features based on the first audio data through the harmonic generation module. By generating first harmonic features and determining first audio features based on the first harmonic features, the possibility of spectral aliasing is effectively reduced, providing better quality input data for the subsequent generation of high-quality audio data. Furthermore, the first audio features are input into the harmonic processing module... In this block, a second harmonic feature is generated based on the first audio feature and filtered to obtain the second audio feature. This achieves further refinement and optimization of the audio feature through the harmonic processing module. Simultaneously, the filtering process removes possible noise or invalid frequency components from the audio feature, making the audio feature clearer and more accurate, thereby improving the quality of the audio data and providing high-quality feature data for subsequent audio reconstruction. Furthermore, the second audio feature is input into the audio reconstruction module to reconstruct the second audio feature and obtain the second audio data. This solves the problem of spectral aliasing and spectral blurring in related technologies, which leads to poor quality of the generated audio data. It achieves the effect of generating, processing, and reconstructing the audio data to be processed through the audio processing model, thereby generating high-quality audio data with fewer artifacts. It significantly improves the fidelity and realism of the final generated audio data while effectively suppressing spectral aliasing and frequency blurring. Attached Figure Description

[0022] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0023] Figure 1 A schematic flowchart illustrating an audio data processing method provided in an embodiment of this disclosure;

[0024] Figure 2 A schematic diagram of the structure of an audio reconstruction module for an audio data processing method according to an embodiment of the present disclosure;

[0025] Figure 3 This is a schematic diagram comparing the effect of audio data processed by the audio data processing method of this disclosure with the expected output audio data and audio data processed by related methods, provided as an embodiment of this disclosure.

[0026] Figure 4 A flowchart illustrating another audio data processing method provided in an embodiment of this disclosure;

[0027] Figure 5 A schematic diagram of the structure of a harmonic generation module for an audio data processing method according to an embodiment of the present disclosure;

[0028] Figure 6 A flowchart illustrating another audio data processing method provided in an embodiment of this disclosure;

[0029] Figure 7 A schematic diagram of the structure of a harmonic processing module for an audio data processing method according to an embodiment of the present disclosure;

[0030] Figure 8 A schematic diagram of the structure of an audio processing model for an audio data processing method provided in an embodiment of the present disclosure;

[0031] Figure 9 This is a schematic diagram of the structure of an audio data processing apparatus provided in an embodiment of the present disclosure;

[0032] Figure 10 This is a schematic diagram of the structure of an electronic device for implementing an embodiment of the present disclosure. Detailed Implementation

[0033] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0034] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0035] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0036] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0037] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0038] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0039] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0040] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0041] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0042] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0043] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0044] Figure 1 This is a flowchart illustrating an audio data processing method provided in an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to scenarios where audio data is processed to obtain high-quality audio data. The method can be executed by an audio data processing device, which can be implemented in the form of software and / or hardware, or optionally by an electronic device, such as a mobile terminal, a PC, or a server.

[0045] like Figure 1 As shown, the method in this embodiment may specifically include:

[0046] S110. Obtain first audio data and determine the audio processing model corresponding to the first audio data; wherein, the audio processing model includes a harmonic generation module, a harmonic processing module and an audio reconstruction module.

[0047] The first audio data can be an audio signal to be processed further. The first audio data can be data characterizing the acoustic features of the audio signal. This data can be represented in any audio signal representation format, optionally including audio waveforms, Mel frequency diagrams, linear spectrum diagrams, Mel-spectral cepstral coefficients, or chromaticity diagrams. The first audio data can be default template audio data, audio data acquired by an audio acquisition device on a terminal device, audio data retrieved from a target storage space (such as an application software audio library, or a local terminal folder) in response to a user's trigger operation, or audio data received from an external device. The audio processing model can be a deep learning model used to process the audio data to obtain audio data that meets the audio processing requirements. The audio processing model can correspond to the audio processing requirements of the first audio data; that is, the first audio data can be processed using the audio processing model to obtain second audio data that meets the audio processing requirements of the first audio data. In this embodiment, the audio processing model can be a deep learning model composed of high-fidelity neural vocoders. This deep learning model can effectively suppress the generation of artifacts in audio data while maintaining high waveform fidelity. The audio processing model can consist of at least one audio processing module. Optionally, at least one audio processing module includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module. The harmonic generation module can be a deep learning model for generating harmonic features. The harmonic processing module can be a deep learning model for processing harmonic features. The audio reconstruction module can be a deep learning model for reconstructing audio features to obtain the final output audio data. In this embodiment, the harmonic generation module, harmonic processing module, and audio reconstruction module in the audio processing model can be connected in series.

[0048] In practical applications, for speech generation scenarios, such as generating speech information from text or speech conversion, deep learning models can be used to process audio data and convert it into audio data that meets the requirements of speech generation. However, although deep learning models can generate high-quality speech signals, there are still cases where high-quality, artifact-free speech signals cannot be generated. Artifacts can be mainly attributed to two aspects: spectral aliasing caused by poor upsampling results; and spectral blurring caused by the loss of spectral details.

[0049] In response to the above situation, this embodiment of the present disclosure employs an audio processing model composed of a filter generation module, a harmonic processing module, and an audio reconstruction module. This audio processing model can generate audio signals with fewer artifacts. Furthermore, the filter in the harmonic processing module can be used to process the artifacts generated in the audio signal. Thus, when processing the first audio data based on the audio processing model, abnormal artifacts in the audio signal can be effectively eliminated, generating higher quality audio data.

[0050] It should be noted that audio generation tasks can typically be performed using deep learning models based on generative adversarial networks (GANs). Compared to other deep learning models, GAN-based models can generate high-quality audio signals at a faster speed. Therefore, in this embodiment, the audio processing model is a generative model trained using a GAN; during training, the model parameters are adjusted based on the audio generation loss.

[0051] The audio processing model can be the generative model within a trained Generative Adversarial Network (GAN). During GAN training, the generative and discriminative models can be trained alternately to obtain the fully trained GAN. During training, the model parameters of the generative model can be adjusted based on the audio generation loss. The audio generation loss can be a composite loss function consisting of multiple losses. Optionally, the audio generation loss includes the discriminative loss of the discriminative model in the GAN regarding the output audio data of the generative model, the feature processing loss of the multiple discriminative layers of the discriminative model regarding the audio data output by the generative model and the expected output data, and the feature processing loss between the input audio data and the output audio data of the generative model.

[0052] In generative adversarial networks (GANs), the discriminative loss of the discriminative model on the output audio data of the generative model can be used to evaluate the degree of difference between the synthesized audio output by the generative model and the real audio. The discriminative loss can be any form of loss function, including, for example, the basic discriminative loss, the least squares GAN loss, or the least squares SAN loss. The output audio data is the audio data generated by the generative model based on the data input to the model.

[0053] Generally, a discriminative model can include multiple discriminative layers. During training, the audio data output by the generator model and the expected output data are input into the discriminative model. Feature processing can be performed sequentially on the audio data and expected output data based on the multiple discriminative layers included in the model, resulting in a first audio feature corresponding to the audio data and a second audio feature corresponding to the expected output data output by each discriminative layer. Further, loss processing can be applied to the first and second audio features output by each discriminative layer, and the losses for each layer can be summed. This summed loss can then be used as the first feature processing loss. Adding the first feature processing loss to the audio generation loss can avoid over-reliance on the original waveform error and enhance the auditory sensory quality. The expected output data can be audio data labels used for model training, which, together with the input audio data of the generator model, constitute the training samples. The input audio data can be the basis for generating audio data based on the generator model. The input audio data is the audio data to be processed. For example, the input audio data can be a Mel spectrogram.

[0054] The second feature processing loss is used to quantify the difference between the input and output audio data, ensuring that the output audio data generated by the generative model is consistent with the expected output data in both the time and frequency domains. During training, the input audio data is fed into the generative model, which then processes the input audio data to obtain the output audio data. Furthermore, loss processing can be applied to both the input audio data and its corresponding output audio data, and the resulting loss can be used as the second feature processing loss.

[0055] In this embodiment of the disclosure, the audio generation loss can be obtained by processing the discrimination loss, the first feature processing loss, and the second feature processing loss through summation or weighted summation.

[0056] In practical implementation, for the training process of generative adversarial networks (GANs), when training the generative model, the input audio data from the training samples can be input into the generative model to obtain the output audio data. Further, loss processing can be applied to the input and output audio data to obtain a second feature loss. Then, the output audio data from the generative model and the expected output data from the training samples are input into the discriminative model. Based on multiple discriminative layers in the discriminative model, feature processing is performed on the output audio data and the expected output data to obtain a first audio feature corresponding to the audio data and a second audio feature corresponding to the expected output data output by each discriminative layer. Further, loss processing can be applied to the first and second audio features output by each discriminative layer separately, and the losses corresponding to each discriminative layer can be summed to obtain a first feature processing loss. Finally, loss processing can be applied to the output audio data and the expected output data based on the loss function deployed in the discriminative model to obtain the discriminative loss. Furthermore, a first weight corresponding to the discriminative loss, a second weight corresponding to the first feature processing loss, and a third weight corresponding to the second feature processing loss can be determined separately. Then, the discriminative loss is multiplied by the first weight, the first feature processing loss is multiplied by the second weight, and the second feature processing loss is multiplied by the third weight. The three products are then summed to obtain the audio generation loss. Further, the model parameters of the generative model can be adjusted based on the audio generation loss, thus obtaining the finally trained generative model, which can then be used as the audio processing model.

[0057] In specific implementation, first audio data to be processed can be obtained, which may be a Mel spectrogram corresponding to the corresponding audio. Further, in order to convert the Mel spectrogram into a speech waveform, an audio processing model capable of audio conversion can be determined, and this audio processing model is used as the audio processing model corresponding to the first audio data. Then, the first audio data can be processed based on the filtering generation module, harmonic processing module, and audio reconstruction module included in the audio processing model to obtain audio data that meets the audio processing requirements.

[0058] S120. Input the first audio data into the harmonic generation module to generate the first harmonic feature for the first audio data, and obtain the first audio feature.

[0059] The harmonic generation module can be used to generate a first harmonic feature based on the first audio data, and to determine the first audio feature based on the first harmonic feature. The harmonic generation module can be used to upsample the first audio data to obtain the desired first audio feature. The first harmonic feature can be a parameterized representation of the first audio data consisting of the fundamental frequency and its integer multiples. The first harmonic feature can include parameterized features such as the fundamental frequency, harmonic energy distribution, and phase. The first harmonic feature can be represented in vector or matrix form. The first audio feature can be the result of fusing the first harmonic feature with other features.

[0060] In practical implementation, given a defined audio processing model, the first audio data can be input into the audio processing model. Furthermore, the first audio data can be processed using a harmonic generation module to generate first harmonic features based on the first audio data, and then the first audio features can be determined based on these first harmonic features. Consequently, the first audio features output by the harmonic generation module can be obtained.

[0061] It should be noted that, in order to help the audio processing model better extract the intrinsic features of the first audio data, before upsampling the first audio data, it can first be mapped to a higher-dimensional latent space to obtain a feature vector. Then, the feature vector is upsampled to obtain the first audio feature. Therefore, the harmonic generation module can consist of at least two sub-modules; one sub-module is a frequency mapping sub-module, used to map the first audio data to a higher-dimensional latent space; the other sub-module can be a harmonic generation sub-module, used to generate the first harmonic feature and obtain the first audio feature.

[0062] S130. Input the first audio feature into the harmonic processing module to generate a second harmonic feature based on the first audio feature and perform filtering processing to obtain the second audio feature.

[0063] The harmonic processing module can be a deep learning model used to generate harmonic features and perform harmonic processing. In this embodiment, the harmonic processing module includes a filter, the filter weights of which are trained based on sample audio data. A filter is a signal processing tool used to selectively enhance or suppress specific frequency components. Filter weights are parameters used in the filter to weight the input signal, characterizing the enhancement or suppression intensity of different frequency components. Generally, the larger the weight value of the filter weight, the more significant the enhancement of the corresponding frequency component; when the weight value of the filter weight is close to zero, the corresponding frequency component is suppressed (e.g., in noise reduction). The filter weights trained based on sample audio data can be used to process artifacts in audio signals. It should be noted that the filter weights can be trained based on sample audio data, meaning that during the training process, the suppression or enhancement intensity of the filter for different frequency components is dynamically adjusted using a backpropagation algorithm or a genetic algorithm to adapt it to the specific filtering task. The dynamically adjusted suppression or enhancement intensity can be represented as filter weights.

[0064] In practical applications, large-scale time-domain vocoders are typically used for audio processing to convert audio data into audio waveforms. Large-scale time-domain vocoders incorporate upsampling-downsampling operations with low-pass filters, which can suppress aliasing and high-frequency artifacts. However, this model framework may suffer from complex structure, high computational complexity, and redundancy.

[0065] To address the above issues, the upsampling-downsampling operation with a low-pass filter is removed from the harmonic processing module provided in this embodiment. This removal is due to the introduction of a filtering sub-unit with better artifact suppression capabilities, which includes a filter. The filtering sub-unit introduced in the harmonic processing module can dynamically adapt to the noise characteristics of different frequency bands through filter weights, making it more flexible than a low-pass filter. Furthermore, compared to a large-scale time-domain vocoder, the filtering processing module provided in this embodiment has a simpler model structure, reducing the number of parameters and computational overhead. This, in turn, improves audio processing efficiency and stability, and helps to simplify the model structure without affecting the output audio quality.

[0066] In this embodiment, upon obtaining a first audio feature, the first audio feature can be input to a harmonic processing module. Then, the harmonic processing module can generate a second harmonic feature based on the first audio feature and perform filtering processing to obtain the second audio feature. The second harmonic feature can be a harmonic feature generated by the harmonic processing module. The second harmonic feature can be represented in vector or matrix form. The filtering processing can be a process of further eliminating residual noise or adjusting timbre balance using a filter. The second audio feature can be the feature representation obtained after performing filtering processing on the second harmonic feature.

[0067] S140. Input the second audio features into the audio reconstruction module to reconstruct the second audio features and obtain the second audio data.

[0068] The audio reconstruction module can be a deep learning model that reconstructs audio based on second audio features. This module can perform post-processing operations on the second audio features to convert them into second audio data based on the target waveform representation. The audio reconstruction module can be a deep learning model consisting of multiple layers. Optionally, the audio reconstruction module may include a periodically enhanced activation function layer, a filtering subunit, a standard convolutional layer, and a hyperbolic tangent function layer connected in series. For example, Figure 2 This is a schematic diagram of the structure of an audio reconstruction module for an audio data processing method according to an embodiment of the present disclosure, as shown in the figure. Figure 2 As shown, the audio reconstruction module may include a periodic enhanced activation function layer, a filter subunit, a standard convolutional layer, and a hyperbolic tangent function layer connected in series.

[0069] The periodically enhanced activation function (PEF) layer is a network layer that deploys PEF functions. PEF functions are activation functions used to process the periodic features of audio signals. PEF layers can improve the model's ability to model the harmonic structure in audio data and avoid the spectral jaggedness caused by ordinary activation functions.

[0070] The filtering subunit can be used to locally adjust features in the frequency or time domain. The filtering subunit includes a frequency domain transformation layer, a filter, and a time domain transformation layer. The filtering subunit can achieve signal enhancement or noise suppression through coordinated operations of the frequency domain, filtering, and time domains. The frequency domain transformation layer can convert the time-domain audio signal into a frequency domain representation, facilitating targeted processing of frequency bands. The implementation of the frequency domain transformation layer can be varied, optionally employing short-time Fourier transform or learnable transform. The filter can be used to selectively enhance or suppress the frequency domain signal. The filter included in the filtering subunit can be a learnable filter, meaning the filter weights can be trained based on sample audio data. The time domain transformation layer can be used to restore the filtered frequency domain signal to a time-domain waveform. The implementation of the time domain transformation layer can be varied, optionally employing inverse short-time Fourier transform or learnable inverse transform. It should be noted that the network structure of the filtering subunit included in the audio reconstruction module is consistent with the network structure of the filtering subunit including the filter in the filtering processing module.

[0071] The standard convolutional layer can be used to extract local temporal or frequency domain features. The standard convolutional layer can be a one-dimensional convolutional layer. This disclosure does not specifically limit the kernel size or the number of channels of the standard convolutional layer.

[0072] The hyperbolic tangent function layer can be used to normalize the data and avoid oversaturation. The advantage of using the hyperbolic tangent function layer is that it can compress features to the range [-1, 1], which matches the standard amplitude distribution of audio waveforms. Compared to the sigmoid function, the gradient of the hyperbolic tangent function layer is more stable, making it suitable for multi-layer stacking.

[0073] In practical implementation, after obtaining the second audio feature, it can be input into the audio reconstruction module. Further, the second audio feature can be sequentially reconstructed based on the periodic enhanced activation function layer, filtering subunit, standard convolutional layer, and hyperbolic tangent function layer in the audio reconstruction module, ultimately outputting the second audio data. The second audio data can be audio data characterized by an audio waveform signal, determined based on the first audio data.

[0074] For example, Figure 3 This disclosure provides an embodiment of audio data processed by the audio data processing method of this disclosure, along with expected output audio data and audio data processed by related methods. Assume the first audio data is a Mel-frequency spectrogram. Figure 3 The left image in the image shows the audio waveform obtained after processing the Mel spectrogram using relevant techniques. Figure 3 The middle image in the diagram is the actual audio waveform corresponding to the Mel spectrogram. Figure 3The right figure in the image is an audio waveform obtained after processing the Mel spectrogram using the audio processing model provided in the embodiments of this disclosure. Figure 3 It can be seen that, compared to Figure 3 The middle image in the middle, Figure 3 The left image shows obvious spectral aliasing, and the spectral blurring indicates artifacts. Compared to... Figure 3 The left image in the middle, Figure 3 The right image in the figure does not show obvious spectral aliasing; the spectrum is relatively clear and there are no artifacts. Compared to... Figure 3 The left image in the middle, Figure 3 The spectrum contained in the right figure is closer to Figure 3 The intermediate image in the diagram, that is, the audio waveform obtained by processing the Mel spectrogram based on the audio processing model provided in this disclosure, is closer to the actual audio waveform than the audio waveform obtained by processing the Mel spectrogram based on related technologies.

[0075] The technical solution of this disclosure, by acquiring first audio data and determining the audio processing model corresponding to the first audio data, provides a data foundation and processing flow framework for subsequent audio data processing. Since the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module, it achieves the effect of applying a three-stage architecture of harmonic generation-harmonic processing-audio reconstruction to process the first audio data, laying a foundation for improving the quality of audio data from an overall architectural perspective. Furthermore, the first audio data is input into the harmonic generation module to generate first harmonic features based on the first audio data, obtaining first audio features. This achieves the effect of feature extraction and feature transformation of audio data by generating first harmonic features based on the first audio data through the harmonic generation module. By generating first harmonic features and determining first audio features based on the first harmonic features, the possibility of spectral aliasing is effectively reduced, providing better quality input data for the subsequent generation of high-quality audio data. Furthermore, the first audio features are input into the harmonic processing module... In this block, a second harmonic feature is generated based on the first audio feature and filtered to obtain the second audio feature. This achieves further refinement and optimization of the audio feature through the harmonic processing module. Simultaneously, the filtering process removes possible noise or invalid frequency components from the audio feature, making the audio feature clearer and more accurate, thereby improving the quality of the audio data and providing high-quality feature data for subsequent audio reconstruction. Furthermore, the second audio feature is input into the audio reconstruction module to reconstruct the second audio feature and obtain the second audio data. This solves the problem of spectral aliasing and spectral blurring in related technologies, which leads to poor quality of the generated audio data. It achieves the effect of generating, processing, and reconstructing the audio data to be processed through the audio processing model, thereby generating high-quality audio data with fewer artifacts. It significantly improves the fidelity and realism of the final generated audio data while effectively suppressing spectral aliasing and frequency blurring.

[0076] Figure 4This is a flowchart illustrating another audio data processing method provided in this embodiment. Based on the above embodiments, the harmonic generation module of this embodiment includes a frequency projection submodule and multiple cascaded harmonic generation submodules connected to the frequency projection submodule. Furthermore, the method for obtaining the first audio feature based on the harmonic generation module is further refined. Optionally, inputting the first audio data into the harmonic generation module to generate a first harmonic feature from the first audio data, thereby obtaining the first audio feature, includes: inputting the first audio data into the frequency projection submodule to obtain a third audio feature; wherein the frequency dimension of the third audio feature is higher than the frequency dimension of the first audio data; inputting the third audio feature into multiple cascaded harmonic generation submodules to add the first harmonic feature to the third audio feature, thereby obtaining the first audio feature. For detailed implementation, please refer to the description of this embodiment. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here. Figure 4 As shown, the method in this embodiment may specifically include:

[0077] S210. Obtain first audio data and determine the audio processing model corresponding to the first audio data; wherein, the audio processing model includes a harmonic generation module, a harmonic processing module and an audio reconstruction module; the harmonic generation module includes a frequency projection submodule and multiple cascaded harmonic generation submodules connected to the frequency projection submodule.

[0078] S220. Input the first audio data into the frequency projection submodule to obtain the third audio feature; wherein the frequency dimension of the third audio feature is higher than the frequency dimension of the first audio data.

[0079] The frequency projection submodule can be used to map low-frequency audio data (i.e., the first audio data) to a higher-dimensional feature space to obtain higher-dimensional audio features, which can then be used as the third audio feature. The advantage of using the frequency projection submodule to process the first audio data is that it enhances the ability of subsequent modules to distinguish frequency domain features, capturing finer high-frequency harmonics or noise details, and providing a denser feature representation for subsequent upsampling modules. The frequency projection submodule can be a module composed of a neural network capable of feature mapping. Optionally, the frequency projection submodule can be composed of a one-dimensional standard convolutional layer. In this embodiment, the first audio data can be data composed of a frequency dimension and a time frame number. The third audio feature can be the audio feature obtained by mapping the features represented by the frequency dimension of the first audio data to a higher-dimensional latent space.

[0080] In a specific implementation, when the first audio data is input into the harmonic generation module, the first audio data can first be input into the frequency projection submodule. Then, the frequency projection submodule can project along the frequency dimension to map the frequency features of the first audio data in the frequency dimension to a higher-dimensional potential space, and the audio features obtained after mapping can be used as the third audio features.

[0081] S230. Input the third audio feature into multiple cascaded harmonic generation submodules to add the first harmonic feature to the third audio feature to obtain the first audio feature.

[0082] The harmonic generation submodule generates harmonic features and adds them to the audio features. This submodule can include a first transposed convolutional layer and a multi-periodic synthesis unit connected to it. The first transposed convolutional layer upsamples the input audio features to expand them into a high-resolution temporal signal. Specifically, it upsamples the third audio feature and restores its temporal resolution. The transposed convolutional layer is a network layer in a deep learning model used to perform upsampling operations, mapping low-dimensional features to a high-dimensional space through inverse convolution. It can be understood as the reverse process of a standard convolutional layer, controlling the size of the output by using padding parameters and stride. The multi-periodic synthesis unit is a deep learning model used to generate harmonic features and add them to the audio features. The multi-periodic synthesis unit can be implemented by generating multiple periodic features in parallel and dynamically mixing them to approximate the physical properties of real acoustic features. Optionally, the multi-periodic synthesis unit includes a periodically enhanced activation function layer, a dilated convolutional layer, and a standard convolutional layer. The periodically enhanced activation function layer can be used to provide a periodic inductive bias, which is suitable for the oscillatory characteristics of audio signals. The advantage of introducing a periodically enhanced activation function layer into the multi-periodic synthesis unit is that it avoids artifacts in the generated harmonic features, thereby improving the generation quality of harmonic features. The dilated convolutional layer, also known as dilated convolution, is a special operation in convolutional neural networks that expands the receptive field by introducing dilation coefficients while keeping the number of parameters constant. Compared to standard convolutional layers, dilated convolutional layers expand the receptive field by skipping sparse sampling through dilation coefficients and without increasing the number of parameters, making them suitable for computationally limited scenarios (such as real-time audio processing).

[0083] In this embodiment of the disclosure, upon obtaining the third audio feature, the third audio feature can be input into multiple cascaded harmonic generation submodules connected to the frequency projection submodule. Furthermore, the third audio feature can be processed sequentially based on the first transposed convolutional layer and the multi-periodic synthesis unit included in the multiple cascaded harmonic generation submodules to obtain the first audio feature.

[0084] Optionally, the third audio feature is input into multiple cascaded harmonic generation sub-modules to add a first harmonic feature to the third audio feature, resulting in the first audio feature. This includes: inputting the third audio feature into the first transposed convolutional layer of the first harmonic generation sub-module to upsample the third audio feature and restore the temporal resolution, resulting in a first upsampled feature; inputting the first upsampled feature into a multi-periodic synthesis unit connected to the first transposed convolutional layer to obtain a sub-module synthesis feature; and inputting the sub-module synthesis feature into a harmonic generation sub-module connected to the first harmonic generation sub-module, determining the sub-module synthesis feature output by the last harmonic generation sub-module as the first audio feature. The advantage of this setup is that upsampling and restoring the temporal resolution enrich the temporal details of the audio feature. Furthermore, by adding the first harmonic feature to the first upsampled feature through the multi-periodic synthesis unit, the final first audio feature is closer to the characteristics of the real audio signal, enriching the timbre and texture of the audio, and making the first audio feature clearer and more accurate.

[0085] The first upsampled feature can be the feature output after upsampling the third audio feature through the first transposed convolutional layer in the first harmonic generation submodule and restoring the temporal resolution. The submodule synthesized feature is the harmonic feature generated after feature synthesis based on the first upsampled feature by the multi-period synthesis unit in the first harmonic generation submodule. The harmonic generation submodule connected to the first harmonic generation submodule can be any harmonic generation submodule that is serially connected to the first harmonic generation submodule, excluding the first harmonic generation submodule. For example, assuming that the harmonic generation module includes three serially connected harmonic generation submodules, the harmonic generation submodule connected to the first harmonic generation submodule can be the second harmonic generation submodule connected to the first harmonic generation submodule and the third harmonic generation submodule connected to the second harmonic generation submodule.

[0086] In specific implementation, after obtaining the third audio feature output by the frequency projection module, the third audio feature can be input into the first harmonic generation submodule. Further, the third audio feature is upsampled based on the first transposed convolutional layer in the first harmonic generation submodule to restore the time dimension resolution, resulting in a first upsampled feature. Further, the first upsampled feature can be input into the multi-periodic synthesis unit of the first harmonic generation submodule, where the multi-periodic synthesis unit performs feature synthesis based on the first upsampled feature to obtain the submodule synthesized feature. Further, the submodule synthesized feature can be further input into the harmonic generation submodule connected to the first harmonic generation submodule, and the operations of upsampling the submodule synthesized feature based on the first transposed convolutional layer in the harmonic generation submodule to restore the time dimension resolution and obtain the first upsampled feature, as well as the operation of inputting the first upsampled feature into the multi-periodic synthesis unit of the first harmonic generation submodule, where the multi-periodic synthesis unit performs feature synthesis based on the first upsampled feature to obtain the submodule synthesized feature, can continue to be performed. Furthermore, if the last harmonic generation submodule outputs the synthesized feature of the submodule, the synthesized feature of that submodule can be used as the first audio feature.

[0087] In this embodiment, the multi-periodic synthesis unit generates harmonic features by generating multiple periodic features in parallel and dynamically mixing them. Furthermore, the multi-periodic synthesis unit includes multiple parallel sub-units capable of generating periodic features. Optionally, the multi-periodic synthesis unit includes multiple parallel feature synthesis sub-units. Each feature synthesis sub-unit includes multiple series-connected first residual blocks.

[0088] The multi-periodic synthesis unit achieves independent modeling of different frequency components through multiple parallel feature synthesis subunits. Each feature synthesis subunit refines harmonic features progressively through a series-connected first residual block. Each feature synthesis subunit can independently process harmonic features of different frequency bands. It should be noted that the advantage of using multiple parallel feature synthesis subunits is that it avoids mutual interference between different frequency components, thereby improving timbre fidelity. Optionally, the number of feature synthesis subunits can be 3, 4, or 5, which can handle low-frequency, mid-frequency, and high-frequency components respectively. The first residual block can be a residual unit used to construct the feature synthesis subunit. This residual unit typically consists of multiple convolutional layers, batch normalization layers, and activation function layers. In the residual unit, the input data is added to the output of the convolutional layer through skip connections to form a residual. In the first residual block, the first periodically enhanced activation function layer, the first dilated convolutional layer, the second periodically enhanced activation function layer, and the standard convolutional layer are sequentially connected in series. In the first residual block, the input data of the first periodically enhanced activation function layer is added to the output of the standard convolutional layer to form a residual. The first dilated convolutional layer can be a one-dimensional dilated convolutional layer. The standard convolutional layer can be a one-dimensional standard convolutional layer. Optionally, the number of first residual blocks included in the feature synthesis subunit can be 3, 4, or 5, etc.

[0089] Optionally, the first upsampled feature is input into a multi-periodic synthesis unit connected to the first transposed convolutional layer to obtain sub-module synthesis features, including: inputting the first upsampled feature into each feature synthesis sub-unit to obtain the unit synthesis feature output by each feature synthesis sub-unit, and adding the unit synthesis features output by multiple feature synthesis sub-units to obtain sub-module synthesis features.

[0090] In specific implementation, when the first upsampled feature is input into a multi-period synthesis unit, the first upsampled feature can be input separately into each feature synthesis subunit connected in parallel. Furthermore, for each feature synthesis subunit, feature synthesis can be performed on the first upsampled feature based on multiple first residual blocks included in the feature synthesis subunit, and the unit-synthesized feature output by the feature synthesis subunit can be obtained. Further, the unit-synthesized features output by multiple feature synthesis subunits can be added together, and the resulting feature can be used as the submodule-synthesized feature.

[0091] Optionally, feature synthesis is performed on the first upsampled feature based on multiple first residual blocks included in the feature synthesis subunit, and the unit synthesized feature output by the feature synthesis subunit is obtained. This includes: inputting the first upsampled feature into the first first residual block in the feature synthesis subunit; processing the first upsampled feature based on the first periodic enhanced activation function layer, the first dilated convolutional layer, the second periodic enhanced activation function layer, and the standard convolutional layer connected in sequence in the first first residual block to obtain the first feature; adding the first feature and the first upsampled feature to obtain the output feature output by the first first residual block; inputting the output feature into the first residual block connected to the first first residual block; and determining the output feature output by the last first residual block as the unit synthesized feature output by the feature synthesis subunit.

[0092] For example, Figure 5 This is a schematic diagram of the structure of a harmonic generation module for an audio data processing method according to an embodiment of this disclosure. Figure 5 As shown, the harmonic generation module includes a frequency projection module and at least three harmonic generation sub-modules. Each harmonic generation sub-module includes a first transposed convolutional layer and a multi-periodic synthesis unit connected to the first transposed convolutional layer. The multi-periodic synthesis unit includes at least three parallel-connected feature synthesis sub-units. Each feature synthesis sub-unit includes at least three serially connected first residual blocks. Each first residual block includes a first periodically enhanced activation function layer, a first dilated convolutional layer, a second periodically enhanced activation function layer, and a standard convolutional layer connected in sequence. This can be combined with... Figure 3The process of generating the first audio feature based on the harmonic generation module is explained as follows: The first audio data is input into the frequency projection submodule. The frequency projection submodule projects along the frequency dimension, mapping the frequency features of the first audio data in the frequency dimension to a higher-dimensional latent space. The resulting audio feature is then used as the third audio feature. Further, the third audio feature is input into the first transposed convolutional layer of the first harmonic generation submodule to upsample the third audio feature and restore the resolution in the time dimension, obtaining the first upsampled feature. Further, the first upsampled feature is input into a multi-periodic synthesis unit connected to the first transposed convolutional layer to obtain the submodule synthesis feature. The submodule synthesis feature is then input into a harmonic generation submodule connected to the first harmonic generation submodule, and the submodule synthesis feature output by the last harmonic generation submodule is determined as the first audio feature. The generation process of the submodule synthesis feature can be as follows: The first upsampled feature is input into each feature synthesis subunit to obtain the unit synthesis feature output by each feature synthesis subunit. The unit synthesis features output by multiple feature synthesis subunits are then added together to obtain the submodule synthesis feature. Furthermore, the generation process of the unit synthesis feature can be as follows: the first upsampled feature is input into the first residual block in the feature synthesis subunit; the first upsampled feature is processed based on the first periodic enhanced activation function layer, the first dilated convolutional layer, the second periodic enhanced activation function layer and the standard convolutional layer connected in sequence in the first residual block to obtain the first feature; the first feature and the first upsampled feature are added together to obtain the output feature output by the first residual block; the output feature is input into the first residual block connected to the first residual block; and the output feature output by the last first residual block is determined as the unit synthesis feature output by the feature synthesis subunit.

[0093] S240. Input the first audio feature into the harmonic processing module to generate a second harmonic feature based on the first audio feature and perform filtering processing to obtain the second audio feature.

[0094] S250. Input the second audio features into the audio reconstruction module to reconstruct the second audio features and obtain the second audio data.

[0095] The technical solution of this embodiment obtains a third audio feature by inputting the first audio data into a frequency projection submodule; wherein the frequency dimension of the third audio feature is higher than that of the first audio data; further, the third audio feature is input into multiple cascaded harmonic generation submodules to add a first harmonic feature to the third audio feature, thereby obtaining the first audio feature. Based on the frequency projection submodule, the first audio data is mapped in the frequency domain, which enhances the representation capability of the first audio data in the frequency domain, making the third audio feature describe the frequency components of the audio feature more precisely, and can capture high-frequency details that may be ignored in the first audio data. Furthermore, by adding harmonic features through the harmonic generation submodule, the final generated first audio feature can more comprehensively describe the characteristics of the audio, realizing further optimization and adjustment of the audio feature, and providing a higher quality audio feature for the subsequent generation of high-quality audio data.

[0096] Figure 6 This is a flowchart illustrating another audio data processing method provided in this embodiment. Based on the above embodiments, the harmonic processing module of this embodiment includes a second transposed convolutional layer and multiple parallel-connected harmonic processing sub-modules connected to the second transposed convolutional layer. Furthermore, the method for obtaining the second audio feature based on the harmonic processing module is further refined. Optionally, inputting the first audio feature into the harmonic processing module to generate a second harmonic feature and perform filtering processing to obtain the second audio feature includes: inputting the first audio feature into the second transposed convolutional layer to upsample the first audio feature and restore the resolution of the time dimension to obtain a second upsampled feature; inputting the second upsampled feature into the harmonic processing sub-module to generate a second harmonic feature and perform filtering processing to obtain the second audio feature. Specific implementation details can be found in the description of this embodiment. Technical features that are the same as or similar to those in the foregoing embodiments will not be repeated here. Figure 6 As shown, the method in this embodiment may specifically include:

[0097] S310. Obtain first audio data and determine the audio processing model corresponding to the first audio data; wherein, the audio processing model includes a harmonic generation module, a harmonic processing module and an audio reconstruction module; the harmonic processing module includes a second transposed convolutional layer and a harmonic processing sub-module connected to the second transposed convolutional layer.

[0098] S320. Input the first audio data into the harmonic generation module to generate the first harmonic feature based on the first audio data, and obtain the first audio feature.

[0099] S330. Input the first audio feature into the second transposed convolutional layer to upsample the first audio feature and restore the resolution of the time dimension, thereby obtaining the second upsampled feature.

[0100] The second transposed convolutional layer is the same as the transposed convolutional layer included in the harmonic processing module. This second transposed convolutional layer can be used to upsample the first audio feature and restore its temporal resolution. The second upsampled feature can be the feature output after the first audio feature has been upsampled by the second transposed convolutional layer and its temporal resolution restored.

[0101] In practical implementation, when the harmonic generation module outputs the first audio feature, the first audio feature can be input into the second transposed convolutional layer in the harmonic generation module. Furthermore, the first audio feature can be upsampled based on the second transposed convolutional layer to restore the temporal resolution. Thus, the second upsampled feature can be obtained.

[0102] S340. Input the second upsampling feature into the harmonic processing submodule to generate a second harmonic feature based on the second upsampling feature and perform filtering processing to obtain the second audio feature.

[0103] The harmonic processing submodule can be used to generate second harmonic features from the second upsampled features and perform filtering. The harmonic processing submodule typically generates harmonic features by generating multiple multi-period features in parallel and dynamically mixing them, and then filtering each generated harmonic feature. Furthermore, the harmonic processing submodule includes multiple parallel units capable of generating periodic features and performing filtering. Optionally, the harmonic processing submodule may include multiple harmonic processing units connected in parallel. Each harmonic processing unit includes multiple second residual blocks connected in series.

[0104] The harmonic processing unit can be used to generate harmonic features and perform filtering. The harmonic processing unit can progressively refine the harmonic features using multiple second residual blocks connected in series. The second residual blocks can be used to generate harmonic features and perform filtering. Optionally, the second residual block includes a periodically enhanced activation function layer, a filtering subunit, a dilated convolutional layer, and a standard convolutional layer; the input of the filtering subunit is connected to the output of the periodically enhanced activation function layer. The filtering subunit can be a neural network used to perform filtering. In this embodiment, introducing a filtering subunit into the harmonic processing submodule can effectively suppress artifact components in the harmonic features. The filtering subunit can be a learnable global artifact filter (such as a genetic algorithm filter), which can operate across the entire frequency band and possesses time-invariant properties. Furthermore, the advantage of introducing a filtering subunit into the harmonic processing submodule is that after processing by the filtering generation module and the second transposed convolutional layer, the temporal resolution of the second upsampled feature has been largely restored. At this point, the second audio feature obtained by frequency domain transformation based on the filtering submodule can be more accurate. This design, which introduces a filtering subunit into the filtering submodule, relaxes the constraints on intermediate feature learning, helps enhance the model's performance potential, and reduces model complexity. Optionally, the filtering subunit includes a frequency domain transformation layer, a filter, and a time domain transformation layer. The filtering subunit can achieve signal enhancement or noise suppression through the coordinated operation of frequency domain transformation, learnable filtering, and time domain recovery. The frequency domain transformation layer can be used to convert the time-domain audio signal into a frequency-domain representation, facilitating targeted frequency band processing. The implementation of the frequency domain transformation layer can include various methods; optionally, it can be implemented using short-time Fourier transform or learnable transform. The filter can be used to selectively enhance or suppress the frequency-domain signal. The filter included in the filtering subunit can be a learnable filter, meaning that the filter weights can be trained based on the sample audio data. A time-domain transformation layer can be used to restore a filtered frequency-domain signal to a time-domain waveform. The implementation of a time-domain transformation layer can take many forms; optionally, it can be implemented using an inverse short-time Fourier transform or a learnable inverse transform.

[0105] For example, a short-time Fourier transform layer can be used as the frequency domain transformation layer in the filtering subunit, a learnable frequency artifact filter can be used as the filter in the filtering subunit, and an inverse short-time Fourier transform layer can be used as the time domain transformation layer in the filtering subunit. Assume a hidden feature z∈R... B×C×T The input is fed into the filtering subunit, where B represents the batch size, C represents the number of channels, and T represents the time dimension. A short-time Fourier transform (SFT) layer is used to perform a SFT on each frame of the input hidden features to achieve a conversion from the time domain to the frequency domain, obtaining the frequency domain features Z∈C. B×C×F×TWhere F represents the number of frequency points. Furthermore, the filter weights of the frequency artifact filter are learnable parameters. These filter weights are applied to the frequency domain features to independently weight the number of channels C and the number of frequency points F in the frequency domain features, and broadcast along the batch B and time dimension T to achieve efficient element-wise multiplication, resulting in a feature Z that removes artifacts. ’ Furthermore, an inverse short-time Fourier transform layer is applied to filter out artifact features Z. ’ Perform an inverse short-time Fourier transform to restore the resolution in the time dimension and output an artifact-free feature z. ’ It can make the artifact-free feature z ’ Features are output by the filter subunit.

[0106] Optionally, the connection relationship between the layers included in the second residual block can be as follows: the third periodic enhanced activation function layer, the first filter subunit, the first dilated convolutional layer, the fourth periodic enhanced activation function layer, the second filter subunit, and the standard convolutional layer in the second residual block are connected in series.

[0107] For example, Figure 7 This is a schematic diagram of the structure of a harmonic processing module for an audio data processing method according to an embodiment of this disclosure. Figure 7 As shown, the harmonic processing module includes a second transposed convolutional layer and a harmonic processing submodule connected to the second transposed convolutional layer. The harmonic processing submodule includes at least three parallel-connected harmonic processing units. Each harmonic processing unit includes at least three serially connected second residual blocks. In each residual block, a third periodic enhanced activation function layer, a first filtering subunit, a first dilated convolutional layer, a fourth periodic enhanced activation function layer, a second filtering subunit, and a standard convolutional layer are sequentially connected in series. The input of the third periodic enhanced activation function layer is residually connected to the output of the standard convolutional layer. Both the first and second filtering subunits include a frequency domain transformation layer, a filter, and a time domain transformation layer sequentially connected in series. Next, we can combine... Figure 4The process of determining the second audio feature based on the harmonic processing module is explained in detail as follows: When the harmonic generation module outputs the first audio feature, this first audio feature can be input into the second transposed convolutional layer in the harmonic processing module to upsample the first audio feature and restore the temporal resolution, resulting in the second upsampled feature. Further, the second upsampled feature is input into each harmonic processing unit in the harmonic processing submodule to obtain a unit-synthesized feature output by each harmonic processing unit. The unit-synthesized features output by multiple harmonic processing units are then added together to obtain the second audio feature. The generation process of the unit synthesis feature can be as follows: the second upsampled feature is input into the first second residual block in the harmonic processing unit; the second upsampled feature is processed based on the third periodic enhanced activation function layer, the first filter sub-unit, the first dilated convolutional layer, the fourth periodic enhanced activation function layer, the first filter sub-unit, and the standard convolutional layer connected in series in the first second residual block to obtain the second feature; the second feature and the second upsampled feature are added together to obtain the output feature output by the first second residual block; the output feature is input into the second residual block connected to the first second residual block; and the output feature output by the last second residual block is determined as the unit synthesis feature output by the harmonic processing unit. When processing the input hidden features based on the first filtering subunit in the second residual block, the input hidden features can first be transformed from the time domain to the frequency domain based on the frequency domain transformation layer to obtain frequency domain features. Further, the frequency domain features are input to the filter, and the filter weights in the filter are multiplied by their corresponding frequency components to independently weight the number of channels and frequency points in the frequency domain features, and broadcast along the batch and time dimensions to obtain the artifact-removed features. Further, the artifact-removed features are input to the time domain transformation layer, and the time domain transformation layer is applied to the artifact-removed features Z. ’ A frequency domain to time domain transformation is performed to restore the resolution of the time dimension and output an artifact-free feature. The output artifact-free feature is then input into the first dilated convolutional layer for processing to finally obtain the feature output of the second residual block.

[0108] S350. Input the second audio feature into the audio reconstruction module to reconstruct the second audio feature and obtain the second audio data.

[0109] The technical solution of this embodiment involves inputting a first audio feature into a second transposed convolutional layer to upsample the first audio feature and restore the resolution of the time dimension, thereby obtaining a second upsampled feature. Further, the second upsampled feature is input into a harmonic processing submodule to generate a second harmonic feature and perform filtering to obtain a second audio feature. By upsampling the first audio feature and restoring the resolution of the time dimension, the second upsampled feature becomes richer in detail in the time dimension, enabling more accurate capture of changes in the audio feature over time. Furthermore, generating the second harmonic feature further enriches the timbre of the audio, resulting in higher realism and naturalness of the generated audio data. Additionally, the filtering effectively removes noise components from the audio feature, making the final audio feature clearer and more accurate, thus improving the quality of the audio data.

[0110] Figure 8 This is a schematic diagram of the structure of an audio processing model for an audio data processing method provided in an embodiment of this disclosure. For example... Figure 8 As shown, the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module. The harmonic generation module includes a frequency projection submodule and at least three serially connected harmonic generation submodules connected to the frequency projection submodule. Each harmonic generation submodule includes a first transposed convolutional layer and a multi-periodic synthesis unit connected to the first transposed convolutional layer. The harmonic processing module includes a second transposed convolutional layer and a harmonic processing submodule connected to the second transposed convolutional layer. Next, combined with... Figure 8 The implementation process of the audio data processing method provided in the embodiments of this disclosure will be described as follows:

[0111] First, first audio data, which can be a Mel spectrogram, is acquired and input into its corresponding audio processing model. Next, the Mel spectrogram is input into a frequency projection submodule to obtain a third audio feature. Then, the third audio feature is input into multiple cascaded harmonic generation submodules. Based on a first transposed convolutional layer and multiple periodic synthesis units in the harmonic generation submodules, a first harmonic feature is generated and added to the third audio feature to obtain the first audio feature. Next, the first audio feature is input into a second transposed convolutional layer in the harmonic processing module to upsample the first audio feature and restore the time dimension resolution, resulting in a second upsampled feature. Next, the second upsampled feature is input into the harmonic processing submodule to generate a second harmonic feature and perform filtering processing, resulting in a second audio feature. Finally, the second audio feature is input into an audio reconstruction module to reconstruct the second audio feature, obtaining second audio data, which can be an audio waveform generated based on the Mel spectrogram.

[0112] Figure 9 This is a schematic diagram of the structure of an audio data processing apparatus provided in an embodiment of the present disclosure, as shown below. Figure 9 As shown, the device includes: an audio data acquisition module 410, an audio harmonic generation module 420, an audio harmonic processing module 430, and an audio data generation module 440.

[0113] The audio data acquisition module 410 is used to acquire first audio data and determine the audio processing model corresponding to the first audio data. The audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module. The audio harmonic generation module 420 is used to input the first audio data into the harmonic generation module to generate a first harmonic feature for the first audio data, thereby obtaining a first audio feature. The audio harmonic processing module 430 is used to input the first audio feature into the harmonic processing module to generate a second harmonic feature for the first audio feature and perform filtering processing to obtain a second audio feature. The audio data generation module 440 is used to input the second audio feature into the audio reconstruction module to reconstruct the second audio feature and obtain second audio data.

[0114] The technical solution of this disclosure, by acquiring first audio data and determining the audio processing model corresponding to the first audio data, provides a data foundation and processing flow framework for subsequent audio data processing. Since the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module, it achieves the effect of applying a three-stage architecture of harmonic generation-harmonic processing-audio reconstruction to process the first audio data, laying a foundation for improving the quality of audio data from an overall architectural perspective. Furthermore, the first audio data is input into the harmonic generation module to generate first harmonic features based on the first audio data, obtaining first audio features. This achieves the effect of feature extraction and feature transformation of audio data by generating first harmonic features based on the first audio data through the harmonic generation module. By generating first harmonic features and determining first audio features based on the first harmonic features, the possibility of spectral aliasing is effectively reduced, providing better quality input data for the subsequent generation of high-quality audio data. Furthermore, the first audio features are input into the harmonic processing module... In this block, a second harmonic feature is generated based on the first audio feature and filtered to obtain the second audio feature. This achieves further refinement and optimization of the audio feature through the harmonic processing module. Simultaneously, the filtering process removes possible noise or invalid frequency components from the audio feature, making the audio feature clearer and more accurate, thereby improving the quality of the audio data and providing high-quality feature data for subsequent audio reconstruction. Furthermore, the second audio feature is input into the audio reconstruction module to reconstruct the second audio feature and obtain the second audio data. This solves the problem of spectral aliasing and spectral blurring in related technologies, which leads to poor quality of the generated audio data. It achieves the effect of generating, processing, and reconstructing the audio data to be processed through the audio processing model, thereby generating high-quality audio data with fewer artifacts. It significantly improves the fidelity and realism of the final generated audio data while effectively suppressing spectral aliasing and frequency blurring.

[0115] Based on any optional technical solution in the embodiments of this disclosure, the harmonic generation module includes a frequency projection submodule and multiple cascaded harmonic generation submodules connected to the frequency projection submodule; the audio harmonic generation module 420 includes: a frequency projection processing unit and a harmonic feature adding unit. The frequency projection processing unit is used to input first audio data into the frequency projection submodule to obtain a third audio feature; wherein, the frequency dimension of the third audio feature is higher than the frequency dimension of the first audio data; the harmonic feature adding unit is used to input the third audio feature into the multiple cascaded harmonic generation submodules to add a first harmonic feature to the third audio feature to obtain a first audio feature.

[0116] Based on any optional technical solution in the embodiments of this disclosure, the harmonic generation submodule includes a first transposed convolutional layer and a multi-periodic synthesis unit connected to the first transposed convolutional layer; the multi-periodic synthesis unit includes a periodically enhanced activation function layer, a dilated convolutional layer, and a standard convolutional layer; the harmonic feature addition unit includes: a feature upsampling processing subunit, a feature synthesis subunit, and a first audio feature determination subunit. Specifically, the feature upsampling processing subunit is used to input a third audio feature into the first transposed convolutional layer of the first harmonic generation submodule to upsample the first audio feature and restore the resolution of the time dimension, obtaining a first upsampled feature; the feature synthesis subunit is used to input the first upsampled feature into the multi-periodic synthesis unit connected to the first transposed convolutional layer to obtain a submodule synthesized feature; the first audio feature determination subunit is used to continue inputting the submodule synthesized feature into the harmonic generation submodule connected to the first harmonic generation submodule, determining the submodule synthesized feature output by the last harmonic generation submodule as the first audio feature.

[0117] Based on any optional technical solution in the embodiments of this disclosure, the multi-periodic synthesis unit includes multiple parallel feature synthesis sub-units; the feature synthesis sub-unit includes multiple serially connected first residual blocks; in the first residual block, a first periodically enhanced activation function layer, a first dilated convolutional layer, a second periodically enhanced activation function layer, and a standard convolutional layer are sequentially connected in series.

[0118] Based on any optional technical solution in the embodiments of this disclosure, the harmonic processing module includes a second transposed convolutional layer and a harmonic processing submodule connected to the second transposed convolutional layer; the harmonic processing submodule includes a periodic enhanced activation function layer, a filtering subunit, a dilated convolutional layer, and a standard convolutional layer; the audio harmonic processing module 430 includes: a feature upsampling unit and a harmonic feature generation unit. The feature upsampling unit is used to input a first audio feature into the second transposed convolutional layer to upsample the first audio feature and restore the resolution of the time dimension, thereby obtaining a second upsampled feature; the harmonic feature generation unit is used to input the second upsampled feature into the harmonic processing submodule to generate a second harmonic feature based on the second upsampled feature and perform filtering processing to obtain a second audio feature.

[0119] Based on any optional technical solution in the embodiments of this disclosure, the harmonic processing submodule includes multiple harmonic processing units connected in parallel; the harmonic processing unit includes multiple second residual blocks connected in series; the second residual block includes a periodically enhanced activation function layer, a filtering subunit, a dilated convolutional layer, and a standard convolutional layer; the input end of the filtering subunit is connected to the output end of the periodically enhanced activation function layer.

[0120] Based on any optional technical solution in the embodiments of this disclosure, the third periodic enhanced activation function layer, the first filter subunit, the first dilated convolutional layer, the fourth periodic enhanced activation function layer, the second filter subunit, and the standard convolutional layer in the second residual block are connected in series.

[0121] Based on any optional technical solution in the embodiments of this disclosure, the filtering subunit includes a frequency domain transformation layer, a filter, and a time domain transformation layer; the filter weights are obtained by training based on sample audio data.

[0122] Based on any optional technical solution in the embodiments of this disclosure, the audio reconstruction module includes a periodically enhanced activation function layer, a filter subunit, a standard convolutional layer, and a hyperbolic tangent function layer connected in series.

[0123] Based on any optional technical solution in the embodiments of this disclosure, the audio processing model is a generative model trained by a generative adversarial network; during the training process, the model parameters of the generative model are adjusted according to the audio generation loss; the audio generation loss includes the discrimination loss of the discriminative model in the generative adversarial network on the output audio data of the generative model, the first feature processing loss between the output audio data of the generative model and the expected output data of the multiple discriminative layers of the discriminative model, and the second feature processing loss between the input audio data and the output audio data of the generative model.

[0124] The audio data processing apparatus provided in this disclosure can execute the audio data processing method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the audio data processing method.

[0125] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0126] The following is for reference. Figure 10 The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 600 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 10 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0127] like Figure 10As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0128] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 10 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0129] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0130] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0131] The electronic device provided in this disclosure and the audio data processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this disclosure can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0132] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the audio data processing method provided in the above embodiments.

[0133] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0134] According to one or more embodiments of this disclosure, [Example 1] provides an audio data processing method, comprising: acquiring first audio data; determining an audio processing model corresponding to the first audio data; wherein the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module; inputting the first audio data into the harmonic generation module to generate a first harmonic feature for the first audio data, thereby obtaining a first audio feature; inputting the first audio feature into the harmonic processing module to generate a second harmonic feature for the first audio feature and performing filtering processing, thereby obtaining a second audio feature; and inputting the second audio feature into the audio reconstruction module to reconstruct the second audio feature, thereby obtaining second audio data.

[0135] According to one or more embodiments of this disclosure, [Example 2] provides the method of Example 1, further comprising: optionally, the harmonic generation module includes a frequency projection submodule and a plurality of cascaded harmonic generation submodules connected to the frequency projection submodule; the step of inputting the first audio data into the harmonic generation module to generate a first harmonic feature for the first audio data to obtain the first audio feature includes: inputting the first audio data into the frequency projection submodule to obtain a third audio feature; wherein the frequency dimension of the third audio feature is higher than the frequency dimension of the first audio data; inputting the third audio feature into the plurality of cascaded harmonic generation submodules to add the first harmonic feature to the third audio feature to obtain the first audio feature.

[0136] According to one or more embodiments of this disclosure, Example 3 provides the method of Example 2, further comprising: optionally, the harmonic generation submodule includes a first transposed convolutional layer and a multi-periodic synthesis unit connected to the first transposed convolutional layer; the multi-periodic synthesis unit includes a periodically enhanced activation function layer, a dilated convolutional layer, and a standard convolutional layer; the step of inputting the third audio feature into multiple cascaded harmonic generation submodules to add a first harmonic feature to the third audio feature to obtain a first audio feature includes: inputting the third audio feature into the first transposed convolutional layer of the first harmonic generation submodule to upsample the first audio feature and restore the resolution of the time dimension to obtain a first upsampled feature; inputting the first upsampled feature into the multi-periodic synthesis unit connected to the first transposed convolutional layer to obtain a submodule synthesis feature; continuing to input the submodule synthesis feature into the harmonic generation submodule connected to the first harmonic generation submodule, and determining the submodule synthesis feature output by the last harmonic generation submodule as the first audio feature.

[0137] According to one or more embodiments of this disclosure, Example 4 provides the method of Example 3, which further includes: optionally, the multi-periodic synthesis unit includes a plurality of parallel feature synthesis sub-units; the feature synthesis sub-unit includes a plurality of serially connected first residual blocks; in the first residual block, a first periodically enhanced activation function layer, a first dilated convolutional layer, a second periodically enhanced activation function layer, and a standard convolutional layer are sequentially connected in series.

[0138] According to one or more embodiments of this disclosure, Example 5 provides the method of Example 1, further comprising: optionally, the harmonic processing module includes a second transposed convolutional layer and a harmonic processing submodule connected to the second transposed convolutional layer; the harmonic processing submodule includes a periodically enhanced activation function layer, a filtering subunit, a dilated convolutional layer, and a standard convolutional layer; the step of inputting the first audio feature into the harmonic processing module to generate a second harmonic feature for the first audio feature and performing filtering processing to obtain the second audio feature includes: inputting the first audio feature into the second transposed convolutional layer to upsample the first audio feature and restore the resolution of the time dimension to obtain a second upsampled feature; inputting the second upsampled feature into the harmonic processing submodule to generate a second harmonic feature for the second upsampled feature and performing filtering processing to obtain the second audio feature.

[0139] According to one or more embodiments of this disclosure, Example Six provides the method of Example Five, further comprising: optionally, a harmonic processing submodule including a plurality of harmonic processing units connected in parallel; the harmonic processing unit including a plurality of second residual blocks connected in series; the second residual block including a periodically enhanced activation function layer, a filtering subunit, a dilated convolutional layer and a standard convolutional layer; the input of the filtering subunit being connected to the output of the periodically enhanced activation function layer.

[0140] According to one or more embodiments of this disclosure, Example 7 provides the method of Example 6, which further includes: optionally, a third periodic enhanced activation function layer, a first filter subunit, a first dilated convolutional layer, a fourth periodic enhanced activation function layer, a second filter subunit, and a standard convolutional layer in the second residual block are connected in series.

[0141] According to one or more embodiments of this disclosure, Example 8 provides the method of Example 5, which further includes: optionally, the filtering subunit includes a frequency domain transformation layer, a filter, and a time domain transformation layer; the filtering weights of the filter are trained based on sample audio data.

[0142] According to one or more embodiments of this disclosure, Example 9 provides the method of Example 1, which further includes: optionally, the audio reconstruction module includes a periodically enhanced activation function layer, a filter subunit, a standard convolutional layer and a hyperbolic tangent function layer connected in series.

[0143] According to one or more embodiments of this disclosure, Example 10 provides the method of Example 1, which further includes: optionally, the audio processing model is a generative model trained by a generative adversarial network; during training, the model parameters of the generative model are adjusted according to the audio generation loss; the audio generation loss includes the discrimination loss of the discriminative model in the generative adversarial network on the output audio data of the generative model, the first feature processing loss between the output audio data of the generative model and the expected output data of the generative model by the multiple discriminative layers of the discriminative model, and the second feature processing loss between the input audio data and the output audio data of the generative model.

[0144] According to one or more embodiments of this disclosure, [Example 11] provides an audio data processing apparatus, including: an audio data acquisition module, configured to acquire first audio data and determine an audio processing model corresponding to the first audio data; wherein, the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module;

[0145] An audio harmonic generation module is used to input the first audio data into the harmonic generation module to generate a first harmonic feature based on the first audio data, thereby obtaining a first audio feature; an audio harmonic processing module is used to input the first audio feature into the harmonic processing module to generate a second harmonic feature based on the first audio feature and perform filtering processing to obtain a second audio feature; an audio data generation module is used to input the second audio feature into the audio reconstruction module to reconstruct the second audio feature and obtain second audio data.

[0146] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0147] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0148] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire first audio data; determine an audio processing model corresponding to the first audio data; wherein the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module; input the first audio data into the harmonic generation module to generate a first harmonic feature for the first audio data, thereby obtaining a first audio feature; input the first audio feature into the harmonic processing module to generate a second harmonic feature for the first audio feature and perform filtering processing, thereby obtaining a second audio feature; and input the second audio feature into the audio reconstruction module to reconstruct the second audio feature, thereby obtaining second audio data.

[0149] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0150] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0151] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The names of the units are not necessarily limiting in certain circumstances; for example, an audio data acquisition module can also be described as a "module for acquiring first audio data".

[0152] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0153] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0155] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0156] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An audio data processing method, characterized in that, include: Acquire first audio data and determine the audio processing model corresponding to the first audio data; wherein, the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module; The first audio data is input into the harmonic generation module to generate a first harmonic feature for the first audio data, thereby obtaining the first audio feature; The first audio feature is input into the harmonic processing module to generate a second harmonic feature based on the first audio feature and perform filtering processing to obtain the second audio feature; The second audio feature is input into the audio reconstruction module to reconstruct the second audio feature and obtain the second audio data.

2. The audio data processing method according to claim 1, characterized in that, The harmonic generation module includes a frequency projection submodule and multiple harmonic generation submodules connected in series with the frequency projection submodule. The step of inputting the first audio data into the harmonic generation module to generate a first harmonic feature from the first audio data, thereby obtaining the first audio feature, includes: The first audio data is input into the frequency projection submodule to obtain a third audio feature; wherein the frequency dimension of the third audio feature is higher than the frequency dimension of the first audio data; The third audio feature is input into multiple cascaded harmonic generation submodules to add a first harmonic feature to the third audio feature, thereby obtaining a first audio feature.

3. The audio data processing method according to claim 2, characterized in that, The harmonic generation submodule includes a first transposed convolutional layer and a multi-periodic synthesis unit connected to the first transposed convolutional layer; the multi-periodic synthesis unit includes a periodically enhanced activation function layer, a dilated convolutional layer, and a standard convolutional layer; The step of inputting the third audio feature into multiple cascaded harmonic generation submodules to add a first harmonic feature to the third audio feature to obtain a first audio feature includes: The third audio feature is input into the first transposed convolutional layer of the first harmonic generation submodule to upsample the third audio feature and restore the resolution of the time dimension, thereby obtaining the first upsampled feature; The first upsampled feature is input into the multi-periodic synthesis unit connected to the first transposed convolutional layer to obtain the sub-module synthesized feature; The submodule synthesis feature is further input into the harmonic generation submodule connected to the first harmonic generation submodule, and the submodule synthesis feature output by the last harmonic generation submodule is determined as the first audio feature.

4. The audio data processing method according to claim 3, characterized in that, The multi-periodic synthesis unit includes multiple parallel characteristic synthesis sub-units; The feature synthesis subunit includes multiple first residual blocks connected in series; in the first residual block, a first periodic enhanced activation function layer, a first dilated convolutional layer, a second periodic enhanced activation function layer, and a standard convolutional layer are connected in series.

5. The audio data processing method according to claim 1, characterized in that, The harmonic processing module includes a second transposed convolutional layer and a harmonic processing submodule connected to the second transposed convolutional layer; the harmonic processing submodule includes a periodically enhanced activation function layer, a filtering subunit, a dilated convolutional layer, and a standard convolutional layer. The step of inputting the first audio feature into the harmonic processing module to generate a second harmonic feature based on the first audio feature and performing filtering processing to obtain the second audio feature includes: The first audio feature is input into the second transposed convolutional layer to upsample the first audio feature and restore the resolution of the time dimension, thus obtaining the second upsampled feature; The second upsampled feature is input into the harmonic processing submodule to generate a second harmonic feature based on the second upsampled feature and perform filtering processing to obtain a second audio feature.

6. The audio data processing method according to claim 5, characterized in that, The harmonic processing submodule includes multiple harmonic processing units connected in parallel; each harmonic processing unit includes multiple second residual blocks connected in series; each second residual block includes a periodically enhanced activation function layer, a filtering subunit, a dilated convolutional layer, and a standard convolutional layer; the input of the filtering subunit is connected to the output of the periodically enhanced activation function layer.

7. The audio data processing method according to claim 6, characterized in that, In the second residual block, the third periodic enhanced activation function layer, the first filter subunit, the first dilated convolutional layer, the fourth periodic enhanced activation function layer, the second filter subunit, and the standard convolutional layer are connected in series.

8. The audio data processing method according to claim 5, characterized in that, The filtering subunit includes a frequency domain transformation layer, a filter, and a time domain transformation layer; the filter weights are obtained by training based on sample audio data.

9. The audio data processing method according to claim 1, characterized in that, The audio reconstruction module includes a periodic enhanced activation function layer, a filter subunit, a standard convolutional layer, and a hyperbolic tangent function layer connected in series.

10. The audio data processing method according to claim 1, characterized in that, The audio processing model is a generative model trained by a generative adversarial network. During training, the model parameters of the generative model are adjusted according to the audio generation loss. The audio generation loss includes the discrimination loss of the discriminative model in the generative adversarial network on the output audio data of the generative model, the first feature processing loss between the output audio data of the generative model and the expected output data of the multiple discriminative layers of the discriminative model, and the second feature processing loss between the input audio data and the output audio data of the generative model.

11. An audio data processing apparatus, characterized in that, include: An audio data acquisition module is used to acquire first audio data and determine an audio processing model corresponding to the first audio data; wherein, the audio processing model includes a harmonic generation module, a harmonic processing module, and an audio reconstruction module; An audio harmonic generation module is used to input the first audio data into the harmonic generation module to generate a first harmonic feature for the first audio data, thereby obtaining the first audio feature; An audio harmonic processing module is used to input the first audio feature into the harmonic processing module to generate a second harmonic feature for the first audio feature and perform filtering processing to obtain the second audio feature; An audio data generation module is used to input the second audio feature into the audio reconstruction module to reconstruct the second audio feature and obtain the second audio data.

12. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the audio data processing method as described in any one of claims 1-10.

13. A storage medium containing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are used to perform the audio data processing method as described in any one of claims 1-10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the audio data processing method as described in any one of claims 1-10.

Images