Efficient audio data generation method and device, computer equipment and storage medium
By using ConvNeXt components in audio processing for feature extraction and fusion, combined with inverse fast Fourier transform and autoencoder network, the problem of low computing efficiency of traditional audio processing methods is solved, and efficient audio data generation is achieved.
Patent Information
- Application Number
- CN202411552350.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-07
AI Technical Summary
Traditional audio processing methods based on generative adversarial networks (GANs) have problems with low computing efficiency.
ConvNeXt component is used for feature extraction and fusion, and target audio data is generated through inverse fast Fourier transform, and information completion and denoising are performed in combination with the autoencoder network.
Significantly improves computing efficiency and speeds by an entire order of magnitude, while achieving an effect that matches the most advanced audio quality.
Smart Images
Figure CN119400189A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio processing technology, applicable to the financial or medical fields, and in particular to efficient audio data generation methods, devices, computer equipment and storage media. Background Art
[0002] A neural vocoder is an audio generation model based on a neural network, whose goal is to convert extracted acoustic features (such as mel-spectrograms) into realistic audio waveforms. This technology has gradually emerged against the backdrop of the rapid development of artificial intelligence and deep learning technologies, bringing revolutionary changes to fields such as speech synthesis, music production, and artificial intelligence interaction. The latest progress of the neural vocoder is mainly driven by generative adversarial networks (GANs) operating in the time domain.
[0003] While this approach is effective, it ignores the inductive bias provided by the time-frequency representation, resulting in redundant and computationally intensive upsampling operations, which shows that traditional audio processing methods driven by generative adversarial networks (GANs) suffer from the problem of low computational efficiency. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose an efficient audio data generation method, apparatus, computer equipment and storage medium to solve the problem of low computational efficiency of traditional audio processing methods based on generative adversarial networks (GANs).
[0005] In order to solve the above technical problems, the present application embodiment provides an efficient audio data generation method, which adopts the following technical solution:
[0006] Obtain MEL spectrum information sent by the user terminal;
[0007] Inputting the MEL spectrum information into a first ConvNeXt component to obtain first variable data;
[0008] Performing a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data;
[0009] Inputting the feature fusion data into a second ConvNeXt component to obtain second variable data;
[0010] An inverse fast Fourier transform operation is performed on the second variable data to obtain target audio data.
[0011] Further, after the step of obtaining the MEL spectrum information sent by the user terminal, the following steps are also included:
[0012] The MEL spectrum information is completed and denoised according to the autoencoder network.
[0013] Furthermore, the step of inputting the MEL spectrum information into the first ConvNeXt component to obtain the first variable data specifically includes the following steps:
[0014] Extracting the time key features and frequency key features of the MEL spectrum information in the time dimension and the frequency dimension according to the convolutional neural network;
[0015] A pooling operation is performed on the time key feature and the frequency key feature to obtain the first variable data.
[0016] Furthermore, the step of performing a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data specifically includes the following steps:
[0017] Calculating the position correlation of the MEL spectrum information and the first variable data to generate an attention weight matrix;
[0018] A weight adjustment operation is performed on the MEL spectrum information and the first variable data according to the attention weight matrix, and a feature fusion operation is performed on the weight-adjusted MEL spectrum information and the first variable data to obtain the feature fusion data.
[0019] Furthermore, before the step of performing an inverse fast Fourier transform operation on the second variable data to obtain the target audio data, the following steps are also included:
[0020] The spectrum distribution and time domain continuity of the second variable data are optimized according to the spectrum constraint technology and the time domain smoothing technology.
[0021] Furthermore, the step of performing an inverse fast Fourier transform operation on the second variable data to obtain target audio data specifically includes the following steps:
[0022] Performing segmentation processing on the second variable data to obtain second variable segmentation data;
[0023] Performing a preprocessing operation on the current second variable segmented data according to the windowing function to obtain windowed segmented data;
[0024] Performing an inverse fast Fourier transform operation on the windowed segmented data to obtain a current time domain signal segment;
[0025] Superimposing adjacent current time domain signal segments according to a preset overlapping ratio to obtain a superimposed time domain signal segment;
[0026] Performing an energy normalization operation on the superimposed time domain signal segment to obtain a normalized signal segment;
[0027] Dynamically adjust the window size and overlap ratio of the next segment of second variable segmented data according to the time domain signal energy distribution characteristics of the normalized signal segment, and repeatedly perform the preprocessing operation, the inverse fast Fourier transform operation, the superposition operation and the energy normalization operation;
[0028] When all the windowed segmented data have completed the preprocessing operation, the inverse fast Fourier transform operation, the superposition operation and the energy normalization operation, the target audio data is obtained.
[0029] Furthermore, after the step of performing an inverse fast Fourier transform operation on the second variable data to obtain target audio data, the following steps are also included:
[0030] Inputting the target audio data into a psychoacoustic model to perform spectrum optimization and energy optimization operations to obtain optimized audio data;
[0031] The optimized audio data is subjected to a noise elimination operation according to an adaptive noise suppression technique to obtain high-quality audio data.
[0032] In order to solve the above technical problems, the embodiment of the present application also provides an efficient audio data generation device, which adopts the following technical solution:
[0033] A spectrum acquisition module, used to acquire MEL spectrum information sent by a user terminal;
[0034] A first variable acquisition module, used for inputting the MEL spectrum information into a first ConvNeXt component to obtain first variable data;
[0035] A feature fusion module, used for performing a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data;
[0036] A second variable acquisition module, used for inputting the feature fusion data into a second ConvNeXt component to obtain second variable data;
[0037] The target audio acquisition module is used to perform an inverse fast Fourier transform operation on the second variable data to obtain target audio data.
[0038] Furthermore, the first variable acquisition module includes:
[0039] A feature extraction submodule, used for extracting the time key features and frequency key features of the MEL spectrum information in the time dimension and the frequency dimension according to a convolutional neural network;
[0040] A pooling submodule is used to perform a pooling operation on the time key feature and the frequency key feature to obtain the first variable data.
[0041] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0042] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned method for generating efficient audio data when executing the computer-readable instructions.
[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0044] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the method for generating efficient audio data as described above are implemented.
[0045] The present application provides an efficient audio data generation method, including: obtaining MEL spectrum information sent by a user terminal; inputting the MEL spectrum information into a first ConvNeXt component to obtain first variable data; performing a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data; inputting the feature fusion data into a second ConvNeXt component to obtain second variable data; performing an inverse fast Fourier transform operation on the second variable data to obtain target audio data. Compared with the prior art, the present application not only achieves audio quality that matches the most advanced, but also significantly improves computational efficiency, and is a full order of magnitude faster than the currently popular time-domain neural vocoder method. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0047] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0048] Figure 2 is a flowchart of an implementation of an efficient audio data generation method provided in an embodiment of the present application;
[0049] Figure 3 is a schematic diagram of the structure of an efficient audio data generating device provided in an embodiment of the present application;
[0050] Figure 4It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0054] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0055] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0056] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.
[0057] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .
[0058] It should be noted that the efficient audio data generation method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the efficient audio data generation device is generally arranged in the server / terminal device.
[0059] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0060] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for generating efficient audio data according to the present application. The method for generating efficient audio data comprises: step S201, step S202, step S203, step S204 and step S205.
[0061] In step S201, MEL spectrum information sent by a user terminal is obtained.
[0062] In step S202, the MEL spectrum information is input into the first ConvNeXt component to obtain the first variable data.
[0063] In step S203, a feature fusion operation is performed on the MEL spectrum information and the first variable data to obtain feature fusion data.
[0064] In step S204, the feature fusion data is input into the second ConvNeXt component to obtain second variable data.
[0065] In step S205, an inverse fast Fourier transform operation is performed on the second variable data to obtain target audio data.
[0066] The above-mentioned MEL spectrum information described in the present application may be "transaction spectrum information or payment spectrum information or business spectrum information or purchase spectrum information" related to financial institutions (such as banks, etc.). The MEL spectrum information may also be medical spectrum information related to medical scenarios, such as personal health records, prescriptions, examination reports and other spectrum information. It should be understood that the examples of MEL spectrum information here are only for the convenience of understanding and are not used to limit this application.
[0067] In the present application, a GAN-based vocoder is proposed to train complex STFT (Short Time Fourier Transform) coefficients for generating audio clips. Unlike the traditional neural vocoder architecture that relies on transposed convolution for upsampling, this work proposes to maintain the same feature time resolution in all layers. Upsampling to the waveform is achieved through the inverse fast Fourier transform.
[0068] In some optional implementations of the embodiments of the present application, in order to estimate the phase angle, the present application proposes a simple activation function, which is defined by the unit circle. This method naturally includes implicit phase wrapping, ensuring that all phase angles have meaningful values.
[0069] In the embodiments of the present application, the proposed GAN model uses a Fourier-based time-frequency representation as the target data distribution of the generator. The construction of this scheme does not use any transposed convolutions; instead, the upsampling operation is only implemented through the fast inverse STFT. Unlike time-domain vocoders that usually use a series of upsampling layers to expand the input features to the target waveform resolution, this approach allows a unique model design, which often requires amplifying the features hundreds of times. In contrast, this proposal maintains the same time domain resolution throughout the network (such as Figure 1 ). This design, called isotropic architecture, has been found to work well in various settings including Transformer. This approach is particularly beneficial for audio synthesis. Traditional methods often use transposed convolutions, which can introduce aliasing artifacts and require additional measures to mitigate the problem. This proposal eliminates the learnable upsampling layers and instead adopts the well-established inverse Fourier transform to reconstruct the waveform at the original scale. In the context of converting Mel-spectrograms to audio signals, the time domain resolution is determined by the jump size of the STFT.
[0070] In the embodiment of the present application, the audio signal is represented in the time-frequency domain using short-time Fourier transform (STFT):
[0071]
[0072] The STFT applies the Fourier transform to successive windowed portions of a signal. In practice, the STFT is computed by taking a series of Fast Fourier Transforms (FFTs) on overlapping, windowed frames of data that are created as the window function advances or "jumps" through time.
[0073] In the embodiment of the present application, ConvNeXt is used as the basic backbone of the generator. It first embeds the input features into a hidden dimension and then applies a series of 1D convolution blocks. Each block consists of a depthwise convolution followed by an inverted bottleneck structure that projects the features to a higher dimension using point-by-point convolution. The GELU (Gaussian Error Linear Unit) activation function is used in the bottleneck, and layer normalization is used between blocks.
[0074] In the embodiment of the present application, the Fourier transform of the real-valued signal is conjugate symmetric, so the present application only uses a single-sided spectrum, and each frame generates nfft / 2+1 coefficients. Since the present application parameterizes the model as output phase and amplitude values, the activation of the hidden dimension is projected into a tensor h with nfft+2 channels and split into:
[0075] m,p=h[1:(n fft / 2+1)],h[(n fft / 2+2):n]
[0076] To express the magnitude, the present application applies an exponential function to m: M = exp(m).
[0077] This application maps p to the unit circle by calculating the cosine and sine of p, and obtains x and y respectively:
[0078] x=cos(p)
[0079] y=sin(p)
[0080] Finally, this application represents the complex-valued coefficients as:
[0081] STFT = M (x + jy)
[0082] Importantly, this simple formulation allows the present application to express the phase angle for any real parameter p And it ensures is correctly wrapped into the required range (-π,π].
[0083] As for the discriminator, this application adopts a multi-period discriminator (MPD) and a multi-resolution discriminator (MRD).
[0084] Loss function:
[0085] The training objectives of this application include reconstruction loss, adversarial loss, and feature matching loss. However, this application adopts the hinge loss formula instead of the least squares GAN objective.
[0086]
[0087] Where Dk is the kth sub-discriminator. The reconstruction loss, denoted as Lmel, is defined as the sum of the real sample x and the synthetic sample The L1 distance between the Mel-scale magnitude spectrograms: Feature matching loss, denoted as L feat , calculated as the average distance between the l-th feature maps of the k-th sub-discriminator:
[0088]
[0089] In an embodiment of the present application, an efficient audio data generation method is provided, including: obtaining MEL spectrum information sent by a user terminal; inputting the MEL spectrum information into a first ConvNeXt component to obtain first variable data; performing a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data; inputting the feature fusion data into a second ConvNeXt component to obtain second variable data; performing an inverse fast Fourier transform operation on the second variable data to obtain target audio data. Compared with the prior art, the present application not only achieves audio quality that matches the most advanced, but also significantly improves computational efficiency, and is a full order of magnitude faster than the currently popular time-domain neural vocoder method.
[0090] In some optional implementations of the embodiments of the present application, after the step of acquiring the MEL spectrum information sent by the user terminal, the following steps are also included:
[0091] The MEL spectrum information is completed and denoised based on the autoencoder network.
[0092] In the embodiment of the present application, the autoencoder network is a deep learning model that realizes feature extraction and dimensionality reduction of data by learning the compressed representation (encoding) of the input data and trying to reconstruct the original data (decoding) from the representation. In the field of signal processing, especially in audio signal processing, the autoencoder network can be used to complete and denoise the MEL spectrum information.
[0093] In the embodiment of the present application, information completion means that when the MEL spectrum information is partially missing due to some reason (such as loss or damage during signal transmission), the autoencoder network can predict and complete the missing part by learning the features of the complete MEL spectrum graph. This process is similar to the interpolation or filling technology in image restoration, but more complex and sophisticated.
[0094] In the embodiment of the present application, denoising refers to the fact that noise is a common problem in audio signal processing. By training the autoencoder network, it can learn to extract the features of pure audio signals from the noisy MEL spectrogram. Specifically, the network takes the noisy MEL spectrogram as input and attempts to reconstruct the original MEL spectrogram without noise. In this process, the network automatically ignores or weakens the influence of noise, thereby achieving denoising of the audio signal.
[0095] In an embodiment of the present application, an autoencoder network is used to perform information completion and denoising. The MEL spectrum information is input into the encoder part of the autoencoder to extract its low-dimensional feature representation, and then the complete MEL spectrum information is reconstructed through the decoder part to reduce the information loss and distortion problems introduced during network transmission.
[0096] In some optional implementations of the embodiments of the present application, the step of inputting the MEL spectrum information into the first ConvNeXt component to obtain the first variable data specifically includes the following steps:
[0097] The time key features and frequency key features of the MEL spectrum information in the time dimension and the frequency dimension are extracted according to the convolutional neural network;
[0098] Pooling operation is performed on the time key features and the frequency key features to obtain the first variable data.
[0099] In the embodiment of the present application, in the field of audio signal processing, especially when using deep learning technology to process audio signals, a convolutional neural network (CNN) can extract key features in the time dimension and frequency dimension from MEL spectrum information, which is crucial for subsequent audio classification, recognition or enhancement tasks. The following is an explanation of “extracting time key features and frequency key features of MEL spectrum information in the time dimension and frequency dimension according to a convolutional neural network; performing pooling operations on the time key features and frequency key features.
[0100] In the embodiment of the present application, the time-critical feature refers to the feature that can reflect the change of the audio signal over time in the audio signal processing. These features are usually related to the dynamic characteristics, rhythm and time point of the event of the audio signal. In the MEL spectrum, the time-critical feature can be extracted from the horizontal direction (i.e., the time axis direction) through the convolution operation of the CNN.
[0101] In the embodiment of the present application, the frequency key feature refers to the feature that can reflect the characteristics of the audio signal at different frequencies. These features are usually related to the timbre, pitch and spectrum structure of the audio signal. In the MEL spectrum diagram, the frequency key feature can be extracted from the vertical direction (i.e., the frequency axis direction) through the convolution operation of the CNN.
[0102] In an embodiment of the present application, a convolutional neural network is used to extract local and global features of the completed MEL spectrum information. By setting convolution kernels of different scales, the key features of the MEL spectrum information in the time and frequency dimensions are captured, and the feature dimensions are reduced through pooling operations to obtain a compact feature representation as the first variable data.
[0103] In the embodiment of the present application, the process of extracting the time key features and frequency key features of the MEL spectrum information in the time dimension and the frequency dimension according to the convolutional neural network, and obtaining the first variable data through the pooling operation is a common and effective method in audio signal processing. This method can make full use of the feature extraction capability of CNN, extract useful feature information from the MEL spectrum diagram, and provide strong support for subsequent audio processing tasks.
[0104] In some optional implementations of the embodiments of the present application, the step of performing a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data specifically includes the following steps:
[0105] Calculate the position correlation of the MEL spectrum information and the first variable data to generate an attention weight matrix;
[0106] A weight adjustment operation is performed on the MEL spectrum information and the first variable data according to the attention weight matrix, and a feature fusion operation is performed on the weight-adjusted MEL spectrum information and the first variable data to obtain feature fusion data.
[0107] In an embodiment of the present application, in audio signal processing tasks, especially when using deep learning technology to process complex audio data, calculating the positional correlation between different data and adjusting the weights accordingly can significantly improve the performance of the model. In this process, the attention mechanism plays an important role. The following is an explanation of "calculating the positional correlation of the MEL spectrum information and the first variable data, generating an attention weight matrix; performing weight adjustment operations on the MEL spectrum information and the first variable data according to the attention weight matrix, and performing feature fusion operations on the weight-adjusted MEL spectrum information and the first variable data.
[0108] In the embodiment of the present application, position correlation refers to the correlation between audio signal features at different time points or frequency points in audio signal processing. This correlation can reflect the temporal structure, spectral structure and interaction between different features of the audio signal.
[0109] In an embodiment of the present application, the attention weight matrix: The attention weight matrix is a matrix used to represent the correlation between features at different positions. In the audio signal processing task, the attention weight matrix can be generated by calculating the positional correlation between the MEL spectrum information and the first variable data. Each element of this matrix represents the strength of the correlation between a certain position in the MEL spectrum information and a certain position in the first variable data.
[0110] In an embodiment of the present application, when fusing the MEL spectrum information and the first variable data, an attention mechanism is used to dynamically adjust the weights of the two data. By calculating the correlation between the MEL spectrum information and the first variable data at different positions, an attention weight matrix is generated and applied to the feature representation of the two data to achieve adaptive feature fusion and reduce the impact of dimensionality and scale differences.
[0111] In an embodiment of the present application, the position correlation of MEL spectrum information and the first variable data is calculated and an attention weight matrix is generated. A weight adjustment operation is performed according to the attention weight matrix, and a feature fusion operation is performed on the weighted feature data. This can make full use of the correlation information between different features and improve the performance and robustness of the model.
[0112] In some optional implementations of the embodiments of the present application, before the step of performing an inverse fast Fourier transform operation on the second variable data to obtain the target audio data, the following steps are also included:
[0113] The spectrum distribution and time domain continuity of the second variable data are optimized according to the spectrum constraint technology and the time domain smoothing technology.
[0114] In the embodiments of the present application, in the field of audio signal processing, spectrum constraint technology and time domain smoothing technology are two commonly used optimization methods, which respectively improve the spectrum distribution and time domain continuity of audio signals.
[0115] In the embodiment of the present application, the spectrum constraint technology is a method for optimizing the quality of an audio signal by limiting or adjusting its spectrum distribution. This technology is usually based on a certain spectrum model or prior knowledge, and aims to ensure that the spectrum characteristics of the audio signal meet specific requirements or standards.
[0116] In the embodiment of the present application, the time domain smoothing technology is a method for optimizing the continuity of an audio signal by reducing the sudden change or jitter of the audio signal in the time domain. This technology aims to make the audio signal smoother in time, thereby improving its auditory effect.
[0117] In an embodiment of the present application, before performing the inverse fast Fourier transform, the second variable data is post-processed and optimized. By introducing techniques such as spectral constraints and time domain smoothing, the spectral distribution and time domain continuity of the second variable data are optimized to improve the reconstruction quality in the inverse transform process and reduce sound quality loss and noise interference.
[0118] In an embodiment of the present application, the spectrum distribution and time domain continuity of the second variable data are optimized according to the spectrum constraint technology and the time domain smoothing technology, which can significantly improve the quality and auditory effect of the audio signal and provide strong support for subsequent audio processing tasks.
[0119] In some optional implementations of the embodiments of the present application, the step of performing an inverse fast Fourier transform operation on the second variable data to obtain the target audio data specifically includes the following steps:
[0120] Performing segmentation processing on the second variable data to obtain second variable segmentation data;
[0121] Performing a preprocessing operation on the current second variable segmented data according to the windowing function to obtain windowed segmented data;
[0122] Performing an inverse fast Fourier transform operation on the windowed segmented data to obtain a current time domain signal segment;
[0123] Superimposing adjacent current time domain signal segments according to a preset overlapping ratio to obtain a superimposed time domain signal segment;
[0124] Performing energy normalization operation on the superimposed time domain signal segment to obtain a normalized signal segment;
[0125] Dynamically adjust the window size and overlap ratio of the next segment of second variable segmented data according to the time domain signal energy distribution characteristics of the normalized signal segment, and repeatedly perform preprocessing operations, inverse fast Fourier transform operations, superposition operations, and energy normalization operations;
[0126] When all the windowed segmented data have completed the preprocessing operation, the inverse fast Fourier transform operation, the superposition operation and the energy normalization operation, the target audio data is obtained.
[0127] In the embodiment of the present application, an improved inverse fast Fourier transform algorithm is adopted, and the accuracy and efficiency of the inverse transform are improved by introducing techniques such as windowing function and overlap-addition. The window size and overlap ratio are dynamically adjusted to adapt to different MEL spectrum information characteristics and generate high-quality target audio data.
[0128] In some optional implementations of the embodiments of the present application, after the step of performing an inverse fast Fourier transform operation on the second variable data to obtain the target audio data, the following step is also included:
[0129] Inputting the target audio data into a psychoacoustic model to perform spectrum optimization and energy optimization operations to obtain optimized audio data;
[0130] The optimized audio data is subjected to a noise elimination operation according to an adaptive noise suppression technique to obtain high-quality audio data.
[0131] In an embodiment of the present application, the generated target audio data is post-processed. By introducing a psychoacoustic model, the spectrum and energy of the audio data are optimized according to the auditory characteristics of the human ear. By using adaptive noise suppression technology, the impact of residual noise on sound quality is further reduced, and the final high-quality audio data output is obtained.
[0132] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0133] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0135] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0136] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of an efficient audio data generation device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0137] like Figure 3 As shown, the efficient audio data generating device 200 of the embodiment of the present application includes:
[0138] The spectrum acquisition module 210 is used to acquire the MEL spectrum information sent by the user terminal;
[0139] A first variable acquisition module 220, used to input the MEL spectrum information into the first ConvNeXt component to obtain first variable data;
[0140] A feature fusion module 230 is used to perform a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data;
[0141] A second variable acquisition module 240 is used to input the feature fusion data into the second ConvNeXt component to obtain second variable data;
[0142] The target audio acquisition module 250 is used to perform an inverse fast Fourier transform operation on the second variable data to obtain target audio data.
[0143] In an embodiment of the present application, an efficient audio data generation device 200 is provided, including: a spectrum acquisition module 210, for acquiring MEL spectrum information sent by a user terminal; a first variable acquisition module 220, for inputting the MEL spectrum information into a first ConvNeXt component to obtain first variable data; a feature fusion module 230, for performing feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data; a second variable acquisition module 240, for inputting the feature fusion data into a second ConvNeXt component to obtain second variable data; a target audio acquisition module 250, for performing an inverse fast Fourier transform operation on the second variable data to obtain target audio data. Compared with the prior art, the present application not only achieves matching with the most advanced audio quality, but also significantly improves computational efficiency, and the speed is increased by a full order of magnitude compared with the currently popular time-domain neural vocoder method.
[0144] In some optional implementations of the embodiments of the present application, the first variable acquisition module includes:
[0145] A feature extraction submodule is used to extract the time key features and frequency key features of the MEL spectrum information in the time dimension and the frequency dimension according to the convolutional neural network;
[0146] The pooling submodule is used to perform a pooling operation on the time key features and the frequency key features to obtain the first variable data.
[0147] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.
[0148] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0149] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0150] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the efficient audio data generation method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.
[0151] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for running the efficient audio data generation method.
[0152] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.
[0153] The computer device provided by this application not only achieves audio quality that matches the most advanced, but also significantly improves computational efficiency, increasing the speed by a full order of magnitude compared to the currently popular time-domain neural vocoder method.
[0154] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the efficient audio data generation method as described above.
[0155] The computer-readable storage medium provided by this application not only achieves audio quality that matches the most advanced audio quality, but also significantly improves computational efficiency, increasing the speed by a full order of magnitude compared to the currently popular time-domain neural vocoder method.
[0156] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0157] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. An efficient audio data generation method, characterized in that: The steps include: Obtain MEL spectrum information sent by the user terminal; Inputting the MEL spectrum information into a first ConvNeXt component to obtain first variable data; Performing a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data; Inputting the feature fusion data into a second ConvNeXt component to obtain second variable data; An inverse fast Fourier transform operation is performed on the second variable data to obtain target audio data.
2. The efficient audio data generation method according to claim 1, characterized in that: After the step of obtaining the MEL spectrum information sent by the user terminal, the following steps are also included: The MEL spectrum information is completed and denoised according to the autoencoder network.
3. The efficient audio data generation method according to claim 1, characterized in that: The step of inputting the MEL spectrum information into the first ConvNeXt component to obtain the first variable data specifically includes the following steps: Extracting the time key features and frequency key features of the MEL spectrum information in the time dimension and the frequency dimension according to the convolutional neural network; A pooling operation is performed on the time key feature and the frequency key feature to obtain the first variable data.
4. The efficient audio data generation method according to claim 1, characterized in that: The step of performing a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data specifically includes the following steps: Calculating the position correlation of the MEL spectrum information and the first variable data to generate an attention weight matrix; A weight adjustment operation is performed on the MEL spectrum information and the first variable data according to the attention weight matrix, and a feature fusion operation is performed on the weight-adjusted MEL spectrum information and the first variable data to obtain the feature fusion data.
5. The efficient audio data generation method according to claim 1, characterized in that: Before the step of performing an inverse fast Fourier transform operation on the second variable data to obtain target audio data, the following steps are also included: The spectrum distribution and time domain continuity of the second variable data are optimized according to the spectrum constraint technology and the time domain smoothing technology.
6. The efficient audio data generation method according to claim 1, characterized in that: The step of performing an inverse fast Fourier transform operation on the second variable data to obtain the target audio data specifically includes the following steps: Performing segmentation processing on the second variable data to obtain second variable segmentation data; Performing a preprocessing operation on the current second variable segmented data according to the windowing function to obtain windowed segmented data; Performing an inverse fast Fourier transform operation on the windowed segmented data to obtain a current time domain signal segment; Superimposing adjacent current time domain signal segments according to a preset overlapping ratio to obtain a superimposed time domain signal segment; Performing an energy normalization operation on the superimposed time domain signal segment to obtain a normalized signal segment; Dynamically adjust the window size and overlap ratio of the next segment of second variable segmented data according to the time domain signal energy distribution characteristics of the normalized signal segment, and repeatedly perform the preprocessing operation, the inverse fast Fourier transform operation, the superposition operation and the energy normalization operation; When all the windowed segmented data have completed the preprocessing operation, the inverse fast Fourier transform operation, the superposition operation and the energy normalization operation, the target audio data is obtained.
7. The efficient audio data generation method according to claim 1, characterized in that: After the step of performing an inverse fast Fourier transform operation on the second variable data to obtain target audio data, the following steps are also included: Inputting the target audio data into a psychoacoustic model to perform spectrum optimization and energy optimization operations to obtain optimized audio data; The optimized audio data is subjected to a noise elimination operation according to an adaptive noise suppression technique to obtain high-quality audio data.
8. An efficient audio data generating device, characterized in that: include: A spectrum acquisition module, used to acquire MEL spectrum information sent by a user terminal; A first variable acquisition module, used for inputting the MEL spectrum information into a first ConvNeXt component to obtain first variable data; A feature fusion module, used for performing a feature fusion operation on the MEL spectrum information and the first variable data to obtain feature fusion data; A second variable acquisition module, used for inputting the feature fusion data into a second ConvNeXt component to obtain second variable data; The target audio acquisition module is used to perform an inverse fast Fourier transform operation on the second variable data to obtain target audio data.
9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the efficient audio data generation method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the efficient audio data generation method according to any one of claims 1 to 7.