A music bandwidth extension method based on an auditory perceptual attention generative adversarial network

Abstract

This invention discloses a music bandwidth expansion method based on auditory perception attention generative adversarial networks (GANs). The method comprises two parts: training and prediction. The prediction part uses the trained network model to complete the bandwidth expansion task. The training part includes the following steps: Step 1, data preprocessing: preprocessing the original music dataset to obtain high-frequency music signals and corresponding low-frequency music signals as training sample pairs; Step 2, model architecture design: training the training sample pairs from Step 1 to obtain a GAN model. The GAN model includes a generator for generating sample data and a multi-discriminator for determining the source of the input sample data. This invention incorporates a multi-discriminator into the network model, which to some extent avoids the problem of overly smooth high-frequency components in the expanded spectrogram.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, and more specifically to a method for extending music bandwidth based on auditory perception attention generative adversarial networks. Background Technology

[0002] With changing consumer habits and the rapid development of digital multimedia technology, streaming music services are gaining increasing support and popularity among users. To ensure a smooth user experience, music signals are compressed during transmission, resulting in the loss of high-frequency information. Music lacking high-frequency components suffers a significant reduction in its layering and naturalness, leading to a substantial decrease in overall listening quality.

[0003] Bandwidth extension technology can recover lost high-frequency information from lossy compressed signals by generating high-frequency signals from low frequencies, making the audio sound brighter. Existing technologies typically use convolutional neural networks to accomplish bandwidth extension tasks, generating higher quality signals than traditional bandwidth extension models such as Gaussian mixture models and hidden Markov models. However, the spectrogram of the generated signal is too smooth in the high-frequency part, lacking detailed information such as harmonic structure, resulting in a poor listening experience.

[0004] Existing speech bandwidth extension methods use attention-based generative adversarial networks (GANs). The architecture of GANs is also generator and multiple discriminators, but they generally use a single attention mechanism. As a result, the spectrogram of the generated signal is not detailed enough and cannot be suitable for music signals containing various instrument sounds, which can easily lead to a large loss of high-frequency energy. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention proposes a music bandwidth extension method based on auditory perception attention generative adversarial networks, which to some extent overcomes the shortcomings of existing technologies, further improves the sound quality of bandwidth extension, and generates high-quality music signals.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a music bandwidth expansion method based on auditory perception attention generative adversarial networks, comprising two parts: training and prediction. The training part includes the following steps:

[0007] Step 1, data preprocessing: The original music dataset is preprocessed. The preprocessing steps include normalization, downsampling and interpolation upsampling operations to obtain high-frequency music signals and corresponding low-frequency music signals as training sample pairs.

[0008] Step 2: Design the model architecture. Train the training sample pairs from Step 1 to obtain the Generative Adversarial Network (GAN) model. The GAN model includes a generator G for generating sample data and a multi-discriminator D for determining the source of the input sample data. The specific steps are as follows:

[0009] S2.1, Constructing the Feature Extraction Layer in Generator G: The generator's role is to generate as realistic a high-frequency music signal as possible. The generator network consists of three main parts: a feature extraction layer, an auditory perception attention layer, and a high-frequency recovery layer. The feature extraction layer includes five sequentially connected feature extraction blocks, forming a progressively smaller structure, with the feature map size decreasing accordingly. After inputting the low-frequency music signal X obtained in step 1 into the feature extraction layer, the feature tensor X is obtained. FE ;

[0010] S2.2, Constructing the auditory perception attention layer in generator G: The auditory perception attention layer includes residual connections from input to final output, one-dimensional convolution, framing, merging, splicing, and harmonic, inter-frame, and intra-frame attention mechanisms. The feature tensor X extracted in S2.1... FE First, it goes through a one-dimensional strided convolutional layer. The purpose of the one-dimensional strided convolution is to further process X... FE The feature map size is compressed to half, and then passed through a ReLU activation function. Before using the attention mechanism, X... FE By using three paths consisting of one-dimensional convolutions, the channel size is adjusted to 1 / 3 of its original size (rounded down). For inter-frame and intra-frame attention mechanisms, the feature tensor needs to be segmented into frames and concatenated into a three-dimensional tensor, finally yielding three feature tensor components T. h T ia T ir , will T h T ia T ir The inputs are respectively fed into the harmonic, inter-frame, and intra-frame attention mechanisms to obtain T. h '、T i ' a T i ' r For T i ' a and T i ' r First, merge them (i.e., the inverse operation of framing and joining), then combine them with T. h 'Concatenate along the channel dimension, then perform a one-dimensional convolution. The purpose of the one-dimensional convolution is to obtain the result with respect to X.' FE Feature tensors X' with the same number of channels FE Finally, add a residual connection to make X FE With X' FE The features are added together, then passed through a one-dimensional strided convolutional layer and a ReLU activation function to restore the feature map size to match X. FE The feature maps are of the same size, resulting in the output X of auditory perception attention. Att ;

[0011] S2.3, Construct the high-frequency recovery layer in generator G: The high-frequency recovery layer consists of five sequentially connected sub-pixel upsampling blocks, forming a progressively enlarged structure to improve the resolution of the feature maps. A skip connection is added between the feature extraction layer and the high-frequency recovery layer, and the corresponding feature maps and feature maps of the same size after passing through the high-frequency recovery layer are concatenated together according to channels; the auditory perception attention output X obtained in step 3 is then used... Att After being input into the high-frequency recovery layer, X' is obtained. Finally, a residual connection is added to add X and X', yielding the final output of the generator.

[0012] S2.4, Constructing the Discriminator D: The discriminator determines whether the input signal is generated data or real data and feeds the result back to the generator. The discriminator consists of four discriminant networks D1, D2, D3, and D4. Each discriminator network has a different input: D1 receives the signal at the original sampling rate, while D2, D3, and D4 receive signals downsampled by 2, 4, and 8 times, respectively. Downsampling is achieved through average pooling. The signal obtained in S2.3... The original high-frequency music signal Y is input to the discriminator D, and the output D(Y) is obtained. They represent the probability that the input signal is real data;

[0013] S2.5 Generator Loss Calculation: Loss Function L of Generator G G Including energy loss L energy Fourier transform loss L F and combat losses L adv Energy loss L energy and Fourier transform loss L F Obtained from S2.3 The adversarial loss L is calculated from the corresponding original high-frequency music signal Y. adv D(Y) obtained from S2.4 and The generator loss L was calculated. G Then, backpropagate and update the generator parameters;

[0014] S2.6: Discriminator Loss Calculation: The loss function L of discriminator D D The sum of the adversarial losses of all discriminative networks, D(Y) obtained from S2.4 and The discriminator loss L was calculated. D Then, backpropagation is performed to update the discriminator parameters;

[0015] The prediction section includes the following steps:

[0016] Step 1: Discard the discriminator and save the parameters of the generator part as the trained network model M;

[0017] Step 2: Bandwidth extension using the trained network model M: Extend the bandwidth of the low-frequency music signal X to be extended. M The input is fed into network model M to obtain a high-frequency music signal after bandwidth expansion.

[0018] The beneficial effects of this invention are as follows: An auditory perception attention layer is embedded between the feature extraction layer and the high-frequency recovery layer of the generator to enhance the model's ability to capture features. Compared to the single attention mechanism used in existing bandwidth expansion methods, the auditory perception attention layer of this invention contains three branches, each extracting and highlighting important features of the music signal from different angles: Harmonic attention weights each feature map, highlighting important signal features such as harmonics in the spectrogram and increasing the naturalness of the generated music; inter-frame attention models the information between frames, enabling the model to better focus on the long-term correlation of the music signal; and intra-frame attention focuses on important information within each frame, making the spectrogram more detailed and the listening experience more refined. An energy loss function is added to the generator's loss function to increase the energy of the high-frequency components of the music signal, making the music sound fuller. Furthermore, a multi-discriminator is added to the network model, applying the multi-discriminator from the field of speech synthesis to music bandwidth expansion, and the number and structure of the multi-discriminators are adjusted to better suit the generator in this invention. Multiple discriminators have receptive fields of different sizes, making them more powerful and avoiding, to some extent, the problem of overly smooth high-frequency parts of the expanded spectrogram. Attached Figure Description

[0019] Figure 1 The overall flowchart of a music bandwidth extension method based on auditory perception attention generative adversarial network provided by the present invention;

[0020] Figure 2 This is a schematic diagram of the generator network structure in this invention;

[0021] Figure 3 This is a schematic diagram of the discriminator network in this invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0023] This embodiment of a music bandwidth expansion method based on auditory perception attention generative adversarial networks includes two parts: training and prediction. The training part provides a bandwidth expansion model by designing a reasonable network structure and objective function. The prediction part completes the bandwidth expansion task by using the trained network model. The training part includes the following steps:

[0024] Step 1: Data preprocessing: The original music dataset is preprocessed. The preprocessing steps include normalization, downsampling and interpolation upsampling, etc., to obtain high-frequency music signal Y and corresponding low-frequency music signal X as training sample pairs.

[0025] The preprocessing process includes the following steps:

[0026] Step 1.1: Randomly select a music segment with a length (i.e., the number of sampling points) of 8000 from the original music dataset with a bandwidth of 8000Hz and a sampling rate of 16000Hz, and normalize it to obtain the high-frequency music signal Y, so that its amplitude range is within the interval [-1,1].

[0027] Step 1.2: Downsample the high-frequency music signal Y to 1 / 4 of the original sampling rate. Since the generator network design requires the input data and output data to have the same shape, the downsampled music signal needs to be interpolated and upsampled to the original sampling rate to obtain the low-frequency music signal X.

[0028] Step 2: Design the model architecture and train the training sample pairs from Step 1 to obtain the Generative Adversarial Network (GAN) model. The GAN model includes a generator G for generating sample data and a multi-discriminator D for determining the source of the input sample data. The specific steps are as follows:

[0029] The feature extraction layer in the generator G is constructed as follows: The generator's role is to generate as realistic a high-frequency music signal as possible. The generator network consists of three main parts: a feature extraction layer, an auditory perception attention layer, and a high-frequency recovery layer. The feature extraction layer includes five sequentially connected feature extraction blocks: feature extraction block 1, feature extraction block 2, feature extraction block 3, feature extraction block 4, and feature extraction block 5, forming a progressively smaller structure with decreasing feature map size. Each feature extraction block defines a one-dimensional strided convolution and a ReLU activation function. The number of convolution kernels in each block are 128, 256, 512, 512, and 512, respectively, and the kernel lengths are 65, 33, 17, 9, and 9, respectively, with a stride of 2 for all blocks. After inputting the low-frequency music signal X obtained in the previous steps into the feature extraction layer, the feature tensor X is obtained. FE ;

[0030] Constructing the auditory perception attention layer in generator G: The auditory perception attention layer includes residual connections from input to final output, one-dimensional convolution, framing, merging, stitching, and harmonic, inter-frame, and intra-frame attention mechanisms. The feature tensor X extracted in the preceding steps... FE First, it goes through a one-dimensional strided convolutional layer. The purpose of the one-dimensional strided convolution is to further process X... FE The feature map size is compressed to half, and then passed through a ReLU activation function. Before using the attention mechanism, X... FE By using three paths consisting of one-dimensional convolutions, the channel size is adjusted to 1 / 3 of its original size (rounded down). For inter-frame and intra-frame attention mechanisms, the feature tensor needs to be segmented into frames and concatenated into a three-dimensional tensor, finally yielding three feature tensor components T. h T ia T ir , will T h T ia T ir The inputs are fed into the harmonic, inter-frame, and intra-frame attention mechanisms respectively, to obtain T′. h 、T′ ia 、T′ ir For T′ ia and T′ ir First, merge them (i.e., the inverse operation of framing and joining), then combine them with T′. h The concatenation is performed along the channel dimension, followed by a one-dimensional convolution. The purpose of the one-dimensional convolution is to obtain the result with respect to X. FE Feature tensors X' with the same number of channels FE Finally, add a residual connection to make X FE With X' FE The features are added together, then passed through a one-dimensional strided convolutional layer and a ReLU activation function to restore the feature map size to match X. FE The feature maps are of the same size, resulting in the output X of auditory perception attention. Att ;

[0031] The auditory perception attention layer includes residual connections from input to final output, one-dimensional convolution, framing, merging, stitching, and harmonic, inter-frame, and intra-frame attention mechanisms. The feature tensor X extracted in the preceding steps... FE First, a one-dimensional strided convolution is applied, with 512 kernels and a stride of 2, to further process X. FE The feature map size is compressed to half, and then passed through a ReLU activation function. Before using the attention mechanism, X... FEBy using three paths consisting of one-dimensional convolutions, the channel size is adjusted to 1 / 3 of its original size (rounded down). For inter-frame and intra-frame attention mechanisms, a 30% overlap setting is also required to segment the feature tensor into frames, concatenate them into a three-dimensional tensor, and finally obtain three feature tensor components T. h T ia T ir , will T h T ia T ir These are respectively input into the harmonic, inter-frame, and intra-frame attention mechanisms;

[0032] Harmonic attention first requires processing the feature T obtained in step 3.1. h Perform global average pooling and global max pooling to obtain the pooled outputs respectively. and Then and The data is continuously fed into linear layers a and b with shared parameters, and the two outputs are summed. Finally, the harmonic attention weight Att is obtained by passing the sigmoid function. h The weighting coefficient Att h With input features T h Multiplying these together yields the harmonic attention output T′. h The calculation formula is as follows:

[0033]

[0034] T′ h =Att h T h ,

[0035] In the formula, W a and W b These represent the parameters of the two linear layers;

[0036] Inter-frame attention first applies the features T obtained in the preceding steps ia Multiply by the three matrices A respectively q A k A v The obtained Q, K, and V are used as inputs to the attention function to calculate the inter-frame attention weights Att. ia Finally, the weighting coefficient Att is... ia Multiply by V and input into a linear layer to obtain the output T′ of the inter-frame attention. ia The calculation formula is as follows:

[0037]

[0038] T′ ia =W(Att)ia V),

[0039] In the formula, represents the scaling factor, and W represents the linear layer weight parameters;

[0040] Intra-frame attention first defines a depthwise separable convolution, and the features T obtained from the preceding steps are... ir After convolution, max pooling and average pooling are performed along the channel dimension to obtain... and Then and The channels are concatenated. Then, a one-dimensional convolutional layer reduces the channel size from 2 to 1. Finally, the intra-frame attention weights Att are obtained through the Sigmoid function. ir The weighting coefficient Att ir With input features T ir Multiply to obtain the output T′ of the intra-frame attention. ir The calculation formula is as follows:

[0041]

[0042] T′ ir =Att ir T ir ,

[0043] In the formula, C represents the weight parameters of the one-dimensional convolutional layer;

[0044] Regarding T′ obtained in the aforementioned steps ia And T′ obtained from the aforementioned steps ir First, merge them (i.e., the inverse operation of framing and joining), then combine them with T′ obtained from the previous steps. h The concatenation is performed along the channel dimension, followed by a one-dimensional convolution with 512 kernels. Its purpose is to obtain the result with respect to X. FE Feature tensors X' with the same number of channels FE Finally, add a residual connection to make X FE With X' FE The features are added together, then passed through a one-dimensional strided convolutional layer and a ReLU activation function to restore the feature map size to match X. FE The feature maps are of the same size, resulting in the output X of auditory perception attention. Att ;

[0045] The high-frequency recovery layer in generator G is constructed as follows: The high-frequency recovery layer consists of five sequentially connected sub-pixel upsampling blocks, forming a progressively enlarging structure to improve the resolution of the feature maps. The five sub-pixel upsampling blocks are designated as Sub-pixel Upsampling Block 1, Sub-pixel Upsampling Block 2, Sub-pixel Upsampling Block 3, Sub-pixel Upsampling Block 4, and Sub-pixel Upsampling Block 5, forming a progressively enlarging structure to improve the resolution of the feature maps. Each sub-pixel upsampling block defines a one-dimensional convolution, a ReLU activation function, and a sub-pixel reconstruction layer. The number of convolutional kernels in each sub-pixel upsampling block are 1024, 1024, 512, 256, and 1, respectively; the kernel lengths are 9, 17, 33, 65, and 9, respectively; and the stride is 1 for all blocks. A skip connection is added between the feature extraction layer and the high-frequency recovery layer, where the corresponding feature map and a feature map of the same size after passing through the high-frequency recovery layer are concatenated by channel. The auditory perception attention output X obtained in the preceding steps is then processed. Att After being input into the high-frequency recovery layer, X' is obtained. Finally, a residual connection is added to add X and X', yielding the final output of the generator.

[0046] Construct a discriminator D: The discriminator determines whether the input signal is generated data or real data and feeds the result back to the generator. The discriminator consists of four discriminant networks D1, D2, D3, and D4. Each discriminator network has a different input: D1 receives the signal at the original sampling rate, while D2, D3, and D4 receive signals downsampled by 2, 4, and 8 times, respectively. Downsampling is achieved through average pooling. The signals obtained in the preceding steps... The original high-frequency music signal Y is input to the discriminator D, and the output D(Y) is obtained. They represent the probability that the input signal is real data;

[0047] Generator loss calculation: Loss function L of generator G G Including energy loss L energy Fourier transform loss L F and combat losses L adv Energy loss L energy and Fourier transform loss L F The results obtained from the aforementioned steps The adversarial loss L is calculated from the corresponding original high-frequency music signal Y. adv D(Y) and obtained from the aforementioned steps The generator loss L was calculated. G Then, backpropagation is performed to update the generator parameters; the loss function of the generator G is specifically expressed as follows:

[0048] L G =L adv +αL energy+βL F ,

[0049] In the formula, α and β are weights, α = 0.0015, β = 0.3;

[0050]

[0051] In the formula, Y(·) represents the amplitude, and N represents the number of sampling points;

[0052]

[0053] In the formula, ||·|| F Let |·|1 represent the Frobenius norm, |STFT(·)| represent the L1 norm, |STFT(·)| represent the amplitude value of the short-time Fourier transform, and N represent the number of elements in the amplitude value.

[0054]

[0055] In the formula, M represents the number of short-time Fourier transform losses with different parameters;

[0056]

[0057] In the formula, K represents the number of discriminant networks;

[0058] Discriminator loss calculation: the loss function L of discriminator D D The sum of the adversarial losses of all discriminative networks, D(Y) and the sum obtained from the preceding steps The discriminator loss L was calculated. D Then, backpropagation is performed to update the discriminator parameters; the loss function of discriminator D is specifically expressed as follows:

[0059]

[0060] In the formula, k represents the index of the discriminant network, k = 1, 2, 3, 4;

[0061] The prediction section includes the following steps:

[0062] Step 1: Discard the discriminator and save the parameters of the generator part as the trained network model M;

[0063] Step 2: Bandwidth expansion using the trained network model M: Prepare a low-frequency music signal X with a bandwidth of 2000Hz. M The signal is then input into the network model M to obtain the high-frequency music signal after bandwidth expansion. Its bandwidth is 8000Hz.

[0064] This invention provides a music bandwidth expansion method based on auditory perception attention generative adversarial networks (GANs). The network model of this method consists of a generator and a multi-discriminator. The generator includes a feature extraction layer, an auditory perception attention layer, and a high-frequency recovery layer, used to generate more detailed music signals. The multi-discriminator contains four discriminator networks, each judging input signals with different sampling rates and feeding the judgment results back to the generator. Energy loss in the objective function is used to enhance the energy of the high-frequency components, and Fourier transform loss is used to capture the frequency distribution of the music signal. After bandwidth expansion, the high-frequency music signal exhibits a clear harmonic structure, and the high-frequency components of the spectrogram are more detailed. Compared to convolutional neural networks, this method improves the signal-to-noise ratio by 1.2 dB and reduces the log-spectral distance by 18% in objective metrics. In terms of subjective scoring, the MOS (Mean Opinion Score) is improved by 0.26 points.

[0065] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A music bandwidth expansion method based on auditory perception attention generative adversarial networks, comprising two parts: training and prediction. The prediction part uses the trained network model to complete the bandwidth expansion task, characterized in that... The training part includes the following steps: Step 1, data preprocessing, preprocessing the original music dataset to obtain high-frequency music signals and corresponding low-frequency music signals as training sample pairs; Step 2: Design the model architecture. Train the training sample pairs from Step 1 to obtain the Generative Adversarial Network (GAN) model. The GAN model includes a generator for generating sample data and a multi-discriminator for determining the source of the input sample data. The generator includes a feature extraction layer, an auditory perception attention layer, and a high-frequency recovery layer. The feature extraction layer gradually reduces the size of the feature map and extracts high-level features. The auditory perception attention layer continues to process the extracted high-level features and highlights important features from three perspectives: harmonic attention, inter-frame attention, and intra-frame attention, to ensure the auditory perception quality of the generated music. The high-frequency recovery layer restores the reduced feature map to the size of the original input feature map and finally outputs the expanded music signal. The construction of the auditory perception attention layer includes residual connections from input to final output, one-dimensional convolution, framing, merging, splicing, and harmonic, inter-frame, and intra-frame attention mechanisms, specifically for low-frequency music signals. After being input into the feature extraction layer, the feature tensor is obtained. Feature tensor First, it goes through a one-dimensional strided convolutional layer. The purpose of the one-dimensional strided convolution is to further... The feature map size is compressed to half, then passed through a ReLU activation function before using the attention mechanism. By using three paths consisting of one-dimensional convolutions, the channel size is adjusted to 1 / 3 of its original size and rounded down. For inter-frame and intra-frame attention mechanisms, the feature tensor needs to be segmented into frames and concatenated into a three-dimensional tensor, finally yielding three feature tensor components. , , ,Will , , The inputs are fed into the harmonic, inter-frame, and intra-frame attention mechanisms respectively to obtain the output of the harmonic attention. Inter-frame attention output Intra-frame attention output ;for and First, they are merged, which is the inverse operation of framing and joining, and then they are combined with... The concatenation is performed along the channel dimension, followed by a one-dimensional convolution. The purpose of the one-dimensional convolution is to obtain the result. Feature tensors with the same number of channels Finally, add a residual join to make and The features are added together, then passed through a one-dimensional strided convolutional layer and a ReLU activation function to restore the feature map size to match the given size. The feature maps are of the same size, resulting in the output of auditory perception attention. .

2. The music bandwidth extension method based on auditory perception attention generative adversarial networks according to claim 1, characterized in that, The preprocessing in step 1 includes normalization, downsampling, and interpolation upsampling operations.

3. The music bandwidth extension method based on auditory perception attention generative adversarial networks according to claim 2, characterized in that, The feature extraction layer in step 2 includes five feature extraction blocks connected in sequence, with an overall structure that gradually shrinks, and the audio feature map becomes smaller accordingly.

4. The music bandwidth extension method based on auditory perception attention generative adversarial networks according to claim 3, characterized in that, The high-frequency recovery layer comprises five sub-pixel upsampling blocks connected in sequence, forming a gradually enlarging structure, which in turn increases the size of the audio feature map.

5. The music bandwidth extension method based on auditory perception attention generative adversarial networks according to claim 4, characterized in that, The multi-discriminator consists of four discriminant networks.

6. The music bandwidth extension method based on auditory perception attention generative adversarial networks according to claim 5, characterized in that, The method also includes a generator loss calculation step, in which the proposed energy loss is added to the generator's loss function.

7. The music bandwidth extension method based on auditory perception attention generative adversarial networks according to claim 6, characterized in that, The method also includes a discriminator loss calculation step.

Images