Sound coder audio generation method and device based on frequency band segmentation, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a vocoder audio generation method based on frequency band segmentation, which converts an input audio signal into a Log Mel spectrum, and performs primary processing on a spectrum signal through a one-dimensional convolutional network. After up-sampling processing is carried out on the signals through transposition convolution, features of different frequency bands are extracted and processed through a multi-receptive-field module. In the frequency band segmentation process, a signal is segmented into a plurality of tensors in the channel dimension, and each tensor corresponds to a specific frequency band range and is independently processed. And the PQMF synthesis filter bank is used for carrying out up-sampling on the signals with the unified number of channels again. And after the signals are combined, processing a finally generated audio signal through another multi-receptive field module. According to the method, tensors of different frequency bands are processed through frequency band segmentation and an independent multi-receptive-field module, and the frequency aliasing phenomenon introduced in the up-sampling process by transposition convolution is reduced, so that the quality and naturalness of audio generation are improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a method, device, equipment and storage medium for generating audio of a vocoder based on frequency band segmentation. Background Art

[0002] In the audio processing systems in the financial field, technologies such as speech recognition and speech synthesis are gradually being widely used in various business scenarios such as intelligent customer service, voice interaction, and risk management. In order to improve the accuracy and response speed of voice interaction, the quality of audio signal processing is crucial. However, the current speech synthesis technology in financial scenarios still faces some challenges, especially in the process of audio restoration. How to accurately reconstruct the phase information of the audio signal has always been a difficulty in the industry.

[0003] In audio signal processing, vocoders, as a system for synthesizing or reconstructing speech signals, are widely used in intelligent speech systems in the financial field. However, the traditional Griffin-Lim algorithm is one of the most common phase recovery methods, but this method mainly targets linear amplitude spectra and does not work well under Mel-spectrogram. This brings significant limitations to intelligent speech synthesis in financial scenarios, such as voice assistants and automatic customer service systems. Due to the limitations of the Griffin-Lim algorithm in phase restoration, the speech generation effect is poor and the generated audio quality is low, which makes it difficult to meet the financial industry's requirements for high-quality and high-authenticity speech synthesis.

[0004] In recent years, with the development of deep learning technology, vocoders based on generative adversarial networks (GANs) have shown significant advantages in speech synthesis. GAN gradually generates high-quality audio during the learning process through adversarial training between the generator and the discriminator. However, in existing financial intelligent speech systems, the generator structure of the GAN vocoder usually uses transposed convolution for upsampling, but because it introduces aliased frequency information in the frequency dimension, defects such as frequency distortion often appear in the final generated audio. This frequency aliasing problem seriously affects the naturalness and comprehensibility of speech synthesis, and reduces the customer experience of financial voice interaction systems.

[0005] In addition, as the complexity of financial services increases, voice systems need to process large amounts of user data and complex language models, which places higher demands on the real-time and efficiency of speech synthesis. However, existing audio generation technologies still face bottlenecks in efficiently processing large amounts of audio data. Therefore, how to effectively solve the phase recovery problem in audio generation in financial scenarios, reduce frequency aliasing, and improve the quality and efficiency of speech synthesis remains a technical challenge to be solved in this field. Summary of the invention

[0006] The main purpose of the present invention is to provide a method, device, equipment and storage medium for generating audio of a vocoder based on frequency band segmentation, aiming to solve the technical problem that when upsampling by transposed convolution is performed in the prior art, aliased frequency information will be generated in the frequency dimension of the generated audio signal, resulting in a decrease in audio quality and authenticity.

[0007] To achieve the above object, the present invention provides a vocoder audio generation method based on frequency band segmentation, comprising:

[0008] Acquire an original audio signal, and convert the original audio signal into a Log mel spectrum, and regard the frequency dimension of the Log mel spectrum as a one-dimensional signal of multiple frequency channels;

[0009] Processing the Log Mel spectrum through a front network including a one-dimensional convolutional layer without non-linear activation;

[0010] The processed signal is passed through multiple upsampling processing modules in sequence, each of which performs upsampling through transposed convolution and uses a multi-receptive field module to perform signal processing;

[0011] After the transposed convolution and multi-receptive field module processing, the upsampled signal is split into multiple tensors in the channel dimension according to a predetermined ratio, and each tensor corresponds to a different frequency band;

[0012] The segmented tensors are processed independently through multi-receptive field modules with unshared parameters, and the number of channels is unified through convolution operations;

[0013] Use PQMF synthesis filter bank to upsample tensors with uniform number of channels;

[0014] Combine the upsampled tensors into one signal and process it through another multi-receptive field module;

[0015] Process the combined signal through a post-network consisting of a one-dimensional convolutional layer and an activation function to adjust the channel dimension;

[0016] The processed signal is passed through the activation function to obtain the generated target audio signal.

[0017] Furthermore, to achieve the above object, the present invention provides a vocoder audio generation device based on frequency band division, comprising:

[0018] An input module is used to obtain an original audio signal, convert the original audio signal into a Log mel spectrum, and regard the frequency dimension of the Log mel spectrum as a one-dimensional signal of multiple frequency channels;

[0019] A front-end network module, used for processing the Log Mel spectrum through a front-end network including a one-dimensional convolutional layer and without non-linear activation;

[0020] An upsampling module is used to sequentially pass the processed signal through multiple upsampling processing modules, each of which performs upsampling through transposed convolution and uses a multi-receptive field module to perform signal processing;

[0021] The frequency band segmentation module is used to segment the upsampled signal into multiple tensors according to a predetermined ratio in the channel dimension after being processed by the transposed convolution and multi-receptive field modules, each tensor corresponding to a different frequency band;

[0022] A multi-receptive field independent processing module, used for independently processing the segmented tensors through multi-receptive field modules whose parameters are not shared, and unifying the number of channels through a convolution operation;

[0023] PQMF upsampling module, used to upsample tensors with uniform number of channels using PQMF synthesis filter banks;

[0024] The signal merging and processing module is used to merge the upsampled tensors into one signal and process it through another multi-receptive field module;

[0025] A post-network module, used to process the combined signal through a post-network including a one-dimensional convolutional layer and an activation function to adjust the channel dimension;

[0026] The output module is used to pass the processed signal through the activation function to obtain the generated target audio signal.

[0027] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a vocoder audio generation program based on frequency band segmentation stored in the memory and executable on the processor, wherein the vocoder audio generation program based on frequency band segmentation, when executed by the processor, implements the steps of the vocoder audio generation method based on frequency band segmentation as described above.

[0028] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which is stored a vocoder audio generation program based on frequency band segmentation, and when the vocoder audio generation program based on frequency band segmentation is executed by a processor, the steps of the vocoder audio generation method based on frequency band segmentation as described above are implemented.

[0029] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a vocoder audio generation method based on frequency band segmentation, which converts the input audio signal into a Log Mel spectrum and performs preliminary processing on the spectrum signal through a one-dimensional convolutional network. After the signal is upsampled by transposed convolution, the multi-receptive field module is used to extract and process the features of different frequency bands. In the process of frequency band segmentation, the signal is divided into multiple tensors in the channel dimension, each tensor corresponds to a specific frequency band range, and is processed independently. The PQMF synthesis filter group is used to upsample the signal with a unified number of channels again to ensure that the high-frequency and low-frequency signals are effectively restored. After the signal is merged, the final generated audio signal is processed by another multi-receptive field module. The present invention processes tensors of different frequency bands through frequency band segmentation and independent multi-receptive field modules, reduces the frequency aliasing phenomenon introduced by transposed convolution in the upsampling process, thereby improving the quality and naturalness of audio generation, ensuring the clarity and efficiency of speech synthesis, and making the application of generated audio in scenarios such as finance and voice interaction more accurate and effective. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0031] Figure 1 A schematic diagram of an application environment of a vocoder audio generation method based on frequency band division in one embodiment of the present invention;

[0032] Figure 2 It is a flow chart of an embodiment of a method for generating audio of a vocoder based on frequency band division according to the present invention;

[0033] Figure 3 It is a functional module diagram of a preferred embodiment of a vocoder audio generation device based on frequency band division of the present invention;

[0034] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0035] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0036] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0037] The vocoder audio generation method based on frequency band division provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the user end communicates with the server end through a network. The server end can convert the input audio signal into a Log Mel spectrum through the user end, and preliminarily process the spectrum signal through a one-dimensional convolutional network. After the signal is upsampled by transposed convolution, the multi-receptive field module is used to extract and process the features of different frequency bands. In the process of frequency band segmentation, the signal is divided into multiple tensors in the channel dimension, each tensor corresponds to a specific frequency band range, and is processed independently. The signal with a unified number of channels is upsampled again using a PQMF synthesis filter group to ensure that the high-frequency and low-frequency signals are effectively restored. After the signal is merged, the final generated audio signal is processed by another multi-receptive field module. The present invention processes tensors of different frequency bands through frequency band segmentation and independent multi-receptive field modules, reduces the frequency aliasing phenomenon introduced by transposed convolution in the upsampling process, thereby improving the quality and naturalness of audio generation, ensuring the clarity and efficiency of speech synthesis, and making the generated audio more accurate and effective in scenarios such as finance and voice interaction. Among them, the user end can be but not limited to various personal computers, laptops, smart phones, tablets and portable wearable devices. The server side can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0038] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a method for generating vocoder audio based on frequency band segmentation provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0039] like Figure 2 As shown, the vocoder audio generation method based on frequency band segmentation proposed by the present invention comprises the following steps:

[0040] S10, acquiring an original audio signal, and converting the original audio signal into a Log mel spectrum, and treating the frequency dimension of the Log mel spectrum as a one-dimensional signal of multiple frequency channels;

[0041] In this embodiment, the original audio signal refers to the sound data collected by an audio device (such as a microphone or other recording device). This audio signal is generally an analog signal and needs to be digitized, usually including sampling, quantization and encoding, to finally obtain discrete digital audio data. The original audio signal is obtained using an audio acquisition device, and the analog signal is converted into a digital signal through an ADC (analog-to-digital converter). Digital audio signals are usually captured using a specific sampling rate (such as 44.1kHz or 48kHz) and a quantization bit number (such as 16 bits or 24 bits). The acquired digital audio signal is represented in the time domain and is subsequently processed.

[0042] Log Mel spectrum is a way to represent the frequency content of audio signals, mainly used for audio signal feature extraction. It is a spectral representation of audio signals, processed by a Mel filter bank and logarithmically transformed, making it more consistent with the human ear's perception of sound. Perform a short-time Fourier transform on the audio signal to convert the time domain signal into a frequency domain signal to obtain frequency distribution information. Fourier transform can generate the amplitude spectrum and phase spectrum of the audio. The spectrum is processed through a series of Mel-scale filters, which are more densely distributed in the low frequency band, reflecting the human ear's greater sensitivity to low frequencies. The STFT spectrum is passed through a Mel filter to obtain a Mel spectrum. In order to conform to the human ear's perception of sound loudness, the Mel spectrum is logarithmically operated to generate a Log Mel spectrum. This step not only compresses the dynamic range of the value, but also increases the importance of the high-frequency part.

[0043] The frequency dimension of the Log Mel spectrum is a discrete frequency point, each of which represents a specific frequency intensity. By treating these frequency points as multiple channels, the entire spectrum is converted into a multi-channel one-dimensional signal. This process is to match the frequency information with the input format of the neural network. The frequency dimension of the Log Mel spectrum is regarded as similar to the "channel" in image processing, and each channel corresponds to information in a different frequency band. After this processing, the Log Mel spectrum becomes a one-dimensional signal with a width of the frequency dimension and a length of the number of time frames. Through this processing, the Log Mel spectrum can be directly used as input to the convolutional neural network (CNN) or recurrent neural network (RNN) for feature extraction.

[0044] Example description: A financial institution has developed an intelligent voice assistant for customer self-service and transaction execution. When a user issues a voice command, such as "check account balance" or "make a stock purchase transaction", the system first obtains the user's original voice signal through a microphone and converts the signal into a Log mel spectrum. The spectrum is processed through a one-dimensional convolutional network, and the frequency information is regarded as a one-dimensional signal of multiple frequency channels, which is then input into a neural network for speech recognition. The system uses this technology to perform efficient audio feature extraction to ensure accurate recognition of the user's intentions. By inputting the Log mel spectrum into a deep learning model, the system is able to generate text based on the voice and make precise voice feedback, such as "Your balance is 5,000 yuan" or "The stock has been successfully purchased."

[0045] By converting the original audio signal into a Log Mel spectrum and treating the frequency dimension as a one-dimensional signal of multiple channels, the audio features can be extracted more effectively and the input format of the neural network can be matched. This can improve the accuracy of speech recognition, reduce frequency aliasing, and improve the naturalness of the generated audio. It is suitable for scenarios such as intelligent voice assistants and real-time voice notifications.

[0046] S20, processing the Log Mel spectrum through a front network including a one-dimensional convolutional layer without non-linear activation;

[0047] In this embodiment, the one-dimensional convolution layer is a convolution layer for processing sequence data, and its convolution operation is performed only in one dimension, which is suitable for processing sequence data such as audio signals. Compared with the two-dimensional convolution layer (usually used for image processing), the application of the one-dimensional convolution layer in audio processing is more focused on the extraction of time series or frequency information. Here, without nonlinear activation means that in the front network, the output of the convolution layer will not be processed by activation functions such as ReLU or Sigmoid, but will be directly passed to the subsequent layers. This design is mainly to retain the original characteristics of the signal without performing nonlinear transformation on the characteristics.

[0048] The main purpose of processing the Log Mel spectrum through a one-dimensional convolutional layer is to perform preliminary feature extraction on the audio spectrum. The convolution operation can capture the local features in the spectrum, but since no nonlinear activation layer is used, the output of the convolution still maintains a linear relationship, making it easier for subsequent networks to process these signals more flexibly.

[0049] The input Log Mel spectrum is usually a two-dimensional matrix with time frames as columns and frequency channels as rows. Each frame contains the frequency characteristics of the audio signal. By performing a one-dimensional convolution operation on the frequency dimension, the characteristics of the local frequency band in the Mel spectrum can be captured. The convolution kernel slides on the frequency dimension to extract the local pattern of each small frequency band. This method ensures that the convolution layer can retain the local frequency characteristics when processing each time frame.

[0050] Since there is no activation function, the output of the convolution layer remains linear, which means that the output signal will not be compressed or amplified. This design is suitable for processing low-level signal features, retaining the linear information in the original data, and helping subsequent nonlinear networks to better process these signals. The convolved signal is passed to the next processing network (such as a more advanced convolutional layer or a multi-receptive field module), which can further extract high-level feature information.

[0051] Example description: In a bank's intelligent voice assistant, the user issues a command through voice, such as "check the account balance". The system first obtains the user's original voice signal and converts it into a Log mel spectrum. The spectrum is processed using a pre-network containing a one-dimensional convolutional layer without nonlinear activation to ensure that the original features of the user's voice signal are preserved, so that the system can accurately recognize the user's voice command. This not only improves the recognition accuracy, but also enhances the user's interactive experience with the voice assistant.

[0052] By using a pre-network containing a one-dimensional convolutional layer without nonlinear activation to process the Log Mel spectrum, the linear characteristics of the audio signal can be effectively retained, avoiding the problem of feature distortion introduced by nonlinear transformation in the initial processing of the signal. The subsequent network's decoding and processing capabilities for the signal are improved, making the generated or recognized speech signal more accurate and natural.

[0053] S30, passing the processed signal through a plurality of upsampling processing modules in sequence, each upsampling processing module performs upsampling through transposed convolution, and uses a multi-receptive field module to perform signal processing;

[0054] In this embodiment, upsampling refers to increasing the sampling rate of data to improve resolution. Here, transposed convolution (also called deconvolution or reverse convolution) is used to achieve upsampling. The main function of transposed convolution is to insert additional values ​​into the feature map, and perform reverse operations through the convolution kernel and step size to enlarge the originally smaller feature map so that the subsequent network can better process it.

[0055] The application of transposed convolution can restore low-resolution signals to high-resolution signals while retaining some feature information. However, transposed convolution may introduce frequency aliasing problems, which need to be dealt with in subsequent processing steps.

[0056] The Multiple Receptive Field (MRF) module refers to a convolutional structure that captures signal features at different scales. By applying convolution kernels of different receptive field sizes in parallel in the same layer, both local and global features of the signal can be captured. The design of this module can effectively process signals that have been upsampled by transposed convolution and compensate for possible frequency aliasing problems. Each receptive field module can focus on feature information at different scales, thereby improving the ability to express details of the signal.

[0057] The input signal (i.e. the signal processed by the front network) is first upsampled by the transposed convolution layer. The transposed convolution increases the resolution by inserting zero values ​​in the input feature map and expanding the size of the feature map. The convolution kernel and stride of each transposed convolution layer are set to achieve a specific upsampling multiple. For example, a convolution kernel size of 4 and a stride of 2 can enlarge the size of the feature map to twice the original size.

[0058] The upsampled signal is passed to the multi-receptive field module. This module contains multiple parallel convolution kernels, each with a different receptive field size to ensure that features at different scales are captured. In order to reduce the information loss caused by upsampling and convolution, the multi-receptive field module can add the original input signal to the convolution output signal through a residual connection. This operation can enhance local features while preserving details.

[0059] After the transposed convolution and multi-receptive field modules in each upsampling processing module complete the signal processing, the signal is passed to the next upsampling processing module to continue the upsampling process until the required resolution is reached.

[0060] By using transposed convolution for upsampling through multiple upsampling processing modules and combining the signal processing of the multi-receptive field module, it is possible to reduce the frequency aliasing problem caused by upsampling while improving the signal resolution. It enhances the ability to retain audio details and ensures that the generated audio signal is natural and clear, which is particularly suitable for applications in the financial field that requires high-quality voice output.

[0061] S40, after the transposed convolution and multi-receptive field module processing, split the upsampled signal into a plurality of tensors according to a predetermined ratio in the channel dimension, each tensor corresponding to a different frequency band;

[0062] In this embodiment, when processing an audio signal, different frequency bands of the signal are considered as different "channels". In this case, the "channel dimension" refers to the signal representation corresponding to each frequency band. By dividing the signal according to the channel dimension, the system can process the information on each frequency band separately.

[0063] The signal is split into multiple tensors in a predetermined ratio in the channel dimension. Different frequency bands have different importance. Usually, the low-frequency part has a greater impact on the perception of audio, so more channels can be allocated, while the high-frequency part is allocated a relatively small number of channels. This predetermined ratio of division makes the amount of information in each frequency band more balanced.

[0064] Each tensor represents the signal characteristics of a certain frequency band. Dividing the upsampled signal into multiple tensors by frequency helps to process each frequency band independently, which can reduce the information interference between different frequency bands and improve the detail retention ability of the generated signal.

[0065] After the signal is processed by the transposed convolution and multi-receptive field module, it has been upsampled to obtain a higher-resolution audio feature map. These feature maps have not been divided into frequency bands in the channel dimension, and they need to be divided according to the predetermined ratio.

[0066] According to the preset frequency band distribution rules, the signal is divided into tensors corresponding to different frequency bands in the channel dimension. For example, the tensor of the low-frequency part may be allocated more channels, while the high-frequency part is relatively less. The specific division ratio can be set according to application requirements, for example, the low-frequency, medium-frequency and high-frequency tensors are divided according to the ratio of 4:2:1.

[0067] Each tensor after segmentation corresponds to a certain frequency band. Through this frequency band division, the features in each frequency range can be processed independently, ensuring that the features of the low-frequency band and the high-frequency band do not affect each other, maintaining higher audio generation quality.

[0068] Example description: In the financial market volatility notification system, the system needs to generate voice prompts based on market data. After the generated voice signal is processed by upsampling and multi-receptive field modules, the system divides the audio signal into multiple tensors by frequency band, and each tensor represents a different frequency band. The tensors in the low-frequency band are allocated with more channels to ensure the clarity of the voice prompts, while the high-frequency bands retain details. This segmentation method makes the generated voice notifications both natural and clear when notifying users of market fluctuations.

[0069] By dividing the upsampled signal into multiple tensors in the channel dimension according to a predetermined ratio, each tensor corresponds to a different frequency band, which can effectively reduce the information interference between different frequency bands. This frequency band division improves the signal's feature retention ability, ensures that both low-frequency and high-frequency bands can be accurately captured, and generates a more natural and clear audio signal, which is particularly suitable for application in financial voice interaction and market notification scenarios.

[0070] S50, independently processing the segmented tensors through multi-receptive field modules that do not share parameters, and unifying the number of channels through a convolution operation;

[0071] In this embodiment, the multi-receptive field module refers to a convolution layer with different receptive fields, which is used to capture features at different scales. The larger the receptive field, the wider the range of features captured by the convolution layer, and the more suitable it is for capturing global features; the convolution layer with a smaller receptive field focuses on local features.

[0072] Parameter non-sharing means that the tensors of each frequency band are processed in different multi-receptive field modules, and the weights and parameters of these modules are independent of each other. This approach ensures that the feature extraction of different frequency bands does not interfere with each other, and can more accurately extract the corresponding feature information for each frequency band.

[0073] After the signal is divided into frequency bands, each tensor corresponds to a frequency band. Through the multi-receptive field module with unshared parameters, the system can extract features for each frequency band independently, prevent information from different frequency bands from interfering with each other, and ensure that the details of each frequency band can be accurately captured.

[0074] After processing each tensor independently, the system unifies the number of channels of all tensors through a convolution operation. The unification of the number of channels ensures that features of different frequency bands can be effectively merged and further processed in subsequent processing steps.

[0075] Each segmented tensor is input into its independent multi-receptive field module. The receptive field sizes of these modules are different, allowing feature extraction at multiple scales. Parameter non-sharing ensures that the feature extraction of each frequency band is independent, ensuring that the information of different frequency bands does not interfere with each other, and each module focuses on feature extraction of the current frequency band.

[0076] The tensors processed by each multi-receptive field module may have different numbers of channels. The number of channels of each tensor is unified to the same number through one-dimensional convolution operation, which facilitates signal fusion in subsequent steps. The convolution kernel size and step size can be set according to requirements to ensure the uniformity of the number of channels and the integrity of feature information.

[0077] Example description: In the financial voice assistant system, the user issues a voice command such as "check the account balance" or "transfer money". The system first obtains and processes the user's voice signal and divides it into tensors of different frequency bands. Through the multi-receptive field module with unshared parameters, the system is able to process the features of each frequency band independently. The low-frequency band processes the main content in the voice, such as the keyword "account balance", while the high-frequency band is responsible for capturing detailed sound information, such as the clarity and intonation of the voice. After independent processing, the system uses convolution operations to unify the number of channels in each frequency band, so that the signals are merged into a single output signal, and finally generate a natural and smooth voice response. For example, when the user asks "What is my account balance", the system accurately identifies the key content in the voice through this step and quickly generates a voice prompt "Your balance is 5,000 yuan". This processing method ensures the high quality of speech synthesis, allowing customers to obtain accurate and clear feedback.

[0078] By using a multi-receptive field module with unshared parameters to independently process the tensors of each frequency band, the features of each frequency band can be extracted independently and accurately, reducing the mutual interference between different frequency bands. By unifying the number of channels through convolution operations, the processed tensors can be effectively merged for subsequent processing. This improves the efficiency and accuracy of feature extraction of audio signals, which is particularly suitable for speech generation and speech recognition scenarios, making the generated audio more natural and smooth.

[0079] S60, upsampling the tensor with a uniform number of channels using a PQMF synthesis filter bank;

[0080] In this embodiment, PQMF (Pseudo-Quadrature Mirror Filter) is a multi-band filter used to perform frequency band division and reconstruction on a signal. A common application of PQMF in audio processing is to perform efficient frequency division and synthesis on an audio signal.

[0081] The PQMF filter bank is used to upsample the tensor with the same number of channels, ensuring that the generated signal has a higher resolution and retains more frequency information. The filter bank can provide fine frequency band division on the signal spectrum, reduce frequency aliasing problems, and effectively reconstruct the signal.

[0082] Upsampling is a technique to increase the sampling rate of a signal through interpolation and reconstruction. In the audio processing process, the resolution of the signal is increased to make the generated audio signal smoother and more natural. The PQMF synthesis filter bank can reduce artifacts and aliasing during the upsampling process by finely processing the frequency domain of the signal, thereby improving the quality of the generated audio.

[0083] After the signal is processed by the multi-receptive field module, the number of channels has been unified. At this point, the tensor contains feature information of different frequency bands and is ready for further upsampling to improve the resolution of the signal. The PQMF synthesis filter bank processes the tensor, divides the signal into different frequency bands, and upsamples each band separately. The filter bank reduces artifacts and frequency aliasing that may be introduced during the upsampling of the signal by fine filtering and reconstruction of each frequency band. Through the PQMF filter bank, the frequency response of the signal is further optimized during upsampling, ensuring that the naturalness and clarity of the signal are maintained while improving the resolution. After processing by the PQMF filter bank, the resolution of the signal is improved and the frequency characteristics are clearer. The upsampled signal will be used in the subsequent generation process to generate high-quality audio output.

[0084] By using the PQMF synthesis filter bank for upsampling, the audio signal is guaranteed to have a higher resolution while reducing artifacts and frequency aliasing. The generated audio signal is smoother and more natural, improving the overall quality of the speech.

[0085] S70, combining the upsampled tensors into one signal and processing it through another multi-receptive field module;

[0086] In this embodiment, after upsampling, the tensor of each frequency band already contains the features of the corresponding frequency range. Next, the system needs to merge these tensors into a whole signal in the channel dimension. The merged signal contains information of all frequency bands and is a comprehensive expression of the entire audio signal. This ensures that the frequency band features of the segmentation process can be further processed as a unified whole in subsequent steps, rather than separate frequency band information.

[0087] The merged signal is processed again by the multi-receptive field module. The different convolution kernel sizes (i.e. the size of the receptive field) of the multi-receptive field module can capture the local and global characteristics of the signal. At this stage, the module can further enhance the details in the signal and correct the frequency inconsistency that may occur during upsampling or merging. The multi-receptive field module is responsible for processing the characteristics of the overall signal to ensure that the output signal has enough details and global consistency.

[0088] After upsampling and filtering, the tensors of each frequency band are merged in the channel dimension. The merging operation combines the feature information of different frequency bands to form a complete signal, ready for subsequent convolution processing. The merged signal is passed to another multi-receptive field module. This module uses parallel convolution operations and different receptive field sizes to extract multi-scale features of the signal. Small receptive fields can capture local details in the signal, while large receptive fields focus on the overall characteristics of the signal. Through the processing of the multi-receptive field module, the consistency of the signal in the frequency and time dimensions is ensured, while the feature expression of the signal is further improved.

[0089] By combining the upsampled tensors into one signal and processing it again through the multi-receptive field module, the system can capture more detailed features while ensuring signal consistency, thus improving the overall quality of speech generation and making the audio signal more natural and smooth.

[0090] S80, processes the combined signal through a post network including a one-dimensional convolutional layer and an activation function to adjust the channel dimension;

[0091] In this embodiment, a one-dimensional convolution layer is used to process time series data, such as an audio signal. In the merged signal, features may come from different frequency bands, and after passing through the multi-receptive field module, the dimension of the signal still needs to be further adjusted. Through the one-dimensional convolution layer, the system can extract or compress specific frequency features from the merged signal according to different convolution kernel sizes and step sizes, thereby achieving the purpose of adjusting the channel dimension. One-dimensional convolution mainly performs convolution operations on the time or frequency dimension to match the number of channels of the signal with the needs of the subsequent network.

[0092] The main function of the activation function is to introduce nonlinear transformations so that the network can learn more complex mapping relationships. In this step, after the combined signal passes through the convolution layer, an activation function (such as ReLU or tanh) is applied to perform a nonlinear transformation on the output of the signal to ensure that the processed signal has stronger expression and feature extraction capabilities.

[0093] The activation function used can be:

[0094] ReLU (Rectified Linear Unit): Only outputs the positive part, and the negative part is output as zero. It is widely used in deep neural networks.

[0095] tanh: compresses the input value to the range of -1 to 1 and is used for processing symmetric data.

[0096] The channel dimension is the characteristic dimension of the signal. After convolution and activation, the number of channels is adjusted to make the number of channels of the signal match the expected input or output dimension of the network. This step is very critical to ensure that the dimensions of the outputs of different convolutional layers are unified, providing preparation for subsequent processing.

[0097] The combined signal is input to a one-dimensional convolutional layer for processing. By setting different convolution kernel sizes and step sizes, the one-dimensional convolutional layer can adjust the channel dimension and compress or expand specific frequency features. The output of the convolutional layer is processed by an activation function, usually ReLU or tanh activation function. The role of the activation function is to introduce nonlinearity and map the linear output after convolution to a nonlinear output, thereby enhancing the feature expression ability of the signal. After convolution and activation, the number of channels of the signal will be uniformly adjusted to the expected dimension. The adjustment of the number of channels is to ensure that the signal can be further processed or directly output by the subsequent network layer.

[0098] Through the post-network including one-dimensional convolutional layers and activation functions, the system can flexibly adjust the channel dimension of the signal to ensure the naturalness and high quality of the generated audio signal. It can effectively improve the expressiveness of the speech signal, reduce frequency aliasing and artifact problems, and make the generated speech clearer and smoother.

[0099] S90, passing the processed signal through an activation function to obtain a generated target audio signal.

[0100] In this embodiment, the activation function is a mathematical function that is usually used to perform nonlinear mapping on the output of linear transformation in a neural network so that the model can process more complex features. In the process of audio generation, the role of the activation function is to ensure that the generated audio signal is within a specified value range, thereby avoiding distortion of the audio signal due to being too large or too small.

[0101] The activation functions used include:

[0102] tanh: Limits the output to between -1 and 1, suitable for generating symmetric data.

[0103] ReLU (Rectified Linear Unit): The output value greater than zero is retained, while the output value less than zero is set to zero.

[0104] Sigmoid: compresses the output value between 0 and 1 and is often used in classification problems.

[0105] In audio generation, the tanh function is often used because audio signals usually fluctuate within a positive and negative range. Tanh can limit the output range of the signal to ensure that the amplitude of the generated audio does not exceed a certain limit. The signal processed by the activation function can generate the final target audio signal. This step ensures that the value of the audio signal is within a reasonable range and avoids distortion caused by excessive or too small values ​​when generating the signal.

[0106] After the processed signal is processed by the one-dimensional convolution layer and the multi-receptive field module, the signal already contains enough audio features. At this time, the value range of the signal may be too large or too small to directly generate a high-quality audio signal.

[0107] The signal is passed through an activation function (usually tanh) to compress the output of the signal to a range of -1 to 1. This step can control the dynamic range of the generated signal, making the generated audio signal more natural and distortion-free. The choice of activation function is key. Tanh can effectively prevent the generated signal from being over-extended in amplitude, resulting in distortion or volume imbalance. The signal processed by the activation function is the generated target audio signal. This signal can be played through a vocoder or other audio processing equipment to ultimately achieve high-quality audio generation.

[0108] Example description: In a bank's intelligent voice assistant system, customers query account information through voice. When the customer issues a voice command "query account balance", the system first processes the customer's voice signal, and the signal obtained through multiple convolutions and feature extraction is processed by the tanh activation function. This process limits the amplitude of the generated voice signal to the range of -1 to 1, ensuring the natural and smooth voice output. The target audio signal generated by this step contains clear and natural sound feedback: "Your balance is 5,000 yuan", and is fed back to the customer in real time to help customers quickly obtain financial information. This speech generation technology controls the signal amplitude through the tanh activation function, avoiding the problem of distortion or excessive volume of the voice signal, making the generated voice more natural and smooth in actual applications, and improving the user experience.

[0109] By processing the generated signal with an activation function (such as tanh), the dynamic range of the audio signal can be effectively controlled to ensure that the target audio signal generated is natural, clear, and distortion-free. This not only improves the quality of speech synthesis, but also ensures the stability of the audio generation process. It is particularly suitable for scenarios that require high-quality audio generation, such as financial voice assistants and real-time market notification systems.

[0110] The present invention relates to the field of artificial intelligence technology and financial technology, and discloses a vocoder audio generation method based on frequency band segmentation, which converts the input audio signal into a Log Mel spectrum and performs preliminary processing on the spectrum signal through a one-dimensional convolutional network. After the signal is up-sampled by transposed convolution, the multi-receptive field module is used to extract and process the features of different frequency bands. In the process of frequency band segmentation, the signal is divided into multiple tensors in the channel dimension, each tensor corresponds to a specific frequency band range, and is processed independently. The PQMF synthesis filter group is used to up-sample the signal with a unified number of channels again to ensure that the high-frequency and low-frequency signals are effectively restored. After the signal is merged, the final generated audio signal is processed by another multi-receptive field module. The present invention processes tensors of different frequency bands through frequency band segmentation and independent multi-receptive field modules, reduces the frequency aliasing phenomenon introduced by transposed convolution in the up-sampling process, thereby improving the quality and naturalness of audio generation, ensuring the clarity and efficiency of speech synthesis, and making the application of generated audio in scenarios such as finance and voice interaction more accurate and effective.

[0111] In one embodiment, the above S20 includes:

[0112] S201, processing the Log Mel spectrum through a one-dimensional convolution layer, wherein the one-dimensional convolution layer has a predetermined convolution kernel size and step size;

[0113] S202, not using a nonlinear activation function after the one-dimensional convolution layer, and using the output of the one-dimensional convolution layer as a processed signal;

[0114] S203, performing batch normalization processing on the processed signals, and transmitting the processed signals to an upsampling processing module.

[0115] In this embodiment, the one-dimensional convolution layer performs convolution operation in the frequency dimension to extract local features of the audio signal. The Log Mel spectrum is the representation of the audio signal after the short-time Fourier transform (STFT) and the Mel filter bank, which contains the spectral characteristics of the audio. Through the convolution operation, the local correlation between different frequency bands can be effectively captured. The predetermined convolution kernel size and step size determine the range and moving step size of the convolution operation. The larger the convolution kernel, the wider the range of features captured; the step size determines the frequency at which the convolution window moves in the input signal.

[0116] After the convolution operation, the design without nonlinear activation function means that the output after convolution remains linear. This design retains the original structure and characteristics of the signal without changing the shape of the signal through nonlinear mapping. Its purpose is not to introduce nonlinear transformation in the initial stage, to ensure that the signal retains the most original characteristics, and to provide high-fidelity input for subsequent network layers.

[0117] After convolution, the signal is processed by batch normalization, a process designed to speed up model training and reduce the problem of vanishing gradients. Batch normalization can adjust the mean and variance of the input signal to zero mean and unit variance, thereby stabilizing the distribution of the signal and ensuring that subsequent layers can process it more effectively. The signal after batch normalization becomes more stable, which helps the neural network converge faster during training.

[0118] After convolution and normalization, the signal is passed to the subsequent upsampling processing module to improve the resolution and perform further signal processing.

[0119] First, obtain the Log Mel spectrum and input it into the one-dimensional convolution layer for processing. The size and step size of the convolution kernel are set according to the requirements. The convolution operation slides along the frequency dimension to extract local features. The convolved signal is passed directly without any nonlinear activation function. The purpose of maintaining linear features is to retain the original spectrum features for subsequent processing. The output after convolution is batch normalized to ensure the uniformity and stability of the signal distribution, which is helpful for the subsequent neural network processing layer. Finally, the normalized signal is passed to the next upsampling processing module to prepare for further upsampling operations on the signal.

[0120] In one embodiment, in a financial market notification system, the kernel size and step size of the convolution layer may need to be adjusted according to the speed of market data changes. For example, a fast-moving market requires a shorter step size to capture fast-changing data features, while a stable market can use a larger step size to reduce the amount of calculation. A dynamic convolution kernel selection mechanism is implemented to adjust the convolution step size based on the market volatility rate so as to more efficiently process market data features.

[0121] In another specific implementation, without using a nonlinear activation function, in order to ensure that the linear features of the speech signal are not distorted, a normalization process (such as Layer Normalization or Instance Normalization) can be added after the convolution operation to maintain the uniformity of the signal, especially when processing complex multilingual speech. A feature alignment algorithm for audio signals in a multilingual environment is added, and the features of speech in different languages ​​are ensured to be evenly processed in the convolution layer through normalization and alignment techniques.

[0122] This embodiment processes the Log Mel spectrum through a pre-network including a one-dimensional convolution layer without nonlinear activation, which can retain the original spectrum characteristics of the signal in the early stage, avoid premature nonlinear transformation, and improve the accuracy of subsequent processing. At the same time, through batch normalization, the distribution of the signal is more stable, ensuring that the neural network maintains higher efficiency and accuracy during training and generation.

[0123] In one embodiment, the above S30 includes:

[0124] S301, in each upsampling processing module, applying transposed convolution to the processed signal for upsampling;

[0125] S302, setting the convolution kernel size, step size and padding method of the transposed convolution to achieve the required upsampling multiple;

[0126] S303, transmitting the upsampled signal to a multi-receptive field module, where the multi-receptive field module includes a plurality of parallel residual blocks with different receptive fields;

[0127] S304, applying dilated convolution in the multi-receptive field module, and using residual connection to fuse the output of the multi-receptive field module with the input signal.

[0128] In this embodiment, transposed convolution is one of the most commonly used operations in the upsampling process, which improves the resolution by expanding the size of the feature map. Compared with conventional convolution, it performs the reverse operation and can upsample the low-resolution feature map to a higher resolution. The convolution kernel size and step size of the transposed convolution determine the upsampling ratio, while the padding method ensures that the output size meets expectations. The upsampling multiple is determined by the size of the convolution kernel, the step size, and the padding method. Larger convolution kernels and step sizes can improve signal resolution faster.

[0129] The multi-receptive field module consists of multiple convolutional layers with different receptive fields, which can capture signal features at different scales. Convolutional layers with smaller receptive fields focus on capturing local details, while larger receptive fields are responsible for capturing global information. Combining different receptive fields can improve the network's ability to focus on both details and overall features when processing audio signals. Dilated convolution is a method of expanding the receptive field by inserting holes between convolution kernels, allowing the convolution operation to capture a wider range of features without increasing the amount of computation. The advantage of dilated convolution is that it can retain both detail information and global features, which is particularly suitable for processing long-term dependencies in audio signals.

[0130] Residual connection is a technique commonly used in deep learning to solve the gradient vanishing problem in deep networks. By performing residual connection between the output of the multi-receptive field module and the input signal, some features of the original signal can be retained, enhancing the robustness and feature transfer capability of the network.

[0131] In each upsampling processing module, the signal is upsampled by transposed convolution. The convolution kernel size, step size, and padding method are set to ensure that each module appropriately improves the resolution of the signal. The multiple of each upsampling depends on the settings of these parameters, and the goal is to gradually restore the low-resolution signal to high resolution.

[0132] The upsampled signal enters the multi-receptive field module. This module consists of multiple convolutional layers with different receptive fields, which are responsible for capturing local and global signal features respectively. The convolution kernel size of each receptive field is different, ensuring that the multi-dimensional features of the signal are captured from different scales.

[0133] In the multi-receptive field module, the dilated convolution further enhances the network's ability to capture global features. The output of the multi-receptive field module is fused with the input signal through the residual connection, which can not only retain some features of the input signal, but also enhance the network's learning ability and improve the overall performance of the signal.

[0134] In one specific implementation, in an intelligent voice assistant, the upsampling multiple of the transposed convolution may be adaptively adjusted according to the user's voice environment. For example, in a noisy environment, a larger step size and convolution kernel are used to improve the noise resistance of the signal; while in a quiet environment, the step size is smaller to capture more details. An environment-adaptive convolution strategy is introduced to adjust the upsampling parameters in real time according to the background noise to ensure the clarity and restoration of the voice signal.

[0135] The multi-receptive field module expansion can be used in voice assistants to optimize in real time according to the user's voice frequency characteristics, adjusting the size of different receptive field convolution kernels to process high-frequency and low-frequency signals. For example, for keywords in voice commands (such as "balance"), high-frequency details can be captured through a smaller receptive field, while for overall voice fluency, a larger receptive field is used. Add a keyword enhancement mechanism and use multi-receptive field modules to enhance the extraction of key information to improve the accuracy and response speed of voice assistant recognition.

[0136] When processing long speech input, the dilated convolution can dynamically adjust its dilation factor according to the length of the speech to ensure that long-term dependent features can be captured, while the residual connection helps maintain the overall consistency of the speech input and prevents information loss during long-term processing. A time dynamic adjustment mechanism is introduced to adjust the dilation factor of the dilated convolution based on the length of the speech signal, and the residual connection is combined to enhance the long-term dependent features.

[0137] This embodiment uses transposed convolution to achieve signal upsampling and feature extraction of multi-receptive field modules, and the system can improve the resolution and feature expression capabilities of audio signals. At the same time, the dilated convolution and residual connection technology effectively retains the details and global information of the signal, improves the expressiveness of the signal, and makes the generated audio signal more natural and clear in voice assistant and market notification scenarios.

[0138] In one embodiment, in the above S40, the upsampled signal is divided into multiple tensors according to a predetermined ratio in the channel dimension, including:

[0139] S401, determining a predetermined ratio of channel division, wherein the predetermined ratio corresponds to different frequency bands;

[0140] S402, dividing the upsampled signal into a plurality of tensors in the channel dimension according to the predetermined ratio;

[0141] S403, assign each tensor to a corresponding frequency band, arrange from low frequency to high frequency, and mark each tensor individually.

[0142] In this embodiment, the predetermined ratio of channel segmentation refers to a reasonable division of the signal according to the importance of different frequency bands or the occupied frequency range. Signals in different frequency bands contribute differently to the overall audio, so different numbers of channels are divided according to the actual application scenario. Usually, the low-frequency part has a greater impact on human ear perception, so more channels are allocated; the high-frequency part is allocated with fewer channels. The purpose of determining this ratio is to ensure that the characteristics of each frequency band can be fully expressed, thereby improving the clarity and restoration of the signal.

[0143] Once the predetermined ratios of the frequency bands are determined, the system will split the upsampled signal in the channel dimension according to these ratios. Each channel corresponds to a specific frequency band, and the split tensor retains the signal characteristics of different frequency ranges. This ensures that the information of each frequency band can be processed independently and further optimized in subsequent processing steps.

[0144] After the segmentation is completed, the system assigns each tensor to different frequency bands such as low frequency, medium frequency or high frequency. This operation clarifies the allocation of each frequency band, making subsequent processing of different frequency bands more efficient and accurate. Each tensor not only corresponds to a frequency range, but also individually identifies these frequency bands according to their importance and characteristics, which facilitates subsequent operations and optimization.

[0145] According to the application scenario (such as speech generation or market notification), the system will set the ratio of channel segmentation through experience or predefined rules. For example, for the low frequency band to which the human ear is more sensitive, more channels are allocated, while for the high frequency band, the number of channels is relatively small. A common segmentation ratio can be 4:2:1, corresponding to the channel allocation of low frequency, medium frequency and high frequency respectively. After the signal is upsampled, a multi-channel signal is obtained. In the channel dimension, the signal is divided into different tensors according to a predetermined ratio. For example, if the signal contains 12 channels, it can be allocated into 4 low frequency channels, 2 medium frequency channels and 1 high frequency channel according to the ratio of 4:2:1. Each tensor corresponds to a different frequency band and retains the information of the corresponding frequency band. Each tensor is clearly identified as a different frequency band, such as low frequency (0Hz-1kHz), medium frequency (1kHz-5kHz) and high frequency (5kHz-20kHz). This identification helps the system to adopt different processing strategies for the characteristics of different frequency bands in subsequent processing, such as different convolution kernels or different activation functions.

[0146] In one specific implementation, in the intelligent voice assistant, the system sets a specific channel segmentation ratio according to the pitch characteristics of the user's voice. For example, for the low-frequency part of the user's voice (usually containing the core content of the voice), more channels can be allocated to ensure the clarity of the voice. The high-frequency part can be allocated fewer channels to save computing resources. A voice frequency adaptive segmentation strategy is introduced to dynamically adjust the channel segmentation ratio according to the spectral characteristics of the user's voice, so that the generated voice is more in line with the actual sound characteristics.

[0147] In this embodiment, by dividing the upsampled signal into multiple tensors in a predetermined ratio in the channel dimension, the system can perform refined processing on the characteristics of different frequency bands. The important information of the low-frequency band can be enhanced, and the details of the high-frequency band can be retained, so that the generated audio signal is more natural and accurate.

[0148] In one embodiment, the above S50 includes:

[0149] S501, input each tensor into an independent multi-receptive field module, perform multi-scale feature extraction on each tensor, the parameters between each multi-receptive field module are not shared, and each multi-receptive field module includes a residual block with different dilation factors and convolution kernel sizes;

[0150] S502, fusing the multi-scale features of each tensor to enhance the frequency band feature expression;

[0151] S503, passing the output of the multi-receptive field module through a one-dimensional convolutional layer to unify the number of channels of the tensor;

[0152] S504, applying an activation function to perform nonlinear transformation on the tensor with the uniform number of channels.

[0153] In this embodiment, each tensor is input into an independent multi-receptive field module. A multi-receptive field module refers to a module that can capture features of different scales with different receptive fields (i.e., convolution kernel size and dilation factor). In this design, each multi-receptive field module focuses on the tensors of a specific frequency band and performs feature extraction according to its specific parameter settings (convolution kernel size and dilation factor). Parameter non-sharing means that the tensors of each frequency band are processed in their own independent multi-receptive field modules without interfering with each other. This ensures that features of different frequency bands can be extracted and optimized independently.

[0154] The multi-receptive field module extracts multi-scale features by applying convolutional layers with different receptive field sizes and dilation factors. Dilated convolution is used to expand the receptive field and capture more global information, while smaller convolution kernels are used to obtain local detail information. Through this multi-scale feature extraction, information at different scales can be extracted without increasing the amount of computation.

[0155] The multi-scale features extracted from each tensor are fused, that is, the features extracted by different convolution kernel sizes and dilation factors are merged, so that the frequency band features can be more abundant at multiple levels. This fusion process improves the system's ability to express frequency band features and ensures that information from different frequency bands can be effectively extracted and fused.

[0156] The feature tensors output by the multi-receptive field module may have different numbers of channels. These tensors are convolved through a one-dimensional convolutional layer to unify their number of channels to ensure the smoothness of subsequent processing steps. This step helps to integrate the features of different frequency bands into one signal.

[0157] After unifying the number of channels, an activation function (such as ReLU or tanh) is used to perform a nonlinear transformation on the signal. The activation function can introduce nonlinear characteristics, enabling the network to learn and express more complex mapping relationships. This step ensures that the signal's expressiveness is further improved and provides a stronger feature representation for subsequent steps.

[0158] Each segmented tensor is input into the corresponding multi-receptive field module. The convolution kernel size and dilation factor of each module are different, ensuring that features can be extracted from different scales. The parameters of these modules are not shared, ensuring that the tensor features of each frequency band are processed independently. After each multi-receptive field module extracts features at different scales, these features are fused. The fused tensor retains both global features and local detail information. The fused features pass through a one-dimensional convolution layer to unify the number of channels of all tensors. This ensures that features of different frequency bands can be further merged and processed without incompatibility issues due to different numbers of channels. After the signal is processed by the one-dimensional convolution layer, the activation function is applied for nonlinear transformation to improve the feature expression capability. The activation function can help the network learn more complex features and ensure that the signal behaves more naturally in different application scenarios.

[0159] In one specific implementation, in the intelligent voice assistant, the speech features of each frequency band are processed by different multi-receptive field modules to extract high-frequency and low-frequency feature information respectively. For the speech information in the low-frequency band, a larger convolution kernel and a smaller expansion factor are used to extract the core content, while for the high-frequency part, a smaller convolution kernel and a larger expansion factor are used to capture more detail information. The speech clarity adaptive enhancement mechanism is introduced to ensure that the features of important parts of the speech (such as keywords) can be extracted more accurately by adjusting the expansion factor and the convolution kernel size, generating clearer speech output.

[0160] This embodiment introduces a multi-receptive field module and multi-scale feature extraction technology with unshared parameters, so that the system can better extract and fuse feature information of different frequency bands. Then, by unifying the number of channels and nonlinear processing of activation functions, it ensures that both signal processing efficiency and feature expression capabilities are improved.

[0161] In one embodiment, the above S70 includes:

[0162] S701, combining the upsampled tensors into one signal by adding;

[0163] S702, inputting the combined signal into an additional multi-receptive field module, where the additional multi-receptive field module includes residual blocks with different receptive fields;

[0164] S703, performing multi-scale feature extraction on the combined signal to capture feature information of the comprehensive frequency band;

[0165] S704, performing feature fusion on the outputs of the multi-receptive field modules to enhance the overall feature expression of the signal;

[0166] S705, passing the output of the fused multi-receptive field module to the post-network.

[0167] In this embodiment, merging upsampled tensors of different frequency bands into one signal is a common operation, which is usually performed by element-wise summation. The advantage of summation is that it can retain the characteristics of each tensor without introducing too much computational complexity. Each tensor represents information of a different frequency band, and a comprehensive signal containing the characteristics of all frequency bands can be generated by summation.

[0168] The merged signal is input into an additional multi-receptive field module for processing. This module is similar to the previous multi-receptive field module and contains multiple convolutional layers and residual blocks with different receptive fields. The residual block can help preserve the original features of the signal and extract feature information at different levels through convolution kernels of multiple scales. The purpose of this step is to further enrich the feature expression ability of the merged signal.

[0169] Through the additional multi-receptive field module, the multi-scale features of the signal are extracted to capture the interaction information between different frequency bands. Compared with the previous individual frequency band processing, this step extracts features from the merged overall signal, enhancing the comprehensive expression ability between frequency bands. Feature fusion is to merge the features extracted by different convolution kernels in the multi-receptive field module to obtain a more expressive feature representation. These features can come from convolution layers with different receptive fields, and after fusion, they can improve the global expression ability of the signal. Finally, the fused signal will be passed to the post-network for further processing or generation of the final audio signal. The post-network may include more one-dimensional convolution layers or non-linear activation functions to complete the final generation of the signal.

[0170] In one specific implementation, in the financial market notification system, market data tensors of different frequency bands are combined into one signal by addition and passed to the multi-receptive field module for multi-scale feature extraction. This module extracts the long-term change information of the market through the convolution kernel in the residual block, while capturing the details of short-term fluctuations. The fused feature signal can generate high-quality voice notifications in the post-network. The market frequency adaptive fusion mechanism is introduced to perform feature fusion through convolution kernels of different receptive field sizes for rapid market fluctuations or long-term changes, ensuring that the generated voice notification is timely and reflects the overall market picture.

[0171] This embodiment can better capture the feature interactions between different frequency bands by adding and merging the upsampled tensors and passing them into an additional multi-receptive field module for processing. The feature fusion process enhances the global expression ability of the signal, making the final generated audio signal have higher clarity and naturalness in the scenarios of intelligent voice assistants and financial market notifications.

[0172] In one embodiment, the above S10 includes:

[0173] S101, obtaining an original audio signal to be processed;

[0174] S102, preprocessing the original audio signal, including denoising and normalization;

[0175] S103, converting the preprocessed original audio signal into a Log mel spectrum;

[0176] S104, treating the frequency dimension of the Log Mel spectrum as multiple frequency channels to form a multi-channel one-dimensional signal;

[0177] S105, adjusting the multi-channel one-dimensional signal to a tensor format that conforms to an input neural network.

[0178] In this embodiment, the original audio signal is first obtained from an external device or data source. These audio signals can come from a voice assistant, a voice recognition system, or other audio processing systems. When obtaining the signal, it is key to ensure the integrity of the signal.

[0179] In audio signal processing, denoising is to remove background noise in the audio to improve the clarity and intelligibility of the audio signal. Common denoising algorithms include frequency domain filters, waveform denoising and other technologies.

[0180] Normalization is used to adjust the amplitude of the audio signal to a uniform range so that audio signals of different volumes can be processed uniformly. This helps the network model process the input signal more stably and avoids the degradation of model performance caused by volume differences.

[0181] The Log Mel spectrum is obtained by performing a short-time Fourier transform (STFT) on the audio signal and converting the result through a Mel filter bank. The Log Mel spectrum is closer to the auditory characteristics of the human ear by taking the logarithm of the frequency representation, and is particularly suitable for speech recognition, generation, and processing. This conversion can effectively capture the frequency components in the audio signal and make more precise distinctions of the frequencies (low frequencies) that are more important to the human ear.

[0182] The frequency dimension of the Log Mel spectrum contains multiple frequency bands. Each frequency band can be regarded as an independent channel, and these channels are combined into a multi-channel one-dimensional signal, so that the spectrum information can be represented in a way that the network can understand. This one-dimensional signal form can better convey audio features and preserve the time-frequency information in the audio.

[0183] In deep learning models, it is usually necessary to convert input data into tensor format. In this step, the obtained multi-channel one-dimensional signal will be organized into a tensor format that the neural network can process for subsequent feature extraction and processing. Tensors can be represented as arrays with multiple dimensions, which are suitable for processing in deep learning models such as convolutional neural networks (CNNs).

[0184] The system obtains the original audio signal from a microphone or other audio input device. These signals are usually represented in the form of waveforms and contain time and amplitude information. The audio is denoised through signal processing technology to remove background noise and improve the clarity of the signal. Then, the amplitude of the audio signal is adjusted to a uniform range through normalization to ensure signal stability during subsequent processing. The audio signal is converted to a frequency domain representation using a short-time Fourier transform (STFT). The frequency domain representation is then converted to a Mel spectrum through a Mel filter bank, and the Log Mel spectrum is obtained through a logarithmic operation. This process retains the frequency characteristics in the audio signal, compresses high-frequency information, and enhances the representation of low-frequency signals. The frequency dimension of the Log Mel spectrum contains multiple frequency bands, each of which can be regarded as an independent frequency channel. These frequency band signals are combined into a multi-channel one-dimensional signal, thereby retaining the time-frequency variation information of the audio features in the time dimension. Finally, the system converts the multi-channel one-dimensional signal into a tensor format suitable for neural network input so that the network can further process the audio features.

[0185] In one specific implementation, for an intelligent voice assistant, the key content of a voice signal may be in a specific frequency band. Different resolutions can be set for different frequency bands of the Log Mel spectrum to enhance sensitivity to key frequency bands, such as enhancing the capture of low-frequency bands of human voices. A weighted frequency band enhancement algorithm is used to pay more attention to the core frequency bands in the voice to improve the accuracy of voice recognition.

[0186] When converting the Log Mel spectrum into a tensor, we can introduce timing enhancement technology to enhance the timing information of speech features. In real-time voice interaction, maintaining the continuity of speech is crucial for the response of the intelligent assistant. Adding a timing convolution layer ensures that the system can accurately extract features from continuous speech signals by processing multiple frames of audio information.

[0187] This embodiment can effectively capture the frequency characteristics of the audio by acquiring the original audio signal and converting it into a Log Mel spectrum, and input it into the neural network in the form of a multi-channel one-dimensional signal. This makes the processing of audio signals more efficient and accurate, and is particularly suitable for use in scenarios such as intelligent voice assistants and financial market notification systems, improving the accuracy of signal processing and the generation effect.

[0188] In one embodiment, a vocoder audio generation device based on frequency band division is provided, and the vocoder audio generation device based on frequency band division corresponds one-to-one to the vocoder audio generation method based on frequency band division in the above embodiment. Figure 3 , Figure 3This is a functional module diagram of a preferred embodiment of a vocoder audio generation device based on frequency band segmentation of the present invention. Input module 10, front network module 20, upsampling module 30, frequency band segmentation module 40, multi-receptive field independent processing module 50, PQMF upsampling module 60, signal merging and processing module 70, post-network module 80 and output module 90. Each functional module is described in detail as follows:

[0189] An input module 10 is used to obtain an original audio signal, convert the original audio signal into a Log mel spectrum, and regard the frequency dimension of the Log mel spectrum as a one-dimensional signal of multiple frequency channels;

[0190] A front-end network module 20, configured to process the Log Mel spectrum through a front-end network including a one-dimensional convolutional layer and without nonlinear activation;

[0191] An upsampling module 30, used to sequentially pass the processed signal through a plurality of upsampling processing modules, each of which performs upsampling by transposed convolution and performs signal processing using a multi-receptive field module;

[0192] A frequency band segmentation module 40, for segmenting the upsampled signal into a plurality of tensors in a predetermined ratio in the channel dimension after being processed by the transposed convolution and multi-receptive field modules, each tensor corresponding to a different frequency band;

[0193] A multi-receptive field independent processing module 50, used to independently process the segmented tensors through multi-receptive field modules whose parameters are not shared, and unify the number of channels through a convolution operation;

[0194] A PQMF upsampling module 60, for upsampling a tensor with a uniform number of channels using a PQMF synthesis filter bank;

[0195] A signal merging and processing module 70, for merging the upsampled tensors into one signal and processing it through another multi-receptive field module;

[0196] A post-network module 80 is used to process the combined signal through a post-network including a one-dimensional convolutional layer and an activation function to adjust the channel dimension;

[0197] The output module 90 is used to pass the processed signal through the activation function to obtain a generated target audio signal.

[0198] In one embodiment, the front-end network module 20 is specifically used for:

[0199] Processing the Log Mel spectrum through a one-dimensional convolution layer, wherein the one-dimensional convolution layer has a predetermined convolution kernel size and a step size;

[0200] No nonlinear activation function is used after the one-dimensional convolution layer, and the output of the one-dimensional convolution layer is used as the processed signal;

[0201] The processed signals are batch normalized and passed to the upsampling processing module.

[0202] In one embodiment, the up-sampling module 30 is specifically configured to:

[0203] In each upsampling processing module, transposed convolution is applied to the processed signal for upsampling;

[0204] Set the kernel size, stride, and padding of the transposed convolution to achieve the desired upsampling factor;

[0205] Passing the upsampled signal to a multi-receptive field module, wherein the multi-receptive field module includes a plurality of parallel residual blocks with different receptive fields;

[0206] Dilated convolution is applied in the multi-receptive field module, and the output of the multi-receptive field module is fused with the input signal using residual connection.

[0207] In one embodiment, the frequency band segmentation module 40 is specifically configured to:

[0208] Determining a predetermined ratio of channel division, the predetermined ratio corresponding to different frequency bands;

[0209] Splitting the upsampled signal into multiple tensors in the channel dimension according to the predetermined ratio;

[0210] Assign each tensor to a corresponding frequency band, arranged from low frequency to high frequency, and identify each tensor individually.

[0211] In one embodiment, the multiple receptive field independent processing module 50 is specifically used for:

[0212] Each tensor is input into an independent multi-receptive field module, and multi-scale feature extraction is performed on each tensor. The parameters between each multi-receptive field module are not shared, and each multi-receptive field module contains residual blocks with different dilation factors and convolution kernel sizes;

[0213] The multi-scale features of each tensor are fused to enhance the frequency band feature expression;

[0214] The output of the multi-receptive field module is passed through a one-dimensional convolutional layer to unify the number of channels of the tensor;

[0215] Apply an activation function to perform a nonlinear transformation on a tensor of uniform number of channels.

[0216] In one embodiment, the signal combining and processing module 70 is specifically used for:

[0217] Combine the upsampled tensors into one signal by adding them together;

[0218] Inputting the combined signal into an additional multi-receptive field module, wherein the additional multi-receptive field module includes residual blocks with different receptive fields;

[0219] Perform multi-scale feature extraction on the combined signal to capture the feature information of the comprehensive frequency band;

[0220] The output of the multi-receptive field module is feature-fused to enhance the overall feature expression of the signal;

[0221] The output of the fused multi-receptive field module is passed to the subsequent network.

[0222] In one embodiment, the input module 10 is specifically used for:

[0223] Obtaining the original audio signal to be processed;

[0224] Preprocessing the original audio signal, including denoising and normalization;

[0225] Convert the preprocessed original audio signal into a Log mel spectrum;

[0226] The frequency dimension of the Log Mel spectrum is regarded as a plurality of frequency channels to form a multi-channel one-dimensional signal;

[0227] The multi-channel one-dimensional signal is adjusted to a tensor format that conforms to the input neural network.

[0228] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, the functions or steps on the service side of a vocoder audio generation method based on frequency band segmentation are implemented.

[0229] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, the functions or steps on the user side of a method for generating audio of a vocoder based on frequency band segmentation are realized.

[0230] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0231] Acquire an original audio signal, and convert the original audio signal into a Log mel spectrum, and regard the frequency dimension of the Log mel spectrum as a one-dimensional signal of multiple frequency channels;

[0232] Processing the Log Mel spectrum through a front network including a one-dimensional convolutional layer without non-linear activation;

[0233] The processed signal is passed through multiple upsampling processing modules in sequence, each of which performs upsampling through transposed convolution and uses a multi-receptive field module to perform signal processing;

[0234] After the transposed convolution and multi-receptive field module processing, the upsampled signal is split into multiple tensors in the channel dimension according to a predetermined ratio, and each tensor corresponds to a different frequency band;

[0235] The segmented tensors are processed independently through multi-receptive field modules with unshared parameters, and the number of channels is unified through convolution operations;

[0236] Use PQMF synthesis filter bank to upsample tensors with uniform number of channels;

[0237] Combine the upsampled tensors into one signal and process it through another multi-receptive field module;

[0238] Process the combined signal through a post-network consisting of a one-dimensional convolutional layer and an activation function to adjust the channel dimension;

[0239] The processed signal is passed through the activation function to obtain the generated target audio signal.

[0240] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0241] Acquire an original audio signal, and convert the original audio signal into a Log mel spectrum, and regard the frequency dimension of the Log mel spectrum as a one-dimensional signal of multiple frequency channels;

[0242] Processing the Log Mel spectrum through a front network including a one-dimensional convolutional layer without non-linear activation;

[0243] The processed signal is passed through multiple upsampling processing modules in sequence, each of which performs upsampling through transposed convolution and uses a multi-receptive field module to perform signal processing;

[0244] After the transposed convolution and multi-receptive field module processing, the upsampled signal is split into multiple tensors in the channel dimension according to a predetermined ratio, and each tensor corresponds to a different frequency band;

[0245] The segmented tensors are processed independently through multi-receptive field modules with unshared parameters, and the number of channels is unified through convolution operations;

[0246] Use PQMF synthesis filter bank to upsample tensors with uniform number of channels;

[0247] Combine the upsampled tensors into one signal and process it through another multi-receptive field module;

[0248] Process the combined signal through a post-network consisting of a one-dimensional convolutional layer and an activation function to adjust the channel dimension;

[0249] The processed signal is passed through the activation function to obtain the generated target audio signal.

[0250] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0251] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0252] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0253] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for generating vocoder audio based on frequency band segmentation, characterized in that: The following steps are involved: Acquire an original audio signal, and convert the original audio signal into a Log mel spectrum, and regard the frequency dimension of the Log mel spectrum as a one-dimensional signal of multiple frequency channels; Processing the Log Mel spectrum through a front network including a one-dimensional convolutional layer without non-linear activation; The processed signal is passed through multiple upsampling processing modules in sequence, each of which performs upsampling through transposed convolution and uses a multi-receptive field module to perform signal processing; After the transposed convolution and multi-receptive field module processing, the upsampled signal is split into multiple tensors in the channel dimension according to a predetermined ratio, and each tensor corresponds to a different frequency band; The segmented tensors are processed independently through multi-receptive field modules with unshared parameters, and the number of channels is unified through convolution operations; Use PQMF synthesis filter bank to upsample tensors with uniform number of channels; Combine the upsampled tensors into one signal and process it through another multi-receptive field module; Process the combined signal through a post-network consisting of a one-dimensional convolutional layer and an activation function to adjust the channel dimension; The processed signal is passed through the activation function to obtain the generated target audio signal.

2. The method for generating vocoder audio based on frequency band division according to claim 1, characterized in that: The Log Mel spectrum is processed by a front network including a one-dimensional convolutional layer without non-linear activation, including: Processing the Log Mel spectrum through a one-dimensional convolution layer, wherein the one-dimensional convolution layer has a predetermined convolution kernel size and a step size; No nonlinear activation function is used after the one-dimensional convolution layer, and the output of the one-dimensional convolution layer is used as the processed signal; The processed signals are batch normalized and passed to the upsampling processing module.

3. The method for generating vocoder audio based on frequency band division according to claim 1, characterized in that: The processed signal is passed through multiple upsampling processing modules in sequence. Each upsampling processing module performs upsampling through transposed convolution and uses a multi-receptive field module for signal processing, including: In each upsampling processing module, transposed convolution is applied to the processed signal for upsampling; Set the kernel size, stride, and padding of the transposed convolution to achieve the desired upsampling factor; Passing the upsampled signal to a multi-receptive field module, wherein the multi-receptive field module includes a plurality of parallel residual blocks with different receptive fields; Dilated convolution is applied in the multi-receptive field module, and the output of the multi-receptive field module is fused with the input signal using residual connection.

4. The method for generating vocoder audio based on frequency band division according to claim 1, characterized in that: The upsampled signal is divided into multiple tensors according to a predetermined ratio in the channel dimension, including: Determining a predetermined ratio of channel division, the predetermined ratio corresponding to different frequency bands; Splitting the upsampled signal into multiple tensors in the channel dimension according to the predetermined ratio; Assign each tensor to a corresponding frequency band, arranged from low frequency to high frequency, and identify each tensor individually.

5. The method for generating vocoder audio based on frequency band division according to claim 1, characterized in that: The segmented tensors are processed independently through multi-receptive field modules with unshared parameters, and the number of channels is unified through convolution operations, including: Each tensor is input into an independent multi-receptive field module, and multi-scale feature extraction is performed on each tensor. The parameters between each multi-receptive field module are not shared, and each multi-receptive field module contains residual blocks with different dilation factors and convolution kernel sizes; The multi-scale features of each tensor are fused to enhance the frequency band feature expression; The output of the multi-receptive field module is passed through a one-dimensional convolutional layer to unify the number of channels of the tensor; Apply an activation function to perform a nonlinear transformation on a tensor of uniform number of channels.

6. The method for generating vocoder audio based on frequency band division as claimed in claim 1, characterized in that: The upsampled tensors are combined into one signal and processed through another multi-receptive field module, including: Combine the upsampled tensors into one signal by adding them together; Inputting the combined signal into an additional multi-receptive field module, wherein the additional multi-receptive field module includes residual blocks with different receptive fields; Perform multi-scale feature extraction on the combined signal to capture the feature information of the comprehensive frequency band; The output of the multi-receptive field module is feature-fused to enhance the overall feature expression of the signal; The output of the fused multi-receptive field module is passed to the subsequent network.

7. The method for generating vocoder audio based on frequency band division according to claim 1, characterized in that: Obtain an original audio signal, and convert the original audio signal into a Log mel spectrum, and regard the frequency dimension of the Log mel spectrum as a one-dimensional signal of multiple frequency channels, including: Obtaining the original audio signal to be processed; Preprocessing the original audio signal, including denoising and normalization; Convert the preprocessed original audio signal into a Log mel spectrum; The frequency dimension of the Log Mel spectrum is regarded as a plurality of frequency channels to form a multi-channel one-dimensional signal; The multi-channel one-dimensional signal is adjusted to a tensor format that conforms to the input neural network.

8. A vocoder audio generation device based on frequency band division, characterized in that: The vocoder audio generation device based on frequency band division comprises: An input module is used to obtain an original audio signal, convert the original audio signal into a Log mel spectrum, and regard the frequency dimension of the Log mel spectrum as a one-dimensional signal of multiple frequency channels; A front-end network module, used for processing the Log Mel spectrum through a front-end network including a one-dimensional convolutional layer and without non-linear activation; An upsampling module is used to sequentially pass the processed signal through multiple upsampling processing modules, each of which performs upsampling through transposed convolution and uses a multi-receptive field module to perform signal processing; The frequency band segmentation module is used to segment the upsampled signal into multiple tensors according to a predetermined ratio in the channel dimension after being processed by the transposed convolution and multi-receptive field modules, each tensor corresponding to a different frequency band; A multi-receptive field independent processing module, used for independently processing the segmented tensors through multi-receptive field modules whose parameters are not shared, and unifying the number of channels through a convolution operation; PQMF upsampling module, used to upsample tensors with uniform number of channels using PQMF synthesis filter banks; The signal merging and processing module is used to merge the upsampled tensors into one signal and process it through another multi-receptive field module; A post-network module, used to process the combined signal through a post-network including a one-dimensional convolutional layer and an activation function to adjust the channel dimension; The output module is used to pass the processed signal through the activation function to obtain the generated target audio signal.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a frequency band splitting based vocoder audio generation program stored in the memory and executable on the processor. When the frequency band splitting based vocoder audio generation program is executed by the processor, the steps of the frequency band splitting based vocoder audio generation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a vocoder audio generation program based on frequency band division, and when the vocoder audio generation program based on frequency band division is executed by the processor, the steps of the vocoder audio generation method based on frequency band division as described in any one of claims 1-7 are implemented.

Images