Band division based vocoder audio generation method, apparatus, device and medium

Abstract

The application relates to the fields of artificial intelligence and financial technology, and discloses a vocoder audio generation method based on frequency band segmentation, which converts input audio signals into Log Mel frequency spectrum, and performs preliminary processing on the frequency spectrum signals through a one-dimensional convolution network. After the signals are subjected to up-sampling processing through transposed convolution, a multi-receptive field module is used to extract and process the features of different frequency bands. In the frequency band segmentation process, the signals are segmented into multiple tensors in the channel dimension, and each tensor corresponds to a specific frequency band range and is independently processed. A PQMF synthesis filter bank is used to perform up-sampling on the signals with unified channel numbers again. After the signals are combined, another multi-receptive field module is used to process the finally generated audio signals. The application processes tensors of different frequency bands through frequency band segmentation and independent multi-receptive field modules, reduces the frequency aliasing phenomenon introduced by transposed convolution in the up-sampling process, and thus improves the quality and naturalness of audio generation.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and financial technology, and in particular to a method, apparatus, device and storage medium for generating audio from a vocoder based on frequency band segmentation. Background Technology

[0002] In audio processing systems within the financial sector, technologies such as speech recognition and speech synthesis are increasingly being widely applied in various business scenarios, including intelligent customer service, voice interaction, and risk management. To improve the accuracy and response speed of voice interaction, the quality of audio signal processing is crucial. However, speech synthesis technology in current financial scenarios still faces some challenges, particularly in the audio reconstruction process, where accurately reconstructing the phase information of the audio signal remains a difficult issue in the industry.

[0003] In audio signal processing, vocoders, as systems used to synthesize or reconstruct speech signals, are widely used in intelligent voice systems in the financial sector. However, the traditional Griffin-Lim algorithm, one of the most common phase retrieval methods, primarily targets linear amplitude spectra and performs poorly with Mel-spectrograms. This poses a significant limitation for intelligent speech synthesis in financial scenarios, such as voice assistants and automated customer service systems. Due to the limitations of the Griffin-Lim algorithm in phase retrieval, the speech generation is poor, resulting in low-quality audio that fails to meet the financial industry's demands for high-quality, highly realistic speech synthesis.

[0004] In recent years, with the development of deep learning technology, vocoders based on Generative Adversarial Networks (GANs) have shown significant advantages in speech synthesis. GANs, through adversarial training between the generator and discriminator, gradually generate high-quality audio during the learning process. However, in existing financial intelligent voice systems, the generator structure of GAN vocoders typically uses transposed convolution for upsampling. This introduces aliasing frequency information in the frequency dimension, often resulting in frequency distortion and other defects in the final generated audio. This frequency aliasing problem severely affects the naturalness and intelligibility of speech synthesis, reducing the customer experience of financial voice interaction systems.

[0005] Furthermore, with the increasing complexity of financial transactions, voice systems need to process large amounts of user data and complex language models, which places higher demands on the real-time performance and efficiency of speech synthesis. However, existing audio generation technologies still face bottlenecks when efficiently processing large-scale audio data. Therefore, how to effectively solve the phase recovery problem in audio generation, reduce frequency aliasing, and improve the quality and efficiency of speech synthesis in financial scenarios remains a technical challenge to be addressed in this field. Summary of the Invention

[0006] The main objective of this invention is to provide a vocoder-based audio generation method, apparatus, device, and storage medium based on frequency band segmentation, aiming to solve the technical problem that in the prior art, when upsampling through transposed convolution, aliasing frequency information is generated in the frequency dimension of the generated audio signal, resulting in a decrease in audio quality and authenticity.

[0007] To achieve the above objectives, the present invention provides a vocoder audio generation method based on frequency band segmentation, comprising:

[0008] The original audio signal is acquired and converted into a Log-Mel spectrum, and the frequency dimension of the Log-Mel spectrum is regarded as a one-dimensional signal with multiple frequency channels.

[0009] The Log-Mel spectrum is processed by a pre-network containing one-dimensional convolutional layers and without non-linear activation.

[0010] The processed signal is sequentially passed through multiple upsampling processing modules. Each upsampling processing module performs upsampling through transposed convolution and uses a multi-receptive field module for signal processing.

[0011] After transposed convolution and multi-receptive field module processing, the upsampled signal is divided into multiple tensors in the channel dimension according to a predetermined ratio, and each tensor corresponds to a different frequency band;

[0012] The segmented tensors are processed independently by multi-receptive field modules with non-shared parameters, and the number of channels is unified by convolution operation;

[0013] The PQMF synthesis filter bank is used to upsample tensors with a uniform number of channels;

[0014] The upsampled tensors are merged into a single signal and then processed by another multi-receptive-field module.

[0015] The merged signal is processed by a post-network containing one-dimensional convolutional layers and activation functions, and the channel dimensions are adjusted.

[0016] The processed signal is passed through an activation function to obtain the generated target audio signal.

[0017] Furthermore, to achieve the above objectives, the present invention provides a vocoder audio generation apparatus based on frequency band segmentation, comprising:

[0018] The input module is used to acquire the original audio signal and convert the original audio signal into a Log-Mel spectrum, treating the frequency dimension of the Log-Mel spectrum as a one-dimensional signal with multiple frequency channels.

[0019] A pre-network module is used to process the Log-Mel spectrum through a pre-network containing one-dimensional convolutional layers and without non-linear activation;

[0020] The upsampling module is used to sequentially pass the processed signal through multiple upsampling processing modules. Each upsampling processing module performs upsampling through transposed convolution and uses a multi-receptive field module for signal processing.

[0021] The frequency band segmentation module is used to divide the upsampled signal into multiple tensors in the channel dimension according to a predetermined ratio after processing by the transposed convolution and multi-receptive field module. Each tensor corresponds to a different frequency band.

[0022] The multi-receptive-field independent processing module is used to process the segmented tensor independently through the multi-receptive-field module with non-shared parameters, and to unify the number of channels through convolution operation;

[0023] The PQMF upsampling module is used to upsample tensors with a uniform number of channels using the PQMF synthesis filter bank.

[0024] The signal merging and processing module is used to merge the upsampled tensors into a single signal, which is then processed by another multi-receptive field module.

[0025] The post-network module is used to process the merged signal through a post-network containing one-dimensional convolutional layers and activation functions, and to adjust the channel dimensions;

[0026] The output module is used to pass the processed signal through an activation function to obtain the generated target audio signal.

[0027] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a frequency band segmentation-based vocoder audio generation program stored in the memory and executable on the processor, wherein when the frequency band segmentation-based vocoder audio generation program is executed by the processor, it implements the steps of the frequency band segmentation-based vocoder audio generation method as described above.

[0028] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a vocoder audio generation program based on frequency band segmentation, wherein when the frequency band segmentation-based vocoder audio generation program is executed by a processor, it implements the steps of the frequency band segmentation-based vocoder audio generation method as described above.

[0029] Beneficial Effects: This invention relates to the fields of artificial intelligence and fintech, and discloses a vocoder audio generation method based on frequency band segmentation. The method converts the input audio signal into a Log-Mel spectrum, and then performs preliminary processing on the spectrum signal using a one-dimensional convolutional network. After upsampling via transposed convolution, the signal is further processed using a multi-receptor field module to extract and process features from different frequency bands. During frequency band segmentation, the signal is divided into multiple tensors along the channel dimension, each tensor corresponding to a specific frequency band range, and each is processed independently. A PQMF synthesis filter bank is used to upsample the signal with a unified number of channels again, ensuring that the high-frequency and low-frequency components are effectively restored. After signal merging, the final generated audio signal is processed by another multi-receptor field module. This invention reduces frequency aliasing introduced by transposed convolution during upsampling by using frequency band segmentation and independent multi-receptor field modules to process tensors of different frequency bands, thereby improving the quality and naturalness of audio generation, ensuring the clarity and efficiency of speech synthesis, and making the generated audio more accurate and effective in financial and voice interaction scenarios. Attached Figure Description

[0030] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0031] Figure 1 This is a schematic diagram of an application environment for a vocoder audio generation method based on frequency band segmentation in one embodiment of the present invention;

[0032] Figure 2 This is a flowchart illustrating an embodiment of the vocoder audio generation method based on frequency band segmentation of the present invention.

[0033] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the vocoder audio generation device based on frequency band segmentation of the present invention.

[0034] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0035] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0036] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0037] The vocoder audio generation method based on frequency band segmentation provided in this invention can be applied to, for example... Figure 1In this application environment, the user terminal communicates with the server via a network. The server can convert the input audio signal into a Log-Mel spectrum from the user terminal and perform preliminary processing on the spectrum signal using a one-dimensional convolutional network. After the signal is upsampled by transposed convolution, a multi-receptive-field module is used to extract and process features from different frequency bands. During frequency band segmentation, the signal is divided into multiple tensors in the channel dimension, each tensor corresponding to a specific frequency band range, and each is processed independently. A PQMF synthesis filter bank is used to upsample the signal with a unified number of channels again to ensure that the high-frequency and low-frequency components are effectively restored. After signal merging, another multi-receptive-field module processes the final generated audio signal. This invention reduces the frequency aliasing phenomenon introduced by transposed convolution during upsampling by using frequency band segmentation and independent multi-receptive-field modules to process tensors of different frequency bands, thereby improving the quality and naturalness of audio generation, ensuring the clarity and efficiency of speech synthesis, and making the generated audio more accurate and effective in scenarios such as finance and voice interaction. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0038] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the vocoder audio generation method based on frequency band segmentation provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0039] like Figure 2 As shown, the vocoder audio generation method based on frequency band segmentation proposed in this invention includes the following steps:

[0040] S10, acquire the original audio signal, and convert the original audio signal into a Log-Mel spectrum, treating the frequency dimension of the Log-Mel spectrum as a one-dimensional signal of multiple frequency channels;

[0041] In this embodiment, the raw audio signal refers to sound data acquired through an audio device (such as a microphone or other recording device). This audio signal is generally an analog signal and needs to be digitized, typically including sampling, quantization, and encoding, ultimately resulting in discrete digital audio data. The raw audio signal is acquired using an audio acquisition device, and the analog signal is converted into a digital signal using an ADC (analog-to-digital converter). The digital audio signal is typically captured using a specific sampling rate (e.g., 44.1kHz or 48kHz) and quantization bit depth (e.g., 16-bit or 24-bit). The acquired digital audio signal is represented in the time domain and subsequently processed.

[0042] The Log Mel spectrum is a method for representing the frequency content of audio signals, primarily used for audio signal feature extraction. It's a spectral representation of the audio signal, processed by a Mel filter bank and subjected to a logarithmic transform to better align with human sound perception. A Short-Time Fourier Transform (STFT) is performed on the audio signal, converting the time-domain signal to the frequency-domain signal to obtain frequency distribution information. The Fourier Transform generates the amplitude and phase spectra of the audio signal. The spectrum is then processed through a series of Mel-scale filters, which are more densely distributed in the low-frequency range, reflecting the human ear's greater sensitivity to low frequencies. The STFT spectrum is then passed through Mel filters to obtain the Mel spectrum. To better align with human perception of sound loudness, the Mel spectrum is logarithmically transformed to generate the Log Mel spectrum. This step not only compresses the dynamic range of the values ​​but also enhances the importance of the high-frequency components.

[0043] The frequency dimension of the Log-Mel spectrum is composed of discrete frequency points, each representing a specific frequency intensity. By treating these frequency points as multiple channels, the entire spectrum is transformed into a multi-channel one-dimensional signal. This process is to match the frequency information with the input format of a neural network. The frequency dimension of the Log-Mel spectrum can be viewed similarly to "channels" in image processing, with each channel corresponding to information in a different frequency band. After this processing, the Log-Mel spectrum becomes a one-dimensional signal with a width equal to the frequency dimension and a length equal to the number of time frames. Through this processing, the Log-Mel spectrum can be directly used as input to a Convolutional Neural Network (CNN) or a Recurrent Neural Network (RNN) for feature extraction.

[0044] Example Description: A financial institution has developed an intelligent voice assistant for customer self-service and transaction execution. When a user issues a voice command, such as "check account balance" or "purchase stocks," the system first acquires the user's raw voice signal through a microphone and converts it into a Log-Mel spectrum. The spectrum is then processed using a one-dimensional convolutional network, treating the frequency information as a one-dimensional signal with multiple frequency channels, and then input into the neural network for speech recognition. The system utilizes this technology for efficient audio feature extraction, ensuring accurate recognition of the user's intent. By inputting the Log-Mel spectrum into a deep learning model, the system can generate text based on the speech and provide precise voice feedback, such as "Your balance is 5000 yuan" or "Stocks have been successfully purchased."

[0045] By converting the raw audio signal into a Log-Mel spectrum and treating the frequency dimension as a one-dimensional signal across multiple channels, audio features can be extracted more effectively and matched to the input format of the neural network. This can improve the accuracy of speech recognition, reduce frequency aliasing, and enhance the naturalness of the generated audio, making it suitable for scenarios such as intelligent voice assistants and real-time voice notifications.

[0046] S20, the Log Mel spectrum is processed by a pre-network containing one-dimensional convolutional layers and without non-linear activation;

[0047] In this embodiment, a one-dimensional convolutional layer is a type of convolutional layer used for processing sequential data. Its convolution operation is performed only in one dimension, making it suitable for processing sequential data such as audio signals. Compared to two-dimensional convolutional layers (typically used in image processing), the application of one-dimensional convolutional layers in audio processing focuses more on extracting time-series or frequency information. Here, the absence of non-linear activation means that in the preceding network, the output of the convolutional layer is not processed by activation functions such as ReLU or Sigmoid, but is directly passed to subsequent layers. This design is primarily to preserve the original features of the signal and avoid non-linear transformations of those features.

[0048] The main purpose of processing the Log Mel spectrum through one-dimensional convolutional layers is to perform preliminary feature extraction of the audio spectrum. Convolutional operations can capture local features in the spectrum, but since no non-linear activation layer is used, the output of the convolution still maintains a linear relationship, which makes it easier for subsequent networks to process these signals more flexibly.

[0049] The input Log Mel spectrum is typically a two-dimensional matrix, with time frames as columns and frequency channels as rows. Each frame contains the frequency features of the audio signal. By performing a one-dimensional convolution operation along the frequency dimension, features of local frequency bands in the Mel spectrum can be captured. The convolution kernel slides along the frequency dimension, extracting local patterns from each small frequency band. This approach ensures that the convolutional layer preserves local frequency features when processing each time frame.

[0050] Because there is no activation function, the output of the convolutional layer remains linear, meaning the output signal is neither compressed nor amplified. This design is suitable for processing low-level signal features, preserving the linear information in the original data, which helps subsequent non-linear networks process these signals better. The convolved signal is then passed to the next processing network (such as higher-level convolutional layers or multi-receptive-field modules), which can further extract higher-level feature information.

[0051] Example Explanation: In a bank's intelligent voice assistant, a user issues a command via voice, such as "Check account balance." The system first acquires the user's raw voice signal and converts it into a Log-Mel spectrum. A pre-processor network containing one-dimensional convolutional layers without non-linear activation is used to process this spectrum, ensuring that the original features of the user's voice signal are preserved, enabling the system to accurately recognize the user's voice commands. This not only improves recognition accuracy but also enhances the user's interactive experience with the voice assistant.

[0052] By using a pre-processing network containing one-dimensional convolutional layers without nonlinear activation to process the Log-Mel spectrum, the linear characteristics of the audio signal can be effectively preserved, avoiding the feature distortion problem introduced by nonlinear transformation in the initial signal processing. This improves the decoding and processing capabilities of subsequent networks, resulting in more accurate and natural generated or recognized speech signals.

[0053] S30, the processed signal is sequentially passed through multiple upsampling processing modules. Each upsampling processing module performs upsampling through transposed convolution and uses a multi-receptive field module for signal processing.

[0054] In this embodiment, upsampling refers to increasing the data sampling rate to improve resolution. Here, transposed convolution (also known as deconvolution or reverse convolution) is used to achieve upsampling. The main function of transposed convolution is to insert additional values ​​into the feature map. By performing a reverse operation through the convolution kernel and stride, the originally small feature map is enlarged so that subsequent networks can process it better.

[0055] Transposed convolution can restore low-resolution signals to high-resolution signals while preserving some feature information. However, transposed convolution may introduce frequency aliasing, which needs to be addressed in subsequent processing steps.

[0056] Multiple receptive field (MRF) modules are convolutional structures that capture signal features at different scales. By applying convolutional kernels of different receptive field sizes in parallel within the same layer, both local and global features of the signal can be captured. This module design effectively processes signals upsampled through transposed convolutions and compensates for potential frequency aliasing issues. Each receptive field module can focus on feature information at different scales, thereby improving the signal's detail representation.

[0057] The input signal (i.e., the signal processed by the preprocessor network) is first upsampled through a transposed convolutional layer. Transposed convolution increases resolution by inserting zeros into the input feature map and enlarging its size. The kernel and stride of each transposed convolutional layer are set to achieve a specific upsampling factor. For example, a kernel size of 4 and a stride of 2 can enlarge the feature map to twice its original size.

[0058] The upsampled signal is then fed into a multi-receptive-field module. This module contains multiple parallel convolutional kernels, each with a different receptive field size to ensure features at different scales are captured. To reduce information loss due to upsampling and convolution, the multi-receptive-field module adds the original input signal to the convolutional output signal using residual connections. This operation enhances local features while preserving details.

[0059] After the transposed convolution and multi-receptive field modules in each upsampling processing module complete signal processing, the signal is passed to the next upsampling processing module to continue upsampling processing until the required resolution is achieved.

[0060] By employing transposed convolution through multiple upsampling processing modules and combining it with signal processing from a multi-receptive-field module, signal resolution can be improved while reducing frequency aliasing issues caused by upsampling. This enhances the ability to preserve audio detail, ensuring that the generated audio signal is natural and clear, making it particularly suitable for applications in the financial sector that require high-quality voice output.

[0061] S40, after transposed convolution and multi-receptive field module processing, divides the upsampled signal into multiple tensors in the channel dimension according to a predetermined ratio, with each tensor corresponding to a different frequency band;

[0062] In this embodiment, when processing audio signals, different frequency bands of the signal are treated as different "channels." In this case, "channel dimension" refers to the signal representation corresponding to each frequency band. By dividing the signal according to the channel dimension, the system can process information in each frequency band separately.

[0063] The signal is divided into multiple tensors along the channel dimension according to a predetermined ratio. Different frequency bands have different importance; generally, the low-frequency portion has a greater impact on audio perception, so it can be allocated more channels, while the high-frequency portion is allocated relatively fewer channels. This predetermined ratio division makes the information content of each frequency band more balanced.

[0064] Each tensor represents the signal characteristics of a certain frequency band. Dividing the upsampled signal into multiple tensors according to frequency helps to process each frequency band independently, which can reduce information interference between different frequency bands and improve the detail preservation of the generated signal.

[0065] The signal processed by transposed convolution and the multi-receptive-field module has been upsampled, resulting in high-resolution audio feature maps. These feature maps have not yet undergone frequency band segmentation in the channel dimension; they need to be segmented according to a predetermined ratio in the next step.

[0066] According to the preset frequency band distribution rules, the signal is divided into tensors corresponding to different frequency bands in the channel dimension. For example, the low-frequency tensor may be allocated more channels, while the high-frequency tensor may be allocated relatively fewer channels. The specific segmentation ratio can be set according to application requirements, such as dividing the low-frequency, mid-frequency, and high-frequency tensors in a 4:2:1 ratio.

[0067] Each segmented tensor corresponds to a specific frequency band. This frequency band division allows features within each frequency range to be processed independently, ensuring that low-frequency and high-frequency features do not interfere with each other, thus maintaining higher audio generation quality.

[0068] Example Explanation: In a financial market volatility notification system, the system needs to generate voice prompts based on market data. After upsampling and processing by a multi-receptive-field module, the generated voice signal is divided into multiple tensors according to frequency bands, with each tensor representing a different frequency band. Tensors in the low-frequency bands are allocated more channels to ensure the clarity of the voice prompts, while the high-frequency bands retain details. This segmentation method makes the generated voice notifications both natural and easy to understand when alerting users to market volatility.

[0069] By dividing the upsampled signal into multiple tensors along the channel dimension according to a predetermined ratio, with each tensor corresponding to a different frequency band, information interference between different frequency bands can be effectively reduced. This frequency band segmentation improves the signal's feature preservation capability, ensuring that both low-frequency and high-frequency bands can be accurately captured, resulting in a more natural and clearer audio signal, making it particularly suitable for applications in financial voice interaction and market notification scenarios.

[0070] S50, the segmented tensors are processed independently by multi-receptive field modules with non-shared parameters, and the number of channels is unified by convolution operation;

[0071] In this embodiment, a multi-receptive-field module refers to a convolutional layer with different receptive fields, used to capture features at different scales. The larger the receptive field, the wider the range of features captured by the convolutional layer, and the more suitable it is for capturing global features; convolutional layers with smaller receptive fields focus on local features.

[0072] Parameter non-sharing means that the tensors of each frequency band are processed in different multi-receptive field modules, and the weights and parameters of these modules are independent of each other. This approach ensures that feature extraction from different frequency bands does not interfere with each other, enabling more accurate extraction of the corresponding feature information for each frequency band.

[0073] After the signal is band-segmented, each tensor corresponds to a frequency band. Through a multi-receptive field module with non-shared parameters, the system can independently extract features from each frequency band, preventing information from different frequency bands from interfering with each other and ensuring that the details of each frequency band can be accurately captured.

[0074] After processing each tensor independently, the system uses convolution operations to unify the number of channels in all tensors. Unifying the number of channels ensures that features from different frequency bands can be effectively merged and further processed in subsequent processing steps.

[0075] Each segmented tensor is input into its independent multi-receptive-field module. These modules have different receptive field sizes, allowing feature extraction at multiple scales. Parameter non-sharing ensures that feature extraction for each frequency band is independent, guaranteeing that information from different frequency bands does not interfere with each other, and that each module focuses on feature extraction for its current frequency band.

[0076] Each tensor processed by a multi-receptive-field module may have a different number of channels. One-dimensional convolution operations are used to unify the number of channels in each tensor to the same value, facilitating signal fusion in subsequent steps. The kernel size and stride can be set according to requirements to ensure uniformity in the number of channels and the integrity of feature information.

[0077] Example Explanation: In a financial voice assistant system, a user issues a voice command, such as "Check account balance" or "Transfer money." The system first acquires and processes the user's voice signal, segmenting it into tensors of different frequency bands. Through a multi-receptive-field module with non-shared parameters, the system can independently process the features of each frequency band. The low-frequency band processes the main content of the speech, such as the keyword "account balance," while the high-frequency band is responsible for capturing detailed audio information, such as the clarity and intonation of the speech. After independent processing, the system uses convolution operations to unify the number of channels in each frequency band, merging the signals into a single output signal, and finally generating a natural and fluent voice response. For example, when a user asks "What is my account balance?", the system accurately identifies the key content in the speech through this step and quickly generates a voice prompt, "Your balance is 5000 yuan." This processing method ensures high-quality speech synthesis, enabling customers to receive accurate and clear feedback.

[0078] By using multi-receptive-field modules with non-shared parameters to independently process tensors in each frequency band, the features of each band can be extracted independently and accurately, reducing mutual interference between different frequency bands. By unifying the number of channels through convolution operations, the processed tensors can be effectively merged, facilitating subsequent processing. This improves the efficiency and accuracy of audio signal feature extraction, making it particularly suitable for speech generation and speech recognition scenarios, resulting in more natural and fluent audio.

[0079] S60 uses a PQMF synthesis filter bank to upsample tensors with a uniform number of channels;

[0080] In this embodiment, PQMF (Pseudo-Quadrature Mirror Filter) is a multi-band filter used for frequency band segmentation and reconstruction of signals. A common application in audio processing is the efficient frequency division and synthesis of audio signals.

[0081] The PQMF filter bank is used to upsample tensors with a uniform number of channels, ensuring that the generated signal has high resolution and retains more frequency information. This filter bank can provide fine-grained frequency band division across the signal's spectrum, reduce frequency aliasing, and effectively reconstruct the signal.

[0082] Upsampling is a technique that increases the sampling rate of a signal through interpolation and reconstruction. In audio processing, by increasing the signal's resolution, the generated audio signal becomes smoother and more natural. PQMF synthesis filter banks can refine the frequency domain of the signal, reducing artifacts and aliasing during upsampling and improving the quality of the generated audio.

[0083] After processing by the multi-receptor field module, the number of channels in the signal has been unified. The tensor at this point contains characteristic information from different frequency bands and is ready for further upsampling to improve signal resolution. The PQMF synthesis filter bank processes this tensor, dividing the signal into different frequency bands and upsampling each band separately. Through fine filtering and reconstruction of each frequency band, the filter bank reduces artifacts and frequency aliasing that may be introduced during signal upsampling. Through the PQMF filter bank, the signal's frequency response is further optimized during upsampling, ensuring that the signal's naturalness and clarity are maintained while improving resolution. After processing by the PQMF filter bank, the signal resolution is improved, and the frequency characteristics are clearer. This upsampled signal will be used in subsequent generation processes to produce high-quality audio output.

[0084] By utilizing a PQMF synthesis filter bank for upsampling, the audio signal resolution is improved while artifacts and frequency aliasing are reduced. The resulting audio signal is smoother and more natural, enhancing the overall speech quality.

[0085] S70 merges the upsampled tensors into a single signal, which is then processed by another multi-receptive-field module.

[0086] In this embodiment, after upsampling, the tensor of each frequency band already contains the features of the corresponding frequency range. Next, the system needs to merge these tensors into a single signal along the channel dimension. The merged signal contains information from all frequency bands and is a comprehensive representation of the entire audio signal. This ensures that the segmented frequency band features can be processed as a unified whole in subsequent steps, rather than as isolated frequency band information.

[0087] The merged signal is then processed again by a multi-receptive-field module. The different kernel sizes (i.e., the receptive field sizes) of this module capture both local and global features of the signal. At this stage, the module further enhances the details in the signal and corrects frequency inconsistencies that may occur during upsampling or merging. This multi-receptive-field module is responsible for processing the overall signal characteristics, ensuring that the output signal possesses sufficient detail and global consistency.

[0088] After upsampling and filtering, the tensors of each frequency band are merged along the channel dimension. This merging operation combines the feature information from different frequency bands into a single, complete signal, ready for subsequent convolution processing. The merged signal is then passed to another multi-receptive-field module. This module extracts multi-scale features of the signal using parallel convolution operations with different receptive field sizes. Small receptive fields capture local details, while large receptive fields focus on the overall features of the signal. Through the processing of the multi-receptive-field module, the consistency of the signal in both frequency and time dimensions is ensured, while further enhancing the signal's feature representation.

[0089] By merging the upsampled tensors into a single signal and processing it again through a multi-receptive-field module, the system can capture more detailed features while maintaining signal consistency. This improves the overall quality of speech generation, making the audio signal more natural and fluent.

[0090] S80 processes the merged signal through a post-network containing one-dimensional convolutional layers and activation functions, adjusting the channel dimensions;

[0091] In this embodiment, a one-dimensional convolutional layer is used to process temporal data, such as audio signals. In the merged signal, features may originate from different frequency bands, and the signal's dimensionality still needs further adjustment after passing through a multi-receptive-field module. Through the one-dimensional convolutional layer, the system can extract or compress specific frequency features from the merged signal based on different kernel sizes and strides, thereby achieving the purpose of adjusting the channel dimension. One-dimensional convolution primarily performs convolution operations on the time or frequency dimension, ensuring that the number of signal channels matches the needs of subsequent networks.

[0092] The main role of activation functions is to introduce nonlinear transformations, enabling the network to learn more complex mapping relationships. In this step, after the merged signals pass through the convolutional layer, activation functions (such as ReLU or tanh) are applied to perform nonlinear transformations on the signal output, ensuring that the processed signal has stronger expressive power and feature extraction capabilities.

[0093] The activation function used can be:

[0094] ReLU (Rectified Linear Unit): Only outputs positive values, and outputs zero for negative values. It is widely used in deep neural networks.

[0095] tanh: Compresses the input value to the range of -1 to 1, used for processing symmetric data.

[0096] Channel dimension refers to the feature dimension of a signal. After convolution and activation, the number of channels is adjusted to match the desired input or output dimension of the network. This step is crucial, ensuring that the output dimension of different convolutional layers is consistent, thus preparing for subsequent processing.

[0097] The merged signal is input into a one-dimensional convolutional layer for processing. By setting different kernel sizes and strides, the one-dimensional convolutional layer can adjust the channel dimension, compressing or expanding specific frequency features. The output of the convolutional layer is processed by an activation function, commonly ReLU or tanh. The activation function introduces nonlinearity, mapping the linear output after convolution to a nonlinear output, thereby enhancing the signal's feature representation capability. The number of channels in the convolutional and activated signals is uniformly adjusted to the desired dimension. This channel number adjustment ensures that the signal can be further processed by subsequent network layers or directly output.

[0098] By employing a post-processor network containing one-dimensional convolutional layers and activation functions, the system can flexibly adjust the channel dimensions of the signal, ensuring the naturalness and high quality of the generated audio signal. This effectively improves the expressiveness of the speech signal, reduces frequency aliasing and artifacts, and results in clearer and smoother generated speech.

[0099] S90 processes the signal through an activation function to obtain the generated target audio signal.

[0100] In this embodiment, the activation function is a mathematical function typically used to non-linearly map the output of a linear transformation in a neural network, enabling the model to handle more complex features. In the audio generation process, the activation function ensures that the generated audio signal is within a specified numerical range, thus preventing distortion due to excessively large or small audio signals.

[0101] The activation functions used include:

[0102] tanh: Limits the output to between -1 and 1, suitable for generating symmetric data.

[0103] ReLU (Rectified Linear Unit): Retains the portion of the output value that is greater than zero, and sets the portion that is less than zero to zero.

[0104] Sigmoid: Compresses the output value to between 0 and 1, and is often used in classification problems.

[0105] In audio generation, the tanh function is frequently used because audio signals typically fluctuate within a positive and negative range. tanh can limit the output range of the signal, ensuring that the amplitude of the generated audio does not exceed a certain limit. The signal processed by the activation function generates the final target audio signal. This step ensures that the audio signal value is within a reasonable range, avoiding distortion problems caused by excessively large or small values ​​in the generated signal.

[0106] After being processed by a one-dimensional convolutional layer and a multi-receptive field module, the signal already contains sufficient audio features. At this point, the numerical range of the signal may be too large or too small to directly generate a high-quality audio signal.

[0107] The signal is then passed through an activation function (usually tanh) to compress the output to a range of -1 to 1. This step controls the dynamic range of the generated signal, resulting in a more natural and distortion-free audio signal. The choice of activation function is crucial; tanh effectively prevents the generated signal from over-expanding in amplitude, leading to distortion or volume imbalance. The signal processed by the activation function is the target audio signal. This signal can be played back using a vocoder or other audio processing devices, ultimately achieving high-quality audio generation.

[0108] Example Description: In a bank's intelligent voice assistant system, customers inquire about account information via voice. When a customer issues the voice command "Check account balance," the system first processes the customer's voice signal. The signal obtained through multiple convolutions and feature extraction is then processed by the tanh activation function. This process limits the amplitude of the generated voice signal to the range of -1 to 1, ensuring a natural and smooth voice output. The target audio signal generated through this step contains a clear and natural sound feedback: "Your balance is 5000 yuan," which is provided to the customer in real time, helping them quickly obtain financial information. This voice generation technology controls the signal amplitude through the tanh activation function, avoiding voice signal distortion or excessive volume, making the generated voice more natural and smooth in practical applications, thus improving the user experience.

[0109] By processing the generated signal using activation functions (such as tanh), the dynamic range of the audio signal can be effectively controlled, ensuring that the final generated target audio signal is natural, clear, and distortion-free. This not only improves the quality of speech synthesis but also ensures stability during the audio generation process, making it particularly suitable for scenarios requiring high-quality audio generation, such as financial voice assistants and real-time market notification systems.

[0110] This invention relates to the fields of artificial intelligence and fintech, and discloses a vocoder audio generation method based on frequency band segmentation. The method converts the input audio signal into a Log-Mel spectrum, and then performs preliminary processing on the spectrum signal using a one-dimensional convolutional network. After upsampling via transposed convolution, the signal is further processed using a multi-receptor field module to extract and process features from different frequency bands. During frequency band segmentation, the signal is divided into multiple tensors along the channel dimension, each tensor corresponding to a specific frequency band range, and each is processed independently. A PQMF synthesis filter bank is used to upsample the signal with a unified number of channels again, ensuring that the high-frequency and low-frequency components are effectively restored. After signal merging, the final generated audio signal is processed by another multi-receptor field module. This invention reduces frequency aliasing introduced by transposed convolution during upsampling by using frequency band segmentation and independent multi-receptor field modules to process tensors of different frequency bands, thereby improving the quality and naturalness of audio generation, ensuring the clarity and efficiency of speech synthesis, and making the generated audio more accurate and effective in financial and voice interaction scenarios.

[0111] In one embodiment, S20 includes:

[0112] S201, the Log Mel spectrum is processed by a one-dimensional convolutional layer having a predetermined kernel size and stride;

[0113] S202, No nonlinear activation function is used after the one-dimensional convolutional layer, and the output of the one-dimensional convolutional layer is used as the processed signal;

[0114] S203 performs batch normalization on the processed signal and then transmits the processed signal to the upsampling processing module.

[0115] In this embodiment, the one-dimensional convolutional layer performs convolution operations in the frequency dimension to extract local features of the audio signal. The Log Mel spectrum is the representation of the audio signal after passing through a short-time Fourier transform (STFT) and a Mel filter bank, containing the spectral features of the audio. Through convolution operations, local correlations between different frequency bands can be effectively captured. The predetermined kernel size and stride determine the range and movement step of the convolution operation; the larger the kernel, the wider the range of features captured; the stride determines the frequency at which the convolution window moves within the input signal.

[0116] Following a convolution operation, the design without a non-linear activation function means that the output remains linear. This design preserves the original structure and features of the signal, without altering its shape through non-linear mapping. Its purpose is to avoid introducing non-linear transformations in the initial stages, ensuring that the signal retains its most original features and providing high-fidelity input for subsequent network layers.

[0117] After convolution, the signal undergoes batch normalization, a process designed to accelerate model training and reduce the vanishing gradient problem. Batch normalization adjusts the mean and variance of the input signal to zero mean and unit variance, thereby stabilizing the signal distribution and ensuring that subsequent layers can process it more effectively. The more stable signal after batch normalization helps the neural network converge faster during training.

[0118] After convolution and normalization, the signal is passed to the subsequent upsampling module to improve the resolution and perform further signal processing.

[0119] First, the Log-Mel spectrum is acquired and input into a one-dimensional convolutional layer for processing. The kernel size and stride are set according to requirements, and the convolution operation slides along the frequency dimension to extract local features. The convolved signal is passed directly without any non-linear activation function. Maintaining linear features is to preserve the original spectral characteristics for easier subsequent processing. The convolutional output is batch normalized to ensure the uniformity and stability of the signal distribution, which is beneficial for subsequent neural network processing layers. Finally, the normalized signal is passed to the next upsampling module, ready for further upsampling operations.

[0120] In one specific implementation, within a financial market notification system, the kernel size and stride of the convolutional layer may need to be adjusted according to the rate of change in market data. For example, rapidly fluctuating markets require shorter strides to capture rapidly changing data features, while stable markets can use larger strides to reduce computational load. A dynamic kernel selection mechanism is implemented to adjust the convolution stride based on the market volatility rate, thereby processing market data features more efficiently.

[0121] In another specific implementation, to ensure that the linear features of the speech signal are not distorted without using a nonlinear activation function, normalization processing (such as layer normalization or instance normalization) can be added after the convolution operation to maintain the uniformity of the signal, especially when processing complex multilingual speech. A feature alignment algorithm for audio signals in multilingual environments can be added, using normalization and alignment techniques to ensure that the features of different languages ​​are processed uniformly in the convolutional layer.

[0122] This embodiment processes the Log-Mel spectrum using a pre-processor network containing one-dimensional convolutional layers but without nonlinear activation. This preserves the original spectral characteristics of the signal in the initial stages, avoiding premature nonlinear transformations and improving the accuracy of subsequent processing. Simultaneously, batch normalization stabilizes the signal distribution, ensuring higher efficiency and accuracy for the neural network during training and generation.

[0123] In one embodiment, S30 includes:

[0124] S301, in each upsampling processing module, the processed signal is upsampled by applying transposed convolution;

[0125] S302, set the kernel size, stride and padding of the transposed convolution to achieve the desired upsampling factor;

[0126] S303, the upsampled signal is transmitted to the multi-receptive field module, which includes multiple parallel residual blocks with different receptive fields;

[0127] S304 applies dilated convolution in the multi-receptive-field module and uses residual connections to fuse the output of the multi-receptive-field module with the input signal.

[0128] In this embodiment, transposed convolution is one of the most commonly used operations in upsampling, which improves resolution by expanding the size of the feature map. Compared to regular convolution, it performs the inverse operation, enabling the upsampling of low-resolution feature maps to higher resolution. The kernel size and stride of the transposed convolution determine the upsampling factor, while the padding method ensures that the output size meets expectations. The upsampling factor is determined by the kernel size, stride, and padding method; larger kernels and strides can improve signal resolution more quickly.

[0129] Multi-receptive-field modules consist of multiple convolutional layers with different receptive fields, enabling them to capture signal features at different scales. Convolutional layers with smaller receptive fields focus on capturing local details, while those with larger receptive fields capture global information. Combining different receptive fields improves the network's ability to simultaneously focus on both detailed and overall features when processing audio signals. Dilated convolution is a method that expands the receptive field by inserting holes between convolutional kernels, allowing the convolutional operation to capture a wider range of features without increasing computational cost. The advantage of dilated convolution is its ability to preserve both detailed information and global features, making it particularly suitable for handling long-term dependencies in audio signals.

[0130] Residual connections are a technique commonly used in deep learning to address the vanishing gradient problem in deep networks. By residually connecting the output of a multi-receptive-field module to the input signal, some features of the original signal can be preserved, enhancing the network's robustness and feature transfer capability.

[0131] In each upsampling module, the signal is upsampled through transposed convolution. The kernel size, stride, and padding are set to ensure each module appropriately upsamples the signal. The upsampling factor depends on these parameter settings, with the goal of progressively restoring the low-resolution signal to high resolution.

[0132] The upsampled signal enters the multi-receptive-field module. This module consists of multiple convolutional layers with different receptive fields, each responsible for capturing local and global signal features. The kernel size for each receptive field is different, ensuring that multi-dimensional features of the signal are captured at different scales.

[0133] In the multi-receptive-field module, dilated convolution further enhances the network's ability to capture global features. By fusing the output of the multi-receptive-field module with the input signal through residual connections, it can both preserve some features of the input signal and enhance the network's learning ability, thereby improving the overall performance of the signal.

[0134] In one specific implementation, within an intelligent voice assistant, the upsampling factor of the transposed convolution may be adaptively adjusted based on the user's speech environment. For example, in noisy environments, a larger stride and convolution kernel are used to improve the signal's noise resistance; while in quiet environments, a smaller stride is used to capture more details. Introducing an environment-adaptive convolution strategy adjusts the upsampling parameters in real time based on background noise to ensure the clarity and fidelity of the speech signal.

[0135] Multi-receptor field modules in voice assistants can perform real-time optimization based on the frequency characteristics of the user's speech, adjusting the size of different receptive field convolutional kernels to process high-frequency and low-frequency signals. For example, for keywords in voice commands (such as "balance"), a smaller receptive field can capture high-frequency details, while a larger receptive field is used for overall speech fluency. A keyword enhancement mechanism is added, utilizing the multi-receptor field module to strengthen and extract key information, thereby improving the accuracy and response speed of the voice assistant's recognition.

[0136] When processing long speech inputs, dilated convolutions can dynamically adjust their dilation factor based on the speech length, ensuring that long-term dependent features are captured. Residual connections, on the other hand, help maintain the overall consistency of the speech input and prevent information loss during long-term processing. A time-dynamic adjustment mechanism is introduced to adjust the dilation factor of the dilated convolution based on the length of the speech signal, combined with residual connections to enhance long-term dependent features.

[0137] This embodiment achieves signal upsampling and feature extraction from multiple receptive field modules through transposed convolution, thereby improving the resolution and feature representation capability of the audio signal. Simultaneously, dilated convolution and residual connection techniques effectively preserve the signal's details and global information, enhancing its expressiveness and making the generated audio signal more natural and clear in voice assistant and market notification scenarios.

[0138] In one embodiment, in S40 above, the upsampled signal is divided into multiple tensors along the channel dimension according to a predetermined ratio, including:

[0139] S401, determine a predetermined ratio for channel segmentation, wherein the predetermined ratio corresponds to different frequency bands;

[0140] S402, the upsampled signal is divided into multiple tensors in the channel dimension according to the predetermined ratio;

[0141] S403 assigns each tensor to a corresponding frequency band, arranged from low frequency to high frequency, and identifies each tensor individually.

[0142] In this embodiment, the predetermined channel segmentation ratio refers to the reasonable division of the signal based on the importance or frequency range occupied by different frequency bands. Different frequency bands contribute differently to the overall audio, therefore, different numbers of channels are allocated according to the actual application scenario. Generally, low-frequency components have a greater impact on human hearing and are allocated more channels; high-frequency components are allocated fewer channels. The purpose of determining this ratio is to ensure that the characteristics of each frequency band can be fully expressed, thereby improving signal clarity and fidelity.

[0143] Once the predetermined frequency band proportions are determined, the system segments the upsampled signal along the channel dimension according to these proportions. Each channel corresponds to a specific frequency band, and the segmented tensors retain the signal characteristics of different frequency ranges. This ensures that the information of each frequency band can be processed independently and further optimized in subsequent processing steps.

[0144] After segmentation, the system assigns each tensor to a different frequency band, such as low-frequency, mid-frequency, or high-frequency. This operation clarifies the allocation of each frequency band, making subsequent processing for different frequency bands more efficient and accurate. Each tensor not only corresponds to a frequency range but is also individually identified based on the importance and characteristics of these frequency bands, facilitating subsequent operations and optimizations.

[0145] Depending on the application scenario (such as speech generation or market notification), the system sets the channel segmentation ratio based on experience or predefined rules. For example, more channels are allocated to the low-frequency band, which is more sensitive to the human ear, while fewer channels are allocated to the high-frequency band. A common segmentation ratio is 4:2:1, corresponding to the channel allocation for low-frequency, mid-frequency, and high-frequency bands, respectively. After upsampling, the signal becomes a multi-channel signal. In the channel dimension, the signal is divided into different tensors according to a predetermined ratio. For example, if the signal contains 12 channels, it can be allocated in a 4:2:1 ratio as 4 low-frequency channels, 2 mid-frequency channels, and 1 high-frequency channel. Each tensor corresponds to a different frequency band and retains the information of that frequency band. Each tensor is explicitly identified as a different frequency band, such as low frequency (0Hz-1kHz), mid-frequency (1kHz-5kHz), and high frequency (5kHz-20kHz). This identification helps the system adopt different processing strategies for different frequency bands in subsequent processing, such as different convolution kernels or different activation functions.

[0146] In one specific implementation, within the intelligent voice assistant, the system sets a specific channel segmentation ratio based on the pitch characteristics of the user's voice. For example, more channels can be allocated to the low-frequency portion of the user's voice (which typically contains the core content) to ensure clarity. Conversely, fewer channels can be allocated to the high-frequency portion to conserve computing resources. This introduces a voice frequency adaptive segmentation strategy, dynamically adjusting the channel segmentation ratio based on the spectral characteristics of the user's voice, resulting in generated speech that more closely matches the actual sound characteristics.

[0147] This embodiment divides the upsampled signal into multiple tensors along the channel dimension according to a predetermined ratio, enabling the system to perform refined processing on the characteristics of different frequency bands. Important information in the low-frequency band is enhanced, while details in the high-frequency band are preserved, resulting in a more natural and accurate audio signal.

[0148] In one embodiment, the above S50 includes:

[0149] S501 inputs each tensor into an independent multi-receptive field module, performs multi-scale feature extraction on each tensor, the parameters between each multi-receptive field module are not shared, and each multi-receptive field module contains residual blocks with different dilation factors and convolution kernel sizes.

[0150] S502 performs multi-scale feature fusion processing on each tensor to enhance the expression of frequency band features;

[0151] S503 passes the output of the multi-receptive field module through a one-dimensional convolutional layer to unify the number of channels in the tensor.

[0152] S504 applies an activation function to perform a nonlinear transformation on a tensor with a uniform number of channels.

[0153] In this embodiment, each tensor is input into an independent multi-receptive-field module. A multi-receptive-field module is one capable of capturing features at different scales with different receptive fields (i.e., kernel size and dilation factor). In this design, each multi-receptive-field module focuses on tensors of a specific frequency band and performs feature extraction based on its specific parameter settings (kernel size and dilation factor). Parameter non-sharing means that tensors of each frequency band are processed in their own independent multi-receptive-field modules without interference. This ensures that features from different frequency bands can be extracted and optimized independently.

[0154] The multi-receptive-field module extracts multi-scale features by applying convolutional layers with different receptive field sizes and dilation factors. Dilated convolutions expand the receptive field, capturing more global information, while smaller convolutional kernels are used to obtain local details. This multi-scale feature extraction enables the extraction of information at different scales without increasing computational cost.

[0155] The multi-scale features extracted from each tensor are fused, that is, features extracted by different convolution kernel sizes and dilation factors are combined, so that the frequency band features can be represented more richly at multiple levels. This fusion process improves the system's ability to express frequency band features and ensures that information from different frequency bands can be effectively extracted and fused.

[0156] The feature tensors output by the multi-receptor field module may have different numbers of channels. These tensors are convolved using a one-dimensional convolutional layer to unify their channel count, ensuring smooth processing in subsequent steps. This step helps integrate features from different frequency bands into a single signal.

[0157] After unifying the number of channels, an activation function (such as ReLU or tanh) is used to perform a nonlinear transformation on the signal. The activation function introduces nonlinear characteristics, enabling the network to learn and express more complex mapping relationships. This step ensures that the signal's expressive power is further enhanced and provides stronger feature representations for subsequent steps.

[0158] Each segmented tensor is input into its corresponding multi-receptive-field module. Each module has a different convolutional kernel size and dilation factor to ensure feature extraction from different scales. These modules do not share parameters, ensuring independent processing of tensor features for each frequency band. After extracting features at different scales, each multi-receptive-field module fuses these features. The fused tensor retains both global features and local details. The fused features are then passed through a one-dimensional convolutional layer to unify the number of channels across all tensors. This ensures that features from different frequency bands can be further merged without incompatibility issues due to varying channel counts. The signal processed by the one-dimensional convolutional layer undergoes a non-linear transformation using an activation function to enhance feature representation. The activation function helps the network learn more complex features, ensuring the signal behaves more naturally in different application scenarios.

[0159] In one specific implementation, within the intelligent voice assistant, speech features of each frequency band are processed through different multi-receptive field modules to extract high-frequency and low-frequency feature information respectively. For low-frequency speech information, a larger convolutional kernel and a smaller dilation factor are used to extract the core content, while for the high-frequency part, a smaller convolutional kernel and a larger dilation factor are used to capture more detailed information. An adaptive speech clarity enhancement mechanism is introduced, which, by adjusting the dilation factor and convolutional kernel size, ensures that the features of important parts of the speech (such as keywords) can be extracted more accurately, generating clearer speech output.

[0160] This embodiment introduces a multi-receptive field module with non-shared parameters and multi-scale feature extraction technology, enabling the system to better extract and fuse feature information from different frequency bands. Furthermore, by unifying the number of channels and applying non-linear processing of the activation function, both signal processing efficiency and feature representation capability are improved.

[0161] In one embodiment, S70 includes:

[0162] S701 combines the upsampled tensors into a single signal through summation.

[0163] S702, the merged signal is input to an additional multi-receptive field module, the additional multi-receptive field module containing residual blocks with different receptive fields;

[0164] S703 performs multi-scale feature extraction on the merged signal to capture the feature information of the integrated frequency band;

[0165] S704 performs feature fusion on the output of the multi-receptive field module to enhance the overall feature representation of the signal;

[0166] The S705 transmits the output of the fused multi-sensory field module to the back-end network.

[0167] In this embodiment, merging upsampled tensors from different frequency bands into a single signal is a common operation, typically performed using element-wise summation. The advantage of summation is that it preserves the characteristics of each tensor without introducing excessive computational complexity. Each tensor represents information from a different frequency band; summation generates a comprehensive signal containing features from all frequency bands.

[0168] The merged signal is then processed in an additional multi-receptive-field module. This module, similar to the previous multi-receptive-field module, contains convolutional layers with multiple receptive fields and residual blocks. The residual blocks help preserve the original features of the signal and extract feature information at different levels through convolutional kernels of multiple scales. The purpose of this step is to further enrich the feature representation capabilities of the merged signal.

[0169] Through additional multi-receptive-field modules, multi-scale features of the signal are extracted, capturing the interaction information between different frequency bands. Compared to previous individual frequency band processing, this step extracts features from the merged overall signal, enhancing the comprehensive expressive power between frequency bands. Feature fusion combines the features extracted by different convolutional kernels in the multi-receptive-field modules to obtain a more expressive feature representation. These features can come from convolutional layers in different receptive fields, and fusion improves the global expressive power of the signal. Finally, the fused signal is passed to a post-processing network for further processing or generation of the final audio signal. The post-processing network may include more one-dimensional convolutional layers or non-linear activation functions to complete the final signal generation.

[0170] In one specific implementation, within the financial market notification system, market data tensors from different frequency bands are summed and merged into a single signal, which is then fed into a multi-receptive-field module for multi-scale feature extraction. This module extracts long-term market change information using convolutional kernels in the residual blocks, while simultaneously capturing details of short-term fluctuations. The fused feature signal can generate high-quality voice notifications in the subsequent network. A market frequency-adaptive fusion mechanism is introduced to fuse features using convolutional kernels of different receptive-field sizes, addressing rapid market fluctuations or long-term changes, ensuring that the generated voice notifications are timely and reflect the overall market picture.

[0171] This embodiment, by summing and merging the upsampled tensors and then feeding them into an additional multi-receptive-field module for processing, can better capture the feature interactions between different frequency bands. The feature fusion process enhances the global expressiveness of the signal, resulting in a higher clarity and naturalness in the final audio signal used in intelligent voice assistants and financial market notification scenarios.

[0172] In one embodiment, S10 includes:

[0173] S101, acquire the raw audio signal to be processed;

[0174] S102, preprocess the original audio signal, including noise reduction and normalization.

[0175] S103, convert the preprocessed original audio signal into a Log Mel spectrum;

[0176] S104, the frequency dimension of the Log Mel spectrum is regarded as multiple frequency channels to form a multi-channel one-dimensional signal;

[0177] S105, adjust the multi-channel one-dimensional signal into a tensor format that conforms to the input neural network.

[0178] In this embodiment, the raw audio signal is first acquired from an external device or data source. This audio signal can come from a voice assistant, a speech recognition system, or other audio processing systems. Ensuring signal integrity is crucial during signal acquisition.

[0179] In audio signal processing, denoising aims to remove background noise from audio signals to improve clarity and intelligibility. Commonly used denoising algorithms include frequency domain filters and waveform denoising techniques.

[0180] Normalization is used to adjust the amplitude of audio signals to a uniform range, allowing audio signals of different volumes to be processed uniformly. This helps the network model process input signals more stably and avoids performance degradation caused by volume differences.

[0181] The Log-Mel spectrum is derived by performing a Short-Time Fourier Transform (STFT) on the audio signal and then transforming the result using a Mel filter bank. The Log-Mel spectrum, with its logarithmic frequency representation, more closely approximates the auditory characteristics of the human ear, making it particularly suitable for speech recognition, generation, and processing. This transformation effectively captures the frequency components of the audio signal and provides finer resolution of frequencies (low frequencies) that are more important to the human ear.

[0182] The frequency dimension of the Log-Mel spectrum comprises multiple frequency bands. Each frequency band can be viewed as an independent channel, and these channels are combined to form a multi-channel one-dimensional signal, thereby representing the spectral information in a way that the network can understand. This one-dimensional signal form can better convey audio characteristics and preserve the time-frequency information in the audio.

[0183] In deep learning models, it is typically necessary to convert input data into tensor format. In this step, the resulting multi-channel one-dimensional signals are formatted into tensor format that the neural network can process, facilitating subsequent feature extraction and processing. Tensors can be represented as arrays with multiple dimensions, making them suitable for processing in deep learning models such as convolutional neural networks (CNNs).

[0184] The system acquires raw audio signals from microphones or other audio input devices. These signals are typically represented as waveforms and contain time and amplitude information. Signal processing techniques are used to denoise the audio, removing background noise and improving signal clarity. Next, normalization adjusts the amplitude of the audio signal to a uniform range, ensuring signal stability during subsequent processing. The audio signal is converted to a frequency domain representation using a Short-Time Fourier Transform (STFT). Then, a Mel filter bank is used to convert the frequency domain representation to a Mel spectrum, and a logarithmic operation is performed to obtain the Log Mel spectrum. This process preserves the frequency characteristics of the audio signal and compresses high-frequency information, enhancing the representation of low-frequency signals. The frequency dimension of the Log Mel spectrum contains multiple frequency bands, each of which can be considered an independent frequency channel. These frequency band signals are combined into a multi-channel one-dimensional signal, thus preserving the time-frequency variation information of the audio features in the time dimension. Finally, the system converts the multi-channel one-dimensional signal into a tensor format suitable for neural network input, allowing the network to further process the audio features.

[0185] In one specific implementation, for intelligent voice assistants, the key content of the voice signal may lie in specific frequency bands. Sensitivity to key frequency bands can be enhanced by setting different resolutions for different frequency bands of the Log-Mel spectrum, for example, by strengthening the capture of low-frequency human voices. A weighted frequency band enhancement algorithm can be employed to give more attention to the core frequency bands in the speech, thereby improving the accuracy of speech recognition.

[0186] When converting the Log-Mel spectrum into a tensor, temporal enhancement techniques can be introduced to enhance the temporal information of speech features. Maintaining the continuity of speech is crucial for the responsiveness of intelligent assistants in real-time voice interaction. Adding temporal convolutional layers, by processing multi-frame audio information, ensures that the system can accurately extract features from continuous speech signals.

[0187] This embodiment acquires the raw audio signal and converts it into a Log-Mel spectrum, effectively capturing the frequency features in the audio. These features are then input into a neural network as multi-channel one-dimensional signals. This makes audio signal processing more efficient and accurate, making it particularly suitable for applications such as intelligent voice assistants and financial market notification systems, improving the accuracy of signal processing and the generation effect.

[0188] In one embodiment, a vocoder audio generation apparatus based on frequency band segmentation is provided, which corresponds one-to-one with the vocoder audio generation method based on frequency band segmentation in the above embodiments. (Refer to...) Figure 3 , Figure 3This is a schematic diagram of the functional modules of a preferred embodiment of the vocoder audio generation device based on frequency band segmentation of the present invention. The modules include an input module 10, a pre-amplifier network module 20, an upsampling module 30, a frequency band segmentation module 40, a multi-receptor field independent processing module 50, a PQMF upsampling module 60, a signal combining and processing module 70, a post-amplifier network module 80, and an output module 90. Detailed descriptions of each functional module are as follows:

[0189] Input module 10 is used to acquire the original audio signal and convert the original audio signal into a Log-Mel spectrum, treating the frequency dimension of the Log-Mel spectrum as a one-dimensional signal of multiple frequency channels;

[0190] The pre-network module 20 is used to process the Log-Mel spectrum through a pre-network containing one-dimensional convolutional layers and without non-linear activation;

[0191] The upsampling module 30 is used to sequentially pass the processed signal through multiple upsampling processing modules. Each upsampling processing module performs upsampling through transposed convolution and uses a multi-receptive field module for signal processing.

[0192] The frequency band segmentation module 40 is used to divide the upsampled signal into multiple tensors in the channel dimension according to a predetermined ratio after processing by the transposed convolution and multi-receptive field module, with each tensor corresponding to a different frequency band.

[0193] The multi-receptive-field independent processing module 50 is used to process the segmented tensor independently through the multi-receptive-field module with non-shared parameters, and to unify the number of channels through convolution operation;

[0194] PQMF upsampling module 60 is used to upsample tensors with a uniform number of channels using a PQMF synthesis filter bank;

[0195] The signal merging and processing module 70 is used to merge the upsampled tensors into a single signal and process it through another multi-receptive field module.

[0196] The post-network module has 80 blocks, which are used to process the merged signal and adjust the channel dimension through a post-network containing one-dimensional convolutional layers and activation functions;

[0197] Output module 90 is used to pass the processed signal through an activation function to obtain the generated target audio signal.

[0198] In one embodiment, the front-end network module 20 is specifically used for:

[0199] The Log-Mel spectrum is processed by a one-dimensional convolutional layer having a predetermined kernel size and stride.

[0200] No nonlinear activation function is used after the one-dimensional convolutional layer, and the output of the one-dimensional convolutional layer is used as the processed signal.

[0201] The processed signals are batch normalized and then passed to the upsampling processing module.

[0202] In one embodiment, the upsampling module 30 is specifically used for:

[0203] In each upsampling module, the processed signal is upsampled by applying transposed convolution;

[0204] Set the kernel size, stride, and padding of the transposed convolution to achieve the desired upsampling factor;

[0205] The upsampled signal is transmitted to a multi-receptive-field module, which contains multiple parallel residual blocks with different receptive fields.

[0206] Dilated convolution is applied in the multi-receptive field module, and residual connections are used to fuse the output of the multi-receptive field module with the input signal.

[0207] In one embodiment, the frequency band segmentation module 40 is specifically used for:

[0208] Determine a predetermined ratio for channel segmentation, wherein the predetermined ratio corresponds to different frequency bands;

[0209] The upsampled signal is divided into multiple tensors along the channel dimension according to the predetermined ratio.

[0210] Each tensor is assigned a corresponding frequency band, arranged from low to high frequency, and each tensor is individually identified.

[0211] In one embodiment, the multi-receptive-field independent processing module 50 is specifically used for:

[0212] Each tensor is input into an independent multi-receptive field module, and multi-scale feature extraction is performed on each tensor. The parameters between each multi-receptive field module are not shared, and each multi-receptive field module contains residual blocks with different dilation factors and convolution kernel sizes.

[0213] Multi-scale features of each tensor are fused to enhance the representation of frequency band features;

[0214] The output of the multi-receptive field module is passed through a one-dimensional convolutional layer to unify the number of channels in the tensor.

[0215] Apply activation functions to perform nonlinear transformations on tensors with a uniform number of channels.

[0216] In one embodiment, the signal combining and processing module 70 is specifically used for:

[0217] The upsampled tensors are combined into a single signal by summation.

[0218] The merged signal is input into an additional multi-receptive field module, which contains residual blocks with different receptive fields.

[0219] Multi-scale feature extraction is performed on the merged signal to capture the feature information of the integrated frequency band;

[0220] The output of the multi-receptive-field module is fused to enhance the overall feature representation of the signal.

[0221] The output of the fused multi-receptor field module is passed to the post-network.

[0222] In one embodiment, the input module 10 is specifically used for:

[0223] Acquire the raw audio signal to be processed;

[0224] The original audio signal is preprocessed, including noise reduction and normalization.

[0225] The preprocessed raw audio signal is converted into a Log Mel spectrum;

[0226] The frequency dimension of the Log Mel spectrum is considered as multiple frequency channels to form a multi-channel one-dimensional signal.

[0227] The multi-channel one-dimensional signal is adjusted to a tensor format that conforms to the input neural network.

[0228] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a vocoder audio generation method based on frequency band division on the server side.

[0229] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the user-side functions or steps of a vocoder audio generation method based on frequency band segmentation.

[0230] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0231] The original audio signal is acquired and converted into a Log-Mel spectrum, and the frequency dimension of the Log-Mel spectrum is regarded as a one-dimensional signal with multiple frequency channels.

[0232] The Log-Mel spectrum is processed by a pre-network containing one-dimensional convolutional layers and without non-linear activation.

[0233] The processed signal is sequentially passed through multiple upsampling processing modules. Each upsampling processing module performs upsampling through transposed convolution and uses a multi-receptive field module for signal processing.

[0234] After transposed convolution and multi-receptive field module processing, the upsampled signal is divided into multiple tensors in the channel dimension according to a predetermined ratio, and each tensor corresponds to a different frequency band;

[0235] The segmented tensors are processed independently by multi-receptive field modules with non-shared parameters, and the number of channels is unified by convolution operation;

[0236] The PQMF synthesis filter bank is used to upsample tensors with a uniform number of channels;

[0237] The upsampled tensors are merged into a single signal and then processed by another multi-receptive-field module.

[0238] The merged signal is processed by a post-network containing one-dimensional convolutional layers and activation functions, and the channel dimensions are adjusted.

[0239] The processed signal is passed through an activation function to obtain the generated target audio signal.

[0240] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0241] The original audio signal is acquired and converted into a Log-Mel spectrum, and the frequency dimension of the Log-Mel spectrum is regarded as a one-dimensional signal with multiple frequency channels.

[0242] The Log-Mel spectrum is processed by a pre-network containing one-dimensional convolutional layers and without non-linear activation.

[0243] The processed signal is sequentially passed through multiple upsampling processing modules. Each upsampling processing module performs upsampling through transposed convolution and uses a multi-receptive field module for signal processing.

[0244] After transposed convolution and multi-receptive field module processing, the upsampled signal is divided into multiple tensors in the channel dimension according to a predetermined ratio, and each tensor corresponds to a different frequency band;

[0245] The segmented tensors are processed independently by multi-receptive field modules with non-shared parameters, and the number of channels is unified by convolution operation;

[0246] The PQMF synthesis filter bank is used to upsample tensors with a uniform number of channels;

[0247] The upsampled tensors are merged into a single signal and then processed by another multi-receptive-field module.

[0248] The merged signal is processed by a post-network containing one-dimensional convolutional layers and activation functions, and the channel dimensions are adjusted.

[0249] The processed signal is passed through an activation function to obtain the generated target audio signal.

[0250] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0251] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0252] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0253] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A vocoder audio generation method based on frequency band segmentation, characterized in that, Includes the following steps: The original audio signal is acquired and converted into a Log-Mel spectrum, and the frequency dimension of the Log-Mel spectrum is regarded as a one-dimensional signal with multiple frequency channels. The Log-Mel spectrum is processed by a pre-network containing one-dimensional convolutional layers and without non-linear activation. The processed signal is sequentially passed through multiple upsampling processing modules. Each upsampling processing module performs upsampling through transposed convolution and uses a multi-receptive field module for signal processing. After transposed convolution and multi-receptive field module processing, the upsampled signal is divided into multiple tensors in the channel dimension according to a predetermined ratio, and each tensor corresponds to a different frequency band; The segmented tensors are processed independently by multi-receptive field modules with non-shared parameters, and the number of channels is unified by convolution operation; The PQMF synthesis filter bank is used to upsample tensors with a uniform number of channels; The upsampled tensors are merged into a single signal and then processed by another multi-receptive-field module. The merged signal is processed by a post-network containing one-dimensional convolutional layers and activation functions, and the channel dimensions are adjusted. The processed signal is passed through an activation function to obtain the generated target audio signal.

2. The vocoder audio generation method based on frequency band segmentation as described in claim 1, characterized in that, Processing the Log-Mel spectrum using a pre-network containing one-dimensional convolutional layers and without non-linear activation includes: The Log-Mel spectrum is processed by a one-dimensional convolutional layer having a predetermined kernel size and stride. No nonlinear activation function is used after the one-dimensional convolutional layer, and the output of the one-dimensional convolutional layer is used as the processed signal. The processed signals are batch normalized and then passed to the upsampling processing module.

3. The vocoder audio generation method based on frequency band segmentation as described in claim 1, characterized in that, The processed signal is sequentially passed through multiple upsampling modules. Each upsampling module performs upsampling through transposed convolution and then utilizes a multi-receptive-field module for signal processing, including: In each upsampling module, the processed signal is upsampled by applying transposed convolution; Set the kernel size, stride, and padding of the transposed convolution to achieve the desired upsampling factor; The upsampled signal is transmitted to a multi-receptive-field module, which contains multiple parallel residual blocks with different receptive fields. Dilated convolution is applied in the multi-receptive field module, and residual connections are used to fuse the output of the multi-receptive field module with the input signal.

4. The vocoder audio generation method based on frequency band segmentation as described in claim 1, characterized in that, The upsampled signal is divided into multiple tensors along the channel dimension according to a predetermined ratio, including: Determine a predetermined ratio for channel segmentation, wherein the predetermined ratio corresponds to different frequency bands; The upsampled signal is divided into multiple tensors along the channel dimension according to the predetermined ratio. Each tensor is assigned a corresponding frequency band, arranged from low to high frequency, and each tensor is individually identified.

5. The vocoder audio generation method based on frequency band segmentation as described in claim 1, characterized in that, The segmented tensors are processed independently using multi-receptive-field modules with non-shared parameters, and the number of channels is unified through convolution operations, including: Each tensor is input into an independent multi-receptive field module, and multi-scale feature extraction is performed on each tensor. The parameters between each multi-receptive field module are not shared, and each multi-receptive field module contains residual blocks with different dilation factors and convolution kernel sizes. Multi-scale features of each tensor are fused to enhance the representation of frequency band features; The output of the multi-receptive field module is passed through a one-dimensional convolutional layer to unify the number of channels in the tensor. Apply activation functions to perform nonlinear transformations on tensors with a uniform number of channels.

6. The vocoder audio generation method based on frequency band segmentation as described in claim 1, characterized in that, The upsampled tensors are merged into a single signal and processed by another multi-receptive-field module, including: The upsampled tensors are combined into a single signal by summation. The merged signal is input into an additional multi-receptive field module, which contains residual blocks with different receptive fields. Multi-scale feature extraction is performed on the merged signal to capture the feature information of the integrated frequency band; The output of the multi-receptive-field module is fused to enhance the overall feature representation of the signal. The output of the fused multi-receptor field module is passed to the post-network.

7. The vocoder audio generation method based on frequency band segmentation as described in claim 1, characterized in that, Acquire the original audio signal and convert it into a Log-Mel spectrum, treating the frequency dimension of the Log-Mel spectrum as a one-dimensional signal with multiple frequency channels, including: Acquire the raw audio signal to be processed; The original audio signal is preprocessed, including noise reduction and normalization. The preprocessed raw audio signal is converted into a Log Mel spectrum; The frequency dimension of the Log Mel spectrum is considered as multiple frequency channels to form a multi-channel one-dimensional signal. The multi-channel one-dimensional signal is adjusted to a tensor format that conforms to the input neural network.

8. A vocoder audio generation device based on frequency band segmentation, characterized in that, The vocoder audio generation device based on frequency band segmentation includes: The input module is used to acquire the original audio signal and convert the original audio signal into a Log-Mel spectrum, treating the frequency dimension of the Log-Mel spectrum as a one-dimensional signal with multiple frequency channels. A pre-network module is used to process the Log-Mel spectrum through a pre-network containing one-dimensional convolutional layers and without non-linear activation; The upsampling module is used to sequentially pass the processed signal through multiple upsampling processing modules. Each upsampling processing module performs upsampling through transposed convolution and uses a multi-receptive field module for signal processing. The frequency band segmentation module is used to divide the upsampled signal into multiple tensors in the channel dimension according to a predetermined ratio after processing by the transposed convolution and multi-receptive field module. Each tensor corresponds to a different frequency band. The multi-receptive-field independent processing module is used to process the segmented tensor independently through the multi-receptive-field module with non-shared parameters, and to unify the number of channels through convolution operation; The PQMF upsampling module is used to upsample tensors with a uniform number of channels using the PQMF synthesis filter bank. The signal merging and processing module is used to merge the upsampled tensors into a single signal, which is then processed by another multi-receptive field module. The post-network module is used to process the merged signal through a post-network containing one-dimensional convolutional layers and activation functions, and to adjust the channel dimensions; The output module is used to pass the processed signal through an activation function to obtain the generated target audio signal.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a band-segmentation-based vocoder audio generation program stored in the memory and executable on the processor. When executed by the processor, the band-segmentation-based vocoder audio generation program implements the steps of the band-segmentation-based vocoder audio generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a vocoder audio generation program based on frequency band segmentation, which, when executed by a processor, implements the steps of the vocoder audio generation method based on frequency band segmentation as described in any one of claims 1-7.

Images