Speech generation method, device, equipment and medium based on prompt coding

Abstract

The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a speech generation method based on prompt coding. The method obtains the original audio signal, extracts the mel-spectrogram features and FBank features, inputs the audio signal into the encoder, and generates an initial feature representation; and compresses the feature representation using vector quantization technology to generate a quantized feature representation. The mel-spectrogram features and FBank features are respectively encoded by a conditional prompt encoder and a voiceprint prompt encoder; feature alignment processing is performed by a multi-layer linear projection module, initial weight coefficients are assigned, and weighted fusion is performed by optimizing the weights to generate a fused feature representation, and finally a target speech signal is generated by a decoder. The present invention extracts multiple audio features and combines them with vector quantization technology for compression, thereby reducing data redundancy while maintaining important audio information. The adaptive prompt coding and weight optimization mechanism ensure that various features are effectively utilized in the fusion process.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a speech generation method, device, equipment and storage medium based on prompt coding. Background Art

[0002] With the development of artificial intelligence and big data technologies, voice technology is increasingly being used in the financial sector. Speech codec technology has garnered significant attention, particularly in scenarios such as intelligent customer service, speech recognition, and speech synthesis. Financial institutions can leverage this technology to provide customers with more efficient and convenient voice interaction services, enabling functions such as customer identity verification, voice operations, and voice report generation. These applications require high-fidelity, high-quality speech processing capabilities, as well as the compression, storage, and efficient transmission of large amounts of voice data to meet the real-time and large-scale data processing requirements of financial systems.

[0003] Existing speech codecs, such as neural speech codec technology, are capable of encoding high-resolution speech signals into compressed discrete codes and reconstructing them. This has led to progress in speech synthesis and reconstruction. These technologies not only reduce the bit rate of voice data while maintaining sound quality, thus reducing data transmission and storage costs, but are also being applied to intelligent voice services and customer support in the financial sector. However, existing technologies still have limitations and cannot fully meet the financial industry's demand for high-quality, reliable, and efficient voice technology.

[0004] In financial scenarios, the transmission and storage costs of voice data are key considerations. To reduce costs, efficient voice data compression is often necessary. However, existing neural voice codecs often struggle to maintain sound quality at low bit rates, resulting in degraded voice reconstruction, impacting both the user experience and system reliability.

[0005] Some neural speech codecs rely on generative adversarial networks (GANs) to improve the quality of reconstructed speech. However, in practical applications, GANs are prone to generating audio artifacts, resulting in unnatural or even distorted speech output. This can lead to serious misunderstandings or security risks in scenarios such as voice authentication or sensitive information transmission in the financial sector.

[0006] Although high compression rates can reduce the storage and transmission costs of voice data, existing technologies experience significant performance degradation when pursuing high compression rates. This is especially true in real-time voice processing scenarios in the financial sector, where voice reconstruction delays and degraded sound quality can seriously impact business continuity and customer experience.

[0007] Therefore, existing neural speech codec technologies still face many challenges in balancing compression rate, speech reconstruction quality, and performance optimization in practical applications. The financial industry urgently needs a speech codec solution that can maintain high-fidelity audio quality at low bit rates, avoid artifacts, and maintain efficient processing performance under high compression rates. Summary of the Invention

[0008] The main purpose of the present invention is to provide a speech generation method, device, equipment and storage medium based on prompt coding, aiming to solve the technical problem that the existing neural speech coding and decoding technology is difficult to simultaneously ensure high-fidelity audio quality and high compression rate at low bit rates.

[0009] To achieve the above object, the present invention provides a method for generating speech based on prompt coding, comprising:

[0010] Obtaining an original audio signal, and extracting a Mel-spectrogram feature and an FBank feature from the original audio signal;

[0011] Inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0012] Compressing the initial feature representation using a quantizer based on vector quantization to generate a quantized feature representation;

[0013] Encoding the mel-spectrogram feature through a conditional prompt encoder to generate a conditional prompt feature;

[0014] Encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0015] Using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation to a unified feature space for feature alignment processing;

[0016] Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment;

[0017] Analyze the similarity index between conditional cue features, voiceprint cue features and quantitative feature representations;

[0018] Optimizing the initial weight coefficients by back propagation according to the similarity index;

[0019] Performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0020] The fused feature representation is input into a decoder to generate a target speech signal.

[0021] Furthermore, to achieve the above-mentioned object, the present invention provides a speech generation device based on prompt coding, comprising:

[0022] An audio feature extraction module is used to obtain an original audio signal and extract Mel-spectrogram features and FBank features from the original audio signal;

[0023] A feature extraction encoder module, configured to input the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0024] a vector quantization module, configured to compress the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation;

[0025] A conditional prompt encoder module, configured to encode the mel-spectrogram features through a conditional prompt encoder to generate conditional prompt features;

[0026] A voiceprint prompt encoder module, configured to encode the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0027] a multi-layer linear projection module, configured to use the multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation to a unified feature space for feature alignment;

[0028] A weight assignment module is used to assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features, and quantized feature representations after feature alignment;

[0029] Similarity analysis module, used to analyze the similarity index between conditional prompt features, voiceprint prompt features and quantitative feature representations;

[0030] A weight optimization module, configured to optimize the initial weight coefficients through back propagation according to the similarity index;

[0031] A feature fusion module is used to perform weighted fusion on the conditional prompt feature, voiceprint prompt feature and quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0032] The decoder module is used to input the fusion feature representation into a decoder to generate a target speech signal.

[0033] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a prompt coding-based speech generation program stored in the memory and runnable on the processor, wherein the prompt coding-based speech generation program, when executed by the processor, implements the steps of the prompt coding-based speech generation method as described above.

[0034] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a speech generation program based on prompt coding is stored. When the speech generation program based on prompt coding is executed by a processor, the steps of the speech generation method based on prompt coding as described above are implemented.

[0035] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a speech generation method based on prompt coding. By acquiring the original audio signal, extracting the mel-spectrogram features and FBank features, the audio signal is input into the encoder to generate an initial feature representation; the feature representation is compressed using vector quantization technology to generate a quantized feature representation. The mel-spectrogram features and FBank features are respectively encoded by the conditional prompt encoder and the voiceprint prompt encoder; feature alignment processing is performed by the multi-layer linear projection module, initial weight coefficients are assigned, and weighted fusion is performed by optimizing the weights to generate a fused feature representation, and finally the target speech signal is generated by the decoder. The present invention extracts multiple audio features and combines them with vector quantization technology for compression, thereby reducing data redundancy while maintaining important audio information. Through adaptive prompt coding and weight optimization mechanisms, it is ensured that various features are effectively utilized in the fusion process, thereby enhancing the fidelity and accuracy of speech reconstruction. It can provide efficient data processing capabilities in high compression rate scenarios, reduce computing resource usage, and significantly improve the quality and naturalness of generated speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0037] Figure 1 Schematic diagram of an application environment of a method for generating speech based on prompt coding according to an embodiment of the present invention;

[0038] Figure 2 1. A flow chart of an embodiment of a method for generating speech based on prompt coding according to the present invention;

[0039] Figure 3 Schematic diagram of functional modules of a preferred embodiment of a speech generation device based on prompt coding of the present invention;

[0040] Figure 4 A schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0041] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0043] The speech generation method based on prompt coding provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the client communicates with the server via a network. The server can obtain the original audio signal from the client, extract mel-spectrogram features and FBank features, input the audio signal into an encoder, and generate an initial feature representation. The feature representation is compressed using vector quantization technology to generate a quantized feature representation. The mel-spectrogram features and FBank features are respectively encoded using a conditional prompt encoder and a voiceprint prompt encoder. Feature alignment is performed using a multi-layer linear projection module, initial weight coefficients are assigned, and weighted fusion is performed by optimizing the weights to generate a fused feature representation. Finally, a decoder is used to generate the target speech signal. This invention extracts multiple audio features and combines them with vector quantization technology for compression, reducing data redundancy while preserving important audio information. Adaptive prompt encoding and weight optimization mechanisms ensure that various features are effectively utilized during the fusion process, enhancing the fidelity and accuracy of speech reconstruction. It can provide efficient data processing capabilities in high compression rate scenarios, reduce computing resource usage, and significantly improve the quality and naturalness of the generated speech. The client can include, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented as a standalone server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific examples.

[0044] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a method for speech generation based on prompt coding provided by the present invention. It should be noted that although a logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0045] like Figure 2 As shown, the speech generation method based on prompt coding proposed by the present invention includes the following steps:

[0046] S10, obtaining an original audio signal, and extracting a Mel-spectrogram feature and an FBank feature from the original audio signal;

[0047] In this embodiment, during the speech generation process, the system first receives the user's voice input or obtains the original audio signal from the audio database. The signal is usually digitized waveform data, which can be captured by an input device such as a microphone, or loaded from an existing audio file (such as WAV, MP3 format). The audio signal is quantized by the sampling rate, and the typical sampling rate is 16kHz or 44.1kHz. The sampled digital audio data is processed in frames, and the length of each frame (frame length) and the frame shift are set according to system requirements. The frame length is usually between 20ms and 40ms, and the frame shift is half or one-third of the frame length. After the audio data is framed, the system is ready for further feature extraction.

[0048] Mel-spectrogram features are a type of frequency domain feature. They obtain spectral information by performing a short-time Fourier transform (STFT) on the framed audio signal. The spectrum is then processed using a Mel filter bank to extract the audio signal's frequency distribution on the Mel scale. An STFT is performed on each frame of the audio signal to convert the time domain signal into a frequency domain signal. The spectrum is then processed using a Mel filter bank. The Mel filter bank uses nonlinearly spaced frequency partitioning to map the spectrum onto the Mel scale, based on the tonal distribution perceived by the human ear. The extracted Mel-spectrogram features reflect the frequency components of the audio signal and serve as an important foundation for subsequent speech coding and generation.

[0049] The FBank (Filter Bank Energy) feature processes the framed audio signal through a filter bank to obtain energy information in different frequency bands. Similar to the Mel-spectrogram, the FBank feature processes the spectrum through a set of filters to extract the energy distribution characteristics of multiple frequency bands. The framed audio signal is passed through a series of filters, each of which integrates a subset of the frequency range to obtain the energy of the corresponding frequency band. A typical FBank feature consists of the energy of multiple frequency bands, which represents the strength of the audio signal in each frequency subset and is often used in scenarios such as voiceprint recognition.

[0050] Example: In the financial industry, voice interaction scenarios (such as intelligent customer service and voice authentication) require high-quality speech generation technology. In intelligent customer service, extracting mel-spectrogram features and FBank features helps the system accurately capture important information in the user's voice, ensuring that the system-generated voice responses have natural and smooth sound quality. In voice authentication scenarios, the voiceprint recognition capability of the FBank feature enables the system to perform reliable identity verification based on the frequency and pronunciation characteristics of the user's voice, effectively preventing fraud.

[0051] By extracting Mel-spectrogram features and FBank features from the original audio signal respectively, we can effectively capture the key information in the speech signal, retaining the spectral distribution of the audio signal and extracting more fine-grained voiceprint information.

[0052] S20, inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0053] In this embodiment, the pre-processed raw audio signal (i.e., the signal that has been framed and relevant audio features extracted) is input into a feature extraction encoder. The feature extraction encoder is typically a deep learning model that converts low-level audio features (such as Mel-spectrograms and FBank features) into high-level initial feature representations. These high-level features can capture richer speech information in the audio signal.

[0054] The initial feature representation refers to the multidimensional features generated by the audio signal after being processed by the encoder. These features not only contain the basic spectral information of the audio, but also contain complex structural information related to speech generation, such as the rhythm, intonation and voiceprint features of the speech.

[0055] After extracting the features of the audio signal, these features are passed as input to a feature extraction encoder. The feature extraction encoder can be based on a deep learning architecture (such as a multi-layer perceptron, convolutional neural network, recurrent neural network, etc.) to handle the conversion from input features to high-level features.

[0056] The encoder gradually abstracts and compresses the input features through multiple layers of nonlinear mapping. This process allows the initial feature representation of the output to capture higher-level audio information by learning feature representation layer by layer.

[0057] The input features can be mapped layer by layer to a higher-dimensional feature space through a multi-layer perceptron (MLP), and each layer performs a nonlinear transformation through an activation function (such as ReLU or Sigmoid).

[0058] Alternatively, a convolutional neural network (CNN) can perform local perception of input audio features through convolution kernels, thereby capturing local patterns or features in the audio.

[0059] Alternatively, a recurrent neural network (RNN / LSTM) can process timing information in audio signals and is particularly suitable for processing continuous audio frame signals.

[0060] After the encoder is processed, the generated initial feature representation is a high-dimensional, multi-level audio description that can provide a basis for subsequent quantization, cue coding and voiceprint analysis.

[0061] Example: In financial scenarios, feature extraction encoders can be used for speech generation of financial reports. For example, in telephone banking or financial report generation scenarios, the raw speech signal, after being processed by the feature extraction encoder, can capture the specific intonation and pronunciation characteristics of the customer's voice. The generated initial feature representation can then be used to generate high-fidelity speech reports or feedback. In customer identity verification scenarios, the feature extraction encoder can capture the customer's unique voiceprint characteristics, ensuring the accuracy and security of identity verification.

[0062] By feeding the raw audio signal into a feature extraction encoder and generating an initial feature representation, the audio data's speech feature representation capabilities are enhanced. Compared to traditional audio feature processing methods, the feature extraction encoder can extract higher-level and more complex speech information, making subsequent quantization and cue encoding more precise and effective. Furthermore, the initial feature representation captures detailed features in the audio, maintaining the fidelity and naturalness of speech generation even at high compression rates, thereby improving the overall performance of the speech generation system.

[0063] S30, compressing the initial feature representation using a quantizer based on vector quantization to generate a quantized feature representation;

[0064] In this embodiment, vector quantization (VQ) is a commonly used data compression technique that effectively reduces the data representation size by mapping continuous high-dimensional features into discrete symbols (codebook indices). The system inputs the initial feature representation obtained from the feature extraction encoder into the vector quantizer. The vector quantizer compresses the high-dimensional initial feature representation into a discrete quantized feature representation based on a predefined codebook. This quantization operation can significantly reduce the data dimension and reduce storage and transmission overhead.

[0065] The core of vector quantization is a codebook, in which each element (i.e., a codebook vector) represents a specific feature representation category. The size of the codebook determines the trade-off between compression accuracy and compression ratio. A larger codebook can retain more information but also increases computational cost.

[0066] The initial feature representation is broken down into several feature vectors, each of which is mapped to a vector in the codebook. Specifically, the codebook vector with the smallest distance to the feature vector is selected as the quantization result. The quantizer's job is to convert all initial feature representations into corresponding discrete indices that can be compactly represented as quantized features.

[0067] The quantized feature indices are called quantized feature representations. These indices represent the key information in the initial feature representation and take up less storage space. The quantized feature representations are used in subsequent hint encoding and decoding operations.

[0068] Example: In financial data processing scenarios, vector quantization can be used to compress financial voice data, such as customer recordings from telephone banking and voice interactions with intelligent voice customer service. Using vector quantization technology, financial institutions can efficiently store and transmit voice data with low bandwidth and storage costs, while maintaining high-quality audio fidelity during voice interactions. For example, when a customer queries their bank account information through a voice system, the system can quickly generate natural, high-fidelity voice responses using limited computing resources, improving the customer experience and conserving computing resources for the financial institution.

[0069] By using a vector quantization-based quantizer, the storage and transmission requirements of the initial feature representation are effectively reduced while retaining the key information required for speech generation. This quantization process significantly reduces the computational overhead of speech processing by selecting the optimal codebook index for feature compression, enabling the speech generation system to maintain high performance in resource-constrained environments.

[0070] S40, encoding the mel-spectrogram feature through a conditional prompt encoder to generate a conditional prompt feature;

[0071] In this embodiment, the conditional cue encoder is a core module in speech generation, used to extract high-level feature representations from the input mel-spectrogram features to generate conditional cue features. Mel-spectrogram features, as input, represent the distribution of the audio signal across the frequency spectrum. Through the encoding process of the conditional cue encoder, the mel-spectrogram features are converted into conditional cue features. These features provide contextual information about the audio signal and are used in the subsequent speech generation process.

[0072] Conditional cue features are features extracted during the encoding process that can capture high-level information such as speech characteristics, contextual information, and spectral changes in the audio.

[0073] The mel-spectrogram features are first input into a conditional cue encoder, which can be a deep learning model such as a multi-layer perceptron (MLP) or a convolutional neural network (CNN).

[0074] The conditional cue encoder processes the input mel-spectrogram features through multiple layers, performing nonlinear mapping on the features at each layer to gradually extract higher-level audio features. The mapping operation at each layer is achieved through weight matrix multiplication and activation functions.

[0075] The Mel-spectrogram features can be mapped layer by layer through a multi-layer perceptron (MLP) to generate intermediate feature representations.

[0076] It is also possible to extract local features through convolution operations using convolutional neural networks (CNNs) to capture local patterns in the audio spectrum.

[0077] After several layers of processing, the conditional cue encoder outputs feature representations known as conditional cue features. These features not only contain the spectral information of the mel-spectrogram but also incorporate context and other cue information, providing important cue signals for subsequent speech generation.

[0078] Example: In financial voice interaction systems, the conditional prompt encoder can be used for speech generation in intelligent voice customer service or telephone banking. For example, when a customer queries their account information through a telephone banking system, the system can extract the user's voice signal using mel-spectrogram features, generate conditional prompt features using the conditional prompt encoder, and generate natural voice feedback based on these prompt features. This encoding process enables the system to accurately capture the details and context of the customer's speech, resulting in more natural and fluent voice responses and an improved customer experience.

[0079] By encoding mel-spectrogram features using a conditional cue encoder, key information can be extracted from the input audio signal, generating high-level conditional cue features. These features integrate the audio spectrum, speech prosody, and context, providing important cues for subsequent speech generation steps. Compared to traditional speech feature extraction methods, the conditional cue encoder can better capture complex audio structural features, thereby maintaining high-quality speech generation even in high compression scenarios.

[0080] S50, encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0081] In this embodiment, FBank features represent the energy distribution of an audio signal across different frequency bands and are commonly used to describe the frequency characteristics of a speech signal. The voiceprint hint encoder encodes the input FBank features and extracts voiceprint hint features from the speech signal—information that characterizes the speaker's unique voice characteristics. These voiceprint hint features capture individual differences in speech, primarily reflecting the speaker's pitch, frequency distribution, and pronunciation characteristics. They are used for authentication or personalization in speech generation or recognition.

[0082] The voiceprint cue encoder receives FBank features extracted from the original audio signal, which describe the energy distribution of the audio signal in different frequency bands.

[0083] The encoder processes the FBank features layer by layer through a multi-layer network structure (such as a convolutional neural network (CNN) or a recurrent neural network (RNN)), gradually extracting voiceprint feature information from the speech signal. Each layer processes the intermediate features through a nonlinear activation function to extract more content related to the speaker's personalized characteristics.

[0084] Convolutional neural networks (CNNs) are suitable for capturing local frequency variations in FBank features and extracting acoustic features of speech. Recurrent neural networks (RNNs / LSTMs) are suitable for processing the temporal information of speech signals and are particularly advantageous in capturing the speaker's pronunciation style and rhythm.

[0085] Finally, through multiple layers of nonlinear mapping and feature compression, the voiceprint hint encoder generates high-level voiceprint hint features. These features contain the speaker's unique voice characteristics and can be used in subsequent voice generation or identity verification steps.

[0086] Example: In the financial industry's authentication and voice interaction, voiceprint prompt encoders can provide personalized and secure voice services for financial voice systems. For example, in a telephone banking system, a customer's voice signal is processed by a voiceprint prompt encoder. The generated voiceprint prompt features can be used to identify the customer, replacing traditional passwords or multi-factor authentication methods. This not only improves customer service efficiency but also enhances system security.

[0087] By encoding FBank features with a voiceprint hint encoder, we can effectively extract speaker-specific voiceprint features from audio signals. These voiceprint hint features provide critical reference information for subsequent speech generation, identity recognition, and personalized voice services. Compared to traditional voiceprint recognition methods, using deep learning networks for voiceprint feature extraction can capture more speech details and provide higher accuracy and adaptability in a variety of speech generation applications.

[0088] S60, using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation to a unified feature space to perform feature alignment processing;

[0089] In this embodiment, during speech generation, the system extracts different types of features from multiple sources, such as conditional cue features, voiceprint cue features, and quantitative feature representations. These features have different dimensions and structures, and directly using them for fusion or further processing can lead to inconsistent or distorted information. To address this issue, a multi-layer linear projection module is used to map these features to the same dimensional space, ensuring that they are represented in a unified feature space for subsequent feature alignment and fusion.

[0090] The multi-layer linear projection module projects conditional features, voiceprint features, and quantitative features into a unified feature space through multiple layers of linear transformations and activation functions. The purpose of feature alignment is to ensure that these features are consistently expressed within the same space, enabling effective weighted fusion and reducing information redundancy or loss.

[0091] The conditional cue features, voiceprint cue features, and quantitative feature representations are fed into the multi-layer linear projection module. Each feature may have different dimensions and distributions and therefore needs to be processed independently.

[0092] Each feature is mapped through multiple layers of linear transformations. The linear mapping of each layer of projection is calculated through the weight matrix and bias term, mapping features of different dimensions to the same target dimension space layer by layer.

[0093] Linear projection formula: Y = W·X + b; where W is the weight matrix, X is the input feature, b is the bias term, and Y is the output projection feature.

[0094] After each layer of linear mapping, a nonlinear activation function (such as ReLU or Sigmoid) is used to process the mapped features to further enhance the representation ability of the features.

[0095] After multiple layers of linear transformation, the conditional cue features, voiceprint cue features, and quantitative feature representations are all projected to the same dimension. This process ensures the consistency of feature representations in the same space, preparing for subsequent feature alignment and fusion.

[0096] A multi-layer linear projection module uniformly maps and aligns conditional cue features, voiceprint cue features, and quantitative feature representations, improving the consistency of feature processing and fusion efficiency. By projecting features from different sources into the same feature space, the system can more effectively perform subsequent feature weighting and fusion, avoiding dimensional mismatches or information conflicts between features. This process ensures the stability and accuracy of the data flow during speech generation, improving the naturalness and fidelity of the generated speech, and significantly improving the quality of speech generation, especially in scenarios where multiple features are fused.

[0097] S70, assigning corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features, and quantized feature representations after feature alignment;

[0098] In this embodiment, after feature alignment, the conditional cue features, voiceprint cue features, and quantized feature representations are projected into a unified feature space. To ensure that these features can play their best role in the subsequent speech generation process, the system needs to assign corresponding initial weight coefficients to each feature. These weight coefficients are used to determine the importance of each feature during fusion, thereby affecting the result of feature fusion.

[0099] Initial weight coefficients are assigned based on the results of feature alignment and can be obtained from preset empirical values ​​or through model training. The weight coefficient assignment can reflect the different roles each feature plays in the speech generation process. For example, conditional cue features may focus more on providing contextual information, while voiceprint cue features may focus more on personalized speaker characteristics.

[0100] Before assigning initial weights, the system first evaluates each feature's contribution to speech generation quality. This can be based on empirical rules or by training a model using historical data. Common evaluation methods include the feature's impact on audio reconstruction quality, personalized generation accuracy, and other factors.

[0101] According to the importance of each feature, the corresponding initial weight coefficient is assigned. The initial weight coefficient can be made in the following ways:

[0102] Use a preset fixed weight template to assign a fixed weight value to each feature based on experience.

[0103] The initial weight value of each feature is dynamically adjusted through an adaptive algorithm, and the specific weight can be obtained through training.

[0104] In order to ensure that the weights of each feature are appropriate, the total initial weight coefficient usually needs to be normalized so that its sum is 1, thereby ensuring that the contributions of each feature are balanced during the fusion process.

[0105] By assigning initial weight coefficients to conditional features, voiceprint features, and quantitative features, the contribution of each feature to speech generation is effectively controlled. This weighted feature fusion ensures that feature information from different sources is properly utilized, avoiding the over-amplification or neglect of certain features. Especially in multi-feature fusion speech generation scenarios, the proper allocation of weights can improve the naturalness, personalization, and fidelity of the generated speech, while reducing data redundancy and improving system processing efficiency.

[0106] S80, analyzing the similarity index between the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation;

[0107] In this embodiment, during the speech generation process, different types of features (such as conditional prompt features, voiceprint prompt features, and quantitative feature representations) may contain redundant or repeated information. In order to optimize the effect of feature fusion, these features need to be analyzed to determine the degree of similarity between them. The similarity index is used to measure the degree of similarity between these features. By analyzing the structural similarity of the feature representations, it is determined which features contain similar information, thereby optimizing the processing in the subsequent weight adjustment and feature fusion process. Commonly used similarity analysis indicators include:

[0108] Structural Similarity Index (SSIM): A metric used to measure the structural similarity between two feature representations. It is typically used to measure the similarity between two features in structure, brightness, and contrast.

[0109] Cosine similarity: Calculates the cosine angle between eigenvectors to measure the similarity in the direction of two eigenvectors.

[0110] Euclidean distance: Calculates the distance between feature vectors to quantify the difference between them.

[0111] Pairwise comparisons are performed on conditional cue features, voiceprint cue features, and quantitative feature representations. Similarity scores between these features are calculated using the Structural Similarity Index (SSIM) or other similarity metrics. A higher similarity score indicates greater redundancy between the two features.

[0112] Through similarity analysis, the system can identify which feature information is redundant and which feature information can provide unique contributions to speech generation. In feature pairs with high redundancy, the system can choose to reduce the weight or perform feature fusion.

[0113] During the optimization process, the system can minimize unnecessary similarities by introducing a penalty term in the loss function. This approach encourages the feature encoder to avoid generating duplicate information when generating hint features.

[0114] By analyzing the similarity index between conditional cue features, voiceprint cue features, and quantitative feature representations, redundant information between features can be effectively identified and reduced. This similarity analysis allows the system to optimize weight distribution during the subsequent feature fusion process, ensuring that each feature's contribution to speech generation is unique and necessary, avoiding the waste of computing resources caused by excessive duplication of information. This improves the quality of speech generation, making the generated speech signal more natural and the collaboration between features more efficient.

[0115] S90, optimizing the initial weight coefficient by back propagation according to the similarity index;

[0116] In this embodiment, during the speech generation process, there may be a certain amount of redundant information or similarity between different features. To more effectively utilize each feature information, it is necessary to optimize the initial weight coefficients of the features based on their similarity index. Backpropagation is a commonly used optimization algorithm that adjusts model parameters by calculating gradients and performing backpropagation. The system uses the backpropagation algorithm to dynamically optimize the initially assigned weight coefficients based on the similarity index results, thereby reducing redundant information and improving the effectiveness of feature fusion.

[0117] The goal of backpropagation is to minimize feature redundancy and ensure that conditional cue features, voiceprint cue features, and quantitative feature representations play their respective unique roles in the feature fusion process, thereby improving the quality of speech generation.

[0118] First, the system calculates a loss function based on the similarity index between features. This loss function can be the structural similarity index (SSIM) or other similarity metrics. This loss function measures the degree of redundancy between features, with larger loss values ​​indicating more redundancy.

[0119] By taking the derivative of the loss function, we calculate the gradient of the similarity index with respect to each weight coefficient. This gradient indicates the direction of change in the feature weight. If a feature has a high similarity, the corresponding gradient will adjust its weight to reduce the contribution of that feature.

[0120] Based on the calculated gradient, the initial weight coefficients are optimized and adjusted using a backpropagation algorithm (such as gradient descent). During each update, the system adjusts the weight coefficients according to the gradient direction to minimize the redundant information between features.

[0121] The backpropagation optimization process is an iterative process. The system gradually optimizes the weight coefficients through multiple iterations until the loss function is minimized, ensuring that the similarity between features is minimized.

[0122] By using a backpropagation algorithm to optimize initial weight coefficients based on similarity indices, the effectiveness of feature fusion during speech generation is enhanced. By minimizing redundant information between features, the system ensures that each feature plays its unique role during fusion, avoiding feature duplication and wasted resources. Furthermore, backpropagation optimization dynamically adjusts feature weights, giving the system greater flexibility and adaptability during speech generation, thereby improving the naturalness and fidelity of the generated speech.

[0123] S100, performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0124] In this embodiment, in the previous step, the system optimized the initial weight coefficients using the similarity index and backpropagation algorithm. Next, based on these optimized weight coefficients, the conditional cue features, voiceprint cue features, and quantitative feature representations are weighted and fused to generate the final fused feature representation. Weighted fusion multiplies each feature by its corresponding optimized weight coefficient and adds the results together to generate a comprehensive feature representation.

[0125] Weighted fusion combines the strengths of various features, enabling the resulting fused features to provide more accurate and comprehensive support for subsequent speech generation. The weight of each feature determines its contribution to the fusion.

[0126] Input optimized weight coefficients and features: The system receives optimized weight coefficients (such as α1, α2, α3) and conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment.

[0127] Multiply each feature by its corresponding optimization weight coefficient. Specifically: z fused =a1·z PC +a2·z PV +a3·z quant ; Among them, z fused is the final fusion feature representation, z PC is the conditional prompt feature, z PV It is the voiceprint prompt feature, z quant It is a quantitative feature representation.

[0128] The weighted fusion feature representation will contain comprehensive information from various features. Depending on the weights, some features will play a greater role in the generation process, while other features may play a supporting role in specific speech generation tasks.

[0129] After generating the fused feature representation, the system can perform regularization on the feature representation to ensure that the value of the feature is in an appropriate range for subsequent decoding.

[0130] By performing weighted fusion based on optimized weight coefficients, the system effectively integrates the advantages of conditional cue features, voiceprint cue features, and quantitative feature representations. This weighted fusion process ensures that the information contribution of each feature is fully utilized, avoiding the problem of certain features being ignored or over-reliant during the generation process. The final fused feature representation contains more comprehensive audio information, making the subsequent speech generation process more accurate and natural. This improves the overall system performance and generation quality, especially in multi-feature fusion speech generation systems.

[0131] S110: Input the fused feature representation into a decoder to generate a target speech signal.

[0132] In this embodiment, after the fused features are generated, the next step is to input the fused feature representation into a decoder, which is responsible for converting these high-level features back into the original target speech signal. The decoder can be a deep learning-based neural network, such as an autoregressive decoder, a convolutional neural network decoder (CNN), or a recurrent neural network (RNN)-based decoder. It reconstructs the audio signal frame by frame, ensuring the naturalness and fidelity of the speech.

[0133] The task of the decoder is to gradually restore the time series signal of the original audio based on the fused feature representation, ensuring high-quality speech reconstruction at a lower bit rate.

[0134] The fused feature representation is generated by weighted fusion of the conditional cue feature, voiceprint cue feature, and quantized feature representation. These fused features are used as input to the decoder.

[0135] The decoder usually processes the audio signal frame by frame. Through an autoregressive model (such as WaveNet or Tacotron), the audio signal of each frame is predicted based on the previously generated frames.

[0136] In the convolutional neural network-based decoder, the system restores the feature representation from low-dimensionality to the high-dimensional audio signal space through deconvolution.

[0137] Through the multi-layer network, the decoder gradually generates the speech signal at each time step until the complete speech signal sequence is generated.

[0138] The decoder generates the target speech signal as its output, which is restored from the fused feature representation to be as close as possible to the original high-quality audio signal. This process ensures that the final generated speech signal has high fidelity and naturalness.

[0139] By feeding the fused feature representation into the decoder, the high-dimensional feature information is restored into a natural, smooth target speech signal. The decoder's frame-by-frame processing ensures that the generated audio signal maintains logical consistency and natural flow across time steps. Furthermore, through an autoregressive or convolutional decoding process, the system maintains high-quality speech reconstruction at a low bit rate.

[0140] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a speech generation method based on prompt coding. The method obtains the original audio signal, extracts the mel-spectrogram features and FBank features, inputs the audio signal into the encoder, and generates an initial feature representation; and uses vector quantization technology to compress the feature representation to generate a quantized feature representation. The mel-spectrogram features and FBank features are respectively encoded by a conditional prompt encoder and a voiceprint prompt encoder; feature alignment processing is performed by a multi-layer linear projection module, initial weight coefficients are assigned, and weighted fusion is performed by optimizing the weights to generate a fused feature representation, and finally a target speech signal is generated by a decoder. The present invention extracts multiple audio features and combines them with vector quantization technology for compression, thereby reducing data redundancy while maintaining important audio information. Through adaptive prompt coding and weight optimization mechanisms, it is ensured that various features are effectively utilized in the fusion process, thereby enhancing the fidelity and accuracy of speech reconstruction. It can provide efficient data processing capabilities in high compression rate scenarios, reduce computing resource usage, and significantly improve the quality and naturalness of generated speech.

[0141] In one embodiment, in the above S10, extracting the Mel-spectrogram features and FBank features from the original audio signal respectively includes:

[0142] S101, performing frame processing on the original audio signal according to a preset frame length and frame shift;

[0143] S102, performing short-time Fourier transform on the framed audio signal to obtain spectrum information of the audio signal;

[0144] S103, performing Mel-scale processing on the spectrum information using a Mel filter bank to extract Mel-spectrogram features of the audio signal;

[0145] S104, using a filter bank to perform linear frequency division or logarithmic frequency division on the framed audio signal to extract FBank features of the audio signal;

[0146] S105 , performing logarithmic processing and feature smoothing processing on the Mel-spectrogram features and FBank features to generate final Mel-spectrogram features and FBank features.

[0147] In this embodiment, two key features are extracted from the original audio signal: Mel-spectrogram features and FBank features. These features are crucial in speech processing systems, describing the frequency information and energy distribution of speech signals. Extracting these features relies on framing the audio signal, applying Fourier transforms, and applying filter banks, ultimately generating high-quality audio features suitable for subsequent speech processing tasks.

[0148] The original audio signal is divided into frames according to a preset frame length and frame shift. The frame length indicates how much audio signal each frame contains, while the frame shift indicates the degree of overlap between frames. Common frame lengths are 20-40ms, and frame shifts are around 10ms. By dividing the continuous audio signal into multiple time segments for further processing, the system can segment the continuous audio signal into multiple time segments.

[0149] A short-time Fourier transform (STFT) is performed on the framed audio signal to convert it from the time domain to the frequency domain. The STFT result is spectral information, which represents the intensity of different frequency components. This step is very critical because spectral information is the basis for Mel-spectrogram feature extraction and FBank feature extraction.

[0150] Mel filter banks are applied to spectral information, using the Mel scale for frequency conversion. The Mel scale simulates the human ear's perception of different frequencies, with lower resolution for higher frequencies and higher resolution for lower frequencies. This processing allows the system to extract Mel spectrogram features that better match human hearing and reflect the perceived spectral distribution of the speech signal.

[0151] Use a filter bank to perform linear or logarithmic frequency division on the framed audio signal to extract FBank features. FBank features are mainly used to represent the energy distribution of the speech signal in different frequency bands. Linear frequency division is suitable for the lower frequency portion of the audio signal, while logarithmic frequency division is suitable for the higher frequency portion.

[0152] Logarithmic processing is performed on the extracted Mel-spectrogram features and FBank features, transforming the feature values ​​to a logarithmic scale. This can better compress the dynamic range and make subsequent processing smoother and more stable. Feature smoothing is performed to reduce noise and mutation points in the features and ensure smooth feature transitions. This improves the robustness of the features, allowing the system to generate high-quality features even when facing speech signals in different environments.

[0153] This embodiment effectively improves the system's ability to process audio signals by extracting mel-spectrogram features and FBank features. Mel-spectrogram features capture more useful speech information by simulating human auditory perception of frequency, while FBank features provide powerful frequency feature support by describing the energy distribution of speech across frequency bands. Logarithmic processing and smoothing further enhance the robustness and stability of the features, enabling the system to generate high-quality speech features even when exposed to audio signals from diverse environments.

[0154] In one embodiment, the above S40 includes:

[0155] S401, inputting the mel-spectrogram feature to a conditional prompt encoder, wherein the conditional prompt encoder includes a multi-layer perceptron;

[0156] S402, using the multilayer perceptron to perform layer-by-layer nonlinear mapping processing on the mel-spectrogram features to generate an intermediate feature representation;

[0157] S403: Perform feature compression processing on the intermediate feature representation to generate conditional prompt features.

[0158] In this embodiment, during speech generation, conditional cue features are high-level feature information extracted from the input mel-spectrogram features, providing context and conditional support for speech generation. The task of the conditional cue encoder is to process and extract the mel-spectrogram features layer by layer using a multi-layer neural network to generate conditional cue features suitable for speech generation.

[0159] The mel-spectrogram features extracted from the original audio are input into the conditional prompt encoder. The mel-spectrogram features are the frequency features of the processed audio signal, which contain the spectral information of the audio signal.

[0160] The conditional cue encoder uses a multilayer perceptron (MLP) to perform layer-by-layer nonlinear mapping on the mel-spectrogram features. The MLP consists of several fully connected layers, each of which maps and transforms the input features through linear transformations (weight matrix multiplication and bias addition) and nonlinear activation functions (such as ReLU and Sigmoid), gradually extracting higher-level feature representations.

[0161] After layer-by-layer processing by the multi-layer perceptron, an intermediate feature representation is generated. The intermediate feature representation is the result of nonlinear mapping of the mel-spectrogram features. It contains higher-level spectral and speech feature information, but has not yet undergone final feature compression processing.

[0162] Perform feature compression on the intermediate feature representation. Feature compression aims to reduce redundant information and further compress the intermediate feature representation into compact conditional cue features. This step reduces unnecessary feature dimensions, ensuring that the output conditional cue features are more representative and compact, facilitating subsequent speech generation tasks.

[0163] The final output conditional prompt feature contains high-level speech feature information extracted by the multi-layer perceptron, which can provide effective prompt signals for the subsequent speech generation step.

[0164] This embodiment encodes mel-spectrogram features using a conditional cue encoder. The system can extract high-level feature representations from frequency information, which can provide contextual cues for subsequent speech generation. The generation of conditional cue features captures important information in the audio signal through layer-by-layer nonlinear mapping. Feature compression reduces redundant data, ensuring the compactness and representativeness of the generated cue features. In speech generation tasks, conditional cue features can effectively help the system generate natural and fluent speech signals, improving the quality of speech generation.

[0165] In one embodiment, the above S50 includes:

[0166] S501, inputting the FBank feature into a voiceprint prompt encoder, wherein the voiceprint prompt encoder includes a deep learning network;

[0167] S502, using a voiceprint prompt encoder to extract individual voiceprint features from the FBank features, where the individual voiceprint features are used to reflect the speaker's speech frequency and pronunciation characteristics;

[0168] S503, performing layer-by-layer nonlinear mapping and activation processing on the individual voiceprint features through the deep learning network;

[0169] S504: Perform feature compression processing on the processed individual voiceprint features to generate voiceprint prompt features.

[0170] In this embodiment, the voiceprint prompt feature is generated based on the FBank features (filter bank band energy features) contained in the input speech signal, and is used to represent the speaker's unique voiceprint information. Through the voiceprint prompt encoder, the system can extract individual voiceprint characteristics from the FBank features, including pitch, pronunciation style, voice frequency, and other characteristics, thereby providing personalized prompt signals for subsequent speech generation.

[0171] The voiceprint hint encoder is usually based on a deep learning network, which processes the FBank features layer by layer, extracts and generates the final voiceprint hint features, ensuring that the personalized and speaker identity-related speech characteristics are preserved.

[0172] The FBank feature extracted from the original audio signal is input into the voiceprint prompt encoder. The FBank feature describes the energy distribution of the audio signal in different frequency bands and is an important representation of speech frequency characteristics.

[0173] The voiceprint hint encoder uses a deep learning network to process the input FBank features and extract individual voiceprint features related to the speaker's identity. These features can reflect the speaker's voice characteristics such as voice frequency, pitch, pronunciation style, etc., and are highly personalized.

[0174] The voiceprint hint encoder is composed of a multi-layer deep learning network. Each layer progressively processes individual voiceprint features through nonlinear mapping, extracting higher-level speech information layer by layer. Activation functions (such as ReLU and Sigmoid) are used to enhance the nonlinear representation capability of features, ensuring that voiceprint features can be fully extracted and optimized.

[0175] The extracted individual voiceprint features are compressed. The purpose of compression is to reduce redundant information and compress high-dimensional feature representations into more compact voiceprint prompt features. This ensures that the generated voiceprint prompt features can effectively reflect the speaker's unique voice characteristics and facilitate subsequent processing and storage.

[0176] The resulting voiceprint prompt features can capture the speaker's personalized voice characteristics. These features will be used as prompt signals in subsequent speech generation tasks to ensure that the generated speech has the speaker's unique timbre and pronunciation characteristics.

[0177] This embodiment encodes FBank features using a voiceprint hint encoder, allowing the system to extract personalized voiceprint features from speech signals. These voiceprint features provide personalized hints for subsequent speech generation, ensuring that the generated speech possesses the unique voice characteristics of the speaker. Through layer-by-layer nonlinear mapping and feature compression, the system extracts highly refined voiceprint hint information from the input FBank features, improving the naturalness and personalization of the generated speech. Furthermore, feature compression makes the generated hint features more compact, making them easier to process and store.

[0178] In one embodiment, the above S60 includes:

[0179] S601, inputting the conditional prompt feature, voiceprint prompt feature and quantitative feature representation into a multi-layer linear projection module;

[0180] S602, using a multi-layer linear projection module to perform linear mapping processing on the input conditional prompt features, voiceprint prompt features, and quantized feature representations, each layer of linear projection includes weight matrix multiplication and bias addition;

[0181] S603, after each layer of linear projection, an activation function is used to activate the mapped features;

[0182] S604: After the mapping and activation processing is completed, the conditional prompt features, voiceprint prompt features and quantitative feature representation aligned in the unified feature space are obtained.

[0183] In this embodiment, in the speech generation system, different types of features (such as conditional cue features, voiceprint cue features, and quantitative feature representations) may have different dimensions and representations. To fuse these features, the system needs to map each type of feature to the same feature space through a multi-layer linear projection module and perform feature alignment. This process ensures consistency and fusion between features, allowing the information of each feature to be effectively utilized in the subsequent weighted fusion process.

[0184] The main task of the multi-layer linear projection module is to map different input features to the same dimension through layer-by-layer linear transformations and nonlinear activation functions, ensuring alignment between features. This alignment is a prerequisite for fusion and avoids issues with incompatibility in feature dimensions or representation.

[0185] Conditional cue features, voiceprint cue features, and quantized feature representations are input into the multi-layer linear projection module. Each feature may come from a different processing module and therefore may have different dimensions and representations before input.

[0186] The multi-layer linear projection module applies linear mapping to each input feature layer by layer. After each layer of linear projection, the mapped features are activated using a nonlinear activation function (such as ReLU or Sigmoid), ensuring that the features can express richer nonlinear relationships. Activation functions enhance the model's nonlinear representation capabilities and help capture complex relationships between features.

[0187] After linear mapping and activation, the conditional cue features, voiceprint cue features, and quantized feature representations are mapped to the same dimension, and all features are aligned in the same feature space. This alignment ensures consistency between different features, allowing subsequent fusion and processing steps to operate in the same space, avoiding feature incompatibility issues.

[0188] This embodiment uses a multi-layer linear projection module to map conditional cue features, voiceprint cue features, and quantitative feature representations to the same feature space, ensuring compatibility and consistency between features. The feature alignment process not only ensures dimensional uniformity among various features but also enhances the expressive power of features through nonlinear activation functions, helping to capture the complex relationships between different features. Ultimately, the aligned features can be used in subsequent weighted fusion, improving the naturalness and accuracy of speech generation.

[0189] In one embodiment, the above S70 includes:

[0190] S701, using error analysis to evaluate the contribution of the conditional prompt feature, voiceprint prompt feature, and quantized feature representation to the audio reconstruction quality based on historical training experience data;

[0191] S702, assigning corresponding initial weight coefficients proportional to the contribution degrees to the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation according to the contribution degrees;

[0192] S703: Normalize all initial weight coefficients.

[0193] In this embodiment, after feature alignment, the system needs to assign appropriate initial weights to different features. This initial weighting is based on an assessment of each feature's contribution to speech generation or reconstruction. To ensure effective feature fusion, the weights must reflect the importance of each feature in audio reconstruction. This process evaluates each feature's contribution to audio reconstruction quality based on historical training data and error analysis, ultimately assigning appropriate weights based on the degree of contribution.

[0194] Accurately assigning the initial weight coefficients is a key step in ensuring that the speech generation process properly utilizes each feature information. Finally, the system normalizes all weight coefficients to ensure that the sum is 1, maintaining a balanced contribution of each feature during the feature fusion process.

[0195] The system evaluates the contribution of conditional cue features, voiceprint cue features, and quantitative feature representations to the quality of audio reconstruction by using historical training data and error analysis of the audio generation model. Error analysis can determine the importance of each feature by comparing the error between the generated audio and the target audio. For example, the system can use MSE (mean square error) or L1 loss to measure the impact of each feature on the accuracy of audio reconstruction, thereby deriving their contribution.

[0196] Based on these evaluation results, the system assigns each feature an initial weight proportional to its contribution. For example, if the conditional cue feature contributes most to audio generation, its weight will be relatively large, while other features, such as voiceprint cue features and quantitative features, will be assigned smaller weights.

[0197] The distribution formula of weight coefficient can be expressed as:

[0198]

[0199] Among them, a i is the weight coefficient of the i-th feature, and the contribution value represents the impact of the feature on the audio reconstruction quality.

[0200] After assigning weight coefficients, the system normalizes all weight coefficients to ensure that their sum is 1. The purpose of normalization is to balance the influence of each feature and avoid excessive influence of a certain feature on the speech generation process.

[0201] By assigning initial weight coefficients based on historical data and error analysis, this embodiment enables the system to rationally utilize information from each feature during the feature fusion process. This ensures that conditional cue features, voiceprint cue features, and quantitative feature representations all play their due roles in audio reconstruction, avoiding overreliance on or neglect of any particular feature. Normalization further ensures that the contribution of each feature remains within a reasonable range, improving the naturalness and accuracy of speech generation.

[0202] In one embodiment, the above S110 includes:

[0203] S1101, inputting the fused feature representation into an autoregressive decoder;

[0204] S1102, processing the fused feature representation time-step by time-step through the autoregressive decoder to gradually generate a speech signal for each time frame;

[0205] S1103: Combine the speech signals of all time frames into a continuous target speech signal.

[0206] In this embodiment, the fused feature representation is input into an autoregressive decoder, which processes it time-step (frame-by-frame) to generate the target speech signal. The autoregressive decoder is a deep neural network model based on a recursive structure. It processes the input features time-step by time-step to generate a frame of the speech signal. It then uses this frame as input to generate the next frame, until the entire speech sequence is generated.

[0207] The generation of each time frame depends on the previously generated frames, so that the temporal continuity and naturalness of the speech signal are maintained. Ultimately, all generated time frames will be combined into a continuous speech signal, that is, the complete target speech.

[0208] The fused feature representation obtained through weighted feature fusion is input into the autoregressive decoder. This fused feature representation, which includes optimized conditional cue features, voiceprint cue features, and quantized feature representations, serves as the basis for generating the target speech signal. The autoregressive decoder processes the fused feature representation time-step by time-step. This means that the system starts with the first time frame, generates an audio frame, and then recursively processes the output of that frame as input to the next time frame. The generation of each time frame depends on the output of the previous frame, ensuring the temporal smoothness and naturalness of the speech signal.

[0209] Each frame of audio generated by the decoder represents a time segment of the target speech signal. By gradually generating the speech signal for each time frame using an autoregressive method, the decoder ensures that the generated speech signal transitions smoothly and naturally.

[0210] The system combines all generated time frame speech signals into a continuous target speech signal. This step stitches together all the individually generated time frame speech data to form the final audio file or stream.

[0211] This embodiment generates a continuous, natural speech signal by feeding the fused feature representation into an autoregressive decoder. The autoregressive decoder generates the speech signal time-step by time, ensuring temporal consistency and coherence of the generated audio signal. Furthermore, through frame-by-frame generation, the system ensures high naturalness and fidelity of the generated speech, making it particularly suitable for applications requiring high-quality audio output, such as speech synthesis and voice assistants.

[0212] In one embodiment, a speech generation device based on prompt coding is provided, and the speech generation device based on prompt coding corresponds to the speech generation method based on prompt coding in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a speech generation device based on prompt coding according to the present invention. It includes an audio feature extraction module 10, a feature extraction encoder module 20, a vector quantization module 30, a conditional prompt encoder module 40, a voiceprint prompt encoder module 50, a multi-layer linear projection module 60, a weight assignment module 70, a similarity analysis module 80, a weight optimization module 90, a feature fusion module 100, and a decoder module 110. Each functional module is described in detail below:

[0213] The audio feature extraction module 10 is used to obtain the original audio signal and extract the Mel-spectrogram feature and FBank feature from the original audio signal;

[0214] A feature extraction encoder module 20 is configured to input the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0215] a vector quantization module 30, configured to compress the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation;

[0216] A conditional prompt encoder module 40 is used to encode the mel-spectrogram feature through a conditional prompt encoder to generate a conditional prompt feature;

[0217] A voiceprint prompt encoder module 50 is used to encode the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0218] A multi-layer linear projection module 60 is used to use a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation to a unified feature space for feature alignment processing;

[0219] A weight assignment module 70 is used to assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features, and quantized feature representations after feature alignment;

[0220] Similarity analysis module 80, for analyzing similarity indexes between conditional prompt features, voiceprint prompt features, and quantitative feature representations;

[0221] A weight optimization module 90 is configured to optimize the initial weight coefficients by back propagation according to the similarity index;

[0222] The feature fusion module 100 is used to perform weighted fusion on the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0223] The decoder module 110 is configured to input the fused feature representation into a decoder to generate a target speech signal.

[0224] In one embodiment, the audio feature extraction module 10 is specifically configured to:

[0225] Performing frame processing on the original audio signal according to a preset frame length and frame shift;

[0226] Perform short-time Fourier transform on the framed audio signal to obtain the spectrum information of the audio signal;

[0227] Performing Mel-scale processing on the spectral information using a Mel filter bank to extract Mel-spectrogram features of the audio signal;

[0228] Using a filter bank to perform linear frequency division or logarithmic frequency division on the framed audio signal to extract FBank features of the audio signal;

[0229] The mel-spectrogram features and FBank features are subjected to logarithmic processing and feature smoothing to generate final mel-spectrogram features and FBank features.

[0230] In one embodiment, the conditional prompt encoder module 40 is specifically configured to:

[0231] Inputting the mel-spectrogram feature into a conditional prompt encoder, wherein the conditional prompt encoder comprises a multi-layer perceptron;

[0232] Using the multilayer perceptron to perform layer-by-layer nonlinear mapping processing on the Mel-spectrogram features to generate intermediate feature representations;

[0233] Perform feature compression processing on the intermediate feature representation to generate conditional prompt features.

[0234] In one embodiment, the voiceprint prompt encoder module 50 is specifically configured to:

[0235] Inputting the FBank features into a voiceprint prompt encoder, wherein the voiceprint prompt encoder comprises a deep learning network;

[0236] Extracting individual voiceprint features from the FBank features using a voiceprint prompt encoder, where the individual voiceprint features are used to reflect the speaker's speech frequency and pronunciation characteristics;

[0237] Performing layer-by-layer nonlinear mapping and activation processing on the individual voiceprint features through the deep learning network;

[0238] The processed individual voiceprint features are subjected to feature compression processing to generate voiceprint prompt features.

[0239] In one embodiment, the multi-layer linear projection module 60 is specifically configured to:

[0240] Inputting the conditional prompt feature, voiceprint prompt feature and quantitative feature representation into a multi-layer linear projection module;

[0241] A multi-layer linear projection module is used to perform linear mapping on the input conditional prompt features, voiceprint prompt features, and quantized feature representations. Each layer of linear projection includes weight matrix multiplication and bias addition.

[0242] After each layer of linear projection, the activation function is used to activate the mapped features;

[0243] After completing the mapping and activation processing, the conditional cue features, voiceprint cue features and quantitative feature representations aligned in a unified feature space are obtained.

[0244] In one embodiment, the weight allocation module 70 is specifically configured to:

[0245] Based on historical training experience data, error analysis is used to evaluate the contribution of the conditional cue features, voiceprint cue features, and quantitative feature representations to the audio reconstruction quality;

[0246] According to the contribution degree, assigning corresponding initial weight coefficients proportional to the contribution degree to the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation;

[0247] Normalize all initial weight coefficients.

[0248] In one embodiment, the decoder module 110 is specifically configured to:

[0249] Inputting the fused feature representation into an autoregressive decoder;

[0250] Processing the fused feature representation time-step by time-step through the autoregressive decoder to gradually generate a speech signal for each time frame;

[0251] The speech signals of all time frames are combined into a continuous target speech signal.

[0252] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a speech generation method based on prompt coding.

[0253] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the user side of a method for generating speech based on prompt coding.

[0254] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0255] Obtaining an original audio signal, and extracting a Mel-spectrogram feature and an FBank feature from the original audio signal;

[0256] Inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0257] Compressing the initial feature representation using a quantizer based on vector quantization to generate a quantized feature representation;

[0258] Encoding the mel-spectrogram feature through a conditional prompt encoder to generate a conditional prompt feature;

[0259] Encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0260] Using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation to a unified feature space for feature alignment processing;

[0261] Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment;

[0262] Analyze the similarity index between conditional cue features, voiceprint cue features and quantitative feature representations;

[0263] Optimizing the initial weight coefficients by back propagation according to the similarity index;

[0264] Performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0265] The fused feature representation is input into a decoder to generate a target speech signal.

[0266] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0267] Obtaining an original audio signal, and extracting a Mel-spectrogram feature and an FBank feature from the original audio signal;

[0268] Inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0269] Compressing the initial feature representation using a quantizer based on vector quantization to generate a quantized feature representation;

[0270] Encoding the mel-spectrogram feature through a conditional prompt encoder to generate a conditional prompt feature;

[0271] Encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0272] Using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation to a unified feature space for feature alignment processing;

[0273] Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment;

[0274] Analyze the similarity index between conditional cue features, voiceprint cue features and quantitative feature representations;

[0275] Optimizing the initial weight coefficients by back propagation according to the similarity index;

[0276] Performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0277] The fused feature representation is input into a decoder to generate a target speech signal.

[0278] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0279] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0280] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0281] It should be noted that if any software tools or components other than those of the Company appear in the embodiments of this application, they are merely for illustration and do not represent actual use. The above embodiments are intended only to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for generating speech based on prompt coding, characterized in that: The following steps are involved: Obtaining an original audio signal, and extracting a Mel-spectrogram feature and an FBank feature from the original audio signal; Inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation; Compressing the initial feature representation using a quantizer based on vector quantization to generate a quantized feature representation; Encoding the mel-spectrogram feature through a conditional prompt encoder to generate a conditional prompt feature; Encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature; Using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation to a unified feature space for feature alignment processing; Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment; Analyze the similarity index between conditional cue features, voiceprint cue features and quantitative feature representations; Optimizing the initial weight coefficients by back propagation according to the similarity index; Performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation; The fused feature representation is input into a decoder to generate a target speech signal.

2. The method for generating speech based on prompt coding according to claim 1, wherein: Mel-spectrogram features and FBank features are extracted from the original audio signal, including: Performing frame processing on the original audio signal according to a preset frame length and frame shift; Perform short-time Fourier transform on the framed audio signal to obtain the spectrum information of the audio signal; Performing Mel-scale processing on the spectral information using a Mel filter bank to extract Mel-spectrogram features of the audio signal; Using a filter bank to perform linear frequency division or logarithmic frequency division on the framed audio signal to extract FBank features of the audio signal; The mel-spectrogram features and FBank features are subjected to logarithmic processing and feature smoothing to generate final mel-spectrogram features and FBank features.

3. The method for generating speech based on prompt coding according to claim 1, wherein: The mel spectrogram feature is encoded by a conditional prompt encoder to generate a conditional prompt feature, including: Inputting the mel-spectrogram feature into a conditional prompt encoder, wherein the conditional prompt encoder comprises a multi-layer perceptron; Using the multilayer perceptron to perform layer-by-layer nonlinear mapping processing on the Mel-spectrogram features to generate intermediate feature representations; Perform feature compression processing on the intermediate feature representation to generate conditional prompt features.

4. The method for generating speech based on prompt coding according to claim 1, wherein: The FBank feature is encoded by a voiceprint prompt encoder to generate a voiceprint prompt feature, including: Inputting the FBank features into a voiceprint prompt encoder, wherein the voiceprint prompt encoder comprises a deep learning network; Extracting individual voiceprint features from the FBank features using a voiceprint prompt encoder, where the individual voiceprint features are used to reflect the speaker's speech frequency and pronunciation characteristics; Performing layer-by-layer nonlinear mapping and activation processing on the individual voiceprint features through the deep learning network; The processed individual voiceprint features are subjected to feature compression processing to generate voiceprint prompt features.

5. The method for generating speech based on prompt coding as claimed in claim 1, wherein: The conditional prompt feature, the voiceprint prompt feature, and the quantitative feature are mapped to a unified feature space using a multi-layer linear projection module to perform feature alignment processing, including: Inputting the conditional prompt feature, voiceprint prompt feature and quantitative feature representation into a multi-layer linear projection module; A multi-layer linear projection module is used to perform linear mapping on the input conditional prompt features, voiceprint prompt features, and quantized feature representations. Each layer of linear projection includes weight matrix multiplication and bias addition. After each layer of linear projection, the activation function is used to activate the mapped features; After completing the mapping and activation processing, the conditional cue features, voiceprint cue features and quantitative feature representations aligned in a unified feature space are obtained.

6. The method for generating speech based on prompt coding according to claim 1, wherein: Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features, and quantized feature representations after feature alignment, including: Based on historical training experience data, error analysis is used to evaluate the contribution of the conditional cue features, voiceprint cue features, and quantitative feature representations to the audio reconstruction quality; According to the contribution degree, assigning corresponding initial weight coefficients proportional to the contribution degree to the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation; Normalize all initial weight coefficients.

7. The method for generating speech based on prompt coding according to claim 1, wherein: Inputting the fused feature representation into a decoder to generate a target speech signal includes: Inputting the fused feature representation into an autoregressive decoder; Processing the fused feature representation time-step by time-step through the autoregressive decoder to gradually generate a speech signal for each time frame; The speech signals of all time frames are combined into a continuous target speech signal.

8. A speech generation device based on prompt coding, characterized in that: The speech generation device based on prompt coding includes: An audio feature extraction module is used to obtain an original audio signal and extract Mel-spectrogram features and FBank features from the original audio signal; A feature extraction encoder module, configured to input the original audio signal into a feature extraction encoder to obtain an initial feature representation; a vector quantization module, configured to compress the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation; A conditional prompt encoder module, configured to encode the mel-spectrogram features through a conditional prompt encoder to generate conditional prompt features; A voiceprint prompt encoder module, configured to encode the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature; a multi-layer linear projection module, configured to use the multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature, and the quantitative feature representation to a unified feature space for feature alignment; A weight assignment module is used to assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features, and quantized feature representations after feature alignment; Similarity analysis module, used to analyze the similarity index between conditional prompt features, voiceprint prompt features and quantitative feature representations; A weight optimization module, configured to optimize the initial weight coefficients through back propagation according to the similarity index; A feature fusion module is used to perform weighted fusion on the conditional prompt feature, voiceprint prompt feature and quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation; The decoder module is used to input the fusion feature representation into a decoder to generate a target speech signal.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a prompt coding-based speech generation program stored in the memory and capable of running on the processor. When the prompt coding-based speech generation program is executed by the processor, the steps of the prompt coding-based speech generation method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that The storage medium stores a speech generation program based on prompt coding, and when the speech generation program based on prompt coding is executed by the processor, the steps of the speech generation method based on prompt coding as described in any one of claims 1 to 7 are implemented.

Images