Voice generation method and device based on prompt codes, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a voice generation method based on prompt coding, and the method comprises the steps: obtaining an original audio signal, extracting a Mel spectrum feature and an FBank feature, inputting the audio signal into an encoder, and generating an initial feature representation; the feature representation is compressed using a vector quantization technique to generate a quantized feature representation. Coding the Mel spectrum features and the FBank features through a condition prompt encoder and a voiceprint prompt encoder respectively; the multi-layer linear projection module is used for carrying out feature alignment processing and distributing an initial weight coefficient, weight fusion is carried out through weight optimization, fusion feature representation is generated, and finally a target voice signal is generated through the decoder. According to the method, multiple audio features are extracted and compressed in combination with a vector quantization technology, so that important audio information is kept while data redundancy is reduced. And through adaptive prompt coding and a weight optimization mechanism, various features are ensured to be effectively utilized in a fusion process.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a speech generation method, device, equipment and storage medium based on prompt coding. Background Art

[0002] With the development of artificial intelligence and big data technology, voice technology is increasingly being used in the financial sector. In particular, voice codec technology has received widespread attention in scenarios such as intelligent customer service, voice recognition, and voice synthesis. Through voice codec technology, financial institutions can provide customers with more efficient and convenient voice interaction services, and implement functions such as customer identity authentication, voice operation, and voice report generation. These applications require high-fidelity, high-quality voice processing capabilities, as well as compressed storage and efficient transmission of large amounts of voice data to meet the real-time and large-scale data processing requirements of the financial system.

[0003] Existing speech codecs, such as neural speech codec technology, have the ability to encode high-resolution speech signals into compressed discrete codes and reconstruct them, and have made progress in speech synthesis and speech reconstruction. These technologies can not only reduce the bit rate of voice data and the cost of data transmission and storage while ensuring sound quality, but can also be applied to intelligent voice services and customer support in the financial field. However, existing technologies still have some limitations and cannot fully meet the financial industry's demand for high quality, high reliability and high efficiency of voice technology.

[0004] In financial scenarios, the transmission and storage costs of voice data are one of the key considerations. In order to reduce costs, voice data usually needs to be efficiently compressed. However, existing neural voice codecs often have difficulty ensuring sound quality at low bit rates, resulting in degraded sound quality during voice reconstruction, affecting customer user experience and system reliability.

[0005] Some neural speech codecs rely on generative adversarial network (GAN) technology to improve the quality of reconstructed speech. However, in practical applications, GAN is prone to generate audio artifacts, resulting in unnatural or even distorted speech output. This may cause serious misunderstandings or security risks in voice authentication or sensitive information transmission scenarios in the financial field.

[0006] Although high compression rates can reduce the storage and transmission costs of voice data, the existing technology suffers from significant performance degradation when pursuing high compression rates. Especially in real-time voice processing scenarios in the financial field, the delay in voice reconstruction and the degradation of sound quality may seriously affect business continuity and customer experience.

[0007] Therefore, existing neural speech codec technologies still face many challenges in balancing compression rate, speech reconstruction quality, and performance optimization in practical applications. The financial industry urgently needs a speech codec solution that can maintain high-fidelity audio quality at low bit rates, while avoiding artifacts and maintaining efficient processing performance under high compression rate conditions. Summary of the invention

[0008] The main purpose of the present invention is to provide a speech generation method, device, equipment and storage medium based on prompt coding, aiming to solve the technical problem that the existing neural speech coding and decoding technology is difficult to simultaneously ensure high-fidelity audio quality and high compression rate at low bit rates.

[0009] To achieve the above object, the present invention provides a method for generating speech based on prompt coding, comprising:

[0010] Acquire an original audio signal, and extract Mel-spectrogram features and FBank features from the original audio signal respectively;

[0011] Inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0012] Compressing the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation;

[0013] Encoding the Mel-spectrogram feature by a conditional prompt encoder to generate a conditional prompt feature;

[0014] Encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0015] Using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation to a unified feature space to perform feature alignment processing;

[0016] Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment;

[0017] Analyze the similarity index between conditional cue features, voiceprint cue features and quantitative feature representations;

[0018] According to the similarity index, optimizing the initial weight coefficient by back propagation;

[0019] Performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0020] The fused feature representation is input into a decoder to generate a target speech signal.

[0021] Furthermore, to achieve the above object, the present invention provides a speech generation device based on prompt coding, comprising:

[0022] An audio feature extraction module is used to obtain an original audio signal and extract a Mel-spectrogram feature and an FBank feature from the original audio signal;

[0023] A feature extraction encoder module, used for inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0024] A vector quantization module, used to compress the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation;

[0025] A conditional prompt encoder module, used for encoding the mel-spectrogram feature through a conditional prompt encoder to generate a conditional prompt feature;

[0026] A voiceprint prompt encoder module, used for encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0027] A multi-layer linear projection module, used for mapping the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation to a unified feature space using the multi-layer linear projection module to perform feature alignment processing;

[0028] A weight assignment module is used to assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantized feature representations after feature alignment;

[0029] A similarity analysis module is used to analyze the similarity index between the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation;

[0030] A weight optimization module, used for optimizing the initial weight coefficient by back propagation according to the similarity index;

[0031] A feature fusion module, used to perform weighted fusion on the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0032] The decoder module is used to input the fused feature representation into a decoder to generate a target speech signal.

[0033] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a prompt coding-based speech generation program stored in the memory and executable on the processor, wherein the prompt coding-based speech generation program, when executed by the processor, implements the steps of the prompt coding-based speech generation method as described above.

[0034] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a speech generation program based on prompt coding is stored, and when the speech generation program based on prompt coding is executed by a processor, the steps of the speech generation method based on prompt coding as described above are implemented.

[0035] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a speech generation method based on prompt coding. By acquiring the original audio signal, extracting the Mel-spectrogram features and FBank features, the audio signal is input into the encoder to generate an initial feature representation; the feature representation is compressed using vector quantization technology to generate a quantized feature representation. The Mel-spectrogram features and FBank features are encoded respectively by a conditional prompt encoder and a voiceprint prompt encoder; feature alignment processing is performed by a multi-layer linear projection module, initial weight coefficients are assigned, and weighted fusion is performed by optimizing weights to generate a fused feature representation, and finally a target speech signal is generated by a decoder. The present invention extracts multiple audio features and compresses them in combination with vector quantization technology, thereby reducing data redundancy while maintaining important audio information. Through adaptive prompt coding and weight optimization mechanisms, it is ensured that various features are effectively utilized in the fusion process, and the fidelity and accuracy of speech reconstruction are enhanced. It can provide efficient data processing capabilities in high compression rate scenarios, reduce computing resource usage, and significantly improve the quality and naturalness of generated speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0037] Figure 1 A schematic diagram of an application environment of a method for generating speech based on prompt coding in an embodiment of the present invention;

[0038] Figure 2 It is a flow chart of an embodiment of a method for generating speech based on prompt coding according to the present invention;

[0039] Figure 3 It is a functional module diagram of a preferred embodiment of a speech generation device based on prompt coding of the present invention;

[0040] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0041] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0043] The speech generation method based on prompt coding provided by the embodiment of the present invention can be applied in Figure 1 In the application environment, the user end communicates with the server end through the network. The server end can obtain the original audio signal through the user end, extract the Mel spectrogram features and FBank features, input the audio signal into the encoder, generate the initial feature representation; use the vector quantization technology to compress the feature representation, and generate the quantized feature representation. The Mel spectrogram features and FBank features are encoded by the conditional prompt encoder and the voiceprint prompt encoder respectively; the feature alignment processing is performed by the multi-layer linear projection module, the initial weight coefficient is assigned, and the weighted fusion is performed by optimizing the weight to generate the fusion feature representation, and finally the target speech signal is generated by the decoder. The present invention extracts a variety of audio features and compresses them in combination with the vector quantization technology, while reducing data redundancy and maintaining important audio information. Through the adaptive prompt coding and weight optimization mechanism, it is ensured that various features are effectively utilized in the fusion process, and the fidelity and accuracy of speech reconstruction are enhanced. It can provide efficient data processing capabilities in high compression rate scenarios, reduce computing resource usage, and significantly improve the quality and naturalness of generated speech. Among them, the user end can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0044] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a method for generating speech based on prompt coding provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0045] like Figure 2 As shown, the speech generation method based on prompt coding proposed by the present invention comprises the following steps:

[0046] S10, obtaining an original audio signal, and extracting a Mel-spectrogram feature and an FBank feature from the original audio signal;

[0047] In this embodiment, during the speech generation process, the system first receives the user's voice input or obtains the original audio signal from the audio database. The signal is usually digitized waveform data, which can be captured by an input device such as a microphone, or loaded from an existing audio file (such as WAV, MP3 format). The audio signal is quantized by the sampling rate, and the typical sampling rate is 16kHz or 44.1kHz. The sampled digital audio data is processed in frames, and the length (frame length) and frame shift of each frame are set according to system requirements. The frame length is usually between 20ms and 40ms, and the frame shift is half or one-third of the frame length. After the audio data is framed, the system is ready for further feature extraction.

[0048] Mel spectrogram feature is a kind of frequency domain feature. By performing short-time Fourier transform (STFT) on the framed audio signal, its spectrum information is obtained, and the spectrum is processed by a Mel filter bank to extract the frequency distribution of the audio signal on the Mel scale. Perform a short-time Fourier transform on each frame of the audio signal to convert the time domain signal into a frequency domain signal. The spectrum is processed by a Mel filter bank. The Mel filter bank uses nonlinear frequency division to map the spectrum to the Mel scale according to the pitch distribution perceived by the human ear. The extracted Mel spectrogram feature reflects the frequency components of the audio signal and is an important basis for subsequent speech coding and generation.

[0049] The FBank (filter bank energy) feature processes the framed audio signal through a filter bank to obtain energy information in different frequency bands. Similar to the Mel-spectrogram, the FBank feature processes the spectrum through a set of filters to extract the energy distribution characteristics of multiple frequency bands. The framed audio signal passes through a series of filters, each filter integrates a subset of the frequency range to obtain the energy of the corresponding frequency band. A typical FBank feature consists of the energy of multiple frequency bands, which represents the strength of the audio signal in each frequency subset and is often used in scenarios such as voiceprint recognition.

[0050] Example description: In the financial industry, high-quality speech generation technology is required in voice interaction scenarios (such as intelligent customer service and voice authentication). In intelligent customer service, extracting Mel-spectrogram features and FBank features helps the system accurately capture important information in the user's voice and ensure that the voice responses generated by the system have natural and smooth sound quality. In the voice authentication scenario, the voiceprint recognition capability of the FBank feature enables the system to perform reliable identity authentication based on the frequency and pronunciation characteristics of the user's voice, effectively preventing fraud.

[0051] By extracting Mel-spectrogram features and FBank features from the original audio signal respectively, the key information in the speech signal can be effectively captured, which not only retains the spectral distribution of the audio signal, but also extracts more fine-grained voiceprint information.

[0052] S20, inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0053] In this embodiment, the original audio signal that has been processed in advance (i.e., the signal that has been framed and relevant audio features extracted) is input into a feature extraction encoder. The feature extraction encoder is usually a deep learning model that converts low-level audio features (such as Mel spectrograms and FBank features) into high-level initial feature representations. These high-level features can capture richer speech information in the audio signal.

[0054] The initial feature representation refers to the multi-dimensional features generated by the audio signal after being processed by the encoder. These features not only contain the basic spectral information of the audio, but also contain complex structural information related to speech generation, such as the rhythm, intonation and voiceprint features of the speech.

[0055] After the feature extraction of the audio signal is completed, these features are passed as input to the feature extraction encoder. The feature extraction encoder can be based on a deep learning architecture (such as multi-layer perceptron, convolutional neural network, recurrent neural network, etc.) to handle the conversion from input features to high-level features.

[0056] The encoder gradually abstracts and compresses the input features through multiple layers of nonlinear mapping. This process learns feature representation layer by layer, so that the output initial feature representation can capture higher-level audio information.

[0057] The input features can be mapped layer by layer to a higher-dimensional feature space through a multi-layer perceptron (MLP), and each layer is nonlinearly transformed through an activation function (such as ReLU or Sigmoid).

[0058] Alternatively, a convolutional neural network (CNN) can locally perceive the input audio features through convolution kernels, thereby capturing local patterns or features in the audio.

[0059] It can also be that the recurrent neural network (RNN / LSTM) can process the timing information in the audio signal, and is particularly suitable for processing continuous audio frame signals.

[0060] After the encoder is processed, the generated initial feature representation is a high-dimensional, multi-level audio description that can provide a basis for subsequent quantization, cue coding, and voiceprint analysis.

[0061] Example description: In financial scenarios, the application of feature extraction encoders can be used for voice generation of financial reports. For example, in the scenario of telephone banking or financial report generation, the original voice signal can be processed by the feature extraction encoder to capture the specific intonation and pronunciation characteristics of the customer's voice, and generate high-fidelity voice reports or feedback through the generated initial feature representation. In the customer identity verification scenario, the feature extraction encoder can capture the customer's unique voiceprint characteristics to ensure the accuracy and security of identity verification.

[0062] By inputting the original audio signal into the feature extraction encoder and generating the initial feature representation, the speech feature representation capability of the audio data is enhanced. Compared with the traditional audio feature processing method, the feature extraction encoder can extract higher-level and more complex speech information, making the subsequent quantization and prompt encoding more accurate and effective. In addition, the initial feature representation captures the detailed features in the audio, and can still maintain the fidelity and naturalness of speech generation under high compression rate, thereby improving the overall performance of the speech generation system.

[0063] S30, compressing the initial feature representation using a quantizer based on vector quantization to generate a quantized feature representation;

[0064] In this embodiment, vector quantization (VQ) is a commonly used data compression technology that effectively reduces the representation size of data by mapping continuous high-dimensional features into discrete symbols (codebook indexes). The system inputs the initial feature representation obtained from the feature extraction encoder into the vector quantizer. The vector quantizer compresses the high-dimensional initial feature representation into a discrete quantized feature representation based on a predefined codebook. This quantization operation can significantly reduce the data dimension and reduce the storage and transmission overhead.

[0065] The core of vector quantization is a codebook, in which each element (i.e., codebook vector) represents a specific category of feature representation. The size of the codebook determines the trade-off between compression accuracy and compression ratio. A larger codebook can retain more information, but it also increases the computational cost.

[0066] The initial feature representation is divided into several feature vectors, each of which is mapped to a vector in the codebook. Specifically, the codebook vector with the smallest distance to the feature vector is selected as the quantization result. The job of the quantizer is to convert all the initial feature representations into corresponding discrete indices that can be compactly represented as quantized features.

[0067] The quantized feature index is the quantized feature representation, which represents the key information in the initial feature representation and takes up less storage space. The quantized feature representation will be used in subsequent hint encoding and decoding operations.

[0068] Example description: In financial data processing scenarios, vector quantizers can be used to compress financial voice data, such as customer recordings of telephone banks, voice interactions of intelligent voice customer service, etc. Through vector quantization technology, financial institutions can efficiently store and transmit voice data under low bandwidth and low storage costs, while ensuring the fidelity of sound quality during voice interactions. For example, when a customer queries bank account information through a voice system, the system can quickly generate natural and high-fidelity voice responses with limited computing resources, thereby improving the customer's user experience and saving computing resources for financial institutions.

[0069] By using a quantizer based on vector quantization, the storage and transmission requirements of the initial feature representation are effectively reduced while retaining the key information required for speech generation. This quantization process significantly reduces the computational overhead of speech processing by selecting the optimal codebook index for feature compression, allowing the speech generation system to maintain high performance in resource-constrained environments.

[0070] S40, encoding the mel-spectrogram feature by a conditional prompt encoder to generate a conditional prompt feature;

[0071] In this embodiment, the conditional cue encoder is a core module in speech generation, which is used to extract high-level feature representations from the input mel-spectrogram features and generate conditional cue features. The mel-spectrogram features are used as input to represent the distribution of the audio signal on the frequency spectrum. Through the encoding process of the conditional cue encoder, the mel-spectrogram features will be converted into conditional cue features, which can provide contextual information of the audio signal and be used in the subsequent speech generation process.

[0072] Conditional cue features are features extracted during the encoding process that can capture high-level information such as speech characteristics, contextual information, and spectral changes in the audio.

[0073] The mel-spectrogram features are first input into a conditional cue encoder, which can be a deep learning model such as a multi-layer perceptron (MLP) or a convolutional neural network (CNN).

[0074] The conditional prompt encoder processes the input Mel spectrogram features in multiple layers, and each layer performs nonlinear mapping on the features to gradually extract higher-level audio features. The mapping operation of each layer can be achieved through weight matrix multiplication and activation function.

[0075] The Mel-spectrogram features can be mapped layer by layer through a multi-layer perceptron (MLP) to generate intermediate feature representations.

[0076] It is also possible to extract local features through convolution operations using convolutional neural networks (CNNs) to capture local patterns in the audio spectrum.

[0077] After several layers of processing, the feature representation output by the conditional cue encoder is the conditional cue feature. These features not only contain the spectral information of the Mel-spectrogram, but also integrate the context and other cue information, which can provide important cue signals for subsequent speech generation.

[0078] Example description: In the financial voice interaction system, the conditional prompt encoder can be used for voice generation of intelligent voice customer service or telephone banking. For example, when a customer queries account information through the telephone banking system, the system can extract the user's voice signal through the Mel-spectrogram feature, generate conditional prompt features through the conditional prompt encoder, and generate natural voice feedback based on these prompt features. This encoding process enables the system to accurately capture the details and contextual information of the customer's voice, thereby generating a more natural and fluent voice response and improving the customer experience.

[0079] By encoding the Mel-spectrogram features through the conditional cue encoder, key information can be extracted from the input audio signal to generate high-level conditional cue features. These features integrate the audio spectrum, speech rhythm and context information, providing important cue signals for the subsequent speech generation steps. Compared with traditional speech feature extraction methods, the conditional cue encoder can better capture complex audio structure features, thereby maintaining high-quality speech generation in high compression scenarios.

[0080] S50, encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0081] In this embodiment, the FBank feature represents the energy distribution information of the audio signal in different frequency bands, and is often used to describe the frequency characteristics of the speech signal. The function of the voiceprint prompt encoder is to encode the input FBank features and extract the voiceprint prompt features in the speech signal, that is, the information used to characterize the unique speech characteristics of the speaker. These voiceprint prompt features can capture individual differences in speech, mainly reflecting the speaker's pitch, frequency distribution, and speech pronunciation characteristics, and are used for identity authentication or personalized processing in speech generation or speech recognition.

[0082] The voiceprint cue encoder receives FBank features extracted from the original audio signal, which describe the energy distribution of the audio signal in different frequency bands.

[0083] The encoder processes the FBank features layer by layer through a multi-layer network structure (such as a convolutional neural network CNN or a recurrent neural network RNN), and gradually extracts the voiceprint feature information in the speech signal. Each layer processes the intermediate features through a nonlinear activation function to extract more content related to the speaker's personalized features.

[0084] Among them, the convolutional neural network (CNN) is suitable for capturing local frequency changes in FBank features and is used to extract the acoustic features of speech. The recurrent neural network (RNN / LSTM) is suitable for processing the temporal information of speech signals, especially in capturing the speaker's pronunciation style and rhythm.

[0085] Finally, through multiple layers of nonlinear mapping and feature compression, the voiceprint hint encoder generates high-level voiceprint hint features. These features contain the speaker's unique voice feature information and can be used in subsequent voice generation or identity verification steps.

[0086] Example description: In the identity authentication and voice interaction of the financial industry, the voiceprint prompt encoder can provide personalized and secure voice services for the financial voice system. For example, in the telephone banking system, the customer's voice signal is processed by the voiceprint prompt encoder, and the generated voiceprint prompt features can be used to identify the customer's identity, thereby replacing the traditional password or multi-factor authentication method. This not only improves the efficiency of customer service, but also enhances the security of the system.

[0087] By encoding the FBank features through the voiceprint cue encoder, the voiceprint features related to the speaker in the audio signal can be effectively extracted. These voiceprint cue features provide key reference information for subsequent speech generation, identity recognition, and personalized voice services. Compared with traditional voiceprint recognition methods, using deep learning networks for voiceprint feature extraction can capture more speech details and provide higher accuracy and adaptability in a variety of speech generation applications.

[0088] S60, using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation to a unified feature space to perform feature alignment processing;

[0089] In this embodiment, during the speech generation process, the system extracts different types of features from multiple sources, such as conditional cue features, voiceprint cue features, and quantitative feature representations. These features have different dimensions and structures, and directly using them for fusion or further processing may lead to inconsistent or distorted information. To solve this problem, it is necessary to map these features to the same dimensional space through a multi-layer linear projection module to ensure that they are represented in a unified feature space for subsequent feature alignment and fusion.

[0090] The multi-layer linear projection module projects conditional cue features, voiceprint cue features, and quantitative feature representations into a unified feature space through multi-layer linear transformations and activation functions. The purpose of feature alignment is to ensure that these features are expressed consistently in the same space, to enable effective weighted fusion, and to reduce information redundancy or loss.

[0091] The conditional cue features, voiceprint cue features, and quantitative feature representations are input into the multi-layer linear projection module respectively. Each feature may have different dimensions and distributions, so it needs to be processed independently.

[0092] Each feature is mapped through multiple layers of linear transformations. The linear mapping of each layer of projection is calculated through the weight matrix and bias term, mapping features of different dimensions to the same target dimensional space layer by layer.

[0093] Linear projection formula: Y = W·X+b; where W is the weight matrix, X is the input feature, b is the bias term, and Y is the output projection feature.

[0094] After each layer of linear mapping, a nonlinear activation function (such as ReLU or Sigmoid) is used to process the mapped features to further enhance the representation ability of the features.

[0095] After multiple layers of linear transformation, the conditional cue features, voiceprint cue features and quantitative feature representations are all projected to the same dimension. This process ensures the consistency of feature representations in the same space and prepares for subsequent feature alignment and fusion.

[0096] The conditional cue features, voiceprint cue features, and quantitative feature representations are uniformly mapped and aligned through a multi-layer linear projection module, improving the consistency of feature processing and fusion efficiency. By projecting features from different sources into the same feature space, the system can more effectively perform subsequent feature weighting and fusion, avoiding the problem of dimensional mismatch or information conflict between features. This process ensures the stability and accuracy of the data flow during speech generation, improves the naturalness and fidelity of the generated speech, and can significantly improve the quality of speech generation, especially in the scenario of multi-feature fusion.

[0097] S70, assigning corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantized feature representations after feature alignment;

[0098] In this embodiment, after feature alignment, conditional cue features, voiceprint cue features, and quantized feature representations have been projected into a unified feature space. In order to enable these features to play the best role in the subsequent speech generation process, the system needs to assign corresponding initial weight coefficients to each feature. These weight coefficients are used to determine the importance of each feature during fusion, thereby affecting the result of feature fusion.

[0099] The initial weight coefficients are allocated based on the results of feature alignment and can be obtained through preset empirical values ​​or through model training. The allocation of weight coefficients can reflect the different roles of each feature in the speech generation process. For example, the conditional cue feature may focus more on providing contextual information, while the voiceprint cue feature focuses more on personalized speaker characteristics.

[0100] Before assigning initial weight coefficients, the system first evaluates the contribution of each feature to the quality of speech generation. This can be based on empirical rules or by training a model with historical data. Common evaluation methods include the impact of features on audio reconstruction quality, personalized generation accuracy, etc.

[0101] According to the importance of each feature, the corresponding initial weight coefficient is assigned. The initial weight coefficient can be made in the following ways:

[0102] Use a preset fixed weight template to assign a fixed weight value to each feature based on experience.

[0103] The initial weight value of each feature can be dynamically adjusted through an adaptive algorithm, and the specific weight can be obtained through training.

[0104] In order to ensure that the weights of each feature are appropriate, the total initial weight coefficient usually needs to be normalized so that its sum is 1, thereby ensuring that the contributions of each feature are balanced during the fusion process.

[0105] By assigning initial weight coefficients to conditional prompt features, voiceprint prompt features, and quantitative features, the contribution of each feature in speech generation can be effectively controlled. This feature fusion based on weight coefficients ensures that feature information from different sources can be reasonably utilized, avoiding the problem of over-amplification or neglect of certain feature information. Especially in the speech generation scenario of multi-feature fusion, reasonable allocation of weights can improve the naturalness, personalization, and fidelity of the generated speech, while reducing data redundancy and improving the processing efficiency of the system.

[0106] S80, analyzing the similarity index between the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation;

[0107] In this embodiment, during the speech generation process, different types of features (such as conditional prompt features, voiceprint prompt features, and quantitative feature representations) may contain redundant or repeated information. In order to optimize the effect of feature fusion, these features need to be analyzed to determine the degree of similarity between them. The similarity index is used to measure the degree of similarity between these features. By analyzing the structural similarity of the feature representations, it is determined which features contain similar information, so as to optimize the processing in the subsequent weight adjustment and feature fusion process. Commonly used similarity analysis indicators include:

[0108] Structural Similarity Index (SSIM): A metric used to measure the structural similarity of two feature representations. It is usually used to measure the similarity between two features in structure, brightness, and contrast.

[0109] Cosine similarity: Calculates the cosine angle between feature vectors to measure the similarity in the direction of two feature vectors.

[0110] Euclidean distance: Calculates the distance between feature vectors to quantify the difference between them.

[0111] Paired comparisons are performed on conditional cue features, voiceprint cue features, and quantitative feature representations, and similarity scores between these features are calculated using the structural similarity index (SSIM) or other similarity measurement methods. The higher the similarity score, the greater the redundancy between the two features.

[0112] Through similarity analysis, the system can identify which feature information is redundant and which feature information can provide unique contributions to speech generation. In feature pairs with high redundancy, the system can choose to reduce the weight or perform feature fusion.

[0113] During the optimization process, the system can minimize unnecessary similarities by introducing a penalty term in the loss function. This method encourages the feature encoder to avoid generating repeated information when generating cue features.

[0114] By analyzing the similarity index between conditional prompt features, voiceprint prompt features, and quantitative feature representations, it is possible to effectively identify and reduce redundant information between features. Through similarity analysis, the system can optimize weight distribution in the subsequent feature fusion process to ensure that the contribution of various features to speech generation is unique and necessary, avoiding the waste of computing resources caused by excessive duplication of information. It can improve the quality of speech generation, making the generated speech signal more natural and the collaboration between features more efficient.

[0115] S90, optimizing the initial weight coefficient by back propagation according to the similarity index;

[0116] In this embodiment, during the speech generation process, there may be certain redundant information or similarities between different features. In order to more effectively utilize each feature information, it is necessary to optimize the initial weight coefficient of the feature according to the similarity index of the feature. Backpropagation is a commonly used optimization algorithm that adjusts the model parameters by calculating the gradient and performing backpropagation. The system uses the backpropagation algorithm to dynamically optimize the initial assigned weight coefficient according to the result of the similarity index to reduce redundant information and improve the effect of feature fusion.

[0117] The goal of back propagation is to minimize feature redundancy and ensure that conditional cue features, voiceprint cue features, and quantized feature representations play their respective unique roles in the feature fusion process, thereby improving the quality of speech generation.

[0118] First, the system calculates a loss function based on the similarity index between features. The loss function can be the structural similarity index (SSIM) or other similarity metrics. The loss function measures the degree of redundancy between features. The larger the loss value, the more redundancy.

[0119] By taking the derivative of the loss function, we calculate the gradient of the similarity index to each weight coefficient. This gradient indicates the direction of change of the feature weight. If the similarity of a feature is high, the corresponding gradient will adjust its weight to reduce the contribution of the feature.

[0120] Based on the calculated gradient, the initial weight coefficient is optimized and adjusted using a back-propagation algorithm (such as the gradient descent method). Each time the system is updated, the weight coefficient is adjusted according to the gradient direction to minimize the redundant information between features.

[0121] The back-propagation optimization process is an iterative process. The system gradually optimizes the weight coefficients through multiple iterations until the loss function is minimized, ensuring that the similarity between features is minimized.

[0122] By using the back-propagation algorithm to optimize the initial weight coefficient according to the similarity index, the effect of feature fusion in the speech generation process is improved. By minimizing the redundant information between features, the system can ensure that each feature can play its unique role when fused, avoiding the reuse of features and wasting resources. At the same time, back-propagation optimization can dynamically adjust the weight of features, making the system more flexible and adaptable during speech generation, thereby improving the naturalness and fidelity of the generated speech.

[0123] S100, performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0124] In this embodiment, in the previous step, the system optimized the initial weight coefficients through the similarity index and the back propagation algorithm. Next, it is necessary to perform weighted fusion on the conditional prompt features, voiceprint prompt features, and quantitative feature representations according to these optimized weight coefficients to generate the final fused feature representation. Weighted fusion is to multiply each feature by its corresponding optimized weight coefficient and add the results together to generate a comprehensive feature representation.

[0125] The role of weighted fusion is to integrate the advantages of various features so that the final fusion features can provide more accurate and comprehensive support in the subsequent speech generation process. The weight of each feature determines the contribution of the feature in the fusion.

[0126] Input optimized weight coefficients and features: The system receives optimized weight coefficients (such as α1, α2, α3) and conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment.

[0127] Multiply each feature by its corresponding optimization weight coefficient. Specifically: z fused =a1·z PC +a2·z PV +a3·z quant ; Among them, z fused is the final fusion feature representation, z PC is the conditional cue feature, z PV It is the voiceprint prompt feature, z quant It is a quantitative feature representation.

[0128] The weighted fusion feature representation will contain comprehensive information of various features. Depending on the weights, some features will play a greater role in the generation process, while other features may play an auxiliary role in specific speech generation tasks.

[0129] After generating the fused feature representation, the system can perform regularization on the feature representation to ensure that the feature value is in an appropriate range for subsequent decoding.

[0130] By performing weighted fusion according to the optimized weight coefficients, the system can effectively integrate the advantages of conditional cue features, voiceprint cue features, and quantitative feature representations. The weighted fusion process ensures that the information contribution of each feature is fully utilized, avoiding the problem of certain features being ignored or over-reliant during the generation process. The final fused feature representation has more comprehensive audio information, making the subsequent speech generation process more accurate and natural, especially in multi-feature fused speech generation systems, improving the overall performance and generation quality of the system.

[0131] S110, inputting the fused feature representation into a decoder to generate a target speech signal.

[0132] In this embodiment, after the fusion feature generation is completed, the next step is to input the fusion feature representation into the decoder, which is responsible for converting these high-level features back to the original target speech signal. The decoder can be a deep learning-based neural network, such as an autoregressive decoder, a convolutional neural network decoder (CNN), or a recurrent neural network (RNN)-based decoder, which ensures the naturalness and fidelity of the speech by reconstructing the audio signal frame by frame.

[0133] The task of the decoder is to gradually restore the time series signal of the original audio based on the fused feature representation, ensuring high-quality speech reconstruction at a lower bit rate.

[0134] The fused feature representation is generated by weighted fusion of the conditional cue feature, voiceprint cue feature and quantized feature representation. These fused features are used as input to the decoder.

[0135] The decoder usually processes the audio signal frame by frame. Through an autoregressive model (such as WaveNet or Tacotron), the audio signal of each frame is predicted based on the previously generated frames.

[0136] In the convolutional neural network-based decoder, the system restores the feature representation from low-dimensional to high-dimensional audio signal space through deconvolution.

[0137] Through the multi-layer network, the decoder gradually generates the speech signal at each time step until the complete speech signal sequence is generated.

[0138] The output generated by the decoder is the target speech signal, which is restored from the fused feature representation and is as close to the original high-quality audio signal as possible. This process ensures that the final generated speech signal has high fidelity and naturalness.

[0139] By inputting the fused feature representation into the decoder, the high-dimensional feature information can be restored to a natural and fluent target speech signal. The frame-by-frame processing of the decoder ensures that the generated audio signal has logical consistency and natural fluidity between each time step. In addition, through the decoding process of autoregression or convolution, the system can maintain high-quality speech reconstruction at a lower bit rate.

[0140] The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a speech generation method based on prompt coding. By acquiring the original audio signal, extracting the Mel-spectrogram feature and FBank feature, the audio signal is input into the encoder to generate the initial feature representation; the feature representation is compressed by using the vector quantization technology to generate the quantized feature representation. The Mel-spectrogram feature and the FBank feature are respectively encoded by the conditional prompt encoder and the voiceprint prompt encoder; the feature alignment processing is performed by the multi-layer linear projection module, the initial weight coefficient is assigned, and the weighted fusion is performed by optimizing the weight to generate the fusion feature representation, and finally the target speech signal is generated by the decoder. The present invention extracts a variety of audio features and compresses them in combination with the vector quantization technology, while reducing data redundancy and maintaining important audio information. Through the adaptive prompt coding and weight optimization mechanism, it is ensured that various features are effectively utilized in the fusion process, and the fidelity and accuracy of speech reconstruction are enhanced. It can provide efficient data processing capabilities in high compression rate scenarios, reduce computing resource usage, and significantly improve the quality and naturalness of generated speech.

[0141] In one embodiment, in the above S10, extracting Mel-spectrogram features and FBank features from the original audio signal respectively includes:

[0142] S101, performing frame processing on the original audio signal according to a preset frame length and frame shift;

[0143] S102, performing short-time Fourier transform on the framed audio signal to obtain frequency spectrum information of the audio signal;

[0144] S103, using a Mel filter bank to perform Mel-scale processing on the frequency spectrum information to extract Mel-spectrogram features of the audio signal;

[0145] S104, using a filter bank to perform linear frequency division or logarithmic frequency division processing on the framed audio signal to extract FBank features of the audio signal;

[0146] S105, performing logarithmic processing and feature smoothing processing on the Mel-spectrogram features and FBank features to generate final Mel-spectrogram features and FBank features.

[0147] In this embodiment, two key features are extracted from the original audio signal: Mel-spectrogram features and FBank features. These features are very important in speech processing systems and can describe the frequency information of speech signals and the energy distribution of audio signals. The extraction of these features depends on the application of audio signal frame processing, Fourier transform and filter bank, and finally generates high-quality audio features suitable for subsequent speech processing tasks.

[0148] The original audio signal is divided into frames according to the preset frame length and frame shift. The frame length indicates how much time the audio signal is contained in each frame, and the frame shift indicates the degree of overlap between frames. The common frame length is 20-40ms, and the frame shift is about 10ms. Through framing, the system can divide the continuous audio signal into multiple time periods for further processing.

[0149] Perform short-time Fourier transform (STFT) on the framed audio signal to convert the audio signal from the time domain to the frequency domain. The result of STFT is spectrum information, which indicates the intensity of different frequency components. This step is very critical because spectrum information is the basis for Mel-spectrogram feature and FBank feature extraction.

[0150] Mel filter banks are applied to the spectral information and frequency conversion is performed using the Mel scale. The Mel scale simulates the human ear's perception of different frequencies, with lower resolution for higher frequencies and higher resolution for lower frequencies. Through this processing, the system is able to extract Mel spectrogram features that are more compatible with human hearing and reflect the perceived spectral distribution of the speech signal.

[0151] Use filter banks to perform linear frequency division or logarithmic frequency division on the framed audio signal to extract FBank features. FBank features are mainly used to represent the energy distribution of speech signals in different frequency bands. Linear frequency division is applicable to the lower frequency part of the audio, while logarithmic frequency division is applicable to the higher frequency part.

[0152] Logarithmic processing is performed on the extracted Mel-spectrogram features and FBank features, and the feature values ​​are transformed into a logarithmic scale, which can better compress the dynamic range and make subsequent processing smoother and more stable. Feature smoothing is performed to reduce noise and mutation points in the features and ensure smooth transition of the features. This can improve the robustness of the features, so that the system can still generate high-quality features when facing speech signals in different environments.

[0153] This embodiment effectively improves the system's ability to process audio signals by extracting Mel-spectrogram features and FBank features. Mel-spectrogram features can capture more useful speech information by simulating human auditory perception frequency, while FBank features provide powerful frequency feature support by describing the frequency band energy distribution of speech. Logarithmic processing and smoothing processing further improve the robustness and stability of the features, allowing the system to generate high-quality speech features when facing audio signals in different environments.

[0154] In one embodiment, the above S40 includes:

[0155] S401, inputting the Mel-spectrogram feature to a conditional prompt encoder, wherein the conditional prompt encoder comprises a multi-layer perceptron;

[0156] S402, using the multilayer perceptron to perform layer-by-layer nonlinear mapping processing on the Mel-spectrogram features to generate an intermediate feature representation;

[0157] S403: Perform feature compression processing on the intermediate feature representation to generate conditional prompt features.

[0158] In this embodiment, during the speech generation process, the conditional cue feature is high-level feature information extracted from the input mel-spectrogram feature, which is used to provide context and conditional support for speech generation. The task of the conditional cue encoder is to process and extract the mel-spectrogram feature layer by layer through a multi-layer neural network, thereby generating conditional cue features suitable for speech generation.

[0159] The mel-spectrogram features extracted from the original audio are input into the conditional prompt encoder. The mel-spectrogram features are the frequency features of the processed audio signal, which contain the spectrum information of the audio signal.

[0160] The conditional cue encoder uses a multi-layer perceptron (MLP) to perform layer-by-layer nonlinear mapping on the Mel-spectrogram features. The multi-layer perceptron consists of several fully connected layers, each of which maps and transforms the input features through linear transformation (weight matrix multiplication and bias addition) and nonlinear activation functions (such as ReLU, Sigmoid), and gradually extracts higher-level feature representations.

[0161] After layer-by-layer processing by the multi-layer perceptron, an intermediate feature representation is generated. The intermediate feature representation is the result of nonlinear mapping of the Mel-spectrogram feature, which contains higher-level spectrum and speech feature information, but has not yet undergone the final feature compression processing.

[0162] Perform feature compression on the intermediate feature representation. Feature compression aims to reduce redundant information and further compress the intermediate feature representation into compact conditional cue features. This step reduces unnecessary feature dimensions to ensure that the output conditional cue features are more representative and compact, which is convenient for subsequent speech generation tasks.

[0163] The final output conditional prompt feature contains high-level speech feature information extracted by the multi-layer perceptron, which can provide effective prompt signals for the subsequent speech generation steps.

[0164] This embodiment encodes the Mel-spectrogram features through a conditional cue encoder, and the system can extract high-level feature representations from the frequency information. These features can provide contextual cues for subsequent speech generation. The generation of conditional cue features can capture important information in the audio signal through layer-by-layer nonlinear mapping, while reducing redundant data through feature compression to ensure that the generated cue features are compact and representative. In the speech generation task, the conditional cue features can effectively help the system generate natural and fluent speech signals and improve the quality of speech generation.

[0165] In one embodiment, the above S50 includes:

[0166] S501, inputting the FBank feature to a voiceprint prompt encoder, wherein the voiceprint prompt encoder includes a deep learning network;

[0167] S502, using a voiceprint prompt encoder to extract individual voiceprint features from the FBank features, where the individual voiceprint features are used to reflect the speech frequency and pronunciation characteristics of the speaker;

[0168] S503, performing layer-by-layer nonlinear mapping and activation processing on the individual voiceprint features through the deep learning network;

[0169] S504, performing feature compression processing on the processed individual voiceprint features to generate voiceprint prompt features.

[0170] In this embodiment, the voiceprint prompt feature is generated based on the FBank feature (filter bank band energy feature) contained in the input voice signal, and is used to represent the speaker's unique voiceprint information. Through the voiceprint prompt encoder, the system can extract individual voiceprint features from the FBank feature, including pitch, pronunciation style, voice frequency and other features, thereby providing a personalized prompt signal for subsequent voice generation.

[0171] The voiceprint cue encoder is usually based on a deep learning network, which processes the FBank features layer by layer, extracts and generates the final voiceprint cue features, ensuring that the personalized and speaker identity-related speech characteristics are preserved.

[0172] The FBank feature extracted from the original audio signal is input into the voiceprint prompt encoder. The FBank feature describes the energy distribution of the audio signal in different frequency bands and is an important representation of speech frequency characteristics.

[0173] The voiceprint prompt encoder uses a deep learning network to process the input FBank features and extract individual voiceprint features related to the speaker's identity. These features can reflect the speaker's voice characteristics such as voice frequency, pitch, pronunciation style, etc., and are highly personalized.

[0174] The voiceprint prompt encoder is composed of multiple layers of deep learning networks. Each layer gradually processes individual voiceprint features through nonlinear mapping, extracting higher-level speech information layer by layer. Activation functions (such as ReLU and Sigmoid) are used to enhance the nonlinear representation ability of features to ensure that voiceprint features can be fully extracted and optimized.

[0175] The extracted individual voiceprint features are compressed. The purpose of compression is to reduce redundant information and compress high-dimensional feature representations into more compact voiceprint prompt features, ensuring that the generated voiceprint prompt features can effectively reflect the speaker's unique voice characteristics and facilitate subsequent processing and storage.

[0176] The final generated voiceprint prompt features can capture the speaker's personalized voice characteristics. These features will be used as prompt signals in subsequent speech generation tasks to ensure that the generated speech has the speaker's unique timbre and pronunciation characteristics.

[0177] This embodiment encodes the FBank features through a voiceprint prompt encoder, and the system can extract individualized voiceprint features from the speech signal. These voiceprint features can provide personalized prompts for subsequent speech generation, ensuring that the generated speech has unique speaker voice characteristics. Through layer-by-layer nonlinear mapping and feature compression, the system can extract highly refined voiceprint prompt information from the input FBank features, improving the naturalness and personalization of the generated speech. In addition, feature compression makes the generated prompt features more compact and easy to process and store.

[0178] In one embodiment, the above S60 includes:

[0179] S601, inputting the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation into a multi-layer linear projection module;

[0180] S602, using a multi-layer linear projection module to perform linear mapping processing on the input conditional prompt features, voiceprint prompt features and quantized feature representations respectively, each layer of linear projection includes weight matrix multiplication and bias addition;

[0181] S603, after each layer of linear projection, use an activation function to activate the mapped features;

[0182] S604, after completing the mapping and activation processing, the conditional prompt features, voiceprint prompt features and quantitative feature representations aligned in the unified feature space are obtained.

[0183] In this embodiment, in the speech generation system, different types of features (such as conditional prompt features, voiceprint prompt features, and quantitative feature representations) may have different dimensions and representations. In order to fuse these features, the system needs to map various types of features to the same feature space through a multi-layer linear projection module and perform feature alignment. This process ensures consistency and fusion between features, so that the information of each feature can be effectively utilized in the subsequent weighted fusion process.

[0184] The main task of the multi-layer linear projection module is to map different input features to the same dimension through layer-by-layer linear transformation and nonlinear activation function, and ensure alignment between features. This alignment is the premise of fusion and avoids the problem of incompatibility of features in dimension or representation.

[0185] Conditional cue features, voiceprint cue features, and quantized feature representations are input into the multi-layer linear projection module respectively. Each feature may come from a different processing module, so it may have different dimensions and representations before input.

[0186] The multi-layer linear projection module performs linear mapping on each input feature layer by layer. After each layer of linear projection, a nonlinear activation function (such as ReLU, Sigmoid) is used to activate the mapped features to ensure that the features can express richer nonlinear relationships. The role of the activation function is to increase the nonlinear representation ability of the model and help capture the complex relationships between features.

[0187] After completing the linear mapping and activation processing, the conditional cue features, voiceprint cue features, and quantized feature representations are mapped to the same dimension, and all features are aligned in the same feature space. This alignment ensures the consistency between different features, allowing subsequent fusion and processing steps to operate in the same space, avoiding feature incompatibility issues.

[0188] This embodiment maps conditional prompt features, voiceprint prompt features, and quantitative feature representations to the same feature space through a multi-layer linear projection module, ensuring compatibility and consistency between features. The feature alignment process not only ensures the dimensional unification of various features, but also enhances the expressiveness of features through nonlinear activation functions, helping to capture the complex relationships between different features. Ultimately, the aligned features can play a role in subsequent weighted fusion, improving the naturalness and accuracy of speech generation.

[0189] In one embodiment, the above S70 includes:

[0190] S701, based on historical training experience data, using error analysis to evaluate the contribution of the conditional prompt feature, voiceprint prompt feature and quantitative feature representation to the audio reconstruction quality;

[0191] S702, according to the contribution degree, assigning corresponding initial weight coefficients proportional to the contribution degree to the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation;

[0192] S703: Normalize all initial weight coefficients.

[0193] In this embodiment, after feature alignment, the system needs to assign appropriate initial weight coefficients to different features. The allocation of initial weight coefficients is based on the evaluation of the contribution of each feature to speech generation or reconstruction. In order to ensure the effect of feature fusion, the weight coefficient must reflect the importance of each feature in audio reconstruction. This process evaluates the contribution of each feature to the quality of audio reconstruction based on historical training experience data and error analysis, and finally assigns corresponding weights according to the degree of contribution.

[0194] The accurate allocation of initial weight coefficients is a key step to ensure that the speech generation process can reasonably utilize each feature information. Finally, the system will normalize all weight coefficients to ensure that the sum is 1, so that the contribution of each feature in the feature fusion process remains balanced.

[0195] The system evaluates the contribution of conditional cue features, voiceprint cue features, and quantized feature representations to the quality of audio reconstruction by using historical training data and error analysis of the audio generation model. Error analysis can determine the importance of each feature by comparing the error between the generated audio and the target audio. For example, the system can use MSE (mean square error) or L1 loss to measure the impact of each feature on the accuracy of audio reconstruction, thereby deriving their contribution.

[0196] Based on the above evaluation results, the system assigns each feature an initial weight coefficient proportional to its contribution. For example, if the conditional cue feature contributes the most to audio generation, its weight coefficient will be relatively large, and other features such as voiceprint cue features and quantitative features will be assigned smaller values ​​accordingly.

[0197] The distribution formula of weight coefficient can be expressed as:

[0198]

[0199] Among them, a i is the weight coefficient of the i-th feature, and the contribution value represents the influence of this feature on the audio reconstruction quality.

[0200] After assigning the weight coefficients, the system normalizes all the weight coefficients to ensure that their sum is 1. The purpose of normalization is to balance the influence of each feature and avoid excessive influence of a certain feature on the speech generation process.

[0201] This embodiment allocates initial weight coefficients based on historical data and error analysis, so that the system can reasonably use the information of each feature in the feature fusion process. It ensures that the conditional prompt feature, voiceprint prompt feature and quantitative feature representation can play their due role in audio reconstruction, avoiding excessive reliance on or neglect of a certain feature. Normalization processing further ensures that the contribution of each feature remains within a reasonable range, improving the naturalness and accuracy of speech generation.

[0202] In one embodiment, the above S110 includes:

[0203] S1101, inputting the fused feature representation into an autoregressive decoder;

[0204] S1102, processing the fused feature representation time-step by time-step through the autoregressive decoder to gradually generate a speech signal for each time frame;

[0205] S1103: Combine the speech signals of all time frames into a continuous target speech signal.

[0206] In this embodiment, the fused feature representation is input into the autoregressive decoder, and the target speech signal is generated by processing it step by step (frame by frame). The autoregressive decoder is a deep neural network model based on a recursive structure, which processes the input features step by step to generate a frame of the speech signal, and then uses the frame information as input to continue generating the next frame until the entire speech sequence is generated.

[0207] The generation of each time frame depends on the previously generated frames, so that the temporal continuity and naturalness of the speech signal are maintained. Finally, all generated time frames will be combined into a continuous speech signal, that is, the complete target speech.

[0208] The fused feature representation obtained by weighted feature fusion is input into the autoregressive decoder. The fused feature representation contains information about the optimized conditional cue features, voiceprint cue features, and quantized feature representations, and is the basis for generating the target speech signal. The autoregressive decoder processes the fused feature representation time-step by time-step. That is, the system starts from the first time frame, generates an audio frame, and then recursively uses the output of the frame as input to process the next time frame. The generation of each time frame depends on the output of the previous frame, ensuring the smoothness and naturalness of the speech signal in time.

[0209] Each frame of audio signal generated by the decoder represents a time segment of the target speech signal. By gradually generating the speech signal of each time frame in an autoregressive manner, the decoder ensures that the generated speech signal can transition naturally and smoothly.

[0210] The system combines all generated time frame speech signals into a continuous target speech signal. This step concatenates all the individually generated time frame speech data to form the final audio file or stream.

[0211] This embodiment can generate a continuous and natural speech signal by inputting the fused feature representation into the autoregressive decoder. The autoregressive decoder generates the speech signal by time step, ensuring the temporal consistency and coherence of the generated audio signal. In addition, through the frame-by-frame generation process, the system can ensure that the generated speech has high naturalness and fidelity, which is particularly suitable for application scenarios that require high-quality audio output, such as speech synthesis and voice assistants.

[0212] In one embodiment, a speech generation device based on prompt coding is provided, and the speech generation device based on prompt coding corresponds to the speech generation method based on prompt coding in the above embodiment. Figure 3 , Figure 3 The figure is a functional module diagram of a preferred embodiment of a speech generation device based on prompt coding of the present invention. Audio feature extraction module 10, feature extraction encoder module 20, vector quantization module 30, conditional prompt encoder module 40, voiceprint prompt encoder module 50, multi-layer linear projection module 60, weight allocation module 70, similarity analysis module 80, weight optimization module 90, feature fusion module 100 and decoder module 110. The functional modules are described in detail as follows:

[0213] The audio feature extraction module 10 is used to obtain the original audio signal and extract the Mel-spectrogram feature and FBank feature from the original audio signal;

[0214] A feature extraction encoder module 20, configured to input the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0215] A vector quantization module 30, configured to compress the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation;

[0216] A conditional prompt encoder module 40, configured to encode the mel-spectrogram feature through a conditional prompt encoder to generate a conditional prompt feature;

[0217] A voiceprint prompt encoder module 50 is used to encode the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0218] A multi-layer linear projection module 60, used for mapping the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation to a unified feature space using a multi-layer linear projection module to perform feature alignment processing;

[0219] A weight assignment module 70, for assigning corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantized feature representations after feature alignment;

[0220] A similarity analysis module 80, for analyzing a similarity index between a conditional prompt feature, a voiceprint prompt feature and a quantitative feature representation;

[0221] A weight optimization module 90, configured to optimize the initial weight coefficient by back propagation according to the similarity index;

[0222] The feature fusion module 100 is used to perform weighted fusion on the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0223] The decoder module 110 is used to input the fused feature representation into a decoder to generate a target speech signal.

[0224] In one embodiment, the audio feature extraction module 10 is specifically used for:

[0225] The original audio signal is processed by dividing the frame into frames according to a preset frame length and frame shift;

[0226] Performing short-time Fourier transform on the framed audio signal to obtain frequency spectrum information of the audio signal;

[0227] Using a Mel filter bank to process the frequency spectrum information in a Mel scale to extract Mel spectrogram features of the audio signal;

[0228] Using a filter bank to perform linear frequency division or logarithmic frequency division on the framed audio signal to extract FBank features of the audio signal;

[0229] The Mel-spectrogram features and FBank features are logarithmically processed and feature smoothed to generate final Mel-spectrogram features and FBank features.

[0230] In one embodiment, the conditional prompt encoder module 40 is specifically used to:

[0231] Inputting the mel-spectrogram feature to a conditional prompt encoder, wherein the conditional prompt encoder comprises a multi-layer perceptron;

[0232] Using the multi-layer perceptron to perform layer-by-layer nonlinear mapping processing on the Mel-spectrogram features to generate an intermediate feature representation;

[0233] The intermediate feature representation is subjected to feature compression processing to generate conditional prompt features.

[0234] In one embodiment, the voiceprint prompt encoder module 50 is specifically used to:

[0235] Inputting the FBank features to a voiceprint prompt encoder, wherein the voiceprint prompt encoder comprises a deep learning network;

[0236] Extracting individual voiceprint features from the FBank features using a voiceprint prompt encoder, wherein the individual voiceprint features are used to reflect the speech frequency and pronunciation characteristics of the speaker;

[0237] Performing layer-by-layer nonlinear mapping and activation processing on the individual voiceprint features through the deep learning network;

[0238] The processed individual voiceprint features are subjected to feature compression processing to generate voiceprint prompt features.

[0239] In one embodiment, the multi-layer linear projection module 60 is specifically used for:

[0240] Inputting the conditional prompt feature, voiceprint prompt feature and quantitative feature representation into a multi-layer linear projection module;

[0241] A multi-layer linear projection module is used to perform linear mapping processing on the input conditional prompt features, voiceprint prompt features and quantitative feature representations respectively. Each layer of linear projection includes weight matrix multiplication and bias addition.

[0242] After each layer of linear projection, an activation function is used to activate the mapped features;

[0243] After completing the mapping and activation processing, the conditional cue features, voiceprint cue features and quantitative feature representations aligned in a unified feature space are obtained.

[0244] In one embodiment, the weight allocation module 70 is specifically configured to:

[0245] Based on historical training experience data, error analysis is used to evaluate the contribution of the conditional cue features, voiceprint cue features, and quantitative feature representations to the quality of audio reconstruction;

[0246] According to the contribution degree, assigning corresponding initial weight coefficients proportional to the contribution degree to the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation;

[0247] Normalize all initial weight coefficients.

[0248] In one embodiment, the decoder module 110 is specifically configured to:

[0249] Inputting the fused feature representation into an autoregressive decoder;

[0250] Processing the fused feature representation time-step by time-step through the autoregressive decoder to gradually generate a speech signal for each time frame;

[0251] The speech signals of all time frames are combined into a continuous target speech signal.

[0252] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a speech generation method based on prompt coding.

[0253] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a user-side method for generating a speech based on prompt coding.

[0254] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0255] Acquire an original audio signal, and extract Mel-spectrogram features and FBank features from the original audio signal respectively;

[0256] Inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0257] Compressing the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation;

[0258] Encoding the Mel-spectrogram feature by a conditional prompt encoder to generate a conditional prompt feature;

[0259] Encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0260] Using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation to a unified feature space to perform feature alignment processing;

[0261] Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment;

[0262] Analyze the similarity index between conditional cue features, voiceprint cue features and quantitative feature representations;

[0263] According to the similarity index, optimizing the initial weight coefficient by back propagation;

[0264] Performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0265] The fused feature representation is input into a decoder to generate a target speech signal.

[0266] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0267] Acquire an original audio signal, and extract Mel-spectrogram features and FBank features from the original audio signal respectively;

[0268] Inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation;

[0269] Compressing the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation;

[0270] Encoding the Mel-spectrogram feature by a conditional prompt encoder to generate a conditional prompt feature;

[0271] Encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature;

[0272] Using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation to a unified feature space to perform feature alignment processing;

[0273] Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment;

[0274] Analyze the similarity index between conditional cue features, voiceprint cue features and quantitative feature representations;

[0275] According to the similarity index, optimizing the initial weight coefficient by back propagation;

[0276] Performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation;

[0277] The fused feature representation is input into a decoder to generate a target speech signal.

[0278] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0279] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0280] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0281] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for generating speech based on prompt coding, characterized in that: The following steps are involved: Acquire an original audio signal, and extract Mel-spectrogram features and FBank features from the original audio signal respectively; Inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation; Compressing the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation; Encoding the Mel-spectrogram feature by a conditional prompt encoder to generate a conditional prompt feature; Encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature; Using a multi-layer linear projection module to map the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation to a unified feature space to perform feature alignment processing; Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantitative feature representations after feature alignment; Analyze the similarity index between conditional cue features, voiceprint cue features and quantitative feature representations; According to the similarity index, optimizing the initial weight coefficient by back propagation; Performing weighted fusion on the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation; The fused feature representation is input into a decoder to generate a target speech signal.

2. The method for generating speech based on prompt coding as claimed in claim 1, characterized in that: Mel-spectrogram features and FBank features are extracted from the original audio signal, including: The original audio signal is processed by dividing the frame into frames according to a preset frame length and frame shift; Performing short-time Fourier transform on the framed audio signal to obtain frequency spectrum information of the audio signal; Using a Mel filter bank to process the frequency spectrum information in a Mel scale to extract Mel spectrogram features of the audio signal; Using a filter bank to perform linear frequency division or logarithmic frequency division on the framed audio signal to extract FBank features of the audio signal; The Mel-spectrogram features and FBank features are logarithmically processed and feature smoothed to generate final Mel-spectrogram features and FBank features.

3. The method for generating speech based on prompt coding as claimed in claim 1, characterized in that: The mel spectrogram feature is encoded by a conditional prompt encoder to generate a conditional prompt feature, including: Inputting the mel-spectrogram feature to a conditional prompt encoder, wherein the conditional prompt encoder comprises a multi-layer perceptron; Using the multi-layer perceptron to perform layer-by-layer nonlinear mapping processing on the Mel-spectrogram features to generate an intermediate feature representation; The intermediate feature representation is subjected to feature compression processing to generate conditional prompt features.

4. The method for generating speech based on prompt coding as claimed in claim 1, characterized in that: The FBank feature is encoded by a voiceprint prompt encoder to generate a voiceprint prompt feature, including: Inputting the FBank features to a voiceprint prompt encoder, wherein the voiceprint prompt encoder comprises a deep learning network; Extracting individual voiceprint features from the FBank features using a voiceprint prompt encoder, wherein the individual voiceprint features are used to reflect the speech frequency and pronunciation characteristics of the speaker; Performing layer-by-layer nonlinear mapping and activation processing on the individual voiceprint features through the deep learning network; The processed individual voiceprint features are subjected to feature compression processing to generate voiceprint prompt features.

5. The method for generating speech based on prompt coding as claimed in claim 1, characterized in that: The conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation are mapped to a unified feature space using a multi-layer linear projection module to perform feature alignment processing, including: Inputting the conditional prompt feature, voiceprint prompt feature and quantitative feature representation into a multi-layer linear projection module; A multi-layer linear projection module is used to perform linear mapping processing on the input conditional prompt features, voiceprint prompt features and quantitative feature representations respectively. Each layer of linear projection includes weight matrix multiplication and bias addition. After each layer of linear projection, an activation function is used to activate the mapped features; After completing the mapping and activation processing, the conditional cue features, voiceprint cue features and quantitative feature representations aligned in a unified feature space are obtained.

6. The method for generating speech based on prompt coding as claimed in claim 1, characterized in that: Assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantized feature representations after feature alignment, including: Based on historical training experience data, error analysis is used to evaluate the contribution of the conditional cue features, voiceprint cue features, and quantitative feature representations to the quality of audio reconstruction; According to the contribution degree, assigning corresponding initial weight coefficients proportional to the contribution degree to the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation; Normalize all initial weight coefficients.

7. The method for generating speech based on prompt coding as claimed in claim 1, characterized in that: Inputting the fused feature representation into a decoder to generate a target speech signal includes: Inputting the fused feature representation into an autoregressive decoder; Processing the fused feature representation time-step by time-step through the autoregressive decoder to gradually generate a speech signal for each time frame; The speech signals of all time frames are combined into a continuous target speech signal.

8. A speech generation device based on prompt coding, characterized in that: The speech generation device based on prompt coding comprises: An audio feature extraction module is used to obtain an original audio signal and extract a Mel-spectrogram feature and an FBank feature from the original audio signal; A feature extraction encoder module, used for inputting the original audio signal into a feature extraction encoder to obtain an initial feature representation; A vector quantization module, used to compress the initial feature representation using a vector quantization-based quantizer to generate a quantized feature representation; A conditional prompt encoder module, used for encoding the mel-spectrogram feature through a conditional prompt encoder to generate a conditional prompt feature; A voiceprint prompt encoder module, used for encoding the FBank feature through a voiceprint prompt encoder to generate a voiceprint prompt feature; A multi-layer linear projection module, used for mapping the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation to a unified feature space using the multi-layer linear projection module to perform feature alignment processing; A weight assignment module is used to assign corresponding initial weight coefficients to the conditional prompt features, voiceprint prompt features and quantized feature representations after feature alignment; A similarity analysis module is used to analyze the similarity index between the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation; A weight optimization module, used for optimizing the initial weight coefficient by back propagation according to the similarity index; A feature fusion module, used to perform weighted fusion on the conditional prompt feature, the voiceprint prompt feature and the quantitative feature representation according to the optimized weight coefficient to generate a final fused feature representation; The decoder module is used to input the fused feature representation into a decoder to generate a target speech signal.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a prompt coding-based speech generation program stored in the memory and executable on the processor, wherein the prompt coding-based speech generation program, when executed by the processor, implements the steps of the prompt coding-based speech generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a speech generation program based on prompt coding, and when the speech generation program based on prompt coding is executed by the processor, the steps of the speech generation method based on prompt coding as described in any one of claims 1 to 7 are implemented.

Images