Speech synthesis methods, devices, equipment, and media based on gating attention mechanisms

Abstract

This invention relates to the fields of artificial intelligence and fintech, and discloses a speech synthesis method based on a gated attention mechanism. The method involves acquiring text and speech data. The text data is converted into a sequence of text symbols using a text encoder, while the speech data has its speech features extracted using a speech encoder trained on a self-supervised learning model and quantized into a discrete sequence of speech symbols. The text and speech symbol sequences are then organized into text and speech sequences, respectively. A preliminary alignment process establishes the correspondence between text and speech symbols, and the gated attention mechanism dynamically adjusts the matching degree between them. Finally, a decoder generates the final speech signal. This invention effectively improves the speech feature extraction capability through a speech encoder trained on a self-supervised learning model, especially in scenarios lacking a large amount of labeled data, where it can still learn effective feature representations from unlabeled data.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and fintech, and particularly to a speech synthesis method, apparatus, device, and storage medium based on a gating attention mechanism. Background Technology

[0002] In the financial sector, with the development of fintech, automated processing and intelligent decision-making are widely used in various scenarios such as customer service, risk assessment, and voice interaction, especially speech synthesis technology. In financial services, speech synthesis is widely used in scenarios such as intelligent customer service, telephone notifications, and transaction confirmation to enhance user experience. However, existing speech synthesis technologies still face many technical limitations in their application.

[0003] First, existing speech synthesis technologies rely on training with a large number of samples, and zero-shot speech synthesis remains a significant challenge. For personalized speech content that needs to be generated quickly in financial scenarios, traditional speech synthesis models that rely on large-scale data training often lag behind and cannot efficiently and accurately generate customized speech content.

[0004] Secondly, while neural network models, especially codec language models, have made significant progress in speech synthesis, their autoregressive nature limits inference speed. In the financial sector, rapid response to customer needs or market changes is crucial, but due to the imperfect implicit alignment between text and audio, the stability and consistency of speech generation are difficult to guarantee, especially under high-concurrency requests, which can easily lead to speech output delays or errors.

[0005] Furthermore, existing technologies occasionally exhibit mismatches between speech synthesis and text prompts, such as inconsistencies between speech and text content. This can lead to lost, repeated, or incorrect information in voice notifications. This can have serious consequences in financial transactions or critical business notifications, especially when errors occur in voice transaction confirmations or compliance reminders, directly impacting customer experience and even posing compliance risks.

[0006] Therefore, how to effectively improve the accuracy and real-time performance of speech synthesis, especially to achieve efficient and stable speech generation in a zero-sample environment, remains a technical bottleneck that urgently needs to be addressed in the financial sector. Summary of the Invention

[0007] The main objective of this invention is to provide a speech synthesis method, apparatus, device, and storage medium based on a gating attention mechanism, aiming to solve the technical problems of slow inference speed and inaccurate matching of speech and text in existing speech synthesis technologies under zero-sample environments, resulting in unstable generated speech signals.

[0008] To achieve the above objectives, this invention provides a speech synthesis method based on a gating attention mechanism, comprising:

[0009] Acquire text and voice data;

[0010] The text data is input into a text encoder, which converts the text data into a sequence of text symbols.

[0011] The speech data is input into a speech encoder, which extracts speech features from the speech data and quantizes the speech features into a discrete sequence of speech symbols.

[0012] The text symbol sequence is organized into a text sequence, and the speech symbol sequence is organized into a speech sequence;

[0013] The text sequence and speech sequence are initially aligned to establish the correspondence between text symbols and speech symbols;

[0014] Gated attention processing is applied to the initially aligned text and speech sequences to adjust the matching degree between text symbols and speech symbols;

[0015] The text sequence and speech sequence, which have undergone gated attention processing, are input into the decoder to generate the final speech signal.

[0016] Furthermore, to achieve the above objectives, the present invention provides a speech synthesis device based on a gating attention mechanism, comprising:

[0017] The data acquisition module is used to acquire text and voice data;

[0018] A text encoding module is used to input the text data into a text encoder, and convert the text data into a sequence of text symbols through the text encoder;

[0019] A speech encoding module is used to input the speech data into a speech encoder, extract speech features from the speech data through the speech encoder, and quantize the speech features into a discrete sequence of speech symbols;

[0020] A sequence organization module is used to organize the text symbol sequence into a text sequence and the speech symbol sequence into a speech sequence;

[0021] The alignment module is used to perform preliminary alignment processing on the text sequence and the speech sequence to establish the correspondence between text symbols and speech symbols;

[0022] The gated attention mechanism module is used to perform gated attention processing on the initially aligned text and speech sequences to adjust the degree of matching between text symbols and speech symbols.

[0023] The decoder module is used to input the gated attention-processed text sequence and speech sequence into the decoder to generate the final speech signal.

[0024] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a speech synthesis program based on a gating attention mechanism stored in the memory and executable on the processor, wherein when the speech synthesis program based on the gating attention mechanism is executed by the processor, it implements the steps of the speech synthesis method based on the gating attention mechanism as described above.

[0025] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a speech synthesis program based on a gating attention mechanism, wherein the speech synthesis program based on the gating attention mechanism, when executed by a processor, implements the steps of the speech synthesis method based on the gating attention mechanism as described above.

[0026] Beneficial Effects: This invention relates to the fields of artificial intelligence and fintech, and discloses a speech synthesis method based on a gated attention mechanism. By acquiring text and speech data, the text data is converted into a sequence of text symbols using a text encoder, and the speech data is processed by a speech encoder trained on a self-supervised learning model to extract speech features, which are then quantized into discrete speech symbol sequences. Next, the text symbol sequences and speech symbol sequences are organized into text sequences and speech sequences, respectively. A preliminary alignment process establishes the correspondence between text and speech symbols, and the gated attention mechanism dynamically adjusts the matching degree between text and speech symbols. Finally, a decoder generates the final speech signal. This invention effectively improves the speech feature extraction capability through a speech encoder trained on a self-supervised learning model, especially in scenarios lacking a large amount of labeled data, still learning effective feature representations from unlabeled data. Combined with the gated attention mechanism, it can adjust the matching relationship between text and speech symbols, ensuring the accuracy and fluency of the synthesized speech. Attached Figure Description

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0028] Figure 1 This is a schematic diagram of an application environment for a speech synthesis method based on a gating attention mechanism according to an embodiment of the present invention;

[0029] Figure 2 This is a flowchart illustrating an embodiment of the speech synthesis method based on the gating attention mechanism of the present invention.

[0030] Figure 3This is a schematic diagram of the functional modules of a preferred embodiment of the speech synthesis device based on the gating attention mechanism of the present invention;

[0031] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0032] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0033] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0034] The speech synthesis method based on gating attention mechanism provided in this invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can obtain text and speech data from the user terminal. The text data is converted into a text symbol sequence by a text encoder, and the speech data has its speech features extracted by a speech encoder trained based on a self-supervised learning model and quantized into a discrete speech symbol sequence. Then, the text symbol sequence and the speech symbol sequence are organized into a text sequence and a speech sequence, respectively. A preliminary alignment process establishes the correspondence between text and speech symbols, and a gating attention mechanism dynamically adjusts the matching degree between text and speech symbols. Finally, a decoder generates the final speech signal. This invention effectively improves the speech feature extraction capability through a speech encoder trained by a self-supervised learning model, especially in scenarios lacking a large amount of labeled data, it can still learn effective feature representations from unlabeled data. Combined with the gating attention mechanism, it can adjust the matching relationship between text and speech symbols, ensuring the accuracy and fluency of the synthesized speech. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0035] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the speech synthesis method based on a gating attention mechanism provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0036] like Figure 2 As shown, the speech synthesis method based on gating attention mechanism proposed in this invention includes the following steps:

[0037] S10, acquire text and voice data;

[0038] In this embodiment, the input data to be processed is obtained from external or internal sources. Text data typically refers to natural language text, such as articles, conversations, or text content in financial reports. Voice data refers to audio files, usually human voice input, including conversations in customer service calls, transaction confirmation notifications, etc.

[0039] Text data is typically formatted as readable strings or sentences and is stored in databases, text files, or retrieved in real-time from external systems via APIs. For example, in the financial sector, text data could be customer conversation logs, transaction information, or contract text.

[0040] Voice data is typically recorded audio files or real-time audio streams, and file formats may include WAV, MP3, etc. Voice data may contain different pronunciations, intonations, and even background noise, requiring processing in subsequent steps.

[0041] Text data can be extracted in real time from databases or financial systems via APIs. For example, in a financial customer service scenario, the system can capture the text entered by the user in real time from customer service conversations, or obtain relevant data from financial statements or contract management systems. Text data can also be obtained from CSV files, databases, or XML files through batch import.

[0042] Voice data can be recorded in real time via an audio interface (such as a microphone) or extracted from an existing voice library. For example, in a financial transaction confirmation call, the system can collect the customer's voice confirmation data in real time through a telephone recording interface. Voice files are stored on a server or cloud storage and can be accessed via file path or the voice signal can be extracted directly from the audio file.

[0043] Example Explanation: When a customer authenticates themselves through a bank's mobile application or online customer service system, the system will require the customer to enter personal information, such as name, account number, and ID card number. This text data is transmitted in real time to the intelligent customer service system via API for subsequent processing. The text data can also include the user's query, such as "I want to view my recent transaction records" or "What should I do if I forget my password?", which is transmitted to the system through the online chat interface.

[0044] In voice customer service scenarios, when a customer calls a bank's customer service hotline to confirm a transaction, the system acquires the customer's voice data through a call recording interface. This voice data can include the customer's voice, answers to questions (such as identity verification questions), and the customer's voice confirmation (such as "I confirm this transaction"). The voice data is captured by a real-time call recording system and transmitted to the backend server as input for subsequent speech synthesis and analysis.

[0045] By acquiring text and speech data in real time or in batches, the speech synthesis system can ensure that it can obtain the necessary data for processing in a timely and accurate manner, thereby improving the system's response speed and data processing capabilities.

[0046] S20, the text data is input into a text encoder, and the text data is converted into a text symbol sequence by the text encoder;

[0047] In this embodiment, natural language text data is converted into a sequence of symbols that a computer can understand and process. A text encoder is a core component that transforms human language into a machine-understandable format. It uses techniques such as word segmentation and encoding to convert text into a lower-level symbolic representation for subsequent processing.

[0048] Text data (such as customer queries and transaction information) is passed to the text encoder through the input interface. Text data is typically a string in natural language form, including words, phrases, or complete sentences.

[0049] A text symbol sequence is a sequence of symbols transformed from natural language text. Symbols can be the smallest units of word, subword, or byte pair encoding (BPE). The generation of symbol sequences ensures that the text can be processed by subsequent models (such as alignment and speech synthesis). Symbol sequences can be discrete sequences of words, or representations at the subword or character level.

[0050] Before being input into a text encoder, text data may need to undergo cleaning and normalization, such as removing invalid characters and symbols. For financial data, there may be specialized processing procedures, such as identifying special terms or financial transaction terms. The first step of a text encoder is tokenization, breaking sentences or phrases down into basic units. This can be achieved using a tokenizer; common tokenization methods include word-based tokenization, sub-word tokenization, or character-level tokenization. The tokenized basic units are then converted into symbol sequences through an encoding process. This step often uses byte-pair encoding (BPE) or other encoding methods to convert words into symbolic forms. The symbol sequences can then be used for subsequent calculations. Encoded text is converted into symbol sequences; for example, the sentence "customer requests transaction record" is converted into a symbol sequence such as [customer, request, transaction, record], or encoded as [101, 204, 309, 412], for subsequent processing.

[0051] By encoding text data into symbol sequences, natural language data can be effectively processed and adapted to the needs of machine learning models. This allows for the rapid and accurate conversion of customers' natural language input (such as customer questions and transaction requests), providing efficient symbol sequence input for subsequent speech synthesis or intelligent processing.

[0052] S30, the speech data is input into a speech encoder trained based on a self-supervised learning model, the speech encoder extracts speech features from the speech data, and the speech features are quantized into discrete speech symbol sequences;

[0053] In this embodiment, voice data is input into the system via an audio interface. The voice data may include the customer's voice or recorded phone calls, and this data needs to be processed to extract relevant features. In the financial sector, voice data typically originates from customer voice confirmations, telephone customer service, and other similar scenarios.

[0054] Self-supervised learning is a method of learning features from unlabeled data. Speech encoders are trained using self-supervised learning models, meaning that the encoder can extract useful speech features from unlabeled data without a large amount of labeled speech data. In the financial sector, customer speech data may lack explicit labels, and self-supervised learning can automatically learn useful speech representations from this data.

[0055] A speech encoder trained using a self-supervised learning model can efficiently extract features from input speech data. It can not only extract key features of speech (such as pitch, frequency, and speech patterns), but also identify details in pronunciation.

[0056] Speech feature extraction involves converting the input speech signal into an understandable feature representation, such as pitch and frequency. Speech encoders extract key feature parameters by analyzing the spectrum and time-domain characteristics of the speech signal.

[0057] After feature extraction, the speech encoder quantizes these features into a discrete sequence of symbols. Quantization is the process of converting a continuous speech signal into a discrete symbolic representation, allowing for further processing. For example, a continuous speech signal might be quantized into a set of discrete symbols (such as [S1, S2, S3]), which can represent different speech segments.

[0058] During training, the speech encoder can be trained using self-supervised learning. The model can learn pronunciation patterns and speech features using a large amount of unlabeled speech data, thereby improving its ability to recognize new speech data. Extracted speech features are captured using techniques such as spectral analysis and short-time Fourier transform. The quantization process uses a codebook to map speech features to discrete symbols. For example, speech data from different frequency bands can be mapped to specific discrete symbols.

[0059] Example Explanation: In financial telephone transaction confirmation, the customer's voice input (such as "I confirm this transfer") is recorded in real time and transmitted to a voice encoder. The voice encoder processes this voice data using a self-supervised learning model, extracting the customer's vocal features and quantifying these features into discrete sequences of speech symbols. These symbol sequences allow the system to further match and analyze the customer's voice input to confirm whether the customer has authorized the transaction. In this scenario, the voice encoder can extract key features from the customer's voice without requiring a large amount of training data. This significantly improves the accuracy of speech recognition, especially in financial transactions, ensuring that every customer confirmation is processed and responded to correctly.

[0060] By using a speech encoder trained based on a self-supervised learning model, speech features can be effectively extracted, improving the processing capability of speech data even without a large amount of labeled data. After quantization into discrete speech symbol sequences, the system can quickly process speech signals, achieving more accurate speech generation and analysis, and improving the system's response speed and recognition ability.

[0061] S40, organize the text symbol sequence into a text sequence, and organize the speech symbol sequence into a speech sequence;

[0062] In this embodiment, the text symbol sequence is a discrete sequence of symbols generated during the preceding text encoding process. Text symbols can be words, subwords, or character-level symbols. Organizing the text symbol sequence into a text sequence involves arranging these symbols according to the grammatical rules of natural language to form a structured text representation. This step ensures that the text symbols can be processed according to the correct language structure for subsequent matching with speech.

[0063] A speech symbol sequence is a sequence of symbols obtained by extracting and quantizing speech features from a speech encoder. Speech symbols represent segments of a speech signal; by organizing a speech symbol sequence, these segments can be arranged according to the temporal order of the speech, forming an ordered speech sequence. This process ensures the continuity of the speech signal and avoids misalignment or information loss in subsequent processing.

[0064] The purpose of organizing text and speech symbols is to ensure their correspondence in subsequent processing. Text sequences and speech sequences need to be arranged in the same logical order to enable subsequent alignment. Text sequences represent corresponding words in the speech, while speech sequences represent specific speech signals.

[0065] In organizing a sequence of text symbols, the system first needs to perform grammatical analysis. By parsing the grammatical roles of the text symbols (such as subject, predicate, object, etc.), the text sequence can be arranged in a semantically logical order. For example, the subject and predicate of a sentence are determined first, and then the word order is adjusted according to language rules.

[0066] Arranging speech symbols in chronological order is achieved by analyzing the frame information of the speech signal. After the speech symbol sequence is generated, the system uses the timestamp of each speech symbol to ensure that these symbols are arranged in the natural order of speech, thus avoiding misalignment of audio segments.

[0067] By organizing text symbol sequences and speech symbol sequences into ordered text sequences and speech sequences respectively, the system can ensure consistency and matching between the two in subsequent processing. The grammatical structure of the text sequence and the temporal order of the speech sequence can correspond better, making the speech synthesis and alignment process smoother and more accurate.

[0068] S50, perform preliminary alignment processing on the text sequence and the speech sequence to establish the correspondence between text symbols and speech symbols;

[0069] In this embodiment, the text sequence and speech sequence are encoded and organized structured symbolic representations. The initial alignment process establishes a correspondence between these text symbols and speech symbols, ensuring that the system can accurately match the content between text and speech in subsequent synthesis or recognition processes. The alignment process includes two levels: temporal alignment and content alignment.

[0070] Time alignment refers to matching the time segment of each symbol in a text sequence with the corresponding symbol in a speech sequence. Since speech symbols are arranged in chronological order, the system analyzes the corresponding durations of text symbols and speech symbols to ensure that the time segment of each text symbol corresponds to the time segment of the corresponding speech symbol.

[0071] Content alignment refers to matching the semantics or meaning of text symbols with those of speech symbols. For example, the text symbol "query" needs to correspond to the pronunciation of "query" in the speech signal. The system ensures that each text symbol has a consistent meaning with its corresponding speech symbol by matching the similarity or semantic correspondence between text and speech symbols.

[0072] Initial alignment is typically achieved through Dynamic Time Warping (DTW) or other alignment algorithms. The DTW algorithm dynamically adjusts the correspondence between text and speech sequences by calculating the temporal differences between them, ensuring that the text symbols and speech symbols are time-consistent. For example, the word "transaction confirmation" in speech might be pronounced faster or slower; DTW dynamically adjusts the time matching between the text and speech.

[0073] In addition to time alignment, the system can also generate preliminary symbol mapping relationships based on the similarity between text symbols and phonetic symbols. By analyzing the semantic or phonemic features between symbols, the system matches text symbols with the most similar phonetic symbols, establishing a preliminary correspondence.

[0074] By performing preliminary alignment processing on the text and speech sequences, the system can effectively establish the correspondence between text symbols and speech symbols. This process ensures the matching accuracy between text and speech, which helps reduce errors and improve the accuracy and consistency of the synthesis results in subsequent speech synthesis and recognition processes.

[0075] S60 performs gated attention processing on the initially aligned text and speech sequences to adjust the matching degree between text symbols and speech symbols;

[0076] In this embodiment, the gated attention mechanism is a dynamic control mechanism used to focus on key information during model processing. In this step, the pre-aligned text and speech sequences are input into the gated attention module, which determines the degree of matching between text and speech symbols by adjusting attention weights.

[0077] The gating attention mechanism automatically assigns attention weights between each pair of text and speech symbols by calculating the feature similarity between them. Based on these attention weights, the system determines which symbols match better and which require further adjustment. This dynamic adjustment improves the alignment between text and speech, ensuring the accuracy of speech synthesis.

[0078] Matching relationships refer to the semantic and phonetic correspondence between text symbols and speech symbols. Through a gating attention mechanism, the system can detect temporal or semantic discrepancies between text and speech symbols and dynamically adjust the matching between them. The gating mechanism helps the system prioritize symbols highly relevant to text or speech features, ignoring less important symbols, thereby optimizing the final synthesized speech result.

[0079] The gating attention module calculates the attention weights between text and speech symbols, determining the relevance between each pair of symbols using the feature vectors of the input text and speech sequences. The gating unit controls the attention allocation between each symbol, ensuring the system can better capture key text-speech alignment relationships.

[0080] After calculating the attention weights, the system updates the dependency relationship between text symbols and speech symbols based on the weights. High-weighted symbol pairs indicate a high degree of matching, while low-weighted symbols may indicate poor matching that requires further adjustment.

[0081] Example Explanation: In a financial intelligent speech synthesis system, a customer inputs "Query the recent transaction records in my account" via voice. At this point, the system has completed the initial alignment of text and speech symbols. Next, a gating attention mechanism is used to dynamically adjust the degree of matching between text and speech symbols.

[0082] The system uses a gating attention mechanism to identify key audio symbols such as "query," "account," and "recent transactions" based on the customer's voice, and then matches these audio symbols with text symbols. By adjusting the attention weights, the system prioritizes key information in the customer's voice and ignores irrelevant background noise or pauses.

[0083] If a customer's pronunciation is unclear or has an accent, the system can make further dynamic adjustments through a gating attention mechanism. For example, if a customer's inaccurate pronunciation causes "account" and "accounts" to sound similar, the system will prioritize the correct matching symbol "account" based on semantic information, ensuring a high-precision match between speech and text symbols.

[0084] By dynamically adjusting the gating attention mechanism, the system can automatically optimize the matching relationship between text symbols and speech symbols, thereby improving the quality of speech synthesis. The gating attention mechanism not only enhances the flexibility of alignment but also reduces matching errors between text and speech, improving the accuracy and naturalness of generated speech. It helps reduce repetition, misalignment, or loss issues in speech synthesis, ensuring accurate speech feedback.

[0085] The S70 inputs the gated attention-processed text sequence and speech sequence into the decoder to generate the final speech signal.

[0086] In this embodiment, after processing by the gating attention mechanism, the matching relationship between the text sequence and the speech sequence has been adjusted and optimized. At this point, a one-to-one association has been formed between text symbols and speech symbols. Next, the system inputs these processed text and speech symbols into the decoder. The decoder's role is to convert these symbol sequences into the final speech signal.

[0087] Decoders typically process the input sequence frame by frame. This means that it decodes each pair of symbols in the text and speech sequences frame by frame to generate a fluent speech signal. Each frame represents a small segment of speech, and frame-by-frame decoding ensures the coherence and naturalness of the speech synthesis process.

[0088] During the decoding process, the system generates corresponding speech segments based on the characteristics of text symbols and their corresponding speech symbols, and gradually splices these segments into a complete speech signal. The decoder matches the content, intonation, and speed of the text and speech to generate a coherent speech output.

[0089] The final speech signal is generated by the decoder; it is a synthesized speech file or stream that can be played or stored. The system uses the decoder to convert all gated symbol sequences into natural speech, and the output speech corresponds to the input text content, ensuring the accuracy and fluency of speech synthesis.

[0090] The decoder generates the speech signal by processing it frame by frame. Each frame consists of a pair of text symbols and speech symbols. The decoder analyzes the time-series features of the text symbols and speech symbols to gradually generate the corresponding speech frames, and finally splices multiple frames into a complete speech signal.

[0091] Speech reconstruction is the process of generating natural speech signals by analyzing the features of input speech symbols (such as pitch, tone, and speed). The decoder needs to ensure the naturalness and coherence of the speech signal during reconstruction. For example, in financial services, the speech rate and intonation of the synthesized speech need to match the customer's voice to ensure the realism of the synthesized speech.

[0092] Example Explanation: In a financial speech synthesis system, a customer's transaction confirmation instruction, such as "confirm transfer of 5000 yuan," requires the system to generate a voice response. After a gating attention mechanism adjusts the matching between text symbols and speech symbols, the system inputs the symbol sequence into the decoder for speech generation.

[0093] The decoder processes the input text and speech symbol sequences frame by frame. For the input "Confirm transfer of 5000 yuan," the system generates a speech signal based on the matching relationship between symbols, ensuring the clarity and appropriate speed of the speech feedback. The decoder generates the corresponding speech signal based on the characteristics of the input symbols. For example, during the transaction confirmation process, the speech synthesis system generates the speech feedback "Transfer of 5000 yuan confirmed," ensuring that the customer hears an accurate and natural speech prompt. Through frame-by-frame processing and speech reconstruction by the decoder, the system can generate high-quality speech feedback, ensuring a more natural and smooth interaction experience between the customer and the financial system.

[0094] By inputting gated attention-processed text and speech symbol sequences into the decoder, the system can generate coherent and natural speech signals. The decoder generates the speech signal based on the correspondence between text symbols and speech symbols, ensuring that the output speech is consistent with the input text content. This effectively improves the fluency and accuracy of speech synthesis.

[0095] This invention relates to the fields of artificial intelligence and fintech, and discloses a speech synthesis method based on a gated attention mechanism. The method involves acquiring text and speech data. The text data is converted into a sequence of text symbols using a text encoder, while the speech data is processed by a speech encoder trained on a self-supervised learning model to extract speech features, which are then quantized into discrete sequences of speech symbols. These sequences are then organized into text and speech sequences, respectively. A preliminary alignment process establishes the correspondence between text and speech symbols. The gated attention mechanism dynamically adjusts the matching degree between the text and speech symbols. Finally, a decoder generates the final speech signal. This invention effectively improves the speech feature extraction capability through a speech encoder trained on a self-supervised learning model, especially in scenarios lacking a large amount of labeled data, where it can still learn effective feature representations from unlabeled data. Combined with the gated attention mechanism, it can adjust the matching relationship between text and speech symbols, ensuring the accuracy and fluency of the synthesized speech.

[0096] In one embodiment, S20 includes:

[0097] S201, Perform a preprocessing operation on the text data, the preprocessing operation including format standardization and text standardization;

[0098] S202, the preprocessed text data is input into the text encoder, and the text data is decomposed into individual words or symbols by the word segmenter in the text encoder;

[0099] S203, the decomposed words or symbols are converted into a predefined sequence of text units by the byte-pair encoding module in the text encoder;

[0100] S204, map the text unit sequence to a predefined symbol table to generate a text symbol sequence;

[0101] S205, the text symbol sequence is indexed and stored in memory.

[0102] In this embodiment, format standardization is the process of converting text data into a standard format that the system can process. For example, text data may come from different data sources and may have problems such as inconsistent encoding formats and inconsistent capitalization. Through format standardization, it can be ensured that all text data is processed in a unified encoding format.

[0103] Text standardization refers to removing unnecessary characters such as extra spaces and punctuation marks from text, enabling the text content to more accurately express its actual meaning. For example, in a financial context, user inputs such as "check balance" and "check balance" can be standardized to a consistent form, ensuring the system can recognize them as the same request.

[0104] A word segmenter breaks down preprocessed text data into individual words or symbols. In natural language processing, word segmentation splits a sentence into individual words, subwords, or symbols to facilitate subsequent encoding operations. The accuracy of word segmentation directly affects the quality of subsequent text encoding and symbol mapping.

[0105] The Byte-Pair Encoding (BPE) module in the text encoder converts segmented words or symbols into a predefined sequence of text units. BPE is a common word segmentation method that further breaks down words into smaller units to improve text compression efficiency and processing speed. This step ensures that the resulting unit sequence contains richer textual information.

[0106] The generated sequence of text units needs to be mapped to a predefined symbol table, which is a mapping dictionary used to convert each text unit into a corresponding symbol number. This allows the text to be represented as a sequence of symbols, facilitating subsequent processing.

[0107] Indexing involves numbering a sequence of text symbols, assigning a unique index number to each symbol. The indexed symbol sequence is then stored in memory for use in subsequent steps. Indexing significantly improves the efficiency of querying and processing symbol sequences.

[0108] Formatting standardization can be achieved through the character encoding libraries built into programming languages ​​(such as Python's UTF-8). Text standardization can use regular expressions to remove unnecessary spaces, punctuation marks, etc., or it can use natural language processing libraries (such as NLTK) to clean up the text content.

[0109] Tokenizers can use rule-based segmentation algorithms or machine learning models for word segmentation. In the financial field, commonly used segmentation tools include Jieba (Chinese) and SpaCy (English). Byte-pair encoding algorithms can generate compact symbol units by progressively merging common character combinations in text. Byte-pair encoding (BPE) is widely used in machine translation, text generation, and other fields.

[0110] Symbol tables can be implemented using HashMap or dictionary data structures to quickly look up the symbol number corresponding to each text unit. In financial data, some specialized financial terms can be predefined in the symbol table to ensure that financial-specific vocabulary is handled accurately.

[0111] Indexing is achieved by mapping symbols to numerical identifiers, the order of which determines the symbol's position in the text. The indexed symbol sequence can be stored using an in-memory database (such as Redis) or a persistent database (such as MySQL) for subsequent processing and querying.

[0112] Example: In an intelligent customer service system, a customer might enter "Check my account balance for last month" as text. The system needs to input this text data into a text encoder for processing.

[0113] The system first performs format standardization, converting all input text to a uniform UTF-8 encoding format to avoid processing errors caused by inconsistencies in data formats from different sources. Then, it performs text standardization, removing unnecessary spaces and punctuation marks to ensure the conciseness and consistency of the input text.

[0114] The system uses a word segmenter to break down "query last month's account balance" into words such as "query," "last month," and "account balance." The BPE module further decomposes these words into text unit sequences. These generated text unit sequences are mapped to the system's symbol table and converted into symbol numbers; for example, "query" is mapped to 101, and "account balance" is mapped to 202. The generated symbol sequences are indexed and stored in memory for use by the subsequent speech synthesis module.

[0115] This embodiment, through preprocessing, word segmentation, encoding, and indexing of text data, enables the system to efficiently convert natural language text into symbol sequences, ensuring that the processed symbol sequences can provide accurate input for subsequent text-to-speech alignment and speech generation.

[0116] In one embodiment, S30 includes:

[0117] S301, perform preprocessing operations on the speech data, the preprocessing operations including background noise removal and signal normalization;

[0118] S302, input the preprocessed speech data into a speech encoder trained based on a self-supervised learning model;

[0119] S303, the speech encoder decomposes the continuous speech signal in the speech data into time frames of fixed length;

[0120] S304, extract features from the speech data in each time frame, extracting key speech features including Mel frequency cepstral coefficients, fundamental frequency and energy;

[0121] S305, the key speech features are quantized and converted into discrete speech symbol sequences.

[0122] In this embodiment, the speech data typically contains background noise, such as ambient sounds and echoes. Before the speech data is input into the encoder, the system needs to process the audio data using filters or noise suppression techniques to remove these interferences. Preprocessing ensures clearer speech data and reduces interference with feature extraction.

[0123] Normalization standardizes the amplitude of an audio signal to ensure consistent volume across different speech data. Through signal normalization, the system can avoid the impact of volume differences on the extraction and processing of speech features.

[0124] The preprocessed speech data is fed into a speech encoder trained using a self-supervised learning model. The self-supervised learning model can automatically learn features from data even without a large amount of labeled data. The speech encoder uses self-supervised learning techniques to extract useful speech features from unlabeled speech data and can adaptively process various types of speech signals.

[0125] The system decomposes continuous speech signals into fixed-length time frames, each representing a segment of the speech signal. This process is called framing. By decomposing the speech signal, the system can process the speech data frame by frame, extracting feature information from each time segment. Each frame is typically 10ms to 30ms in length.

[0126] For each time frame, the speech encoder extracts key features from the speech signal. Commonly used feature extraction techniques include Mel-frequency cepstral coefficients (MFCC), pitch, and energy. These features represent the frequency distribution, pitch, and energy variations of the speech signal and are important parameters for speech recognition and speech synthesis.

[0127] The extracted speech features need to be quantized, converting continuous feature values ​​into discrete symbol sequences. The quantization process maps speech features to a discrete set of symbols, enabling further processing and storage. The quantized discrete symbol sequences simplify subsequent calculations, improving processing speed and efficiency.

[0128] Noise removal can be achieved using digital filters (such as low-pass filters) or noise suppression techniques based on spectral subtraction to remove background noise from speech signals. Volume normalization techniques standardize the amplitude range of all speech data. For example, by adjusting the signal to a uniform amplitude range, the volume is made consistent. Self-supervised learning speech encoders can automatically extract useful speech features by learning from unlabeled data. Common implementations include using models such as Wav2Vec to learn speech representations from massive amounts of raw speech data without manually labeled data.

[0129] Feature extraction techniques such as MFCC, fundamental frequency, and energy are used for the audio data of each time frame. Short-Time Fourier Transform (STFT) is used to analyze the spectral characteristics of the speech signal and extract speech features such as Mel-frequency cepstral coefficients. Quantization processing uses methods such as vector quantization to map the feature values ​​to discrete symbols.

[0130] This embodiment utilizes a speech encoder trained with a self-supervised learning model. The system can automatically learn speech features and process them effectively even without labeled speech data. Combined with preprocessing techniques (such as noise removal and signal normalization), the system can generate high-quality speech symbol sequences, ensuring a more accurate and natural speech synthesis process. In the financial sector, particularly in telephone customer service or voice confirmation, the speech encoder can improve the accuracy of speech recognition and enhance the customer's voice interaction experience.

[0131] In one embodiment, S40 includes:

[0132] S401, The text symbol sequence is analyzed by a syntax analyzer to identify and label the grammatical function of each text symbol;

[0133] S402, based on the grammatical function and contextual relationship of each text symbol, the text symbols are arranged sequentially to generate an ordered text sequence that conforms to semantic logic. The contextual relationship refers to the coordination relationship between the text symbol and its adjacent symbols in terms of grammatical structure, semantic association and word order.

[0134] S403, the speech symbol sequence is arranged in chronological order to generate an ordered speech sequence.

[0135] In this embodiment, the system first uses a parser to analyze the sequence of text symbols. The parser identifies and annotates the grammatical function of each text symbol. For example, the system identifies which symbols are subjects, predicates, objects, etc., and generates structured annotation information based on the grammatical role of each symbol in the sentence. The purpose of this step is to ensure that the text symbols are organized according to the correct grammatical structure, facilitating subsequent processing.

[0136] After identifying the grammatical function of each text symbol, the system arranges the text symbols sequentially based on these grammatical functions and contextual relationships. Contextual relationships refer to the grammatical structure, semantic connections, and word order coordination between text symbols and adjacent symbols. Through this arrangement, the system can generate an ordered text sequence that conforms to semantic logic, ensuring that the text symbols can coherently and correctly express information. The analysis of contextual relationships guarantees the semantic accuracy of the text sequence.

[0137] The speech symbol sequence is generated based on the previously extracted speech features. To ensure the coherence of the speech synthesis, the system needs to arrange these speech symbols in chronological order. Each speech symbol corresponds to a specific time frame of the speech signal. By arranging them in chronological order, the system can generate an ordered speech sequence, ensuring that the speech signal can correctly represent the text content.

[0138] Parser typically employs dependency parsing-based natural language processing techniques to identify grammatical components (such as subject, predicate, and object) in a sentence. This type of parser can perform structured analysis on the input text symbols using a pre-trained grammatical model and annotate their grammatical functions.

[0139] The system determines the order of text symbols by analyzing the semantic relationships and grammatical structure between each text symbol and its adjacent symbols. For example, in the sentence "Query account balance", the system recognizes "Query" as a verb and "account balance" as an object, and arranges them in the correct word order to generate an ordered text sequence.

[0140] Each speech symbol corresponds to a specific time frame, and the system arranges the speech symbols sequentially based on the time information. For example, if a speech symbol corresponds to an audio segment during pronunciation, the system ensures that these segments are arranged according to the actual pronunciation order to ensure the continuity of the speech.

[0141] Example Explanation: In an intelligent customer service system, customers may input different forms of query requests via voice or text, such as "check account balance" or "account balance inquiry". The system needs to process these inputs in different sequences to ensure accurate voice responses. For example:

[0142] Customer A enters: "Check account balance".

[0143] Customer B enters: "Account balance inquiry".

[0144] Despite the different order of the two customers' input, the system, through its parser, recognized "query" as a verb and "account balance" as a noun, and reordered the text symbols based on grammatical functions. The system then parsed "query account balance" into a correct request according to the subject-verb-object structure.

[0145] For each customer's voice input, the system extracts speech segments from the sequence of speech symbols and arranges them according to the actual pronunciation order. For example, if a customer's voice input is "account balance inquiry," the system can still arrange the speech symbols reasonably based on their timing information to ensure natural speech feedback during speech synthesis.

[0146] This embodiment organizes text symbols and speech symbols into ordered text sequences and speech sequences respectively, ensuring the matching of text and speech. This improves the accuracy and fluency of speech synthesis, especially in the financial field where precise information needs to be conveyed, effectively avoiding misalignment between text and speech.

[0147] In one embodiment, the above S50 includes:

[0148] S501, Analyze the time length of each text symbol in the text sequence;

[0149] S502, analyze the time frame of each speech symbol in the speech sequence;

[0150] S503 uses a dynamic time warping module to time-align the time length of text symbols with the time frame of speech symbols.

[0151] S504, based on time alignment results, analyzes the similarity between text symbols and speech symbols;

[0152] S505, Generate a preliminary symbol mapping relationship based on the similarity and establish a correspondence table between text symbols and speech symbols.

[0153] In this embodiment, the time length corresponding to each text symbol (such as a word or character) can be estimated based on the text symbol's position and length within the sentence. For example, the text symbol "query" might be longer or shorter than "balance". The system calculates the time it might occupy in speech synthesis by analyzing the length of each text symbol.

[0154] Each speech symbol in a speech sequence corresponds to a time segment in the speech signal, called a time frame. Each time frame may represent a sound segment in the speech (such as the duration of the pronunciation of a vowel or consonant). The system analyzes the time frame of each speech symbol to correspond it with the time length of the text symbol.

[0155] Dynamic Time Warping (DTW) is a commonly used time series alignment technique that can handle the time differences between text symbols and speech symbols. DTW calculates the optimal time alignment path by comparing the time length of the text symbol and the time frame of the speech symbol. For example, if a text symbol is pronounced quickly, DTW will adjust the time to ensure that the alignment between the text symbol and the speech symbol remains consistent.

[0156] Once the time alignment of text symbols and speech symbols is complete, the system calculates their similarity. This similarity can be based on the phonemes of the text symbols and the pronunciation features of the speech symbols. For example, the system evaluates how well the pronunciation of a symbol in the text matches its corresponding speech symbol, ensuring accuracy in speech synthesis.

[0157] Finally, based on time alignment and similarity analysis between text and speech symbols, the system generates a preliminary symbol mapping relationship. This mapping relationship shows which speech symbol corresponds to each text symbol. The system records these mapping relationships in a correspondence table for use in subsequent speech synthesis and further adjustments.

[0158] The duration of text symbols can be estimated based on the symbol's length or the distribution of symbols within a sentence. Using natural language processing (NLP) techniques, the system analyzes the position and function of symbols within the text, thus reasonably estimating their duration. The system decomposes the speech signal into several time frames. The time frame corresponding to each speech symbol can be obtained by analyzing the frequency and amplitude of the speech signal, ensuring that the pronunciation process of each symbol is correctly mapped to the text symbol. The DTW module dynamically adjusts the matching path between text and speech symbols based on the differences in duration and time frames. The system finds the optimal alignment path through recursive calculation, minimizing the time difference between text and speech symbols. Using the phoneme features of text and speech symbols (such as the phonemes and frequencies of pronunciation), the system calculates their similarity. Symbol pairs with higher similarity indicate a higher degree of matching, ensuring accuracy in the speech synthesis process. A symbol mapping table records the mapping pairs between text and speech symbols for subsequent speech synthesis or processing steps.

[0159] This embodiment effectively solves the time mismatch problem in speech synthesis by aligning the time of text and speech symbols, ensuring a high degree of correspondence between text and speech. The use of a dynamic time warping module and similarity analysis further improves the accuracy and naturalness of speech synthesis.

[0160] In one embodiment, S60 includes:

[0161] S601, extract features from the text sequence and the speech sequence respectively, and generate high-dimensional feature vectors corresponding to the text symbols and speech symbols;

[0162] S602 inputs the high-dimensional feature vectors of text symbols and speech symbols into the gated attention mechanism module;

[0163] S603, the gating attention mechanism module adjusts the attention weights based on the feature differences between the high-dimensional feature vectors of text symbols and speech symbols, and generates an association matrix between text symbols and speech symbols based on the attention weights;

[0164] S604, the gated attention mechanism module adjusts the matching relationship between text symbols and corresponding speech symbols based on the association matrix, and updates the dependency relationship between text symbols and speech symbols based on the adjusted matching relationship;

[0165] S605, the gating attention mechanism module generates the final updated matching result of text symbols and speech symbols based on the dependency relationship between the updated text symbols and speech symbols.

[0166] In this embodiment, each symbol in the text sequence and speech sequence can be represented as multiple feature dimensions. The feature extraction process extracts important information such as syntax, semantics, phonemes, and intonation from the text and speech data, and combines these features into a high-dimensional vector. The high-dimensional feature vector is a mathematical representation used to describe the complex attributes of each symbol, containing multi-dimensional information about the symbol. This step ensures that the information between text symbols and speech symbols is input into subsequent modules in the form of feature vectors of the same dimension.

[0167] After extraction, the high-dimensional feature vectors corresponding to the text symbols and speech symbols are input into the gating attention mechanism module. The gating attention mechanism can dynamically adjust the model's attention to each symbol, ensuring that the system focuses on the most important symbols in the speech synthesis process and reduces the influence of irrelevant symbols.

[0168] The gating attention mechanism calculates the attention weight between each pair of symbols by comparing the feature differences between the high-dimensional feature vectors of text symbols and speech symbols. The smaller the feature difference, the higher the degree of matching between the text symbol and its corresponding speech symbol, and the greater the attention weight. By adjusting the attention weight, the system can dynamically control the model's attention to different symbols, ensuring that the focus is concentrated on those symbols that are crucial to speech synthesis.

[0169] An association matrix is ​​a matrix that describes the relationship between text symbols and speech symbols, with values ​​representing the degree of matching between each pair of symbols. By generating the association matrix through a gated attention mechanism, the system can intuitively represent the strength of the association between text symbols and their corresponding speech symbols. Based on the values ​​in the association matrix, the system can adjust the matching relationships between symbols to ensure that high-weighted symbol pairs match more closely.

[0170] After adjusting the matching relationships, the system needs to update the dependencies between text symbols and speech symbols. Dependencies describe the influence of each symbol on other symbols; contextual dependencies between symbols are particularly important in natural language processing and speech synthesis. By updating dependencies, the system can ensure the temporal and semantic consistency of text and speech symbols.

[0171] Finally, based on the adjusted matching relationships, the system generates the final text-to-speech symbol matching results. These results will be used in subsequent speech synthesis steps to ensure that the text content can be accurately converted into natural and coherent speech signals.

[0172] Deep learning models, such as Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs), are used to extract features from text and speech data. The feature extraction model considers the contextual information of symbols, generating high-dimensional vectors containing syntactic, semantic, phonemic, and pronunciation features. A gated attention mechanism based on the Transformer architecture is used to generate attention weights by calculating the dot product of the high-dimensional feature vectors, and the matching priority of symbols is adjusted according to the magnitude of the weights. The attention mechanism of the Transformer model is particularly well-suited for handling long-distance dependencies between symbols. By calculating the attention weights for each pair of symbols, an association matrix is ​​generated, ensuring that the system can visualize the matching between each symbol. The values ​​of the association matrix reflect the temporal and semantic matching degree between text symbols and speech symbols. By modeling the dependencies between symbols, the system ensures the transmission of contextual information between symbols. The updated dependencies help the system preserve word order and semantic associations in natural language during speech generation.

[0173] This embodiment utilizes a gated attention mechanism for dynamic adjustment, enabling the system to precisely control the matching degree between text and speech symbols, ensuring a high degree of semantic and phonological consistency in the generated speech synthesis results. This mechanism not only improves the accuracy of speech synthesis but also enhances the system's ability to handle complex contexts and symbolic dependencies, making it particularly suitable for scenarios such as financial data report generation and voice queries, providing accurate voice feedback.

[0174] In one embodiment, S70 includes:

[0175] S701, input the text sequence and speech sequence into the decoder;

[0176] S702, the decoder processes the text sequence and speech sequence frame by frame and generates a speech signal corresponding to each text symbol;

[0177] During the decoding process, S703 adjusts the generated speech signal based on the dependency relationship between text symbols and speech symbols to obtain the final speech signal.

[0178] In this embodiment, the consistency of the text sequence and the speech sequence in the temporal dimension is ensured. Through preprocessing, the system can synchronize the timing of text symbols and speech symbols, avoiding time misalignment during speech generation. Based on timing alignment, the semantic correspondence between text symbols and speech symbols is further established to ensure that the generated speech signal accurately reflects the meaning of the text content.

[0179] Time synchronization algorithms, such as Dynamic Time Warping (DTW), are used to align the timelines of text and speech sequences. By analyzing the temporal characteristics of text and speech symbols, the sequence length and alignment path are adjusted to ensure temporal consistency. Natural Language Processing (NLP) techniques are combined with contextual semantic analysis tools to identify the semantic relationships between text and speech symbols. Semantic embedding-based methods can be employed to map text and speech symbols to the same semantic space, ensuring accurate semantic correspondence.

[0180] The decoder receives input sequences from two different modalities: text and speech. It fuses these two information sources to generate a natural, coherent speech signal. The decoder's input interface is designed to simultaneously receive data streams of both text and speech sequences. A dual-stream input structure can be used, with text and speech sequences input to the decoder's fusion module through separate channels. Fusion algorithms, such as weighted averaging, concatenation, or attention mechanisms, are used to effectively integrate the information from the text and speech sequences as input features for the decoder. The decoder processes the text and speech sequences frame by frame, generating a speech signal corresponding to each text symbol.

[0181] The decoder processes the input sequence frame by frame to ensure the coherence and naturalness of the speech signal. Each frame is generated based on the input of the current frame and the context information of the previous frames. This ensures that each text symbol corresponds to a specific speech signal segment, achieving accurate text-to-speech conversion.

[0182] Decoders employing architectures based on recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or Transformers are capable of processing sequential data. When generating each frame of speech signal, the decoder progressively constructs a complete speech signal using the generated result of the previous frame and the input symbols of the current frame. A symbol-to-speech signal mapping model, such as vocoders, is used to convert the symbol sequence into a specific audio waveform. During the decoding process, the generated speech signal is adjusted based on the dependency between text symbols and speech symbols to obtain the final speech signal.

[0183] By leveraging the dependencies between text and speech symbols, the speech signal generation process is dynamically adjusted to ensure a high degree of match between the intonation, rhythm, and pronunciation of the speech and the text content. Adjusting these dependencies improves the naturalness and clarity of the generated speech, reducing errors and incoherence in speech synthesis.

[0184] A dependency model is embedded in the decoder, utilizing attention mechanisms or conditional generation models to capture the complex dependencies between text and speech symbols. Based on the output of the dependency model, the decoder's generation parameters, such as intonation variations, pause positions, and volume control, are adjusted in real time to optimize the quality of the speech signal. Speech enhancement and noise suppression techniques are applied to further improve the clarity and naturalness of the generated speech.

[0185] This embodiment achieves high-precision alignment and matching between text and speech sequences through gated attention processing. Utilizing the gated attention mechanism, the system can dynamically adjust the matching degree between text and speech symbols, ensuring that the generated speech signal is highly consistent with the input text in terms of intonation, rhythm, and pronunciation. This not only improves the accuracy and naturalness of speech synthesis but also enhances the system's adaptability in complex financial scenarios, such as handling diverse customer inquiries and transaction confirmation needs, significantly improving the user experience and service quality of the intelligent customer service system.

[0186] In one embodiment, a speech synthesis device based on a gated attention mechanism is provided, which corresponds one-to-one with the speech synthesis method based on a gated attention mechanism described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the speech synthesis device based on a gating attention mechanism of the present invention. The modules include a data acquisition module 10, a text encoding module 20, a speech encoding module 30, a sequence organization module 40, an alignment module 50, a gating attention mechanism module 60, and a decoder module 70. Detailed descriptions of each functional module are as follows:

[0187] Data acquisition module 10 is used to acquire text data and voice data;

[0188] The text encoding module 20 is used to input the text data into the text encoder and convert the text data into a text symbol sequence through the text encoder;

[0189] The speech encoding module 30 is used to input the speech data into the speech encoder, extract speech features from the speech data through the speech encoder, and quantize the speech features into a discrete sequence of speech symbols.

[0190] The sequence organization module 40 is used to organize the text symbol sequence into a text sequence and the speech symbol sequence into a speech sequence;

[0191] The preliminary alignment module 50 is used to perform preliminary alignment processing on the text sequence and the speech sequence to establish the correspondence between text symbols and speech symbols;

[0192] The gated attention mechanism module 60 is used to perform gated attention processing on the initially aligned text sequence and speech sequence to adjust the degree of matching between text symbols and speech symbols;

[0193] Decoder module 70 is used to input the gated attention-processed text sequence and speech sequence into the decoder to generate the final speech signal.

[0194] In one embodiment, the text encoding module 20 is specifically used for:

[0195] The text data is preprocessed, including format normalization and text normalization.

[0196] The preprocessed text data is input into the text encoder, and the text data is decomposed into individual words or symbols by the word segmenter in the text encoder.

[0197] The byte-pair encoding module in the text encoder converts the decomposed words or symbols into a predefined sequence of text units.

[0198] The text unit sequence is mapped to a predefined symbol table to generate a text symbol sequence;

[0199] The text symbol sequence is indexed, and the indexed text symbol sequence is stored in memory.

[0200] In one embodiment, the speech encoding module 30 is specifically used for:

[0201] The speech data is preprocessed, including background noise removal and signal normalization.

[0202] The preprocessed speech data is input into a speech encoder trained based on a self-supervised learning model;

[0203] The speech encoder decomposes the continuous speech signal in the speech data into time frames of fixed length.

[0204] Feature extraction is performed on the speech data in each time frame to extract key speech features including Mel frequency cepstral coefficients, fundamental frequency, and energy;

[0205] The key speech features are quantized and converted into discrete speech symbol sequences.

[0206] In one embodiment, the sequence organization module 40 is specifically used for:

[0207] The text symbol sequence is analyzed by a parser to identify and label the grammatical function of each text symbol;

[0208] Based on the grammatical function and contextual relationship of each text symbol, the text symbols are arranged sequentially to generate an ordered text sequence that conforms to semantic logic. The contextual relationship refers to the coordination relationship between the text symbol and its adjacent symbols in terms of grammatical structure, semantic association, and word order.

[0209] The speech symbol sequence is arranged in chronological order to generate an ordered speech sequence.

[0210] In one embodiment, the preliminary alignment module 50 is specifically used for:

[0211] Analyze the time length of each text symbol in the text sequence;

[0212] Analyze the time frame of each speech symbol in the speech sequence;

[0213] The dynamic time warping module is used to align the time length of text symbols with the time frame of speech symbols.

[0214] Based on the time alignment results, the similarity between text symbols and speech symbols is analyzed;

[0215] A preliminary symbol mapping relationship is generated based on the similarity, and a correspondence table between text symbols and speech symbols is established.

[0216] In one embodiment, the gating attention mechanism module 60 is specifically used for:

[0217] Feature extraction is performed on the text sequence and the speech sequence respectively to generate high-dimensional feature vectors corresponding to the text symbols and speech symbols;

[0218] The high-dimensional feature vectors of text symbols and speech symbols are input into the gated attention mechanism module;

[0219] The gating attention mechanism module adjusts the attention weights based on the feature differences between the high-dimensional feature vectors of text symbols and speech symbols, and generates an association matrix between text symbols and speech symbols based on the attention weights.

[0220] The gated attention mechanism module adjusts the matching relationship between text symbols and corresponding speech symbols based on the association matrix, and updates the dependency relationship between text symbols and speech symbols based on the adjusted matching relationship;

[0221] The gated attention mechanism module generates the final updated matching result of text symbols and speech symbols based on the dependency relationship between the updated text symbols and speech symbols.

[0222] In one embodiment, the decoder module 70 is specifically used for:

[0223] The text sequence and speech sequence are input into the decoder;

[0224] The decoder processes the text and speech sequences frame by frame and generates a speech signal corresponding to each text symbol.

[0225] During the decoding process, the generated speech signal is adjusted based on the dependency relationship between text symbols and speech symbols to obtain the final speech signal.

[0226] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side speech synthesis method based on a gating attention mechanism.

[0227] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the user-side functions or steps of a speech synthesis method based on a gated attention mechanism.

[0228] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0229] Acquire text and voice data;

[0230] The text data is input into a text encoder, which converts the text data into a sequence of text symbols.

[0231] The speech data is input into a speech encoder trained based on a self-supervised learning model. The speech encoder extracts speech features from the speech data and quantizes the speech features into discrete speech symbol sequences.

[0232] The text symbol sequence is organized into a text sequence, and the speech symbol sequence is organized into a speech sequence;

[0233] The text sequence and speech sequence are initially aligned to establish the correspondence between text symbols and speech symbols;

[0234] Gated attention processing is applied to the initially aligned text and speech sequences to adjust the matching degree between text symbols and speech symbols;

[0235] The text sequence and speech sequence, which have undergone gated attention processing, are input into the decoder to generate the final speech signal.

[0236] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0237] Acquire text and voice data;

[0238] The text data is input into a text encoder, which converts the text data into a sequence of text symbols.

[0239] The speech data is input into a speech encoder trained based on a self-supervised learning model. The speech encoder extracts speech features from the speech data and quantizes the speech features into discrete speech symbol sequences.

[0240] The text symbol sequence is organized into a text sequence, and the speech symbol sequence is organized into a speech sequence;

[0241] The text sequence and speech sequence are initially aligned to establish the correspondence between text symbols and speech symbols;

[0242] Gated attention processing is applied to the initially aligned text and speech sequences to adjust the matching degree between text symbols and speech symbols;

[0243] The text sequence and speech sequence, which have undergone gated attention processing, are input into the decoder to generate the final speech signal.

[0244] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0245] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0246] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0247] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech synthesis method based on a gating attention mechanism, characterized in that, Includes the following steps: Acquire text and voice data; The text data is input into a text encoder, which converts the text data into a sequence of text symbols. The speech data is input into a speech encoder trained based on a self-supervised learning model. The speech encoder extracts speech features from the speech data and quantizes the speech features into discrete speech symbol sequences. The text symbol sequence is organized into a text sequence, and the speech symbol sequence is organized into a speech sequence; The text sequence and speech sequence are initially aligned to establish the correspondence between text symbols and speech symbols; Gated attention processing is applied to the initially aligned text and speech sequences to adjust the matching degree between text symbols and speech symbols. This includes: extracting features from the text and speech sequences to generate high-dimensional feature vectors corresponding to the text and speech symbols; inputting these high-dimensional feature vectors into a gated attention mechanism module; adjusting attention weights based on the feature differences between the high-dimensional feature vectors of the text and speech symbols, and generating an association matrix between the text and speech symbols based on these attention weights; adjusting the matching relationship between the text and speech symbols based on the association matrix, and updating the dependency relationship between the text and speech symbols based on the adjusted matching relationship; and generating the final updated matching result between the text and speech symbols based on the updated dependency relationship. The text sequence and speech sequence, which have undergone gated attention processing, are input into the decoder to generate the final speech signal.

2. The speech synthesis method based on gating attention mechanism as described in claim 1, characterized in that, The text data is input into a text encoder, which converts the text data into a sequence of text symbols, including: The text data is preprocessed, including format normalization and text normalization. The preprocessed text data is input into the text encoder, and the text data is decomposed into individual words or symbols by the word segmenter in the text encoder. The byte-pair encoding module in the text encoder converts the decomposed words or symbols into a predefined sequence of text units. The text unit sequence is mapped to a predefined symbol table to generate a text symbol sequence; The text symbol sequence is indexed, and the indexed text symbol sequence is stored in memory.

3. The speech synthesis method based on gating attention mechanism as described in claim 1, characterized in that, The speech data is input into a speech encoder trained based on a self-supervised learning model. The speech encoder extracts speech features from the speech data and quantizes the speech features into discrete speech symbol sequences, including: The speech data is preprocessed, including background noise removal and signal normalization. The preprocessed speech data is input into a speech encoder trained based on a self-supervised learning model; The speech encoder decomposes the continuous speech signal in the speech data into time frames of fixed length. Feature extraction is performed on the speech data in each time frame to extract key speech features including Mel frequency cepstral coefficients, fundamental frequency, and energy; The key speech features are quantized and converted into discrete speech symbol sequences.

4. The speech synthesis method based on gating attention mechanism as described in claim 1, characterized in that, Organizing the text symbol sequence into a text sequence and the speech symbol sequence into a speech sequence includes: The text symbol sequence is analyzed by a parser to identify and label the grammatical function of each text symbol; Based on the grammatical function and contextual relationship of each text symbol, the text symbols are arranged sequentially to generate an ordered text sequence that conforms to semantic logic. The contextual relationship refers to the coordination relationship between the text symbol and its adjacent symbols in terms of grammatical structure, semantic association, and word order. The speech symbol sequence is arranged in chronological order to generate an ordered speech sequence.

5. The speech synthesis method based on gating attention mechanism as described in claim 1, characterized in that, The text sequence and speech sequence undergo preliminary alignment processing to establish the correspondence between text symbols and speech symbols, including: Analyze the time length of each text symbol in the text sequence; Analyze the time frame of each speech symbol in the speech sequence; The dynamic time warping module is used to align the time length of text symbols with the time frame of speech symbols. Based on the time alignment results, the similarity between text symbols and speech symbols is analyzed; A preliminary symbol mapping relationship is generated based on the similarity, and a correspondence table between text symbols and speech symbols is established.

6. The speech synthesis method based on gating attention mechanism as described in claim 1, characterized in that, The gated attention-processed text and speech sequences are input into the decoder to generate the final speech signal, including: The text sequence and speech sequence are input into the decoder; The decoder processes the text and speech sequences frame by frame and generates a speech signal corresponding to each text symbol. During the decoding process, the generated speech signal is adjusted based on the dependency relationship between text symbols and speech symbols to obtain the final speech signal.

7. A speech synthesis device based on a gating attention mechanism, characterized in that, The speech synthesis device based on the gated attention mechanism includes: The data acquisition module is used to acquire text and voice data; A text encoding module is used to input the text data into a text encoder, and convert the text data into a sequence of text symbols through the text encoder; A speech encoding module is used to input the speech data into a speech encoder, extract speech features from the speech data through the speech encoder, and quantize the speech features into a discrete sequence of speech symbols; A sequence organization module is used to organize the text symbol sequence into a text sequence and the speech symbol sequence into a speech sequence; The preliminary alignment module is used to perform preliminary alignment processing on the text sequence and the speech sequence to establish the correspondence between text symbols and speech symbols; A gated attention mechanism module is used to perform gated attention processing on pre-aligned text and speech sequences to adjust the matching degree between text symbols and speech symbols. This includes: extracting features from the text and speech sequences respectively to generate high-dimensional feature vectors corresponding to the text and speech symbols; inputting the high-dimensional feature vectors of the text and speech symbols into the gated attention mechanism module; adjusting the attention weights based on the feature differences between the high-dimensional feature vectors of the text and speech symbols, and generating an association matrix between the text and speech symbols based on the attention weights; adjusting the matching relationship between the text symbols and corresponding speech symbols based on the association matrix, and updating the dependency relationship between the text and speech symbols based on the adjusted matching relationship; and generating the final updated matching result between the text and speech symbols based on the updated dependency relationship between the text and speech symbols. The decoder module is used to input the gated attention-processed text sequence and speech sequence into the decoder to generate the final speech signal.

8. A computer device, characterized in that, The computer device includes a memory, a processor, and a gated attention mechanism-based speech synthesis program stored in the memory and executable on the processor. When executed by the processor, the gated attention mechanism-based speech synthesis program implements the steps of the gated attention mechanism-based speech synthesis method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a speech synthesis program based on a gating attention mechanism, which, when executed by a processor, implements the steps of the speech synthesis method based on a gating attention mechanism as described in any one of claims 1-6.

Images