Voice synthesis method and device based on gated attention mechanism, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a speech synthesis method based on a gated attention mechanism, and the method comprises the steps: obtaining text data and speech data, converting the text data into a text symbol sequence through a text encoder, speech features of the speech data are extracted through a speech encoder trained based on a self-supervised learning model, and the speech features are quantized into a discrete speech symbol sequence; then, respectively organizing the text symbol sequence and the voice symbol sequence into a text sequence and a voice sequence; the corresponding relation between the text symbols and the voice symbols is established through preliminary alignment processing, the matching degree between the text symbols and the voice symbols is dynamically adjusted through a gating attention mechanism, and finally a final voice signal is generated through a decoder. According to the invention, through the voice encoder trained by the self-supervised learning model, the extraction capability of voice features is effectively improved, and effective feature representation can still be learned from unlabeled data especially in a scene lacking a large amount of labeled data.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and financial technology, and in particular to a speech synthesis method, device, equipment and storage medium based on a gated attention mechanism. Background Art

[0002] In the financial field, with the development of financial technology, automated processing and intelligent decision-making have been widely used in multiple scenarios such as customer service, risk assessment, and voice interaction, especially speech synthesis technology. In financial services, speech synthesis is widely used in scenarios such as intelligent customer service, phone notifications, and transaction confirmation to improve user experience. However, existing speech synthesis technology still faces many technical limitations in the application process.

[0003] First, existing speech synthesis technology relies on a large number of sample training, and zero-sample speech synthesis is still a huge challenge. For personalized speech content that needs to be generated quickly in financial scenarios, traditional speech synthesis models that rely on large-scale data training often lag behind and cannot efficiently and accurately generate customized speech content.

[0004] Secondly, although neural network models, especially codec language models, have made significant progress in speech synthesis, their autoregressive characteristics limit the speed of inference. In the financial field, it is crucial to respond quickly to customer needs or market changes, but because the implicit alignment between text and audio in the model is not ideal, the stability and consistency of speech generation is difficult to guarantee, especially under high concurrent requests, speech output delays or errors are prone to occur.

[0005] Furthermore, in the existing technology, there is an occasional mismatch between speech synthesis and text prompts, such as inconsistency between speech content and text content, which may lead to loss, duplication or errors in the voice notification. This may have serious consequences in financial transactions or key business notifications, especially when errors occur in voice transaction confirmations or compliance reminders, which will directly affect customer experience and even bring compliance risks.

[0006] Therefore, how to effectively improve the accuracy and real-time performance of speech synthesis, especially to achieve efficient and stable speech generation in a zero-sample environment, remains a technical bottleneck that needs to be urgently solved in the financial field. Summary of the invention

[0007] The main purpose of the present invention is to provide a speech synthesis method, apparatus, device and storage medium based on a gated attention mechanism, aiming to solve the technical problems that the existing speech synthesis technology has slow inference speed and inaccurate matching of speech and text in a zero-sample environment, resulting in unstable generated speech signals.

[0008] To achieve the above object, the present invention provides a speech synthesis method based on a gated attention mechanism, comprising:

[0009] Acquire text data and voice data;

[0010] Inputting the text data into a text encoder, and converting the text data into a text symbol sequence by the text encoder;

[0011] Inputting the speech data into a speech encoder, extracting speech features from the speech data through the speech encoder, and quantizing the speech features into a discrete speech symbol sequence;

[0012] Organizing the text symbol sequence into a text sequence and organizing the voice symbol sequence into a voice sequence;

[0013] Performing preliminary alignment processing on the text sequence and the voice sequence to establish a corresponding relationship between the text symbols and the voice symbols;

[0014] Perform gated attention processing on the preliminarily aligned text sequence and speech sequence to adjust the matching degree between text symbols and speech symbols;

[0015] The text sequence and speech sequence processed by gated attention are input into the decoder to generate the final speech signal.

[0016] Furthermore, to achieve the above object, the present invention provides a speech synthesis device based on a gated attention mechanism, comprising:

[0017] A data acquisition module, used to acquire text data and voice data;

[0018] A text encoding module, used for inputting the text data into a text encoder, and converting the text data into a text symbol sequence through the text encoder;

[0019] A speech coding module, used for inputting the speech data into a speech encoder, extracting speech features from the speech data through the speech encoder, and quantizing the speech features into a discrete speech symbol sequence;

[0020] A sequence organization module, used for organizing the text symbol sequence into a text sequence and organizing the voice symbol sequence into a voice sequence;

[0021] An alignment module, used for performing preliminary alignment processing on the text sequence and the voice sequence to establish a corresponding relationship between the text symbols and the voice symbols;

[0022] The gated attention mechanism module is used to perform gated attention processing on the preliminarily aligned text sequence and speech sequence to adjust the matching degree between text symbols and speech symbols;

[0023] The decoder module is used to input the text sequence and speech sequence processed by gated attention into the decoder to generate the final speech signal.

[0024] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a speech synthesis program based on a gated attention mechanism stored in the memory and executable on the processor, wherein the speech synthesis program based on the gated attention mechanism, when executed by the processor, implements the steps of the speech synthesis method based on the gated attention mechanism as described above.

[0025] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a speech synthesis program based on a gated attention mechanism is stored. When the speech synthesis program based on the gated attention mechanism is executed by a processor, the steps of the speech synthesis method based on the gated attention mechanism as described above are implemented.

[0026] Beneficial effects: The present invention relates to the fields of artificial intelligence technology and financial technology, and discloses a speech synthesis method based on a gated attention mechanism. By acquiring text data and speech data, the text data is converted into a text symbol sequence through a text encoder, and the speech data is extracted through a speech encoder trained based on a self-supervised learning model. Voice features are quantified into discrete speech symbol sequences; then, the text symbol sequence and the speech symbol sequence are organized into text sequences and speech sequences respectively; the correspondence between text symbols and speech symbols is established through preliminary alignment processing, and the gated attention mechanism dynamically adjusts the matching degree between text symbols and speech symbols, and finally generates the final speech signal through a decoder. The present invention effectively improves the ability to extract speech features through a speech encoder trained by a self-supervised learning model, especially in scenarios where a large amount of labeled data is lacking, effective feature representations can still be learned from unlabeled data. Combined with the gated attention mechanism, the matching relationship between text and speech symbols can be adjusted to ensure the accuracy and fluency of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0028] Figure 1 A schematic diagram of an application environment of a speech synthesis method based on a gated attention mechanism in an embodiment of the present invention;

[0029] Figure 2 It is a flow chart of an embodiment of a speech synthesis method based on a gated attention mechanism of the present invention;

[0030] Figure 3It is a functional module diagram of a preferred embodiment of a speech synthesis device based on a gated attention mechanism of the present invention;

[0031] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0032] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0033] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0034] The speech synthesis method based on the gated attention mechanism provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the user end communicates with the server end through the network. The server end can obtain text data and voice data from the user end, the text data is converted into a text symbol sequence through a text encoder, and the voice data is extracted by a voice encoder trained based on a self-supervised learning model, and the voice features are quantized into a discrete voice symbol sequence; then, the text symbol sequence and the voice symbol sequence are organized into a text sequence and a voice sequence respectively; the correspondence between the text symbol and the voice symbol is established through a preliminary alignment process, and the gated attention mechanism dynamically adjusts the matching degree between the text symbol and the voice symbol, and finally generates the final voice signal through a decoder. The present invention effectively improves the ability to extract voice features through a voice encoder trained by a self-supervised learning model, especially in the scenario where a large amount of labeled data is lacking, effective feature representation can still be learned from unlabeled data. Combined with the gated attention mechanism, the matching relationship between text and voice symbols can be adjusted to ensure the accuracy and fluency of the synthesized speech. Among them, the user end can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server end can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0035] See also Figure 2 , Figure 2 A flowchart of an embodiment of a speech synthesis method based on a gated attention mechanism provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0036] like Figure 2 As shown, the speech synthesis method based on the gated attention mechanism proposed in the present invention includes the following steps:

[0037] S10, acquiring text data and voice data;

[0038] In this embodiment, the input data to be processed is obtained from an external or internal source. Text data usually refers to natural language text, such as text content in articles, conversations, or financial reports. Voice data refers to audio files, usually human voice input, including conversations in customer service calls, transaction confirmation notifications, etc.

[0039] The format of text data is usually readable strings or sentences, stored in databases, text files, or obtained from external systems in real time through API interfaces. For example, text data in the financial field can be customer conversation records, transaction information, or contract text.

[0040] Voice data is usually a recorded audio file or a real-time voice stream, and the file format may include WAV, MP3, etc. Voice data may contain different pronunciations, intonations, and even background noise, which needs to be processed in subsequent steps.

[0041] Text data can be extracted from databases or financial systems in real time through APIs. For example, in a financial customer service scenario, the system will capture the text entered by the user in real time from the customer service conversation, or obtain relevant data from financial statements and contract management systems. Text data can also be obtained from CSV files, databases, or XML files through batch import.

[0042] Voice data can be recorded in real time through an audio interface (such as a microphone) or extracted from an existing voice library. For example, in a financial transaction confirmation call, the system can collect the customer's voice confirmation data in real time through the phone recording interface. Voice files are stored in a server or cloud storage, and voice signals can be accessed through a file path or directly extracted from an audio file.

[0043] Example: When a customer authenticates through a bank’s mobile app or online customer service system, the system asks the customer to enter personal information, such as name, account number, ID number, etc. These text data are transmitted to the intelligent customer service system in real time through the API interface for subsequent processing. The text data can also include the user’s query content, such as “I want to view recent transaction records” or “What should I do if I forget my password?” These contents are passed to the system through the online chat interface.

[0044] In the voice customer service scenario, when a customer calls the bank's customer service hotline to confirm a transaction, the system will obtain the customer's voice data through the telephone recording interface. This voice data can be the customer's voice, answers to questions (such as identity verification questions), and the customer's voice confirmation (such as "I confirm this transaction"). The voice data is captured by the real-time telephone recording system and transmitted to the backend server as input for subsequent speech synthesis and analysis.

[0045] By acquiring text and voice data in real time or in batches, it can be ensured that the speech synthesis system can obtain the necessary data for processing in a timely and accurate manner, which can improve the system's response speed and data processing capabilities.

[0046] S20, inputting the text data into a text encoder, and converting the text data into a text symbol sequence through the text encoder;

[0047] In this embodiment, natural language text data is converted into a symbol sequence that can be understood and processed by a computer. The text encoder is the core component that converts human language into a format that can be understood by a machine. It can convert text into a lower-level symbol representation through techniques such as word segmentation and encoding for subsequent processing.

[0048] Text data (e.g., customer inquiries, transaction information) is passed to the text encoder through an input interface. The text data is usually a string in the form of natural language, including words, phrases, or complete sentences.

[0049] A text symbol sequence is a symbol sequence converted from a natural language text. A symbol can be a word, a subword, or the smallest unit after byte pair encoding (BPE). The generation of a symbol sequence ensures that the text can be processed by subsequent models (such as alignment and speech synthesis). A symbol sequence can be either a discrete word sequence or a subword or character-level representation.

[0050] Before being fed into a text encoder, the text data may need to be cleaned and normalized, such as removing invalid characters, symbols, etc. For financial data, there may be specialized processing procedures, such as identifying special terms or financial transaction terms. The first step of a text encoder is tokenization, which breaks sentences or phrases into basic units. This can be achieved through a tokenizer, and common tokenization methods include word-based tokenization, subword tokenization, or character-level tokenization. The tokenized basic units are converted into a sequence of symbols through an encoding process. In this step, byte pair encoding (BPE) or other encoding methods are often used to convert vocabulary into symbolic form. The symbol sequence can be used for subsequent calculations. The encoded text is converted into a sequence of symbols, such as converting the sentence "customer request transaction record" into a symbol sequence, such as [customer, request, transaction, record], or encoded as [101,204,309,412] for subsequent processing.

[0051] By encoding text data into a sequence of symbols, natural language data can be effectively processed and adapted to the needs of machine learning models. Customers' natural language input (such as customer questions, transaction requests) can be quickly and accurately converted to provide efficient symbol sequence input for subsequent speech synthesis or intelligent processing.

[0052] S30, inputting the speech data into a speech encoder trained based on a self-supervised learning model, extracting speech features from the speech data through the speech encoder, and quantizing the speech features into a discrete speech symbol sequence;

[0053] In this embodiment, voice data is input into the system through an audio interface. Voice data may include the voice of the customer or a telephone recording, which needs to be processed to extract relevant features. In the financial field, voice data usually comes from scenarios such as customer voice confirmation and telephone customer service.

[0054] Self-supervised learning is a way to learn features from unlabeled data. The speech encoder is trained using a self-supervised learning model, which means that the encoder can extract useful speech features from unlabeled data without a large amount of labeled speech data. In the financial field, customers' speech data may not have clear labels, and self-supervised learning can automatically learn useful speech representations from this data.

[0055] The speech encoder trained by the self-supervised learning model can efficiently extract features from the input speech data. It can not only extract the key features of the speech (such as pitch, frequency, voice pattern, etc.), but also identify the details in the pronunciation.

[0056] Speech feature extraction is to convert the input speech signal into understandable feature representations, such as pitch, frequency, etc. The speech encoder extracts key feature parameters from the speech signal by analyzing its spectrum and time domain characteristics.

[0057] After extracting the features, the speech encoder quantizes these features into a discrete sequence of symbols. The process of quantization is to convert the continuous speech signal into a discrete symbol representation so that it can be further processed. For example, a continuous speech signal may be quantized into a set of discrete symbols (such as [S1, S2, S3]) that can represent different speech segments.

[0058] During the training process, the speech encoder can be trained through self-supervised learning. The model can use a large amount of unlabeled speech data to learn pronunciation patterns and speech features, thereby improving the ability to recognize new speech data. The extracted speech features capture the characteristics of the speech signal through spectral analysis, short-time Fourier transform and other technologies. The quantization process uses a codebook to map speech features to discrete symbols. For example, speech data in different frequency bands are mapped to specific discrete symbols.

[0059] Example description: In the confirmation of a financial telephone transaction, the customer's voice input (such as "I confirm this transfer") is recorded by the system in real time and transmitted to the voice encoder. The voice encoder processes the voice data through a self-supervised learning model, extracts the customer's voice features, and quantifies these features into discrete voice symbol sequences. Through these symbol sequences, the system can further match and analyze the customer's voice input to confirm whether the customer has authorized it. In this scenario, the voice encoder can extract key features from the customer's voice without a large amount of training data. This greatly improves the accuracy of speech recognition, especially when it comes to financial transactions, ensuring that every confirmation from the customer is correctly processed and responded to.

[0060] By using a speech encoder trained based on a self-supervised learning model, speech features can be effectively extracted, and the ability to process speech data can be improved even without a large amount of labeled data. After quantization into a discrete sequence of speech symbols, the system can quickly process speech signals, achieve more accurate speech generation and analysis, and improve the system's response speed and recognition capabilities.

[0061] S40, organizing the text symbol sequence into a text sequence, and organizing the voice symbol sequence into a voice sequence;

[0062] In this embodiment, the text symbol sequence is a discrete symbol generated in the previous text encoding process. The text symbol can be a word, a subword or a character-level symbol. The process of organizing the text symbol sequence into a text sequence is to arrange these symbols reasonably according to the grammatical rules of natural language to form a structured text representation. This step ensures that the symbols of the text can be processed according to the correct language structure so as to be matched with the speech later.

[0063] The speech symbol sequence is the symbol extracted and quantized by the speech encoder. The speech symbol represents the fragments in the speech signal. By organizing the speech symbol sequence, these fragments can be arranged in the time sequence of the speech to form an orderly speech sequence. This process ensures the coherence of the speech signal and avoids dislocation or loss of information in subsequent processing.

[0064] The purpose of organizing text and speech symbols is to ensure the correspondence between the two in subsequent processing. The text sequence and the speech sequence need to be arranged in the same logical order so that subsequent alignment processing can be performed. The text sequence is used to represent the corresponding words in the speech, and the speech sequence is used to represent the specific speech signal.

[0065] In the process of organizing text symbol sequences, the system needs to perform grammatical analysis first. By parsing the grammatical roles of text symbols (such as subject, predicate, object, etc.), the text sequence can be arranged in a semantically logical order. For example, the subject and predicate of the sentence are determined first, and then the word order is adjusted according to the language rules.

[0066] The voice symbols are arranged in time order by analyzing the frame information of the voice signal. After the voice symbol sequence is generated, the system ensures that these symbols are arranged in the natural sounding order of the voice signal according to the timestamp of each voice symbol to avoid the misalignment of the audio segments.

[0067] By organizing the text symbol sequence and the voice symbol sequence into an ordered text sequence and voice sequence respectively, the system can ensure the consistency and matching of the two in subsequent processing. The grammatical structure of the text sequence and the temporal order of the voice sequence can be better matched, making the speech synthesis and alignment process smoother and more accurate.

[0068] S50, performing preliminary alignment processing on the text sequence and the voice sequence to establish a corresponding relationship between the text symbols and the voice symbols;

[0069] In this embodiment, the text sequence and the speech sequence are structured symbol representations that have been encoded and organized. The initial alignment process is to establish a correspondence between these text symbols and speech symbols to ensure that the system can accurately match the content between text and speech in the subsequent synthesis or recognition process. The alignment process includes two levels: time alignment and content alignment.

[0070] Time alignment means that each symbol in the text sequence should match the time segment corresponding to the corresponding symbol in the speech sequence. Since the speech symbols are arranged in time order, the system ensures that the time segment of each text symbol can correspond to the corresponding speech symbol time segment by analyzing the corresponding duration of the text symbol and the speech symbol.

[0071] Content alignment refers to matching the semantics or meaning of text symbols with speech symbols. For example, the text symbol "query" needs to correspond to the pronunciation "query" in the speech signal. The system ensures that each text symbol has the same meaning as its corresponding speech symbol by matching the similarity or semantic correspondence between text and speech symbols.

[0072] Initial alignment is usually achieved through dynamic time warping (DTW) or other alignment algorithms. The DTW algorithm can dynamically adjust the correspondence between the text sequence and the speech sequence by calculating the time difference between them to ensure that the time of the text symbols is consistent with the speech symbols. For example, the "transaction confirmation" in the speech may be pronounced faster or slower, and DTW will dynamically adjust the time matching of the text and speech.

[0073] In addition to time alignment, the system can also generate preliminary symbol mapping relationships based on the similarity between text symbols and voice symbols. By analyzing the semantic or phonetic features between symbols, the system matches the text symbol with the most similar voice symbol and establishes a preliminary correspondence.

[0074] By initially aligning the text sequence and the speech sequence, the system can effectively establish the correspondence between text symbols and speech symbols. This process ensures the matching accuracy between text and speech, helps reduce errors in the subsequent speech synthesis and recognition process, and improves the accuracy and consistency of the synthesis results.

[0075] S60, performing gated attention processing on the preliminarily aligned text sequence and speech sequence to adjust the matching degree between the text symbols and the speech symbols;

[0076] In this embodiment, the gated attention mechanism is a dynamic control mechanism used to focus on key information in model processing. In this step, the preliminarily aligned text sequence and speech sequence are input to the gated attention module, which determines the degree of match between text symbols and speech symbols by adjusting the attention weights.

[0077] The gated attention mechanism automatically assigns attention weights between each pair of symbols by calculating the feature similarity between text symbols and speech symbols. Based on the size of the attention weights, the system determines which symbols match better and which symbols need further adjustment. This dynamic adjustment can improve the alignment between text and speech and ensure the accuracy of speech synthesis.

[0078] The matching relationship refers to the semantic and phonetic correspondence between text symbols and speech symbols. Through the gated attention mechanism, the system can detect the temporal or semantic deviations between text symbols and speech symbols, and dynamically adjust the matching between symbols. The gating mechanism helps the system prioritize those symbols that are highly related to text or speech features, ignore those unimportant symbols, and thus optimize the final synthesized speech results.

[0079] The gated attention module calculates the attention weights between text symbols and speech symbols, and determines the correlation between each pair of symbols through the feature vectors of the input text sequence and speech sequence. The gated unit controls the attention allocation between each symbol to ensure that the system can better capture the key text-speech alignment relationship.

[0080] After calculating the attention weights, the system updates the dependency between the text and speech symbols based on the weights. High-weighted symbol pairs indicate that they are well matched, while low-weighted symbols may indicate a poor match that requires further adjustment.

[0081] Example: In the financial intelligent speech synthesis system, the customer inputs "query the latest transaction records in my account" through voice. At this point, the system has completed the preliminary alignment of text and speech symbols, and then dynamically adjusts the matching degree of text and speech symbols through the gated attention mechanism.

[0082] Based on the voice of the customer, the system uses the gated attention mechanism to identify the key content in the voice, such as "query", "account", "recent transaction", and other voice symbols, and matches these voice symbols with text symbols. By adjusting the attention weight, the system prioritizes the key information in the customer's voice and ignores irrelevant background noise or pauses.

[0083] If the customer's pronunciation is unclear or has an accent, the system can make further dynamic adjustments through the gated attention mechanism. For example, when the customer's pronunciation is inaccurate and the pronunciations of "account" and "account" are similar, the system will prioritize the correct matching symbol "account" based on semantic information to ensure high-precision matching of speech and text symbols.

[0084] Through the dynamic adjustment of the gated attention mechanism, the system can automatically optimize the matching relationship between text symbols and voice symbols, thereby improving the quality of speech synthesis. The gated attention mechanism not only improves the flexibility of alignment, but also reduces the matching error between text and speech, improves the accuracy and naturalness of generated speech, helps reduce repetition, misalignment or loss problems in speech synthesis, and ensures the accuracy of speech feedback.

[0085] S70, inputting the text sequence and speech sequence processed by gated attention into the decoder to generate a final speech signal.

[0086] In this embodiment, after being processed by the gated attention mechanism, the matching relationship between the text sequence and the voice sequence has been adjusted and optimized. At this point, the text symbols and the voice symbols have formed a one-to-one association. Next, the system inputs these processed text and voice symbols into the decoder. The role of the decoder is to convert these symbol sequences into the final voice signal.

[0087] The decoder usually processes the input sequence frame by frame. This means that it decodes each pair of symbols in the text sequence and the speech sequence frame by frame to generate a smooth speech signal. Each frame represents a small segment of speech, and frame-by-frame decoding can ensure the coherence and naturalness of speech synthesis.

[0088] During the decoding process, the system generates corresponding speech segments based on the characteristics of the text symbols and the corresponding speech symbols, and gradually splices these segments into a complete speech signal. The decoder matches the content, intonation, speed, etc. of the text and speech to generate a coherent speech output.

[0089] The final speech signal is generated by the decoder, which is a synthesized speech file or speech stream that can be used for playback or storage. The system converts all gated symbol sequences into natural speech through the decoder. The output speech corresponds to the input text content, ensuring the accuracy and fluency of speech synthesis.

[0090] The decoder generates speech signals by processing them frame by frame. Each frame consists of a pair of text symbols and speech symbols. The decoder gradually generates corresponding speech frames by analyzing the time series characteristics of text symbols and speech symbols, and finally splices multiple frames into a complete speech signal.

[0091] Speech reconstruction is the process of generating natural speech signals by analyzing the characteristics of input speech symbols (such as pitch, tone, speed, etc.). The decoder needs to ensure the naturalness and coherence of the speech when reconstructing the speech signal. For example, in financial services, the speed and tone of speech synthesis need to match the customer's voice to ensure the realism of the synthesized speech.

[0092] Example: In a financial speech synthesis system, the transaction confirmation instruction entered by the customer, such as "confirm transfer of 5,000 yuan", needs to be generated by the system. After adjusting the matching of text symbols and speech symbols through the gated attention mechanism, the system inputs the symbol sequence into the decoder for speech generation.

[0093] The decoder processes the input sequence of text and voice symbols frame by frame. For the input "Confirm transfer of 5,000 yuan", the system generates a voice signal based on the matching relationship between the symbols to ensure the clarity and moderate speed of the voice feedback. The decoder generates the corresponding voice signal based on the characteristics of the input symbols. For example, during the transaction confirmation process, the speech synthesis system will generate voice feedback of "Confirmed transfer of 5,000 yuan" to ensure that the customer can hear accurate and natural voice prompts. Through the decoder's frame-by-frame processing and voice reconstruction, the system can generate high-quality voice feedback to ensure that the customer's interaction experience with the financial system is more natural and smooth.

[0094] By inputting the sequence of text and speech symbols processed by gated attention into the decoder, the system can generate coherent and natural speech signals. The decoder generates speech signals based on the correspondence between text symbols and speech symbols, ensuring that the output speech is consistent with the input text content. This effectively improves the fluency and accuracy of speech synthesis.

[0095] The present invention relates to the field of artificial intelligence technology and financial technology, and discloses a speech synthesis method based on a gated attention mechanism, by acquiring text data and speech data, the text data is converted into a text symbol sequence through a text encoder, and the speech data is extracted from the speech features through a speech encoder trained based on a self-supervised learning model, and quantized into a discrete speech symbol sequence; then, the text symbol sequence and the speech symbol sequence are organized into a text sequence and a speech sequence respectively; the corresponding relationship between the text symbol and the speech symbol is established through a preliminary alignment process, and the gated attention mechanism dynamically adjusts the matching degree between the text symbol and the speech symbol, and finally generates a final speech signal through a decoder. The present invention effectively improves the ability to extract speech features through a speech encoder trained by a self-supervised learning model, especially in a scenario where a large amount of labeled data is lacking, effective feature representations can still be learned from unlabeled data. Combined with the gated attention mechanism, the matching relationship between text and speech symbols can be adjusted to ensure the accuracy and fluency of the synthesized speech.

[0096] In one embodiment, the above S20 includes:

[0097] S201, performing a preprocessing operation on the text data, the preprocessing operation including format standardization and text standardization;

[0098] S202, inputting the preprocessed text data into a text encoder, and decomposing the text data into separate words or symbols through a word segmenter in the text encoder;

[0099] S203, converting the decomposed words or symbols into a predefined text unit sequence through a byte pair encoding module in a text encoder;

[0100] S204, mapping the text unit sequence to a predefined symbol table to generate a text symbol sequence;

[0101] S205, indexing the text symbol sequence, and storing the indexed text symbol sequence in a memory.

[0102] In this embodiment, format standardization is the process of converting text data into a standard format that can be processed by the system. For example, the text data may come from different data sources, and there may be problems such as inconsistent encoding formats and inconsistent capitalization. Through format standardization, it can be ensured that all text data is processed in a unified encoding format.

[0103] Text standardization refers to processing unnecessary characters such as extra spaces and punctuation marks in text so that the text content can more accurately express the actual meaning. For example, in a financial scenario, the user input "check balance" and "check balance" can be standardized into a consistent form to ensure that the system can recognize them as the same request.

[0104] The tokenizer breaks down the preprocessed text data into individual words or symbols. In natural language processing, tokenization is the process of breaking sentences into individual words, subwords, or symbols for subsequent encoding operations. The accuracy of tokenization directly affects the quality of subsequent text encoding and symbol mapping.

[0105] The byte pair encoding (BPE) module in the text encoder converts the segmented words or symbols into a predefined sequence of text units. BPE is a common word segmentation method that further breaks down words into smaller units to improve the compression efficiency and processing speed of the text. This step ensures that the generated unit sequence can contain richer text information.

[0106] The generated text unit sequence needs to be mapped to a predefined symbol table, which is a mapping dictionary that converts each text unit into a corresponding symbol number. This allows the text to be represented in the form of a symbol sequence, which is convenient for subsequent processing.

[0107] Indexing is to number the text symbol sequence so that each symbol has a unique index number. The indexed symbol sequence is stored in the memory for use in subsequent steps. Indexing greatly improves the efficiency of querying and processing symbol sequences.

[0108] Format standardization can be uniformly processed through the built-in character encoding library of the programming language (such as Python's utf-8). Text standardization can use regular expressions to remove unnecessary spaces, punctuation marks, etc., or it can be cleaned up through a natural language processing library (such as NLTK).

[0109] The word segmenter can use a rule-based word segmentation algorithm or a machine learning model for word segmentation. In the financial field, commonly used word segmentation tools include Jieba word segmentation (Chinese) or SpaCy (English). The byte pair encoding algorithm can generate compact symbol units by gradually merging common character combinations in the text. BPE is widely used in machine translation, text generation and other fields.

[0110] The symbol table can be implemented by HashMap or dictionary data structure, which is used to quickly find the symbol number corresponding to each text unit. In financial data, some special financial terms can be pre-defined in the symbol table to ensure that the vocabulary unique to the financial field is accurately processed.

[0111] Indexing can be achieved by mapping symbols to digital numbers, and the order of the numbers determines the position of the symbols in the text. Storage can use an in-memory database (such as Redis) or a persistent database (such as MySQL) to store the indexed symbol sequence for subsequent processing and query.

[0112] Example description: In the intelligent customer service system, a customer may enter the text "query last month's account balance". The system needs to input this text data into the text encoder for processing.

[0113] The system first performs format standardization, converting all input text into a unified UTF-8 encoding format to avoid processing errors caused by inconsistent data formats from different sources. Then it performs text standardization to remove unnecessary spaces and punctuation marks to ensure the simplicity and consistency of the input text.

[0114] The system decomposes "query last month's account balance" into the words "query", "last month", and "account balance" through the word segmenter, and further decomposes these words into text unit sequences through the BPE module. The generated text unit sequence is mapped to the system's symbol table and converted into symbol numbers, for example, "query" is mapped to 101 and "account balance" is mapped to 202. The generated symbol sequence is indexed and stored in memory for use by the subsequent speech synthesis module.

[0115] Through preprocessing, word segmentation, encoding and indexing of text data, the system of this embodiment can efficiently convert natural language text into symbol sequences, ensuring that the processed symbol sequences can provide accurate input for subsequent text and speech alignment and speech generation.

[0116] In one embodiment, the above S30 includes:

[0117] S301, performing a preprocessing operation on the voice data, wherein the preprocessing operation includes removing background noise and signal normalization processing;

[0118] S302, inputting the pre-processed speech data into a speech encoder trained based on a self-supervised learning model;

[0119] S303, decomposing the continuous speech signal in the speech data into time frames of fixed length by the speech encoder;

[0120] S304, performing feature extraction on the speech data in each time frame to extract key speech features including Mel-frequency cepstral coefficients, fundamental frequency and energy;

[0121] S305, quantizing the key speech features and converting them into discrete speech symbol sequences.

[0122] In this embodiment, the speech data usually contains background noise, such as ambient sound, echo, etc. Before the speech data is input into the encoder, the system needs to process the audio data through a filter or noise suppression technology to remove these interferences. Preprocessing ensures that the speech data is clearer and reduces interference with feature extraction.

[0123] Normalization is to standardize the amplitude of the audio signal to ensure that the volume of different voice data is consistent. Through signal normalization processing, the system can avoid the influence of volume differences on the extraction and processing of voice features.

[0124] The preprocessed speech data is input into a speech encoder trained by a self-supervised learning model. The self-supervised learning model can automatically learn features from data without a large amount of labeled data. The speech encoder uses self-supervised learning technology to extract useful speech features from unlabeled speech data and can adaptively process various types of speech signals.

[0125] The system decomposes the continuous speech signal into time frames of fixed length, each frame representing a segment of the speech signal. This process is called framing. By decomposing the speech signal, the system can process the speech data frame by frame and extract the feature information in each time period. Each frame is usually 10ms to 30ms in length.

[0126] For each time frame, the speech encoder extracts the key features of the speech signal. Commonly used techniques for feature extraction include Mel Frequency Cepstral Coefficients (MFCC), fundamental frequency (Pitch), energy (Energy), etc. These features represent the frequency distribution, intonation and energy changes of speech, and are important parameters for speech recognition and speech synthesis.

[0127] The extracted speech features need to be quantized to convert the continuous feature values ​​into discrete symbol sequences. The quantization process maps the speech features into discrete symbol sets, making them suitable for further processing and storage. The quantized discrete symbol sequences can simplify subsequent calculations and improve processing speed and efficiency.

[0128] Noise removal can remove background noise from speech signals using digital filters (such as low-pass filters) or noise suppression techniques based on spectral subtraction. The amplitude range of all speech data is standardized through volume normalization technology. For example, the volume is consistent by adjusting the signal to a uniform amplitude range. Self-supervised learning speech encoders can automatically extract useful speech features by learning from unlabeled data. Common implementation methods include using models such as Wav2Vec to learn speech representations from massive amounts of raw speech data without manually labeled data.

[0129] The audio data of each time frame is subjected to feature extraction techniques such as MFCC, fundamental frequency and energy. The spectral characteristics of the speech signal can be analyzed through the short-time Fourier transform (STFT), and speech features such as Mel-frequency cepstral coefficients can be extracted. Quantization processing uses methods such as vector quantization to map feature values ​​into discrete symbols.

[0130] In this embodiment, the speech encoder is trained by the self-supervised learning model, and the system can automatically learn speech features and perform effective processing without annotating speech data. Combined with preprocessing techniques (such as noise removal and signal normalization), the system can generate high-quality speech symbol sequences to ensure that the subsequent speech synthesis process is more accurate and natural. In the financial field, especially in telephone customer service or voice confirmation, the speech encoder can improve the accuracy of speech recognition and improve the customer's voice interaction experience.

[0131] In one embodiment, the above S40 includes:

[0132] S401, analyzing the text symbol sequence by a grammatical analyzer, identifying and marking the grammatical function of each text symbol;

[0133] S402, based on the grammatical function and contextual relationship of each text symbol, arranging the text symbols in sequence to generate an ordered text sequence that conforms to semantic logic, wherein the contextual relationship represents the coordination relationship between the text symbol and adjacent symbols in terms of grammatical structure, semantic association and word order;

[0134] S403, arranging the voice symbol sequence in chronological order to generate an ordered voice sequence.

[0135] In this embodiment, the system first uses a grammatical analyzer to analyze the text symbol sequence. The grammatical analyzer will identify and annotate the grammatical function of each text symbol. For example, the system will identify which symbols are subjects, predicates, objects, etc., and generate structured annotation information based on the grammatical role of each symbol in the sentence. The purpose of this step is to ensure that the text symbols can be organized according to the correct grammatical structure for subsequent processing.

[0136] After identifying the grammatical function of each text symbol, the system arranges the text symbols in sequence based on these grammatical functions and contextual relationships. Contextual relationships refer to the grammatical structure, semantic association, and word order coordination between a text symbol and adjacent symbols. Through this arrangement, the system can generate an ordered text sequence that conforms to semantic logic, ensuring that the text symbols can express information coherently and correctly. The analysis of contextual relationships ensures the semantic accuracy of the text sequence.

[0137] The speech symbol sequence is generated based on the speech features extracted previously. In order to ensure the coherence of speech synthesis, the system needs to arrange these speech symbols in chronological order. Each speech symbol corresponds to a time frame of the speech signal. By arranging them in chronological order, the system can generate an orderly speech sequence to ensure that the speech signal can correctly express the text content.

[0138] Syntax analyzers usually use natural language processing technology based on dependency analysis to identify grammatical components in sentences (such as subject, predicate, and object). This type of analyzer can perform structural analysis on input text symbols and annotate their grammatical functions through a pre-trained grammar model.

[0139] The system determines the order of text symbols by analyzing the semantic association and grammatical structure between each text symbol and adjacent symbols. For example, in the sentence "Query account balance", the system will recognize "query" as a verb and "account balance" as an object, and arrange them in the correct word order to generate an orderly text sequence.

[0140] Each phonetic symbol corresponds to a time frame, and the system arranges the phonetic symbols in order according to the time information. For example, if a phonetic symbol corresponds to an audio clip during the pronunciation process, the system will ensure that these clips are arranged in the actual order of pronunciation to ensure the coherence of the speech.

[0141] Example: In an intelligent customer service system, customers may enter different forms of inquiry requests through voice or text, such as "query account balance" or "account balance inquiry". The system needs to process these different input sequences to ensure accurate voice responses. For example:

[0142] Customer A inputs: "Query account balance".

[0143] Customer B inputs: "Account balance inquiry".

[0144] Although the two customers input in different orders, the system recognizes that "query" is a verb and "account balance" is a noun through the grammatical analyzer, and reorders the text symbols based on grammatical functions. The system will parse "query account balance" into a correct request according to the subject-predicate-object structure.

[0145] For each customer's voice input, the system extracts the voice fragments through the voice symbol sequence and arranges them according to the actual pronunciation order. For example, when a customer inputs "account balance inquiry", the system can still arrange them reasonably based on the time information of the voice symbols to ensure natural voice feedback during speech synthesis.

[0146] This embodiment organizes text symbols and voice symbols into orderly text sequences and voice sequences respectively, so that the system can ensure the matching of text and voice, improve the accuracy and fluency of speech synthesis, and effectively avoid the misalignment problem between text and voice, especially in the financial field where accurate information needs to be conveyed.

[0147] In one embodiment, the above S50 includes:

[0148] S501, analyzing the time length of each text symbol in the text sequence;

[0149] S502, analyzing the time frame where each speech symbol in the speech sequence is located;

[0150] S503, using a dynamic time warping module to perform time alignment processing on the time length of the text symbol and the time frame of the voice symbol;

[0151] S504, analyzing the similarity between the text symbol and the voice symbol based on the time alignment result;

[0152] S505, generating a preliminary symbol mapping relationship according to the similarity, and establishing a correspondence table between text symbols and voice symbols.

[0153] In this embodiment, the time length corresponding to each text symbol (such as a word or character) can be estimated based on the position and length of the text symbol in the sentence. For example, the text symbol "query" may be longer or shorter than "balance". The system calculates the time it may occupy in speech synthesis by analyzing the length of each text symbol.

[0154] Each speech symbol in a speech sequence corresponds to a time segment in the speech signal, called a time frame. Each time frame may represent a sound segment in speech (such as the duration of a vowel or consonant). The system analyzes the time frame of each speech symbol to correspond to the time length of the text symbol.

[0155] Dynamic Time Warping (DTW) is a commonly used time series alignment technique that can handle the time difference between text symbols and speech symbols. DTW calculates the best time alignment path by comparing the time length of text symbols and the time frame of speech symbols. For example, if a text symbol is pronounced faster, DTW will adjust the time to ensure that the alignment relationship between the symbol and the speech symbol remains consistent.

[0156] Once the time alignment of the text symbol and the speech symbol is completed, the system will calculate the similarity between them. This similarity can be based on the phonemes of the text symbol and the pronunciation characteristics of the speech symbol. For example, the system will evaluate the degree of match between the pronunciation of a symbol in the text and the corresponding speech symbol to ensure accuracy in speech synthesis.

[0157] Finally, the system generates a preliminary symbol mapping relationship based on the time alignment and similarity analysis between the text symbols and the voice symbols. This mapping relationship shows which voice symbol each text symbol corresponds to. The system records these mapping relationships in the correspondence table for subsequent speech synthesis and further adjustments.

[0158] The time length of a text symbol can be estimated based on the length of the symbol or the distribution of the symbol in the sentence. Through natural language processing technology, the system can analyze the position and role of the symbol in the text, so as to make a reasonable estimate of its time length. The system decomposes the speech signal and divides it into several time frames. The time frame corresponding to each speech symbol can be obtained by analyzing the frequency and amplitude of the speech signal to ensure that the pronunciation process of each symbol can be correctly mapped to the text symbol. The DTW module dynamically adjusts the matching path of the text symbol and the speech symbol according to the difference in time length and time frame. The system finds the optimal alignment path by recursive calculation to minimize the time difference between the text symbol and the speech symbol. Using the phoneme features of the text symbol and the speech symbol (such as the phoneme, frequency of pronunciation, etc.), the system calculates their similarity. The higher the similarity of the symbol pair, the higher the matching degree, which ensures the accuracy in the speech synthesis process. The symbol mapping relationship table records the mapping pairs of text symbols and speech symbols for subsequent speech synthesis or processing steps.

[0159] By aligning the time of text symbols and voice symbols, the system can effectively solve the time mismatch problem in the speech synthesis process and ensure a high degree of correspondence between text and speech. The use of the dynamic time warping module and similarity analysis can further improve the accuracy and naturalness of speech synthesis.

[0160] In one embodiment, the above S60 includes:

[0161] S601, extracting features from the text sequence and the speech sequence respectively, generating high-dimensional feature vectors corresponding to the text symbols and the speech symbols;

[0162] S602, inputting the high-dimensional feature vectors of the text symbol and the voice symbol into a gated attention mechanism module;

[0163] S603, adjusting the attention weight according to the feature difference between the high-dimensional feature vectors of the text symbol and the voice symbol through the gated attention mechanism module, and generating a correlation matrix between the text symbol and the voice symbol according to the attention weight;

[0164] S604, the gated attention mechanism module adjusts the matching relationship between the text symbol and the corresponding voice symbol based on the association matrix, and updates the dependency relationship between the text symbol and the voice symbol based on the adjusted matching relationship;

[0165] S605, the gated attention mechanism module generates a final matching result between the updated text symbol and the voice symbol according to the dependency relationship between the updated text symbol and the voice symbol.

[0166] In this embodiment, each symbol of the text sequence and the voice sequence can be represented as multiple feature dimensions. The feature extraction process is to extract important information such as grammar, semantics, phonemes, intonation, etc. from the text and voice data, and combine these features into a high-dimensional vector. The high-dimensional feature vector is a mathematical representation used to describe the complex properties of each symbol, which contains the multi-dimensional information of the symbol. This step ensures that the information between the text symbol and the voice symbol is input into the subsequent module in the form of a feature vector of the same dimension.

[0167] After extraction, the high-dimensional feature vectors corresponding to the text symbols and speech symbols are input into the gated attention mechanism module. The gated attention mechanism can dynamically adjust the model's attention to each symbol, ensuring that the system focuses on the most important symbols in the speech synthesis process and reduces the impact of irrelevant symbols.

[0168] The gated attention mechanism calculates the attention weight between each pair of symbols by comparing the feature differences between the high-dimensional feature vectors of text symbols and speech symbols. The smaller the feature difference, the higher the match between the text symbol and the corresponding speech symbol, and the greater the attention weight. By adjusting the attention weight, the system can dynamically control the model's attention to different symbols, ensuring that the focus is on those symbols that are critical to speech synthesis.

[0169] The association matrix is ​​a matrix that describes the relationship between text symbols and voice symbols. The values ​​in the matrix represent the degree of match between each pair of symbols. By generating the association matrix through the gated attention mechanism, the system can intuitively represent the strength of the association between the text symbols and the corresponding voice symbols. According to the values ​​in the association matrix, the system can adjust the matching relationship of the symbols to ensure that the high-weighted symbol pairs can match more closely.

[0170] After the matching relationship is adjusted, the system needs to update the dependency between text symbols and voice symbols. The dependency describes the impact of each symbol on other symbols. Especially in natural language processing and speech synthesis, the contextual dependency between symbols is very important. By updating the dependency, the system can ensure the temporal and semantic consistency of text symbols and voice symbols.

[0171] Finally, the system generates the final text symbol and voice symbol matching result based on the adjusted matching relationship. This result will be used in the subsequent speech synthesis step to ensure that the text content can be accurately converted into a natural and coherent speech signal.

[0172] Use deep learning models, such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs), to extract features from text and speech data. The feature extraction model considers the contextual information of the symbol and generates a high-dimensional vector containing grammatical, semantic, phonemic, and pronunciation features. Using a gated attention mechanism based on the Transformer architecture, attention weights are generated by calculating the dot product of the high-dimensional feature vector, and the matching priority of the symbol is adjusted according to the size of the weight. The attention mechanism of the Transformer model is particularly suitable for handling long-distance dependencies between symbols. By calculating the attention weights for each pair of symbols, an association matrix is ​​generated to ensure that the system can visualize the matching between each symbol. The values ​​of the association matrix reflect the temporal and semantic matching between the text symbols and the speech symbols. The system ensures the transfer of contextual information between symbols by modeling the dependencies between symbols. The updated dependencies can help the system preserve the word order and semantic associations in natural language when generating speech.

[0173] In this embodiment, through the dynamic adjustment of the gated attention mechanism, the system can accurately control the matching degree between text symbols and voice symbols, ensuring that the generated speech synthesis results are highly consistent in semantics and phonetics. This mechanism not only improves the accuracy of speech synthesis, but also enhances the system's ability to handle complex contexts and symbol dependencies, and is particularly suitable for scenarios such as financial data report generation and voice query, providing accurate voice feedback.

[0174] In one embodiment, the above S70 includes:

[0175] S701, inputting the text sequence and the speech sequence into a decoder;

[0176] S702, processing the text sequence and the speech sequence frame by frame through the decoder, and generating a speech signal corresponding to each text symbol;

[0177] S703, during the decoding process, adjusting the generated speech signal based on the dependency relationship between the text symbols and the speech symbols to obtain a final speech signal.

[0178] In this embodiment, the consistency of the text sequence and the voice sequence in the time dimension is ensured. Through preprocessing, the system can synchronize the timing of text symbols and voice symbols to avoid time dislocation during voice generation. On the basis of time alignment, the semantic correspondence between text symbols and voice symbols is further established to ensure that the generated voice signal can accurately reflect the meaning of the text content.

[0179] Use time synchronization algorithms, such as dynamic time warping (DTW), to align the time axes of text and speech sequences. By analyzing the temporal features of text symbols and speech symbols, adjust the sequence length and alignment path to ensure the temporal consistency of the two. Combined with natural language processing (NLP) technology, use contextual semantic analysis tools to identify the semantic relationship between text symbols and speech symbols. A semantic embedding-based method can be used to map text and speech symbols to the same semantic space to ensure accurate semantic correspondence.

[0180] The decoder receives input sequences from two different modalities, text and speech, and fuses these two information sources to generate natural and coherent speech signals. The input interface of the decoder is designed so that it can receive data streams of text sequences and speech sequences at the same time. A dual-stream input structure can be used to input the text and speech sequences into the fusion module of the decoder through different channels. Fusion algorithms such as weighted averaging, concatenation, or attention mechanisms are used to effectively integrate the information of text and speech sequences as input features of the decoder. The decoder processes the text sequence and speech sequence frame by frame and generates a speech signal corresponding to each text symbol.

[0181] The decoder processes the input sequence step by step according to the time frame to ensure the coherence and naturalness of the speech signal. The generation of each frame is based on the input of the current frame and the context information of the previous frame. It ensures that each text symbol corresponds to a specific speech signal segment, achieving accurate conversion from text to speech.

[0182] The decoder based on recurrent neural network (RNN), long short-term memory network (LSTM) or Transformer architecture has the ability to process sequence data. When generating each frame of speech signal, the decoder uses the generation result of the previous frame and the input symbol of the current frame to gradually build a complete speech signal. Use a symbol-to-speech signal mapping model, such as vocoders, to convert the symbol sequence into a specific audio waveform. During the decoding process, the generated speech signal is adjusted based on the dependency between text symbols and speech symbols to obtain the final speech signal.

[0183] By using the dependency between text symbols and speech symbols, the generation process of speech signals is dynamically adjusted to ensure that the intonation, rhythm and pronunciation of the speech are highly matched with the text content. By adjusting the dependency, the naturalness and clarity of the generated speech are improved, and errors and incoherence in speech synthesis are reduced.

[0184] The dependency model is embedded in the decoder, and the attention mechanism or conditional generation model is used to capture the complex dependency between text symbols and speech symbols. According to the output of the dependency model, the generation parameters of the decoder, such as intonation, pause position, and volume control, are adjusted in real time to optimize the quality of the speech signal. Speech enhancement and noise suppression techniques are applied to further improve the clarity and naturalness of the generated speech.

[0185] This embodiment achieves high-precision alignment and matching between text and speech by gated attention processing of text sequences and speech sequences. Using the gated attention mechanism, the system can dynamically adjust the matching degree between text symbols and speech symbols to ensure that the generated speech signal is highly consistent with the input text in intonation, rhythm and pronunciation. This not only improves the accuracy and naturalness of speech synthesis, but also enhances the system's adaptability in complex financial scenarios, such as handling diverse customer inquiries and transaction confirmation requirements, significantly improving the user experience and service quality of the intelligent customer service system.

[0186] In one embodiment, a speech synthesis device based on a gated attention mechanism is provided, and the speech synthesis device based on the gated attention mechanism corresponds one-to-one to the speech synthesis method based on the gated attention mechanism in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a speech synthesis device based on a gated attention mechanism of the present invention. Data acquisition module 10, text encoding module 20, speech encoding module 30, sequence organization module 40, alignment module 50, gated attention mechanism module 60 and decoder module 70. Each functional module is described in detail as follows:

[0187] A data acquisition module 10, used to acquire text data and voice data;

[0188] A text encoding module 20, used for inputting the text data into a text encoder, and converting the text data into a text symbol sequence through the text encoder;

[0189] A speech coding module 30, configured to input the speech data into a speech encoder, extract speech features from the speech data through the speech encoder, and quantize the speech features into a discrete speech symbol sequence;

[0190] A sequence organization module 40, used for organizing the text symbol sequence into a text sequence and organizing the voice symbol sequence into a voice sequence;

[0191] A preliminary alignment module 50, used for performing preliminary alignment processing on the text sequence and the speech sequence to establish a corresponding relationship between the text symbols and the speech symbols;

[0192] A gated attention mechanism module 60 is used to perform gated attention processing on the preliminarily aligned text sequence and speech sequence to adjust the matching degree between the text symbols and the speech symbols;

[0193] The decoder module 70 is used to input the text sequence and speech sequence processed by gated attention into the decoder to generate the final speech signal.

[0194] In one embodiment, the text encoding module 20 is specifically used for:

[0195] Performing a preprocessing operation on the text data, the preprocessing operation including format standardization and text standardization;

[0196] Inputting the preprocessed text data into a text encoder, and decomposing the text data into separate words or symbols by a word segmenter in the text encoder;

[0197] The decomposed words or symbols are converted into a predefined sequence of text units through a byte pair encoding module in a text encoder;

[0198] Mapping the text unit sequence to a predefined symbol table to generate a text symbol sequence;

[0199] The text symbol sequence is indexed and the indexed text symbol sequence is stored in a memory.

[0200] In one embodiment, the speech encoding module 30 is specifically configured to:

[0201] Performing a preprocessing operation on the speech data, wherein the preprocessing operation includes removing background noise and signal normalization processing;

[0202] Input the preprocessed speech data into a speech encoder trained based on a self-supervised learning model;

[0203] Decomposing a continuous speech signal in the speech data into time frames of fixed length by the speech encoder;

[0204] Perform feature extraction on the speech data in each time frame to extract key speech features including Mel frequency cepstral coefficients, fundamental frequency and energy;

[0205] The key speech features are quantized and converted into discrete speech symbol sequences.

[0206] In one embodiment, the sequence organization module 40 is specifically configured to:

[0207] Analyzing the text symbol sequence by a grammatical analyzer to identify and annotate the grammatical function of each text symbol;

[0208] Arrange the text symbols in sequence based on the grammatical function and contextual relationship of each text symbol to generate an ordered text sequence that conforms to semantic logic, wherein the contextual relationship represents the coordination relationship between the text symbol and adjacent symbols in terms of grammatical structure, semantic association and word order;

[0209] The speech symbol sequence is arranged in chronological order to generate an ordered speech sequence.

[0210] In one embodiment, the preliminary alignment module 50 is specifically configured to:

[0211] Analyze the duration of each text symbol in the text sequence;

[0212] Analyze the time frame of each speech symbol in the speech sequence;

[0213] Use the dynamic time warping module to time-align the time length of text symbols with the time frame of speech symbols;

[0214] Based on the time alignment results, analyze the similarity between text symbols and speech symbols;

[0215] A preliminary symbol mapping relationship is generated according to the similarity, and a correspondence table between text symbols and voice symbols is established.

[0216] In one embodiment, the gated attention mechanism module 60 is specifically configured to:

[0217] Perform feature extraction on the text sequence and the speech sequence respectively to generate high-dimensional feature vectors corresponding to the text symbols and the speech symbols;

[0218] Input the high-dimensional feature vectors of text symbols and speech symbols into the gated attention mechanism module;

[0219] The gated attention mechanism module adjusts the attention weight according to the feature difference between the high-dimensional feature vectors of the text symbol and the voice symbol, and generates a correlation matrix between the text symbol and the voice symbol according to the attention weight;

[0220] The gated attention mechanism module adjusts the matching relationship between the text symbol and the corresponding voice symbol based on the association matrix, and updates the dependency relationship between the text symbol and the voice symbol based on the adjusted matching relationship;

[0221] The gated attention mechanism module generates a final matching result between the updated text symbols and the voice symbols according to the dependency relationship between the updated text symbols and the voice symbols.

[0222] In one embodiment, the decoder module 70 is specifically configured to:

[0223] Input the text sequence and the speech sequence into a decoder;

[0224] Processing the text sequence and the speech sequence frame by frame through the decoder, and generating a speech signal corresponding to each text symbol;

[0225] During the decoding process, the generated speech signal is adjusted based on the dependency between the text symbols and the speech symbols to obtain the final speech signal.

[0226] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a speech synthesis method based on a gated attention mechanism.

[0227] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the user side of a speech synthesis method based on a gated attention mechanism

[0228] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0229] Acquire text data and voice data;

[0230] Inputting the text data into a text encoder, and converting the text data into a text symbol sequence by the text encoder;

[0231] Inputting the speech data into a speech encoder trained based on a self-supervised learning model, extracting speech features from the speech data through the speech encoder, and quantizing the speech features into a discrete speech symbol sequence;

[0232] Organizing the text symbol sequence into a text sequence and organizing the voice symbol sequence into a voice sequence;

[0233] Performing preliminary alignment processing on the text sequence and the voice sequence to establish a corresponding relationship between the text symbols and the voice symbols;

[0234] Perform gated attention processing on the preliminarily aligned text sequence and speech sequence to adjust the matching degree between text symbols and speech symbols;

[0235] The text sequence and speech sequence processed by gated attention are input into the decoder to generate the final speech signal.

[0236] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0237] Acquire text data and voice data;

[0238] Inputting the text data into a text encoder, and converting the text data into a text symbol sequence by the text encoder;

[0239] Inputting the speech data into a speech encoder trained based on a self-supervised learning model, extracting speech features from the speech data through the speech encoder, and quantizing the speech features into a discrete speech symbol sequence;

[0240] Organizing the text symbol sequence into a text sequence and organizing the voice symbol sequence into a voice sequence;

[0241] Performing preliminary alignment processing on the text sequence and the voice sequence to establish a corresponding relationship between the text symbols and the voice symbols;

[0242] Perform gated attention processing on the preliminarily aligned text sequence and speech sequence to adjust the matching degree between text symbols and speech symbols;

[0243] The text sequence and speech sequence processed by gated attention are input into the decoder to generate the final speech signal.

[0244] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0245] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0246] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0247] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech synthesis method based on a gated attention mechanism, characterized in that: The following steps are involved: Acquire text data and voice data; Inputting the text data into a text encoder, and converting the text data into a text symbol sequence by the text encoder; Inputting the speech data into a speech encoder trained based on a self-supervised learning model, extracting speech features from the speech data through the speech encoder, and quantizing the speech features into a discrete speech symbol sequence; Organizing the text symbol sequence into a text sequence and organizing the voice symbol sequence into a voice sequence; Performing preliminary alignment processing on the text sequence and the voice sequence to establish a corresponding relationship between the text symbols and the voice symbols; Perform gated attention processing on the preliminarily aligned text sequence and speech sequence to adjust the matching degree between text symbols and speech symbols; The text sequence and speech sequence processed by gated attention are input into the decoder to generate the final speech signal.

2. The speech synthesis method based on the gated attention mechanism according to claim 1, characterized in that: Inputting the text data into a text encoder, and converting the text data into a text symbol sequence by the text encoder, comprises: Performing a preprocessing operation on the text data, the preprocessing operation including format standardization and text standardization; Inputting the preprocessed text data into a text encoder, and decomposing the text data into separate words or symbols by a word segmenter in the text encoder; The decomposed words or symbols are converted into a predefined sequence of text units through a byte pair encoding module in a text encoder; Mapping the text unit sequence to a predefined symbol table to generate a text symbol sequence; The text symbol sequence is indexed and the indexed text symbol sequence is stored in a memory.

3. The speech synthesis method based on the gated attention mechanism according to claim 1, characterized in that: The speech data is input into a speech encoder trained based on a self-supervised learning model, speech features are extracted from the speech data by the speech encoder, and the speech features are quantized into a discrete speech symbol sequence, including: Performing a preprocessing operation on the speech data, wherein the preprocessing operation includes removing background noise and signal normalization processing; Input the preprocessed speech data into a speech encoder trained based on a self-supervised learning model; Decomposing a continuous speech signal in the speech data into time frames of fixed length by the speech encoder; Perform feature extraction on the speech data in each time frame to extract key speech features including Mel frequency cepstral coefficients, fundamental frequency and energy; The key speech features are quantized and converted into discrete speech symbol sequences.

4. The speech synthesis method based on the gated attention mechanism according to claim 1, characterized in that: Organizing the text symbol sequence into a text sequence and organizing the voice symbol sequence into a voice sequence, comprising: Analyzing the text symbol sequence by a grammatical analyzer to identify and annotate the grammatical function of each text symbol; Arrange the text symbols in sequence based on the grammatical function and contextual relationship of each text symbol to generate an ordered text sequence that conforms to semantic logic, wherein the contextual relationship represents the coordination relationship between the text symbol and adjacent symbols in terms of grammatical structure, semantic association and word order; The speech symbol sequence is arranged in chronological order to generate an ordered speech sequence.

5. The speech synthesis method based on gated attention mechanism as described in claim 1, characterized in that: Performing preliminary alignment processing on the text sequence and the voice sequence to establish a corresponding relationship between the text symbols and the voice symbols, including: Analyze the duration of each text symbol in the text sequence; Analyze the time frame of each speech symbol in the speech sequence; Use the dynamic time warping module to time-align the time length of text symbols with the time frame of speech symbols; Based on the time alignment results, analyze the similarity between text symbols and speech symbols; A preliminary symbol mapping relationship is generated according to the similarity, and a correspondence table between text symbols and voice symbols is established.

6. The speech synthesis method based on gated attention mechanism as claimed in claim 1, characterized in that: Perform gated attention processing on the preliminarily aligned text sequence and speech sequence to adjust the matching degree between text symbols and speech symbols, including: Perform feature extraction on the text sequence and the speech sequence respectively to generate high-dimensional feature vectors corresponding to the text symbols and the speech symbols; Input the high-dimensional feature vectors of text symbols and speech symbols into the gated attention mechanism module; The gated attention mechanism module adjusts the attention weight according to the feature difference between the high-dimensional feature vectors of the text symbol and the voice symbol, and generates a correlation matrix between the text symbol and the voice symbol according to the attention weight; The gated attention mechanism module adjusts the matching relationship between the text symbol and the corresponding voice symbol based on the association matrix, and updates the dependency relationship between the text symbol and the voice symbol based on the adjusted matching relationship; The gated attention mechanism module generates a final matching result between the updated text symbols and the voice symbols according to the dependency relationship between the updated text symbols and the voice symbols.

7. The speech synthesis method based on gated attention mechanism as claimed in claim 1, characterized in that: The text sequence and speech sequence processed by gated attention are input into the decoder to generate the final speech signal, including: Input the text sequence and the speech sequence into a decoder; Processing the text sequence and the speech sequence frame by frame through the decoder, and generating a speech signal corresponding to each text symbol; During the decoding process, the generated speech signal is adjusted based on the dependency between the text symbols and the speech symbols to obtain the final speech signal.

8. A speech synthesis device based on a gated attention mechanism, characterized in that: The speech synthesis device based on the gated attention mechanism includes: A data acquisition module, used to acquire text data and voice data; A text encoding module, used for inputting the text data into a text encoder, and converting the text data into a text symbol sequence through the text encoder; A speech coding module, used for inputting the speech data into a speech encoder, extracting speech features from the speech data through the speech encoder, and quantizing the speech features into a discrete speech symbol sequence; A sequence organization module, used for organizing the text symbol sequence into a text sequence and organizing the voice symbol sequence into a voice sequence; A preliminary alignment module, used for performing preliminary alignment processing on the text sequence and the voice sequence to establish a corresponding relationship between the text symbols and the voice symbols; The gated attention mechanism module is used to perform gated attention processing on the preliminarily aligned text sequence and speech sequence to adjust the matching degree between text symbols and speech symbols; The decoder module is used to input the text sequence and speech sequence processed by gated attention into the decoder to generate the final speech signal.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a speech synthesis program based on a gated attention mechanism stored in the memory and executable on the processor. When the speech synthesis program based on the gated attention mechanism is executed by the processor, the steps of the speech synthesis method based on the gated attention mechanism as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a speech synthesis program based on a gated attention mechanism, and when the speech synthesis program based on a gated attention mechanism is executed by a processor, the steps of the speech synthesis method based on a gated attention mechanism as described in any one of claims 1-7 are implemented.

Images