Voice synthesis method and device based on voice token fusion, equipment and medium

Abstract

The invention relates to the technical field of speech semantics, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses a speech synthesis method, device, equipment and medium based on speech token fusion. Performing text coding on the initial potential representation to obtain a target text feature; generating a semantic token corresponding to the initial text according to the target text feature, and performing time sequence alignment on the semantic token and the target text feature to obtain a target semantic token; obtaining user voice of a reference user, extracting timbre characteristics of the user voice, and generating a Mel spectrogram frame by frame according to the timbre characteristics and the target semantic token; and performing speech synthesis according to the Mel spectrogram to obtain target speech. The speech synthesis efficiency and quality can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech semantics, and in particular to a speech synthesis method and device based on speech token fusion, equipment and medium. BACKGROUND

[0002] With the continuous progress of science and technology, speech synthesis technology has been gradually applied to business scenarios such as financial technology and medical health. However, the existing technology mostly uses a "forward-looking mechanism" for speech synthesis, but this mechanism requires the advance acquisition of predicted text information, resulting in high processing delay in the synthesis process. For example, in scenarios such as real-time consultation in financial transactions and emergency information broadcasting in medical emergency scenarios, which have extremely high requirements for real-time interaction, high delay will seriously affect the timeliness of speech information transmission, resulting in incoherent dialogue and delayed information feedback, greatly reducing user experience. In addition, most current speech synthesis models are based on non-streaming architecture, which will cause unstable speech generation and degraded sound quality when streaming conversion.

[0003] For example, during the peak period of financial stock transactions, customers urgently need to understand the real-time market situation and operation suggestions of a certain stock. The speech synthesis model needs to acquire subsequent related text information in advance to synthesize speech, resulting in a delay of several seconds before the customer can hear the complete reply after asking, and the customer service personnel also need to confirm repeatedly, resulting in serious lag in information feedback and greatly reducing user experience.

[0004] For example, in the emergency information broadcasting scenario in the medical emergency scenario, the problems of most speech synthesis models based on non-streaming architecture are highlighted. When the emergency personnel acquire the patient's condition and emergency measures through the speech system, the non-streaming model is extremely unstable in the streaming conversion process, the sound quality is greatly degraded, there are obvious noise and distortion, and the inappropriate emergency action may be taken due to information errors, delaying the treatment opportunity of the injured.

[0005] Therefore, how to improve the speech quality and synthesis efficiency of target speech synthesis has become a problem to be solved. SUMMARY

[0006] The present application provides a speech synthesis method, device, equipment and medium based on speech token fusion, which mainly aims to solve the problems of low semantic synthesis efficiency and low synthesis speech quality.

[0007] In a first aspect, to achieve the above-mentioned purpose, the present application provides a speech synthesis method based on speech token fusion, comprising: acquiring an initial text, converting the initial text into an initial latent representation, and performing text encoding on the initial latent representation to obtain target text features; A semantic token for the initial text is generated based on the target text features, and the semantic token is temporally aligned with the target text features to obtain the target semantic token. Acquire the user's voice from the reference user, extract the timbre features of the user's voice, and generate a Mel spectrogram frame by frame based on the timbre features and the target semantic token; Speech synthesis is performed based on the Mel spectrogram to obtain the target speech.

[0008] Secondly, the present invention also provides a speech synthesis device based on voice token fusion, comprising: The text feature encoding module is used to acquire initial text, convert the initial text into an initial latent representation, and perform text encoding on the initial latent representation to obtain target text features; A semantic token generation module is used to generate a semantic token for the initial text based on the target text features, and to perform temporal alignment between the semantic token and the target text features to obtain a target semantic token; The Mel spectrogram construction module is used to acquire user speech from a reference user, extract the timbre features of the user speech, and generate a Mel spectrogram frame by frame based on the timbre features and the target semantic token. The target speech synthesis module is used to synthesize speech based on the Mel spectrogram to obtain the target speech.

[0009] Thirdly, the present invention also provides an electronic device, the electronic device comprising: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the speech synthesis method based on voice token fusion as described above.

[0010] Fourthly, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the above-described speech synthesis method based on voice token fusion.

[0011] In this embodiment of the invention, converting the initial text into an initial latent representation can accurately capture the semantic associations and positional information of words in the text, effectively extracting the preliminary semantic features of the text; text encoding of the initial latent representation can significantly improve the accuracy and efficiency of subsequent speech generation, enhancing the computer's ability to understand and process text data; generating a semantic token for the initial text based on the target text features achieves a high degree of abstraction and condensation of text semantics, enhancing the accuracy and completeness of semantic expression; acquiring the user's voice from a reference user and extracting timbre features lays the foundation for personalized speech processing, enabling a deep understanding of the unique attributes of the user's voice; generating a Mel spectrogram frame by frame based on timbre features and the target semantic token further greatly improves the quality and efficiency of speech processing; the Mel spectrogram can accurately capture the spectral features of speech, highly restoring the timbre, pitch, and other details of the original speech, making the generated target speech more natural and realistic, greatly enhancing the auditory experience of speech; the Mel spectrogram provides a structured data foundation for speech synthesis, improving synthesis speed, reducing computational resource consumption, and enhancing the flexibility and scalability of speech synthesis. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of an application environment for a speech synthesis method based on voice token fusion according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating a speech synthesis method based on voice token fusion, provided as an embodiment of the present invention. Figure 3 This is a schematic diagram of a process for generating a Mel spectrogram frame by frame based on the timbre features and the target semantic token, according to an embodiment of the present invention. Figure 4 A schematic diagram of a speech synthesis device based on voice token fusion according to an embodiment of the present invention; Figure 5 A schematic diagram of the structure of an electronic device that implements a speech synthesis method based on voice token fusion, according to an embodiment of the present invention; Figure 6 This is another structural schematic diagram of an electronic device that implements a speech synthesis method based on voice token fusion, as provided in an embodiment of the present invention.

[0014] The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0015] To enable those skilled in the art to better understand the technical solutions of this disclosure, and to fully understand and implement the process of how this disclosure applies technical means to solve technical problems and achieve corresponding technical effects, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, not all embodiments. The embodiments of this disclosure and the various features within them can be combined with each other without conflict, and the resulting technical solutions are all within the protection scope of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort should fall within the protection scope of this disclosure.

[0016] It should be noted that the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.

[0017] This application provides a speech synthesis method based on voice token fusion. The execution entity of this speech synthesis method includes, but is not limited to, at least one electronic device that can be configured to execute the device provided in this application, such as a server or a terminal. In other words, the speech synthesis method based on voice token fusion can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0018] This invention discloses a speech synthesis method based on voice token fusion, which can be applied to applications such as... Figure 1In this application environment, the client communicates with the server via a network. The server can obtain initial text from the client, convert the initial text into an initial latent representation, encode the initial latent representation to obtain target text features, generate a semantic token for the initial text based on the target text features, and align the semantic token with the target text features temporally to obtain the target semantic token. The server can also obtain the user's speech from a reference user, extract the timbre features of the user's speech, and generate a Mel spectrogram frame by frame based on the timbre features and the target semantic token. Speech synthesis is then performed based on the Mel spectrogram to obtain the target speech, thereby improving speech synthesis efficiency and speech quality. Finally, the target speech is output and fed back to the client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates this invention.

[0019] Reference Figure 2 The diagram shown is a flowchart illustrating a speech synthesis method based on voice token fusion according to an embodiment of the present invention. In this embodiment, the speech synthesis method based on voice token fusion includes: S1. Obtain the initial text, convert the initial text into an initial latent representation, and perform text encoding on the initial latent representation to obtain the target text features.

[0020] In this embodiment of the invention, the initial text refers to the original text material that has not been processed, modified, or analyzed in any way. It can be in various forms such as books, articles, reports, dialogue records, and social media posts. Among them, relevant texts can be downloaded by searching academic databases using keywords, or the platform's own search function or web crawler tools can be used to crawl relevant texts published by users according to specific topics and keywords.

[0021] In this embodiment of the invention, converting the initial text into an initial latent representation includes: The initial text is segmented to obtain a text word sequence; Add positional encoding to each subword in the text word sequence to obtain the target input sequence; The semantic association between each word in the target input sequence and all words in the target input sequence is analyzed using the multi-head self-attention function in the preset encoder to obtain the text association sequence; A nonlinear transformation is performed on the text association sequence to obtain the initial potential representation corresponding to the initial text.

[0022] In detail, the word segmentation process is the process of dividing a continuous initial text into words with independent semantics. A dictionary containing a large number of words can be pre-constructed. When processing the initial text, starting from the beginning of the initial text, words that match the continuous characters in the initial text are searched in the dictionary according to certain rules (such as maximum matching method, minimum matching method, etc.).

[0023] Among them, the maximum matching method will match the longest word possible, while the minimum matching method will do the opposite. For example, for the initial text "I love natural language processing", if the word "natural language processing" exists in the dictionary, the maximum matching method will segment it as a whole, thereby achieving word segmentation of the initial text.

[0024] Specifically, positional encoding is used to identify the location information of words in text. A common positional encoding method is to assign a unique vector representation to each word at each position. The vector representation is usually generated by sine and cosine functions, and different positions are distinguished by different combinations of frequencies.

[0025] For example, for a word at position 1, its position encoding vector may be calculated from a set of sine and cosine values ​​of a specific frequency; for a word at position 2, another set of values ​​of different frequencies is used. Based on the position encoding vector, the position of each word in the text can be accurately determined, thereby better understanding the semantics and structure of the text.

[0026] Furthermore, the multi-head self-attention mechanism is the core part of the encoder, which can capture the complex semantic relationships between words in the text. In the target input sequence, each sub-word will interact with all other sub-words and calculate the attention weights between them. The calculation of the attention weights is based on the similarity between sub-words. The encoder maps the vector representations of sub-words to different spaces through a series of linear transformations, calculates the dot product between these sub-word vectors, and then obtains the attention weight matrix after normalization.

[0027] The attention weight matrices of multiple heads are concatenated and linearly transformed to obtain the comprehensive semantic association information between each subword and all subwords, thereby generating a text association sequence. The nonlinear transformation of the text association sequence is to map the text association sequence to a new space to obtain a more abstract and meaningful representation. Commonly used nonlinear transformation methods include activation functions such as ReLU (Rectified Linear Unit), Sigmoid, and Tanh.

[0028] Specifically, the ReLU function sets the less than zero part of the input value to zero, while keeping the greater than zero part unchanged. This can effectively increase the expressive power of the encoder, while avoiding the gradient vanishing problem. It can further refine and compress the semantic information in the text to obtain the initial latent representation corresponding to the initial text.

[0029] Furthermore, text encoding of the initial latent representation to obtain target text features can be achieved using a convolutional neural network (CNN). By sliding convolutional kernels over the initial latent representation, local feature extraction is performed. Convolutional kernels of different sizes can capture text features of different granularities. Small convolutional kernels focus on local details, while large convolutional kernels can obtain broader contextual information. After pooling operations, such as max pooling, key features are preserved and dimensionality is reduced, ultimately yielding more representative target text features.

[0030] For example, in a healthcare scenario, assuming the initial text is "The patient has recently experienced persistent cough and fever symptoms, and has not seen improvement after taking regular cold medicine", it is segmented into words to obtain a text word sequence of "patient, recently, appeared, persistent, cough, fever, symptoms, and, took, regular, cold medicine, after, not seen, improved". Then, positional encoding is added to each sub-word to clarify its position information in the sentence, forming the target input sequence. The multi-head self-attention mechanism in the preset encoder is used to analyze the semantic relationship between each sub-word and all sub-words in the sequence.

[0031] For example, "cough" and "fever" are strongly associated because they belong to the same symptom description, and "taking" and "cold medicine" are closely linked because of the semantic relationship between the action and the object, thus obtaining a text association sequence. Then, a non-linear transformation is performed on the text association sequence to transform the linear relationship into a complex non-linear relationship, mining deeper semantic information and finally obtaining an initial potential representation. This helps medical AI systems to understand the patient's condition description more accurately and provides a reliable semantic foundation for subsequent diagnostic suggestions, disease prediction, etc.

[0032] For example, in a fintech scenario, if the initial text is "The stock market has been fluctuating significantly recently, and tech stocks have fallen sharply due to policy influences. Investors should exercise caution," after word segmentation, a sequence of text words is obtained: "recently, stock market, fluctuations, significant, and, tech stocks, affected by, policy, influence, fall, obvious, investors, need, caution, operation." After adding positional encoding to form the target input sequence, semantic associations are analyzed through a multi-head self-attention mechanism. After nonlinear transformation, an initial latent representation is obtained, which can provide rich semantic information for financial analysis models, assist in market trend analysis, investment risk assessment, etc., help financial institutions and investors make more informed decisions, and improve the intelligence level of fintech services.

[0033] In this embodiment of the invention, the initial text is converted into an initial latent representation. Through mechanisms such as word segmentation, positional encoding, and multi-head self-attention, the semantic associations and positional information of words in the text can be accurately captured, and the preliminary semantic features of the text can be effectively extracted. Text encoding of the initial latent representation can further mine the deeper semantic information of the text, obtain more representative and distinguishable target text features, significantly improve the accuracy and efficiency of subsequent speech generation, and enhance the computer's ability to understand and process text data.

[0034] S2. Generate a semantic token for the initial text based on the target text features, and perform temporal alignment between the semantic token and the target text features to obtain the target semantic token.

[0035] In this embodiment of the invention, text is transformed into a fused feature matrix through feature enhancement, semantic dimension mapping and feature fusion, then encoded into a semantic token, and the semantic token is time-aligned to enhance the semantic representation and temporal consistency of the token.

[0036] In this embodiment of the invention, generating the semantic token of the initial text based on the target text features includes: The target text features are enhanced to obtain an enhanced text feature set; The semantic dimension corresponding to the enhanced text feature set is determined based on a preset semantic hierarchy system; Based on the semantic dimension, feature mapping is performed on the enhanced text feature set to generate a semantic feature vector under each semantic dimension; The semantic feature vectors are fused to obtain a fused feature matrix; The fusion feature matrix is ​​encoded and converted based on a preset token generation strategy to generate a semantic token for the initial text.

[0037] In detail, feature enhancement of target text features aims to improve the quality and richness of target text features. Attention mechanisms can be used to calculate the correlation weights between target text feature vectors, assign higher weights to important features, and suppress irrelevant features, thereby obtaining an enhanced text feature set.

[0038] Specifically, the pre-defined semantic hierarchy is a division of the semantic structure of a text, which usually includes different levels such as words, phrases, sentences, and paragraphs. To enhance the text feature set, it is necessary to map it into this hierarchy to determine the corresponding semantic dimensions. This can be achieved by analyzing the degree of correlation between features and semantic units at different levels.

[0039] For example, clustering algorithms can be used to group features according to similarity, with each group corresponding to a semantic dimension. Alternatively, rule matching methods can be used to match features with predefined rules in the semantic hierarchy (such as specific word combinations corresponding to phrase levels) to determine the semantic dimension to which a feature belongs.

[0040] Furthermore, for each semantic dimension, a linear transformation can be used for mapping. The linear transformation transforms the feature vector to a new semantic dimension space through matrix multiplication. For example, a weight matrix can be used to transform the feature vector so that the transformed vector can better represent the features under that semantic dimension. The semantic feature vectors under different semantic dimensions can be integrated into a unified feature representation. Specifically, the various semantic feature vectors can be connected in a certain order to form a longer vector, i.e., a fused feature matrix.

[0041] Specifically, the preset token generation strategy specifies how to convert the fused feature matrix into a semantic token. The feature vectors in the fused feature matrix can be clustered, with each cluster center corresponding to a semantic token. By calculating the distance between the feature vector and each cluster center, the feature vector is assigned to the nearest cluster center. The resulting semantic token can effectively map similar fused feature matrices to the same semantic token, improving the representativeness and interpretability of the semantic token.

[0042] In this embodiment of the invention, the step of temporally aligning the semantic token with the target text features to obtain the target semantic token includes: Based on a preset alignment granularity rule, the temporal alignment levels between the semantic token and the target text features are divided, and the temporal tagging information of the semantic token is extracted. The target text features are analyzed temporally to obtain temporal structure features, and the time span and temporal sequence relationship of each time feature unit in the temporal structure features are extracted. In the fine alignment layer of the time alignment hierarchy, the time sequence marker information and the time feature unit are matched point by point according to the time span and the time sequence relationship to obtain misalignment tokens and misalignment features; In the coarse-grained alignment layer of the time alignment hierarchy, the misaligned token and the misaligned feature are subjected to time compensation processing to generate a time alignment matrix; The target semantic token is obtained by filtering the time alignment matrix according to the preset token confidence threshold.

[0043] In detail, the preset alignment granularity rule is a specification for the temporal alignment process, which determines the level of detail of the temporal alignment and can divide the temporal alignment of semantic tokens and target text features into different levels; for example, in the character-level alignment level, the focus is on the position and temporal order of each character in the text; in the sentence-level alignment level, more attention is paid to the temporal correspondence of the entire sentence.

[0044] Specifically, temporal marker information can be obtained by adding a timestamp or position index to each semantic token. The timestamp records the time point when the semantic token appears in the original text sequence, while the position index indicates its position order in the text. For text features, temporal structure may be reflected in the logical order between sentences, the order of events, etc. The temporal structure in the text can be identified through natural language processing techniques, such as syntactic analysis and semantic role labeling.

[0045] Among them, time span refers to the length of time that an event or sentence occupies in the time sequence, which can be determined by analyzing time expressions in the text (such as "three days later", "10 a.m.", etc.) or by logical inference based on the text content; chronological order is to determine the order of different time feature units in the time sequence, for example, by identifying conjunctions in the text (such as "then", "after", etc.) or causal relationships of events.

[0046] Furthermore, the fine alignment layer focuses on the details of temporal alignment. Point-by-point matching is the process of matching the temporal tag information of semantic tokens with the temporal feature units in the target text features. Based on the previously extracted time span and temporal sequence, the temporal tag information of each semantic token and its corresponding temporal feature unit are compared and matched.

[0047] Specifically, if the temporal tag information of a semantic token does not match the temporal feature unit in terms of time span or temporal sequence, misaligned tokens and misaligned features will be generated. For example, if the temporal tag of a semantic token indicates that it should appear in a certain time period, but the corresponding temporal feature unit appears in another time period, then this semantic token is regarded as a misaligned token, and the corresponding temporal feature unit is a misaligned feature.

[0048] Furthermore, the coarse-grained alignment layer adjusts and compensates for temporal alignment as a whole. Temporal compensation is to solve the misalignment problem that occurs in the fine alignment layer. It can be achieved by interpolation, that is, by inserting new semantic tokens or adjusting temporal feature units between misaligned tokens and misaligned features, so that they can correspond better in time. For example, if a semantic token is found to appear too early in time, a semantic token indicating delay can be inserted in front of it to make it conform to the overall temporal pattern.

[0049] In detail, the temporal correspondence between the processed semantic tokens and the target text features is represented in matrix form. The rows and columns of the matrix can represent semantic tokens and temporal feature units, respectively, and the elements in the matrix represent the correspondence or similarity between them. In the temporal alignment matrix, each semantic token has a corresponding confidence value, which reflects the degree of temporal and semantic matching between the semantic token and the target text features. Semantic tokens with confidence values ​​lower than a preset threshold in the temporal alignment matrix are filtered out, and only semantic tokens with confidence values ​​higher than the threshold are retained to obtain the target semantic token. This ensures that the obtained target semantic token has high reliability and accuracy and can better represent the temporal and semantic information of the original text.

[0050] In this embodiment of the invention, a semantic token for the initial text is generated based on the features of the target text, which achieves a high degree of abstraction and condensation of the text semantics. The semantic token captures the core semantic information of the text in a concise and representative form, which greatly reduces the data dimensionality, improves the efficiency of computer text processing, and enhances the accuracy and completeness of semantic expression.

[0051] S3. Obtain the user's voice from the reference user, extract the timbre features of the user's voice, and generate a Mel spectrogram frame by frame based on the timbre features and the target semantic token.

[0052] In this embodiment of the invention, a dedicated voice acquisition device, such as a high-quality microphone, can be used to record the user's voice. During the acquisition, it is necessary to ensure that the environment is relatively quiet and reduce background noise interference in order to obtain clear and accurate voice. At the same time, voice acquisition software can be used to perform preliminary processing on the acquired voice, such as noise reduction and gain adjustment, to improve the voice quality.

[0053] For example, in the field of healthcare, obtaining user voice data from reference users plays a crucial role. For instance, in a project to build an intelligent consultation system in a large general hospital, the system needs to provide personalized services for different patients. For an elderly patient with chronic obstructive pulmonary disease (COPD), medical staff will guide him to engage in voice interaction.

[0054] First, in a quiet consultation room with relatively minimal echo interference, the patient's voice is captured using professional voice acquisition equipment, such as a high-sensitivity microphone array. During the acquisition process, medical staff patiently guide the patient to describe their symptoms naturally and clearly, such as the frequency of coughing, the color and texture of sputum, and the degree of shortness of breath. After the voice is acquired, the system preprocesses it to remove background noise and background sounds, ensuring the purity of the voice. This voice data is then stored in the hospital's dedicated database as the patient's reference voice. When the patient uses the intelligent consultation system again, the system can better identify the patient's vocal characteristics, including tone, speech rate, and accent, based on the previously acquired reference voice, thus more accurately understanding the patient's intentions and assisting doctors in determining whether the condition has improved or worsened, providing strong support for accurate diagnosis and treatment.

[0055] For example, in a fintech scenario, taking a bank's intelligent customer service system as an example, when a customer calls the bank's customer service hotline to inquire about complex financial product information, such as the calculation method of returns and risk levels of wealth management products, the intelligent customer service system will initiate a voice acquisition process. The acquired voice will be encrypted to ensure the security and privacy of customer information, and then stored in the bank's financial big data platform to form the customer's reference voice. For example, based on the customer's past consultation preferences, the system can proactively recommend suitable financial products. Moreover, the reference voice can also be used to analyze the customer's emotional state. If the customer's tone is urgent or angry, the system can promptly transfer the call to a human customer service representative, improving customer satisfaction and enhancing the bank's competitiveness in the fintech service field.

[0056] In this embodiment of the invention, extracting the timbre features of the user's speech includes: Extract the raw audio data from the user's speech and perform audio enhancement processing on the raw audio data to obtain an enhanced speech signal; The enhanced speech signal is divided into frames according to a preset duration to obtain multiple signal frames, and the temporal features of each signal frame are extracted. Perform a Fourier transform on each signal frame to obtain the spectral characteristics of the signal frame, and extract the formant parameters from the spectral characteristics; The formant parameters and the time-domain features are fused to obtain the timbre features of the user's speech.

[0057] In detail, the extraction of raw audio data usually relies on audio acquisition devices, such as microphones. When a user speaks, the microphone converts the sound waves into electrical signals, and then the analog electrical signals are converted into digital signals by an analog-to-digital converter, thus obtaining the raw audio data. Audio enhancement processing aims to improve the quality of the speech signal and reduce noise and interference. By analyzing the noise components in the audio, filters (such as adaptive filters) are used to filter them out, thereby obtaining a clean speech signal, i.e., an enhanced speech signal.

[0058] Specifically, framing processing divides a continuous speech signal into multiple shorter signal frames. The preset duration is usually determined based on the characteristics of the speech signal and subsequent processing requirements, generally around tens of milliseconds. Overlapping framing can be used, meaning that there is some overlap between adjacent frames, which can avoid information loss and abrupt changes caused by framing. Extracting temporal features involves analyzing the characteristics of each signal frame in the time domain. Common temporal features include short-time energy, which reflects the energy level of the signal frame and can be obtained by calculating the average of the squares of the sampling points within the signal frame.

[0059] Furthermore, the Fourier transform is an important tool for converting time-domain signals into frequency-domain signals. For each signal frame, its spectral characteristics can be obtained through the Fourier transform, which shows the energy distribution of the signal at different frequencies. Formants are regions of concentrated energy in the speech spectrum, corresponding to the resonance characteristics of the vocal tract. We can find high-energy peak points in the spectrum and obtain formant parameters by further analyzing the frequency, bandwidth, and other parameters of these peak points. Formant parameters can reflect the timbre characteristics of speech. The formant parameters of different people's speech are different, so they can be used for tasks such as speech synthesis.

[0060] Specifically, fusing formant parameters and temporal features aims to comprehensively utilize information from speech signals in different domains to obtain more comprehensive and accurate timbre features. This involves assigning different weights to formant parameters and temporal features, and then adding them together to obtain the fused features. The weights can be set according to the specific task and the importance of the features.

[0061] like Figure 3 As shown in this embodiment of the invention, the step of generating a Mel spectrogram frame by frame based on the timbre features and the target semantic token includes: Extract the frame-level decomposition data of the timbre features and the temporal frame identifier of the target semantic token; The frame-level decomposed data is matched with the temporal frame identifier to obtain a speech frame sequence. Based on a preset Mel spectrum parameter system, the corresponding initial values ​​of spectrum parameters are matched for each speech frame in the speech frame sequence; The semantic weight information in the target semantic token is mapped to a spectral energy adjustment coefficient, and the initial value of the spectral parameter is corrected using the spectral energy adjustment coefficient to obtain the target spectral parameter. A single frame of Mel spectrum data is generated based on the target spectrum parameters using the Mel transform algorithm. The Mel spectrum data of adjacent frames are smoothed to obtain a smoothed Mel spectrum; The smoothed Mel spectra are spliced ​​together according to the frame order in the speech frame sequence to obtain a Mel spectrogram.

[0062] In detail, frame-level decomposition of timbre features involves splitting timbre features according to certain time intervals to obtain timbre data corresponding to each frame. In audio processing, timbre features may be decomposed at intervals of tens of milliseconds per frame to obtain a series of frame-level decomposed data. Temporal frame identification of target semantic tokens determines the frame position to which each semantic token belongs in time. Since there is a certain temporal correspondence between semantic tokens and speech signals, temporal frame identification can clarify in which frame or frames of speech each semantic token embodies its semantic information. For example, in a speech synthesis task, the semantic token "hello" may correspond to the first few frames of the speech signal, and temporal frame identification can accurately associate them.

[0063] Specifically, frame-level association matching is the process of corresponding and integrating the frame-level decomposition data of timbre features with the temporal frame identifier of the target semantic token. By analyzing the semantic content of the semantic token and the changing trend of timbre features, the timbre feature data is arranged and combined according to the temporal requirements of the semantic token, thereby obtaining a complete speech frame sequence.

[0064] Furthermore, the preset Mel spectrum parameter system is a predefined set of parameters used to generate the Mel spectrum, including the center frequency and bandwidth of the filter. For each speech frame in the speech frame sequence, the corresponding initial values ​​of the spectrum parameters are matched from the preset Mel spectrum parameter system according to its own characteristics (such as timbre features, semantic information, etc.).

[0065] For example, a speech frame with a brighter timbre and more high-frequency components may be matched with an initial filter parameter value that has a higher center frequency and narrower bandwidth; while a speech frame with a deeper timbre and more low-frequency components may be matched with an initial filter parameter value that has a lower center frequency and wider bandwidth.

[0066] Specifically, semantic weight information reflects the importance of the target semantic token in semantic expression. Mapping semantic weight information to spectral energy adjustment coefficients means converting the importance of semantics into the degree of adjustment of spectral energy. For example, if a semantic token is very important semantically, the corresponding spectral energy adjustment coefficient may be large, indicating that the spectral energy of the speech frame needs to be enhanced; conversely, if the semantic token is less important, the spectral energy adjustment coefficient may be small, indicating that the spectral energy of the speech frame needs to be weakened.

[0067] Among them, energy correction of the initial value of the spectral parameters based on the spectral energy adjustment coefficient is achieved by adjusting the gain of the filter, etc. In this way, the generated Mel spectrum can better reflect the importance of semantics, thereby generating speech that is more in line with semantic requirements in tasks such as speech synthesis.

[0068] Furthermore, the Mel transform algorithm is an algorithm that converts linear frequencies into Mel frequencies, which can better simulate the human ear's perception of sound. When generating single-frame Mel spectrum data, the spectrum of the speech frame is filtered and weighted according to the target spectrum parameters (such as the center frequency, bandwidth, gain, etc. of the filter).

[0069] Specifically, the spectrum of the speech frame is filtered through a set of Mel filters, each filter corresponding to a Mel frequency range. The filtered results are then weighted and summed to obtain the single-frame Mel spectrum data. The Mel spectrum data generated in this way can more accurately reflect the energy distribution of the speech frame in the Mel frequency domain.

[0070] Furthermore, there may be abrupt changes or discontinuities in the Mel spectrogram data of adjacent frames. The purpose of smoothing is to eliminate these abrupt changes and discontinuities, making the Mel spectrogram data of adjacent frames transition more smoothly. For each frame's Mel spectrogram data, the average value of the Mel spectrogram data of several surrounding frames can be taken as the smoothed Mel spectrogram data of that frame. This can reduce the impact of noise and abrupt changes on the Mel spectrogram and improve the quality of the spectrogram. After obtaining the smoothed single-frame Mel spectrogram data, it is necessary to splice the single-frame Mel spectrogram data according to the frame order of the speech frame sequence to form a complete two-dimensional matrix. This matrix is ​​the final Mel spectrogram.

[0071] Specifically, the horizontal axis of the Mel spectrogram represents time (frame number), the vertical axis represents Mel frequency, and each element in the matrix represents the energy value at the corresponding time and Mel frequency. In this way, the time-frequency characteristics of the speech signal in the Mel frequency domain can be intuitively displayed.

[0072] In this embodiment of the invention, the acquisition of user voice data from a reference user and the extraction of timbre features lay the foundation for personalized voice processing, enabling a deep understanding of the unique attributes of user voice. Furthermore, the generation of Mel spectrograms frame by frame based on timbre features and target semantic tokens significantly improves the quality and efficiency of voice processing. When generating voice data, the resulting data not only has high timbre fidelity but also accurately conveys semantics, resulting in more natural and fluent semantic expression. This promotes the greater value of voice technology in numerous computer application scenarios such as intelligent customer service and voice assistants, facilitating more natural and efficient interaction between computers and humans.

[0073] S4. Speech synthesis is performed based on the Mel spectrogram to obtain the target speech.

[0074] In this embodiment of the invention, the Mel spectrum is converted into a linear spectrum by inverse Mel transform, and the fundamental frequency, amplitude and duct parameters are predicted by a vocoder to generate an initial waveform. After noise reduction, timing alignment and loudness equalization, the target speech is finally synthesized, which improves the quality and efficiency of speech synthesis.

[0075] In this embodiment of the invention, the step of performing speech synthesis based on the Mel spectrogram to obtain the target speech includes: A frame-level spectrum database is constructed based on the Mel spectrogram, and the spectral energy values ​​in the frame-level spectrum database are subjected to inverse Mel transform to obtain linear spectrum features. Based on a preset vocoder parameter model, the linear spectral features are predicted to generate corresponding fundamental frequency parameters, amplitude parameters, and channel filtering parameters. The initial speech waveform data is obtained by performing waveform generation processing based on the fundamental frequency parameter, the amplitude parameter, and the vocal tract filtering parameter. The initial speech waveform data is subjected to noise reduction processing to obtain a noise-reduced speech waveform; The denoised speech waveform is time-aligned to obtain an aligned speech waveform; The aligned speech waveform is subjected to loudness equalization processing to obtain the target speech.

[0076] In detail, building a frame-level spectrum database involves splitting and storing the Mel spectrogram according to the frame dimension. By building a frame-level spectrum database, the spectrum data of each frame is extracted separately to form an ordered database set, which facilitates independent processing of each frame of data in the future.

[0077] The inverse Mel transform is used to convert the Mel frequency scale back to the linear frequency scale. The Mel frequency is designed to simulate the nonlinear perception of sound frequency by the human ear, while the linear spectral characteristics are more consistent with the true frequency representation of sound in the physical world. The inverse Mel transform uses a specific algorithm to map the spectral energy value of each frame in the Mel spectrogram back to the linear frequency domain, thereby obtaining the linear spectral characteristics.

[0078] Furthermore, the preset vocoder parameter model is a model trained with a large amount of speech data. It can learn the complex relationship between linear spectral features and various parameters required for speech synthesis. The fundamental frequency parameter reflects the pitch information of the sound. Different people speak with different pitches, and the pitch of the same person will also change under different emotions or contexts. The amplitude parameter determines the loudness of the sound, reflecting the strength of the sound energy. The vocal tract filtering parameter simulates the filtering effect of the human vocal tract on the sound. Different shapes of vocal tracts will produce different sound characteristics.

[0079] Specifically, based on the vocoder parameter model, the computer analyzes and processes the linear spectral features of the input, and through complex calculations and inferences within the model, predicts the fundamental frequency parameters, amplitude parameters, and vocal tract filtering parameters corresponding to each frame. These parameters are like the "parts" that construct speech, and together they determine the pitch, loudness, and timbre of the final synthesized speech.

[0080] In the waveform generation process, a basic sine wave or similar sine wave signal is generated based on the fundamental frequency parameter. This signal represents the basic pitch of the sound. Then, the amplitude of this basic signal is modulated according to the amplitude parameter, so that the loudness of the sound changes according to the set rule. Finally, the modulated signal is filtered using the channel filtering parameter to simulate the effect of the channel on the sound, thereby obtaining the initial speech waveform data.

[0081] Furthermore, the initial speech waveform data may be affected by various noises during the generation process. These noises may come from environmental noise, model errors, etc. The purpose of noise reduction is to remove these unwanted noise components and improve the quality of speech. By performing Fourier transform on the speech waveform data to convert the time domain signal into a frequency domain signal, and then suppressing the noise components in the frequency domain, the impact of noise can be effectively reduced, and the clarity of speech is significantly improved.

[0082] During speech synthesis, time errors may occur in the processing and generation of various parameters, leading to inaccuracies in the timing of the denoised speech waveform. This can cause temporal shifts or misalignments in different parts of the synthesized speech. Timing alignment adjustment uses algorithms and techniques to correct and adjust the timing of the denoised speech waveform. A standard speech signal with known accurate timing is selected as a reference. The denoised speech waveform is compared and analyzed with the reference signal to identify the time differences. Then, interpolation, shifting, and other operations are used to adjust the denoised speech waveform to ensure that its timing is consistent with the reference signal. After timing alignment adjustment, the resulting aligned speech waveform is more accurate in time, and the coherence and fluency of the speech are improved.

[0083] Specifically, the loudness of aligned speech waveforms may vary significantly across different time periods. For example, the loudness may be lower at the beginning of the speech and higher in the middle. This imbalance in loudness can make listeners uncomfortable. The purpose of loudness equalization processing is to adjust the loudness of the speech waveform across different time periods, so that it maintains a relatively uniform loudness level throughout the speech process. That is, by analyzing the loudness distribution of the speech waveform through a loudness equalization algorithm, a suitable loudness target value is determined, and then the gain of the speech waveform is dynamically adjusted based on the difference between the current loudness of the speech waveform and the target value.

[0084] For example, for parts with lower loudness, the gain is increased appropriately; for parts with higher loudness, the gain is decreased appropriately. Through loudness equalization, the final target speech has a more stable and comfortable loudness, which can provide listeners with a better auditory experience.

[0085] Optionally, in order to balance naturalness and latency in speech streaming generation, a "Bos mechanism" is introduced. While a pure causal autoregressive model can achieve low latency, it may result in unnatural prosody due to the lack of future context. The traditional "look-ahead" mechanism can improve naturalness, but it introduces latency.

[0086] The deletion mechanism accesses a small amount of future textual information in a controlled manner (e.g., by adding some “starting marks” or special control tokens to the input sequence and allowing the model to “delete” them under specific conditions, i.e., not directly converting them into speech, but utilizing the future context they contain). This “deletion” operation is learned by the model and dynamically adjusted during inference to find the optimal balance between latency and naturalness. It provides the encoder model with a limited future context to improve the naturalness of the generated speech.

[0087] For example, in the field of healthcare, taking an intelligent medical consultation assistant as an example, when a patient describes their symptoms by voice, such as "I have been having headaches lately, accompanied by nausea, and I don't sleep well," the system first processes the voice to obtain a Mel spectrogram, and then synthesizes the target voice based on this spectrogram.

[0088] During the synthesis process, taking into account the rigor and professionalism of the medical scenario, the system will accurately reproduce the key information in the patient's voice, such as the tone and intonation changes when the patient describes the symptoms. If the patient's tone is urgent, the synthesized voice will also reflect the sense of urgency, allowing the doctor to better understand the severity of the patient's condition.

[0089] The obtained target speech can be used in various scenarios. On the one hand, it can be used as an audio recording of patient medical records, facilitating doctors' subsequent review and comparison of changes in the patient's condition. On the other hand, in telemedicine consultations, clear and accurate target speech allows experts from different regions to better understand the patient's condition, improving diagnostic accuracy. Moreover, for doctors with hearing impairments, target speech combined with subtitles allows them to obtain more comprehensive patient information and develop more suitable treatment plans.

[0090] For example, in a fintech scenario, taking intelligent financial customer service as an example, when a customer inquires about a wealth management product, they say, "I would like to know about the risks of this wealth management product with an annualized return of 5%." The system processes the speech to obtain a Mel spectrogram and then performs speech synthesis. During synthesis, attention is paid to the friendliness and professionalism of the voice, using a gentle and patient tone to explain the product risks to the customer, making the customer feel attentive to the service. The resulting target speech not only accurately conveys information but also enhances the customer experience. At the same time, in financial marketing activities, the synthesized high-quality target speech can be used to create promotional audio, attracting more customers to learn about financial products and promoting the development of fintech business.

[0091] In this embodiment of the invention, Mel spectrograms can accurately capture the spectral features of speech, and can highly restore the timbre, pitch and other details of the original speech, making the generated target speech more natural and realistic, and greatly improving the auditory experience of speech; Mel spectrograms provide a structured data foundation for speech synthesis, improve the synthesis speed, reduce the consumption of computing resources, and enhance the flexibility and scalability of speech synthesis.

[0092] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0093] like Figure 4 The diagram shown is a functional block diagram of a speech synthesis device based on voice token fusion provided in an embodiment of the present invention.

[0094] In this embodiment of the disclosure, a speech synthesis device based on voice token fusion is provided, which corresponds one-to-one with the speech synthesis method based on voice token fusion described in the above embodiment. For example... Figure 4 As shown, the speech synthesis device 100 based on voice token fusion can be installed in an electronic device. According to its functions, the speech synthesis device 100 includes a text feature encoding module 101, a semantic token generation module 102, a Mel spectrum construction module 103, and a target speech synthesis module 104. Detailed descriptions of each functional module are as follows: The text feature encoding module 101 is used to acquire initial text, convert the initial text into an initial latent representation, and perform text encoding on the initial latent representation to obtain target text features; The semantic token generation module 102 is used to generate a semantic token for the initial text based on the target text features, and to perform temporal alignment between the semantic token and the target text features to obtain a target semantic token; The Mel spectrogram construction module 103 is used to acquire the user speech of the reference user, extract the timbre features of the user speech, and generate a Mel spectrogram frame by frame based on the timbre features and the target semantic token. The target speech synthesis module 104 is used to synthesize speech based on the Mel spectrogram to obtain the target speech.

[0095] In one embodiment, the text feature encoding module 101, when performing the conversion of the initial text into an initial latent representation, is configured to: The initial text is segmented to obtain a text word sequence; Add positional encoding to each subword in the text word sequence to obtain the target input sequence; The semantic association between each word in the target input sequence and all words in the target input sequence is analyzed using the multi-head self-attention function in the preset encoder to obtain the text association sequence; A nonlinear transformation is performed on the text association sequence to obtain the initial potential representation corresponding to the initial text.

[0096] In one embodiment, when the semantic token generation module 102 generates a semantic token for the initial text based on the target text features, it is used to: The target text features are enhanced to obtain an enhanced text feature set; The semantic dimension corresponding to the enhanced text feature set is determined based on a preset semantic hierarchy system; Based on the semantic dimension, feature mapping is performed on the enhanced text feature set to generate a semantic feature vector under each semantic dimension; The semantic feature vectors are fused to obtain a fused feature matrix; The fusion feature matrix is ​​encoded and converted based on a preset token generation strategy to generate a semantic token for the initial text.

[0097] In one embodiment, when the semantic token generation module 102 performs temporal alignment between the semantic token and the target text features to obtain the target semantic token, it is used to: Based on a preset alignment granularity rule, the temporal alignment levels between the semantic token and the target text features are divided, and the temporal tagging information of the semantic token is extracted. The target text features are analyzed temporally to obtain temporal structure features, and the time span and temporal sequence relationship of each time feature unit in the temporal structure features are extracted. In the fine alignment layer of the time alignment hierarchy, the time sequence marker information and the time feature unit are matched point by point according to the time span and the time sequence relationship to obtain misalignment tokens and misalignment features; In the coarse-grained alignment layer of the time alignment hierarchy, the misaligned token and the misaligned feature are subjected to time compensation processing to generate a time alignment matrix; The target semantic token is obtained by filtering the time alignment matrix according to the preset token confidence threshold.

[0098] In one embodiment, the Mel spectrum construction module 103, when performing the extraction of timbre features from the user's speech, is used to: Extract the raw audio data from the user's speech and perform audio enhancement processing on the raw audio data to obtain an enhanced speech signal; The enhanced speech signal is divided into frames according to a preset duration to obtain multiple signal frames, and the temporal features of each signal frame are extracted. Perform a Fourier transform on each signal frame to obtain the spectral characteristics of the signal frame, and extract the formant parameters from the spectral characteristics; The formant parameters and the time-domain features are fused to obtain the timbre features of the user's speech.

[0099] In one embodiment, the Mel spectrogram construction module 103, when performing frame-by-frame generation of a Mel spectrogram based on the timbre features and the target semantic token, is configured to: Extract the frame-level decomposition data of the timbre features and the temporal frame identifier of the target semantic token; The frame-level decomposed data is matched with the temporal frame identifier to obtain a speech frame sequence. Based on a preset Mel spectrum parameter system, the corresponding initial values ​​of spectrum parameters are matched for each speech frame in the speech frame sequence; The semantic weight information in the target semantic token is mapped to a spectral energy adjustment coefficient, and the initial value of the spectral parameter is corrected using the spectral energy adjustment coefficient to obtain the target spectral parameter. A single frame of Mel spectrum data is generated based on the target spectrum parameters using the Mel transform algorithm. The Mel spectrum data of adjacent frames are smoothed to obtain a smoothed Mel spectrum; The smoothed Mel spectra are spliced ​​together according to the frame order in the speech frame sequence to obtain a Mel spectrogram.

[0100] In one embodiment, when the target speech synthesis module 104 performs speech synthesis based on the Mel spectrogram to obtain the target speech, it is configured to: A frame-level spectrum database is constructed based on the Mel spectrogram, and the spectral energy values ​​in the frame-level spectrum database are subjected to inverse Mel transform to obtain linear spectrum features. Based on a preset vocoder parameter model, the linear spectral features are predicted to generate corresponding fundamental frequency parameters, amplitude parameters, and channel filtering parameters. The initial speech waveform data is obtained by performing waveform generation processing based on the fundamental frequency parameter, the amplitude parameter, and the vocal tract filtering parameter. The initial speech waveform data is subjected to noise reduction processing to obtain a noise-reduced speech waveform; The denoised speech waveform is time-aligned to obtain an aligned speech waveform; The aligned speech waveform is subjected to loudness equalization processing to obtain the target speech.

[0101] In this invention, the specific limitations of a speech synthesis device based on voice token fusion can be found in the above-described limitations of a speech synthesis method based on voice token fusion, and will not be repeated here. Each module in the aforementioned speech synthesis device based on voice token fusion can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0102] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a voice token fusion-based speech synthesis method on the server side.

[0103] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of a voice token-based speech synthesis method.

[0104] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the initial text, convert the initial text into an initial latent representation, and perform text encoding on the initial latent representation to obtain the target text features; A semantic token for the initial text is generated based on the target text features, and the semantic token is temporally aligned with the target text features to obtain the target semantic token. Acquire the user's voice from the reference user, extract the timbre features of the user's voice, and generate a Mel spectrogram frame by frame based on the timbre features and the target semantic token; Speech synthesis is performed based on the Mel spectrogram to obtain the target speech.

[0105] In the several embodiments provided by this invention, it should be understood that the disclosed devices and apparatuses can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0106] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0107] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0108] In some embodiments of this example, a computer-readable storage medium is provided, on which a computer program is stored, characterized in that the computer program, when executed by a processor, implements the steps of the method described in the above embodiments.

[0109] The readable storage medium of the present invention stores a computer program, which, when executed by a processor of an electronic device, can perform the following: Obtain the initial text, convert the initial text into an initial latent representation, and perform text encoding on the initial latent representation to obtain the target text features; A semantic token for the initial text is generated based on the target text features, and the semantic token is temporally aligned with the target text features to obtain the target semantic token. Acquire the user's voice from the reference user, extract the timbre features of the user's voice, and generate a Mel spectrogram frame by frame based on the timbre features and the target semantic token; Speech synthesis is performed based on the Mel spectrogram to obtain the target speech.

[0110] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0111] Computer-readable storage media may also store at least one computer-executable program / instruction, such as computer-readable instructions. Computer-readable storage media include, but are not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Computer-readable storage media may include, for example, read-only memory (ROM), hard disk, flash memory, etc. For example, a non-transitory computer-readable storage medium may be connected to a computing device such as a computer, and then, when the computing device executes the computer-readable instructions stored on the computer-readable storage medium, the various methods described above can be performed.

[0112] In one embodiment, the at least one computer-executable instruction may also be compiled into or comprise a software product / computer program product, wherein one or more computer-executable instructions are executed by a processor to perform the steps of the various functions and / or methods in the embodiments described herein.

[0113] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Furthermore, any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory.

[0114] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0115] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0116] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0117] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely for illustrative purposes and do not represent actual use.

Claims

1. A speech synthesis method based on voice token fusion, characterized in that, The method includes: Obtain the initial text, convert the initial text into an initial latent representation, and perform text encoding on the initial latent representation to obtain the target text features; A semantic token for the initial text is generated based on the target text features, and the semantic token is temporally aligned with the target text features to obtain the target semantic token. Acquire the user's voice from the reference user, extract the timbre features of the user's voice, and generate a Mel spectrogram frame by frame based on the timbre features and the target semantic token; Speech synthesis is performed based on the Mel spectrogram to obtain the target speech.

2. The speech synthesis method based on voice token fusion as described in claim 1, characterized in that, The process of converting the initial text into an initial latent representation includes: The initial text is segmented to obtain a text word sequence; Add positional encoding to each subword in the text word sequence to obtain the target input sequence; The semantic association between each word in the target input sequence and all words in the target input sequence is analyzed using the multi-head self-attention function in the preset encoder to obtain the text association sequence; A nonlinear transformation is performed on the text association sequence to obtain the initial potential representation corresponding to the initial text.

3. The speech synthesis method based on voice token fusion as described in claim 1, characterized in that, The step of generating the semantic token of the initial text based on the target text features includes: The target text features are enhanced to obtain an enhanced text feature set; The semantic dimension corresponding to the enhanced text feature set is determined based on a preset semantic hierarchy system; Based on the semantic dimension, feature mapping is performed on the enhanced text feature set to generate a semantic feature vector under each semantic dimension; The semantic feature vectors are fused to obtain a fused feature matrix; The fusion feature matrix is ​​encoded and converted based on a preset token generation strategy to generate a semantic token for the initial text.

4. The speech synthesis method based on voice token fusion as described in claim 1, characterized in that, The step of temporally aligning the semantic token with the target text features to obtain the target semantic token includes: Based on a preset alignment granularity rule, the temporal alignment levels between the semantic token and the target text features are divided, and the temporal tagging information of the semantic token is extracted. The target text features are analyzed temporally to obtain temporal structure features, and the time span and temporal sequence relationship of each time feature unit in the temporal structure features are extracted. In the fine alignment layer of the time alignment hierarchy, the time sequence marker information and the time feature unit are matched point by point according to the time span and the time sequence relationship to obtain misalignment tokens and misalignment features; In the coarse-grained alignment layer of the time alignment hierarchy, the misaligned token and the misaligned feature are subjected to time compensation processing to generate a time alignment matrix; The target semantic token is obtained by filtering the time alignment matrix according to the preset token confidence threshold.

5. The speech synthesis method based on voice token fusion as described in claim 1, characterized in that, The extraction of the timbre features of the user's speech includes: Extract the raw audio data from the user's speech and perform audio enhancement processing on the raw audio data to obtain an enhanced speech signal; The enhanced speech signal is divided into frames according to a preset duration to obtain multiple signal frames, and the temporal features of each signal frame are extracted. Perform a Fourier transform on each signal frame to obtain the spectral characteristics of the signal frame, and extract the formant parameters from the spectral characteristics; The formant parameters and the time-domain features are fused to obtain the timbre features of the user's speech.

6. The speech synthesis method based on voice token fusion as described in claim 1, characterized in that, The step of generating a Mel spectrogram frame by frame based on the timbre features and the target semantic token includes: Extract the frame-level decomposition data of the timbre features and the temporal frame identifier of the target semantic token; The frame-level decomposed data is matched with the temporal frame identifier to obtain a speech frame sequence. Based on a preset Mel spectrum parameter system, the corresponding initial values ​​of spectrum parameters are matched for each speech frame in the speech frame sequence; The semantic weight information in the target semantic token is mapped to a spectral energy adjustment coefficient, and the initial value of the spectral parameter is corrected using the spectral energy adjustment coefficient to obtain the target spectral parameter. A single frame of Mel spectrum data is generated based on the target spectrum parameters using the Mel transform algorithm. The Mel spectrum data of adjacent frames are smoothed to obtain a smoothed Mel spectrum; The smoothed Mel spectrogram is obtained by splicing the smoothed Mel spectrograms according to the frame order in the speech frame sequence.

7. The speech synthesis method based on voice token fusion as described in claim 1, characterized in that, The step of synthesizing speech based on the Mel spectrogram to obtain the target speech includes: A frame-level spectrum database is constructed based on the Mel spectrogram, and the spectral energy values ​​in the frame-level spectrum database are subjected to inverse Mel transform to obtain linear spectrum features. Based on a preset vocoder parameter model, the linear spectral features are predicted to generate corresponding fundamental frequency parameters, amplitude parameters, and channel filtering parameters. The initial speech waveform data is obtained by performing waveform generation processing based on the fundamental frequency parameter, the amplitude parameter, and the vocal tract filtering parameter. The initial speech waveform data is subjected to noise reduction processing to obtain a noise-reduced speech waveform; The denoised speech waveform is time-aligned to obtain an aligned speech waveform; The aligned speech waveform is subjected to loudness equalization processing to obtain the target speech.

8. A speech synthesis device based on voice token fusion, characterized in that, The device includes: The text feature encoding module is used to acquire initial text, convert the initial text into an initial latent representation, and perform text encoding on the initial latent representation to obtain target text features; A semantic token generation module is used to generate a semantic token for the initial text based on the target text features, and to perform temporal alignment between the semantic token and the target text features to obtain a target semantic token; The Mel spectrogram construction module is used to acquire user speech from a reference user, extract the timbre features of the user speech, and generate a Mel spectrogram frame by frame based on the timbre features and the target semantic token. The target speech synthesis module is used to synthesize speech based on the Mel spectrogram to obtain the target speech.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the speech synthesis method based on voice token fusion as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech synthesis method based on voice token fusion as described in any one of claims 1 to 7.

Images