Speech synthesis methods, devices, computer equipment and storage media

Abstract

This application relates to the field of artificial intelligence technology and discloses a speech synthesis method, apparatus, computer device, and storage medium, comprising: acquiring speech synthesis text and a speaker identifier code of a target speaker; encoding the speech synthesis text to obtain text embedding features; acquiring a speaker vector of the target speaker based on the speaker identifier code; serializing the speaker vector based on the text embedding features to obtain a target speaker embedding sequence; and performing speech synthesis based on the text embedding features and the target speaker embedding sequence to obtain target speech data. It can be applied to speech synthesis scenarios in the fintech and healthcare fields, improving the accuracy of speech synthesis.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence technology and natural language processing technology, and is applicable to the fields of financial technology and healthcare. In particular, it relates to a speech synthesis method, device, computer equipment and storage medium. Background Technology

[0002] Speech synthesis technology is a technique that converts text information into speech output. Given speaker embedding features of a specific speaker, it combines the text to be synthesized with the speaker embedding features to perform speech synthesis, resulting in speech containing the speaker's specific information. Speech synthesis technology can be applied in various scenarios. For example, in the financial sector, it can be used to synthesize the speech of specific business personnel in scenarios such as intelligent customer service, voice assistants, and multilingual customer service. In the healthcare sector, it can be used to synthesize the speech of specific medical personnel in scenarios such as intelligent customer service, telemedicine, and health education.

[0003] Currently, given speaker embedding features for a specific speaker, the common approach is to concatenate or add the speaker embedding features and text features, or to fuse them using global scalar weighting. However, in practical applications, these methods fail to accurately fuse speaker features and text features. For example, when speaker features are incorporated into text features, it becomes impossible to distinguish which dimensions of the speaker features (e.g., representing timbre or speech rate) are beneficial supplements to specific words or phrases at the current time step, and which are interfering noise or redundant information. This leads to irrelevant speaker information interfering with or causing semantic drift in the core text semantics, thus affecting the accuracy of speech synthesis. Summary of the Invention

[0004] This application provides a speech synthesis method, apparatus, computer device, and storage medium to solve the technical problem that the inability to accurately fuse speaker features and text features leads to irrelevant speaker information interfering with or causing semantic drift in the core text semantics, thus affecting the accuracy of speech synthesis.

[0005] Firstly, a speech synthesis method is provided, including: Obtain the synthesized text and the speaker identifier code of the target speaker; The synthesized speech text is encoded to obtain text embedding features; The speaker vector of the target speaker is obtained based on the speaker identifier code; The speaker vector is serialized based on the text embedding features to obtain the target speaker embedding sequence. Speech synthesis is performed based on the text embedding features and the target speaker embedding sequence to obtain target speech data.

[0006] Secondly, a speech synthesis device is provided, comprising: The data acquisition module is used to acquire the synthesized speech text and the speaker identification code of the target speaker; The text encoding module is used to encode the speech-synthesized text to obtain text embedding features; The speaker vector acquisition module is used to acquire the speaker vector of the target speaker based on the speaker identifier code; The serialization processing module is used to serialize the speaker vector based on the text embedding features to obtain the target speaker embedding sequence; The target speech synthesis module is used to synthesize speech based on the text embedding features and the target speaker embedding sequence to obtain target speech data.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described speech synthesis method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described speech synthesis method.

[0009] The aforementioned speech synthesis methods, devices, computer equipment, and storage media, specifically designed for speech synthesis scenarios in the fintech and healthcare fields, firstly, acquire the speech synthesis text and the speaker identifier code of the target speaker. The speech synthesis text is then encoded to obtain text embedding features, which accurately capture the text's semantics and grammatical structure. Based on the speaker identifier code, the speaker vector of the target speaker is obtained, effectively representing the unique acoustic features of the target speaker. Next, the speaker vector is serialized based on the text embedding features to obtain the target speaker embedding sequence, enabling better adaptation between acoustic and text features. Finally, speech synthesis is performed based on the text embedding features and the target speaker embedding sequence to obtain target speech data. This accurately conveys the text content and highly reproduces the target speaker's timbre, intonation, and other features, improving the naturalness and personalization of the speech synthesis. This solves the technical problem that the inability to accurately integrate speaker and text features leads to irrelevant speaker information interfering with or causing semantic drift in the core text semantics, thus affecting the accuracy of speech synthesis. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a speech synthesis method according to an embodiment of this application; Figure 2 This is a flowchart illustrating a speech synthesis method in one embodiment of this application; Figure 3 yes Figure 2 A schematic diagram of a specific implementation method for step S20; Figure 4 yes Figure 2 A schematic diagram of a specific implementation of step S40; Figure 5 yes Figure 4 A schematic diagram of a specific implementation method for step S41; Figure 6 yes Figure 4 A schematic diagram of a specific implementation method for step S42; Figure 7 yes Figure 2 A schematic diagram of a specific implementation method for step S50; Figure 8 This is a schematic diagram of a speech synthesis device in one embodiment of this application; Figure 9 This is a schematic diagram of the structure of a computer device according to one embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] First, let's analyze some of the terms used in this application: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0014] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0015] Speech synthesis technology is a technique that converts text information into speech output. Given speaker embedding features of a specific speaker, it combines the text to be synthesized with the speaker embedding features to perform speech synthesis, resulting in speech containing the speaker's specific information. Speech synthesis technology can be applied in various scenarios. For example, in the financial sector, it can be used to synthesize the speech of specific business personnel in scenarios such as intelligent customer service, voice assistants, and multilingual customer service. In the healthcare sector, it can be used to synthesize the speech of specific medical personnel in scenarios such as intelligent customer service, telemedicine, and health education.

[0016] Currently, given speaker embedding features for a specific speaker, the common approach is to concatenate or add the speaker embedding features and text features, or to fuse them using global scalar weighting. However, in practical applications, these methods fail to accurately fuse speaker features and text features. For example, when speaker features are incorporated into text features, it becomes impossible to distinguish which dimensions of the speaker features (e.g., representing timbre or speech rate) are beneficial supplements to specific words or phrases at the current time step, and which are interfering noise or redundant information. This leads to irrelevant speaker information interfering with or causing semantic drift in the core text semantics, thus affecting the accuracy of speech synthesis.

[0017] Based on this, embodiments of this application provide a speech synthesis method, apparatus, computer device, and storage medium to solve the technical problem that the inability to accurately fuse speaker features and text features leads to irrelevant speaker information interfering with or causing semantic drift in the core text semantics, thus affecting the accuracy of speech synthesis.

[0018] The speech synthesis method, apparatus, computer equipment, and storage medium provided in the embodiments of this application are specifically described through the following embodiments. First, the speech synthesis method in the embodiments of this application is described.

[0019] The speech synthesis method provided in this application embodiment can be applied to, for example, Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain the speech-synthesized text and the speaker identifier code of the target speaker from the client; encode the speech-synthesized text to obtain text embedding features; obtain the speaker vector of the target speaker based on the speaker identifier code; serialize the speaker vector based on the text embedding features to obtain the target speaker embedding sequence; perform speech synthesis based on the text embedding features and the target speaker embedding sequence to obtain the target speech data, and feed the target speech data back to the client for speech playback.

[0020] In this application, targeting speech synthesis scenarios in the fintech and healthcare fields, the process begins by first obtaining the speech synthesis text and the speaker identifier code of the target speaker. The speech synthesis text is then encoded to obtain text embedding features, which accurately capture the text's semantics and grammatical structure. Based on the speaker identifier code, the speaker vector of the target speaker is obtained, effectively representing the unique acoustic features of the target speaker. Next, the speaker vector is serialized based on the text embedding features to obtain the target speaker embedding sequence, enabling better adaptation between acoustic and text features. Finally, speech synthesis is performed based on the text embedding features and the target speaker embedding sequence to obtain target speech data. This accurately conveys the text content and highly reproduces the target speaker's timbre, intonation, and other features, improving the naturalness and personalization of the speech synthesis. This addresses the technical problem that the inability to accurately integrate speaker and text features leads to irrelevant speaker information interfering with or causing semantic drift in the core text semantics, thus affecting the accuracy of speech synthesis.

[0021] The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description uses specific embodiments to illustrate this application.

[0022] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the speech synthesis method provided in this application embodiment includes the following steps: S10. Obtain the synthesized speech text and the speaker identification code of the target speaker; Speech-synthesized text refers to text content that needs to be converted into speech, and this text can take various forms. For example, in the financial sector, speech-synthesized text can be used for voice-broadcast financial information, such as stock quotes and financial news, or for voice-introduced insurance product information. In the healthcare sector, speech-synthesized text can be medical records, health education articles, and other textual information.

[0023] The target speaker is the object that the synthesized speech is intended to imitate; that is, the final synthesized speech should reproduce the speaker's vocal characteristics. The speaker identifier code is a unique identifier for the target speaker, used to accurately identify and locate a specific speaker within the system. For example, in a financial context, the target speaker could be a specific financial news anchor, insurance agent, or product development manager; in a healthcare context, the target speaker could be a medical professional or expert.

[0024] Specifically, the text to be synthesized and the speaker identifier code of the target speaker can be input by a user on the client side, while the speaker vector corresponding to the speaker identifier code is stored in the speaker database on the server side. By clearly defining the text content and target speaker of the synthesized speech, basic information is provided for subsequent speech synthesis, ensuring that the synthesized speech not only meets the specific text requirements but also presents the voice characteristics of the target speaker.

[0025] S20. Encode the speech-synthesized text to obtain text embedding features; It is important to understand that the encoding process here refers to the process of converting the synthesized speech text from its original character form into a numerical vector form that a computer can understand and process.

[0026] Among them, such as Figure 3 As shown, step S20, which involves encoding the speech-synthesized text to obtain text embedding features, includes the following steps: S21. Perform text cleaning on the speech-synthesized text to obtain the target synthesized text; S22. Perform word encoding on the target synthesized text to obtain word embedding features; S23. Perform context encoding on the word embedding features to obtain text embedding features.

[0027] For steps S21 to S23, the speech-synthesized text is first cleaned to obtain a purer and more accurate target synthesized text. Next, word encoding is performed on the target synthesized text to capture lexical information and obtain word embedding features. Finally, context encoding is performed on the word embedding features to further explore the relationships between words, integrate the information to obtain text embedding features, and improve the ability to understand and express speech-synthesized text.

[0028] Specifically, various rules (such as regular expressions) and algorithms are used, including removing special symbols, correcting spelling errors, and filtering irrelevant advertising information, to perform text cleaning on the speech synthesis text. This removes irrelevant and erroneous information, resulting in the target synthesized text. This target synthesized text is the clean, accurate, and compliant text obtained after text cleaning, avoiding the adverse effects of these interfering factors on subsequent encoding and speech synthesis. For example, in fintech scenarios, the original financial information text may contain web page markup symbols, repetitive advertising paragraphs, etc., which can be removed through text cleaning, ensuring the accuracy of stock data, financial opinions, and other information broadcast in the speech. In healthcare scenarios, medical record text may contain handwritten recognition errors, redundant annotations, etc., which can be removed through text cleaning, resulting in clean medical record text that helps in more accurate analysis and diagnosis of conditions.

[0029] After obtaining the target synthetic text, specific word encoding algorithms (such as Word2Vec and GloVe) are used to map each word in the target synthetic text into a fixed-dimensional vector. This converts the vocabulary in the text into numerical form, enabling the computer to perform mathematical operations and analysis on the text, providing a foundation for subsequent context encoding and speech synthesis. For example, in fintech scenarios, computers can analyze the semantic relationships between different financial terms by calculating word embedding features. Simultaneously, word embedding features can preserve the semantic information of words; similar words have similar word embedding features. In healthcare scenarios, this helps computers understand the semantic connections between different medical terms; for example, the word embedding features of "hypertension" and "hyperlipidemia" may have certain similarities.

[0030] It should be understood that word embedding features are numerical vectors obtained by encoding words in the target synthesized text. Word embedding features can capture semantic, grammatical and other information of words, and different words have different word embedding features.

[0031] Furthermore, after obtaining the word embedding features, the word embedding feature sequence is processed by a pre-trained text encoder (such as a Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), or Transformer model). The text encoder considers the position and relationships of words within the text, thus more accurately capturing the overall semantics of the text. This provides a more comprehensive and accurate input for subsequent speech synthesis, making the synthesized speech more consistent with the semantics and context of the text, improving the naturalness and accuracy of the speech synthesis. For example, in a fintech scenario, for a text analyzing the macroeconomic situation, context encoding can understand the logical relationships between different economic indicators and policy descriptions, resulting in an accurate semantic representation of the text. In a healthcare scenario, accurate text embedding features allow synthesized speech to more clearly convey medical record information and health advice.

[0032] It is understandable that speech-synthesized text usually contains multiple sentences, and each sentence will obtain a corresponding embedding feature after encoding. Therefore, text embedding features include multiple text sequence embedding features, and one text sequence embedding feature corresponds to one sentence.

[0033] Understandably, suppose a synthesized speech text is "The weather is so nice today, I'm going out for a bike ride, I can call Xiaoming to come along." This synthesized speech text contains three sentences: "The weather is so nice today," "I'm going out for a bike ride," and "I can call Xiaoming to come along." After encoding this synthesized speech text, the resulting text embedding features can be [A, B, C]. Here, A, B, and C are text sequence embedding features; A is the text sequence embedding feature corresponding to the sentence "The weather is so nice today," B is the text sequence embedding feature corresponding to the sentence "I'm going out for a bike ride," and C is the text sequence embedding feature corresponding to the sentence "I can call Xiaoming to come along."

[0034] S30. Obtain the speaker vector of the target speaker based on the speaker identifier code; It should be noted that a speaker vector is a numerical vector used to represent the vocal characteristics of a target speaker, including information such as timbre, intonation, and speech rate. This speaker vector is stored in a server-side speaker database, which contains speaker vectors for multiple pre-collected speakers, and each speaker vector is assigned a unique speaker identifier.

[0035] After identifying the speaker identifier code corresponding to the target speaker, the speaker vector of the target speaker is retrieved from the speaker database on the server based on the speaker identifier code. The voice feature representation of the target speaker is obtained, so that the subsequent speech synthesis can imitate the voice of the target speaker and realize personalized speech synthesis.

[0036] S40. Based on the text embedding features, the speaker vector is serialized to obtain the target speaker embedding sequence; It's important to understand that the serialization process here refers to adjusting and processing the speaker vector according to the text sequence corresponding to the embedded text features. This allows for sequence-level matching and fusion of the speaker vector and text information, ensuring that each text part corresponds to the appropriate target speaker's voice features during speech synthesis, thus improving the naturalness and personalization of the synthesized speech. For example, Figure 4 As shown, step S40, which involves serializing the speaker vector based on text embedding features to obtain the target speaker embedding sequence, includes the following steps: S41. Perform sequence alignment processing on speaker vectors based on text embedding features to obtain aligned speaker embedding sequences; It's important to understand that the sequence alignment process here refers to data padding. The purpose of data padding is to copy speaker features to each valid time step in the entire text sequence. A time step can be understood as a segment of time in the speech synthesis process (e.g., a word is a time step). This ensures that speaker features accurately correspond to text information in the time dimension, avoiding situations where speaker features and text semantics do not match due to feature misalignment during speech synthesis.

[0037] Among them, such as Figure 5 As shown, step S41, which involves performing sequence alignment processing on the speaker vector based on text embedding features to obtain the aligned speaker embedding sequence, includes the following steps: S411. Obtain the text length feature of each text sequence embedding feature, and select the feature with the longest length from the text length features as the maximum length feature; It should be noted that text length features refer to the number of basic units such as characters, words, or sub-words contained in a text sequence. Text length features can intuitively reflect the length information of the text. For example, in the financial technology text "buy stocks", the length feature is 3 (in characters); in the medical and health text "cold", the length feature is 2 (in characters).

[0038] Understandably, the lengths of the embedded features for each text sequence are not the same. Therefore, it is necessary to select the longest feature from the text length features as the maximum length feature, providing a benchmark for constructing a speaker matrix with a unified structure. Because different texts have different lengths, only by knowing the length of the longest text can a sufficiently large matrix be constructed to accommodate the speaker features corresponding to all texts, ensuring the completeness and accuracy of the processing.

[0039] S412. Construct a speaker matrix based on the maximum length feature and the number of text sequence embedding features; It should be noted that the number of text sequence embedding features refers to the number of text sequences. For example, if the synthesized speech text contains 10 sentences, then there are 10 corresponding text sequence embedding features, and the total number of text sequence embedding features is 10.

[0040] Specifically, the number of rows in the matrix is ​​determined based on the number of embedded features in the text sequence, and the number of columns is determined based on the maximum length feature. Then, a matrix of appropriate size is initialized, which is the speaker matrix. Constructing the speaker matrix facilitates the unified storage and processing of speaker features corresponding to all texts. For example, assuming the maximum length feature is 10 and the number of embedded features in the text sequence is 8, an 8-row, 10-column speaker matrix is ​​constructed to store the speaker features corresponding to these texts. This means that each row of the speaker matrix corresponds to one embedded feature in the text sequence.

[0041] S413. Perform vector filling on the speaker matrix based on the speaker vector to obtain the initial speaker embedding matrix; It is important to understand that vector padding here refers to using a copying method to assign a speaker vector to each element in the speaker matrix, thereby obtaining an initial speaker embedding matrix.

[0042] Specifically, assuming the speaker matrix is ​​a 2x5 matrix, the speaker matrix can be represented as: ; Assuming the speaker vector is denoted as A, the initial speaker embedding matrix obtained after vector imputation of the speaker matrix can be denoted as: .

[0043] S414. Mask the initial speaker embedding matrix to obtain the aligned speaker embedding sequence.

[0044] It's important to note that since the length of each text sequence embedding feature is not uniform, and the number of columns in the speaker matrix is ​​determined based on the longest feature, there are some invalid positions for shorter text sequence embedding features. These invalid positions should be ignored during speech synthesis. Therefore, the masking process here masks the invalid poses corresponding to the shorter text sequence embedding features in the initial speaker embedding matrix, ensuring they do not interfere with subsequent fusion. This allows the speaker vector to correspond and fuse only with the semantically meaningful parts of the text sequence, improving the accuracy and quality of speech synthesis.

[0045] Specifically, step S414, which involves masking the initial speaker embedding matrix to obtain the aligned speaker embedding sequence, includes the following steps: For each text sequence embedding feature, the position of invalid characters is determined based on the text length feature of the text sequence embedding feature; The initial speaker embedding matrix is ​​masked based on the position of invalid characters to obtain an aligned speaker embedding sequence.

[0046] Understandably, because different texts have different lengths, when a shorter text is padded to the same length as the longest text, the positions corresponding to the padded portion are invalid character positions, which have no actual textual semantics. For example, when a text of length 20 characters is padded to 30 characters, the last 10 character positions are invalid character positions.

[0047] Specifically, for each text sequence embedding feature, i.e., for each row in the initial speaker embedding matrix, invalid character positions (e.g., row and column numbers) are determined based on its text length feature. Then, the elements corresponding to these positions are found in the initial speaker embedding matrix and set as invalid mask values, ultimately resulting in the aligned speaker embedding sequence. The invalid mask value is typically a specific value (e.g., 0) to mask the speaker vectors at these positions, preventing them from interfering with the results in subsequent calculations and fusion.

[0048] This aligned speaker embedding sequence can accurately match speaker vectors with the semantically meaningful parts of the text sequence, eliminating the interference of speaker vectors at invalid character positions on the speech synthesis results. This helps to improve the accuracy and quality of speech synthesis, making the synthesized speech more consistent with the semantics expressed by the actual text content.

[0049] For example: Suppose the initial speaker embedding matrix can be denoted as: In this matrix, the first row (the first text sequence embedding feature) has an effective length of 5, and the second row (the second text sequence embedding feature) has an effective length of 3. Therefore, the invalid positions are the fourth and fifth columns of the second row. Thus, masking is applied to the fourth and fifth columns of the second row, resulting in the masked speaker embedding matrix: Finally, the masked speaker embedding matrix is ​​converted into a one-dimensional vector to obtain the aligned speaker embedding sequence. .

[0050] It should be noted that the dimension of the final aligned speaker embedding sequence needs to be the same as the dimension of the text embedding features.

[0051] S42. Calculate speaker weights based on text embedding features and aligned speaker embedding sequences; It should be understood that the weight calculation here refers to analyzing the text embedding features and aligned speaker embedding sequences through a pre-trained gating network to determine which parts of the text are key information and calculate the speaker weight corresponding to each text. This can dynamically adjust the influence of speaker features under different text contents, making speech synthesis more in line with actual needs.

[0052] It's important to note that a gating network is a special type of neural network structure that typically incorporates a gating mechanism, enabling it to dynamically control the flow of information based on different input conditions. In speech synthesis, gating networks can adjust the degree to which different speaker vectors influence the final result based on varying concatenated features. For example, in fintech scenarios, gating networks can dynamically adjust the representation of speaker voice features in synthesized speech based on the importance of different financial terms in the text. In healthcare scenarios, gating networks can adjust features such as speaker intonation based on the severity of different symptoms in medical records.

[0053] Among them, such as Figure 6 As shown, step S42, which involves calculating the speaker weights based on the text embedding features and the aligned speaker embedding sequence, includes the following steps: S421. Concatenate the text embedding features and the aligned speaker embedding sequence to obtain the concatenated features; S422. Perform a linear transformation on the spliced ​​features to obtain linear features; S423. Perform activation calculations on the linear features to obtain speaker weights.

[0054] For steps S421 to S423, the text embedding features and the aligned speaker embedding sequence are first concatenated to obtain concatenated features. This integrates the information from both, providing a richer and more comprehensive feature base for subsequent processing. Next, a linear transformation is applied to the concatenated features through a linear layer of a gating network, changing the feature dimension and distribution to obtain linear features. Finally, the activation function of the gating network is used to calculate the activation of the linear features, yielding speaker weights. This accurately measures the influence of the speaker vector on the text, improving the accuracy and effectiveness of speaker feature fusion in speech synthesis.

[0055] Specifically, the text embedding features and the aligned speaker embedding sequence are merged according to their feature dimensions to form a new feature vector, namely the concatenated feature. This merges text information and speaker feature information, allowing subsequent weight calculations to simultaneously consider both text content and speaker characteristics. This results in speech synthesis that generates speech that is more consistent with the speaker's style and has accurate semantic expression.

[0056] After obtaining the concatenated features, these features are used as input to a gating network, processed sequentially through linear layers and activation functions. Specifically, the concatenated features are input into the linear layers of the gating network, where the weight matrix and bias vector of the linear layer perform a linear transformation to obtain linear features. This process reduces or increases the dimensionality of the concatenated features, extracting more representative features. Simultaneously, by adjusting the weight matrix and bias vector, the importance and relationships between different dimensions of information in the concatenated features can be learned, enabling subsequent activation calculations to better capture the nonlinear relationships between features.

[0057] Furthermore, the linear features are input into the activation function of the gating network. The activation function calculates speaker weights based on the values ​​of the linear features. The non-linear nature of the activation function makes the calculation of speaker weights more flexible and accurate, thereby dynamically adjusting the size of the speaker weights according to different value ranges of the linear features, thus better simulating the performance of speaker voice features under different text content.

[0058] It should be noted that activation functions such as Sigmoid, ReLU, and Tanh can be used, and the specific choice should be made based on the actual application scenario. This application does not impose any restrictions.

[0059] Understandably, gating networks can adjust the contribution of aligned speaker embedding sequences step-by-step and feature-dimension-by-feature based on text content and speaker features, accurately identifying and utilizing valuable style information while enhancing robustness and resistance to interference. Specifically, gating networks can effectively filter noise and potential semantic interference in text because they can learn to set the gating weights corresponding to irrelevant information to zero.

[0060] S43. Based on speaker weights and aligned speaker embedding sequences, perform aggregation calculations to obtain the target speaker embedding sequence.

[0061] After obtaining the speaker weights, the speaker weights are used as modulation weights to perform element-wise multiplication (Hadamard product) on the aligned speaker embedding sequence to obtain the weighted target speaker embedding sequence, which can better integrate speaker features and text information, making speech synthesis more natural and accurate.

[0062] S50. Based on text embedding features and target speaker embedding sequences, speech synthesis is performed to obtain target speech data.

[0063] It is important to understand that speech synthesis here refers to using a pre-trained speech synthesis model (such as Tacotron, WaveNet, etc.) as input, taking text embedding features and target speaker embedding sequences as input, and generating corresponding speech waveform data, i.e., target speech data, through the model's calculation and processing.

[0064] Among them, such as Figure 7 As shown, step S50, which involves speech synthesis based on text embedding features and the target speaker embedding sequence to obtain target speech data, includes the following steps: S51. Perform residual fusion on the text embedding features and the target speaker embedding sequence to obtain fused features; S52. Based on the fusion features, perform spectrum synthesis to obtain the target Mel spectrum; S53. Speech conversion is performed based on the target Mel spectrum to obtain the target speech data.

[0065] For steps S51 to S53, firstly, residual fusion is performed on the text embedding features and the target speaker embedding sequence to obtain fused features, which fully integrates the information from both, making the fused features possess both text content and speaker characteristics. Next, spectral synthesis is performed based on the fused features to obtain the target Mel spectrum. Finally, speech conversion is performed based on the target Mel spectrum to obtain high-quality, accurate target speech data that expresses the text and possesses the characteristics of the target speaker.

[0066] Specifically, residual connections are made between the text embedding features and the target speaker embedding sequence, which means adding the text embedding features and the target speaker embedding sequence together to obtain fused features. The fused features can not only retain the key content information of the text and ensure the semantic integrity of the backbone, but also incorporate the speech features of the target speaker. This can give full play to the advantages of text information and speaker features and avoid the semantic dilution that may occur in traditional weighting or splicing.

[0067] After residual fusion, the pre-trained speech synthesis model predicts the corresponding target Mel spectrum based on the fusion features of the input, and the generated target Mel spectrum provides an accurate frequency and energy information basis for subsequent speech conversion.

[0068] It should be noted that the pre-trained speech synthesis model can be a general speech synthesis model. Therefore, it is understood that steps S10 to S40 illustrated above in this application can be generalized to many speech synthesis model tasks, thereby helping to solve the technical problem that the inability to accurately fuse speaker features and text features leads to irrelevant speaker information interfering with or causing semantic drift in the core text semantics, thus affecting the accuracy of speech synthesis.

[0069] After obtaining the target Mel spectrum, a vocoder (such as WaveGlow or Griffin-Lim) is used to convert the target Mel spectrum into speech, resulting in target speech data. This achieves a complete conversion from text and speaker features to actual speech. In fintech scenarios, the final target speech data can convey financial information to users in a natural, fluent manner that matches the target speaker's style, improving user experience and information dissemination effectiveness. In healthcare scenarios, accurate and professional target speech data can help patients better understand medical advice, improving the quality and efficiency of healthcare services.

[0070] As can be seen, in the above solution, for speech synthesis scenarios in the fintech and healthcare fields, the following steps are taken: First, the speech synthesis text and the speaker's identifier code of the target speaker are obtained. The speech synthesis text is then encoded to obtain text embedding features, which accurately capture the text's semantics and grammatical structure. Based on the speaker identifier code, the speaker vector of the target speaker is obtained, effectively representing the unique acoustic features of the target speaker. Next, the speaker vector is serialized based on the text embedding features to obtain the target speaker embedding sequence, enabling better adaptation between acoustic and text features. Finally, speech synthesis is performed based on the text embedding features and the target speaker embedding sequence to obtain the target speech data. This accurately conveys the text content and highly restores the target speaker's timbre, intonation, and other features, improving the naturalness and personalization of the speech synthesis. This solves the technical problem that the inability to accurately integrate speaker and text features leads to irrelevant speaker information interfering with or causing semantic drift in the core text semantics, thus affecting the accuracy of speech synthesis.

[0071] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0072] In one embodiment, a speech synthesis device is provided, which corresponds one-to-one with the speech synthesis methods described in the above embodiments. For example... Figure 8 As shown, the speech synthesis device includes a data acquisition module 101, a text encoding module 102, a speaker vector acquisition module 103, a serialization processing module 104, and a target speech synthesis module 105. Detailed descriptions of each functional module are as follows: Data acquisition module 101 is used to acquire the speech-synthesized text and the speaker identification code of the target speaker; Text encoding module 102 is used to encode the speech-synthesized text to obtain text embedding features; Speaker vector acquisition module 103 is used to acquire the speaker vector of the target speaker based on the speaker identifier code; The serialization processing module 104 is used to serialize the speaker vector based on the text embedding features to obtain the target speaker embedding sequence; The target speech synthesis module 105 is used to synthesize speech based on text embedding features and target speaker embedding sequence to obtain target speech data.

[0073] In one embodiment, the text encoding module 102 is specifically used for: Text cleaning is performed on the speech-synthesized text to obtain the target synthesized text; Word encoding is performed on the target synthesized text to obtain word embedding features; Context encoding is performed on the word embedding features to obtain the text embedding features.

[0074] In one embodiment, the serialization processing module 104 is specifically used for: The speaker vectors are sequence aligned based on text embedding features to obtain the aligned speaker embedding sequence. Speaker weights are calculated based on text embedding features and aligned speaker embedding sequences. The target speaker embedding sequence is obtained by performing aggregation calculations based on speaker weights and aligned speaker embedding sequences.

[0075] In one embodiment, the text embedding features include multiple text sequence embedding features; the serialization processing module 104 is further configured to: Based on text embedding features, the speaker vectors are sequence aligned to obtain the aligned speaker embedding sequence, including: Obtain the text length feature of each text sequence embedding feature, and select the feature with the longest length from the text length features as the maximum length feature; Construct a speaker matrix based on the maximum length feature and the number of text sequence embedding features; The speaker matrix is ​​filled with vectors based on the speaker vectors to obtain the initial speaker embedding matrix; The initial speaker embedding matrix is ​​masked to obtain the aligned speaker embedding sequence.

[0076] In one embodiment, the serialization processing module 104 is specifically used for: The initial speaker embedding matrix is ​​masked to obtain the aligned speaker embedding sequence, including: For each text sequence embedding feature, the position of invalid characters is determined based on the text length feature of the text sequence embedding feature; The initial speaker embedding matrix is ​​masked based on the positions of invalid characters to obtain an aligned speaker embedding sequence.

[0077] In one embodiment, the serialization processing module 104 is specifically used for: The text embedding features and the aligned speaker embedding sequence are concatenated to obtain the concatenated features. A linear transformation is performed on the spliced ​​features to obtain linear features; Activation calculations are performed on the linear features to obtain speaker weights.

[0078] In one embodiment, the target speech synthesis module 105 is specifically used for: Residual fusion is performed on the text embedding features and the target speaker embedding sequence to obtain fused features; The target Mel spectrum is obtained by synthesizing the spectrum based on the fusion features; Speech conversion is performed based on the target Mel spectrum to obtain the target speech data.

[0079] This application provides a speech synthesis device for speech synthesis scenarios in the fintech and healthcare fields. First, it obtains the speech synthesis text and the speaker identifier code of the target speaker. The speech synthesis text is then encoded to obtain text embedding features, which accurately capture the text's semantics and grammatical structure. Based on the speaker identifier code, the speaker vector of the target speaker is obtained, effectively representing the unique acoustic features of the target speaker. Next, the speaker vector is serialized based on the text embedding features to obtain the target speaker embedding sequence, enabling better adaptation between acoustic and text features. Finally, speech synthesis is performed based on the text embedding features and the target speaker embedding sequence to obtain target speech data. This accurately conveys the text content and highly reproduces the target speaker's timbre, intonation, and other features, improving the naturalness and personalization of the speech synthesis. This solves the technical problem that the inability to accurately integrate speaker and text features leads to irrelevant speaker information interfering with or causing semantic drift in the core text semantics, thus affecting the accuracy of speech synthesis.

[0080] For specific limitations regarding the speech synthesis device, please refer to the limitations on the speech synthesis method above, which will not be repeated here. Each module in the aforementioned speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the operations corresponding to each module.

[0081] Please see Figure 9 , Figure 9 The hardware structure of a computer device according to another embodiment is illustrated. The computer device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the speech synthesis method of the embodiments of this application, including: Obtain the synthesized text and the speaker identifier code of the target speaker; The speech-synthesized text is encoded to obtain text embedding features; Obtain the speaker vector of the target speaker based on the speaker identifier code; The speaker vector is serialized based on text embedding features to obtain the target speaker embedding sequence. Speech synthesis is performed based on text embedding features and target speaker embedding sequences to obtain target speech data.

[0082] The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0083] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the synthesized text and the speaker identifier code of the target speaker; The speech-synthesized text is encoded to obtain text embedding features; Obtain the speaker vector of the target speaker based on the speaker identifier code; The speaker vector is serialized based on text embedding features to obtain the target speaker embedding sequence. Speech synthesis is performed based on text embedding features and target speaker embedding sequences to obtain target speech data.

[0084] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0086] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0087] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0088] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the synthesized text and the speaker identifier code of the target speaker; The synthesized speech text is encoded to obtain text embedding features; The speaker vector of the target speaker is obtained based on the speaker identifier code; The speaker vector is serialized based on the text embedding features to obtain the target speaker embedding sequence. Speech synthesis is performed based on the text embedding features and the target speaker embedding sequence to obtain target speech data.

2. The speech synthesis method according to claim 1, characterized in that, The step of serializing the speaker vector based on the text embedding features to obtain the target speaker embedding sequence includes: Based on the text embedding features, the speaker vector is sequence aligned to obtain an aligned speaker embedding sequence; Speaker weights are calculated based on the text embedding features and the aligned speaker embedding sequence. The target speaker embedding sequence is obtained by performing aggregation calculations based on the speaker weights and the aligned speaker embedding sequence.

3. The speech synthesis method according to claim 2, characterized in that, The text embedding features include multiple text sequence embedding features; The step of performing sequence alignment processing on the speaker vector based on the text embedding features to obtain an aligned speaker embedding sequence includes: Obtain the text length feature of each of the text sequence embedding features, and select the feature with the longest length from the text length features as the maximum length feature; Construct a speaker matrix based on the maximum length feature and the number of text sequence embedding features; The speaker matrix is ​​filled with vectors based on the speaker vectors to obtain an initial speaker embedding matrix; The initial speaker embedding matrix is ​​masked to obtain the aligned speaker embedding sequence.

4. The speech synthesis method according to claim 3, characterized in that, The step of masking the initial speaker embedding matrix to obtain the aligned speaker embedding sequence includes: For each of the text sequence embedding features, the position of the invalid character is determined based on the text length feature of the text sequence embedding feature; The initial speaker embedding matrix is ​​masked based on the position of the invalid character to obtain the aligned speaker embedding sequence.

5. The speech synthesis method according to claim 2, characterized in that, The step of calculating speaker weights based on the text embedding features and the aligned speaker embedding sequence includes: The text embedding features and the aligned speaker embedding sequence are concatenated to obtain concatenated features; Perform a linear transformation on the splicing features to obtain linear features; The speaker weights are obtained by performing activation calculations on the linear features.

6. The speech synthesis method according to any one of claims 1 to 5, characterized in that, The process of synthesizing speech based on the text embedding features and the target speaker embedding sequence to obtain target speech data includes: Residual fusion is performed on the text embedding features and the target speaker embedding sequence to obtain fused features; Based on the fusion features, a spectrum is synthesized to obtain the target Mel spectrum; Speech conversion is performed based on the target Mel spectrum to obtain the target speech data.

7. The speech synthesis method according to any one of claims 1 to 5, characterized in that, The encoding process of the synthesized speech text to obtain text embedding features includes: The synthesized text is cleaned to obtain the target synthesized text; The target synthesized text is word-encoded to obtain word embedding features; The word embedding features are then context-encoded to obtain the text embedding features.

8. A speech synthesis device, characterized in that, The device includes: The data acquisition module is used to acquire the synthesized speech text and the speaker identification code of the target speaker; The text encoding module is used to encode the speech-synthesized text to obtain text embedding features; The speaker vector acquisition module is used to acquire the speaker vector of the target speaker based on the speaker identifier code; The serialization processing module is used to serialize the speaker vector based on the text embedding features to obtain the target speaker embedding sequence; The target speech synthesis module is used to synthesize speech based on the text embedding features and the target speaker embedding sequence to obtain target speech data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the speech synthesis method as described in any one of claims 1 to 7.

Images