Artificial intelligence-based speech synthesis method, device, computer equipment and medium

Abstract

The present invention relates to the field of artificial intelligence technology, and more particularly to an artificial intelligence-based speech synthesis method, apparatus, computer equipment, and medium. The method forms a subtext from a text term and its adjacent terms in a text to be synthesized, inputs the text to be synthesized and the subtext into a semantic extraction model to obtain global semantic features and local semantic features, inputs the text to be synthesized into an event classification model, outputs a first classification probability, modifies the first classification probability based on the similarity between the local semantic features and the global semantic features to obtain a second classification probability, performs colloquial processing on text terms corresponding to the second classification probability that meet conditions to obtain an updated term, performs speech synthesis based on the colloquial text formed from the updated term, modifies the classification probability based on feature similarity, obtains a colloquial event type that conforms to language habits, and performs colloquial processing, thereby improving the simulation level of speech synthesis and thereby enhancing the user's inquiry experience on the financial service platform.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based speech synthesis method, apparatus, computer equipment, and medium. Background Art

[0002] With the rise of artificial intelligence technology, speech synthesis technology based on artificial intelligence models has been widely used in financial service platforms. Financial service platforms can be insurance systems, banking systems, transaction systems, order systems, etc. Financial service platforms can support shopping, social networking, interactive games, resource transfer and other functions. They can also have functions such as applying for loans, credit cards, or purchasing insurance and financial products.

[0003] Speech synthesis technology converts text into speech. In financial services platforms, this task involves synthesizing machine customer service voices to enhance the user experience. Existing speech synthesis methods typically analyze the rhythm and sentiment of text and synthesize emotionally responsive speech based on these rhythms and sentiments.

[0004] However, in scenarios where high realism is required for synthesized speech, such as customer service scenarios on financial service platforms, synthesized speech that only reflects emotions still differs significantly from natural speech produced by real people, and the degree of simulation of synthesized speech is low. Therefore, how to improve the degree of simulation of synthesized speech has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a speech synthesis method, apparatus, computer equipment, and medium based on artificial intelligence to solve the problem of low simulation degree of synthesized speech.

[0006] In a first aspect, an embodiment of the present invention provides a speech synthesis method based on artificial intelligence, the speech synthesis method comprising:

[0007] Obtain a text to be synthesized consisting of N text terms, and for any text term, determine adjacent terms of the text term from the text to be synthesized based on the distance between the text terms, and form a subtext corresponding to the text term from all adjacent terms and the text term, where N is an integer greater than zero;

[0008] Input the text to be synthesized into a trained semantic extraction model to obtain global semantic features, and input all sub-texts into the trained semantic extraction model to obtain local semantic features of the corresponding sub-texts;

[0009] Input the text to be synthesized into a trained event classification model, and output the colloquial event type corresponding to each text term and its first classification probability;

[0010] For any local semantic feature corresponding to a text term, calculating the similarity between the local semantic feature and the global semantic feature, multiplying the similarity by the first classification probability corresponding to the text term, and determining the multiplication result as a second classification probability; if the second classification probability is greater than a preset threshold, performing colloquial processing on the text term using the colloquial event type corresponding to the text term, and determining the processing result as an updated term corresponding to the text term;

[0011] A spoken text is formed based on all updated terms, and the spoken text is input into a trained speech synthesis model to obtain a speech synthesis result.

[0012] In a second aspect, an embodiment of the present invention provides a speech synthesis device based on artificial intelligence, the speech synthesis device comprising:

[0013] a subtext construction module for obtaining a text to be synthesized consisting of N text terms, and for any text term, determining adjacent terms of the text term from the text to be synthesized based on the distance between the text terms, and forming a subtext corresponding to the text term from all adjacent terms and the text term, where N is an integer greater than zero;

[0014] A feature extraction module is used to input the text to be synthesized into a trained semantic extraction model to obtain global semantic features, and input all sub-texts into the trained semantic extraction model to obtain local semantic features of the corresponding sub-texts;

[0015] An event classification module, configured to input the text to be synthesized into a trained event classification model and output the colloquial event type corresponding to each text term and its first classification probability;

[0016] a term update module configured to calculate, for each text term, a local semantic feature corresponding to the local semantic feature and the global semantic feature, multiply the similarity by a first classification probability corresponding to the text term, determine the multiplication result as a second classification probability, and, if the second classification probability is greater than a preset threshold, perform colloquial processing on the text term using the colloquial event type corresponding to the text term, and determine the processing result as an updated term corresponding to the text term;

[0017] The speech synthesis module is used to form a spoken text based on all updated terms, and input the spoken text into a trained speech synthesis model to obtain a speech synthesis result.

[0018] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the speech synthesis method as described in the first aspect when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech synthesis method as described in the first aspect is implemented.

[0020] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0021] Obtain a text to be synthesized consisting of N text terms. For any text term, determine the adjacent terms of the text term from the text to be synthesized based on the distance between the text terms. Form a subtext of the corresponding text term from all adjacent terms and the text terms. Input the text to be synthesized into a trained semantic extraction model to obtain global semantic features. Input all subtexts into the trained semantic extraction model to obtain local semantic features of the corresponding subtexts. Input the text to be synthesized into a trained event classification model to output the colloquial event type corresponding to each text term and its first classification probability. For the local semantic features corresponding to any text term, calculate the similarity between the local semantic features and the global semantic features, and multiply the similarity by the first classification probability corresponding to the text term. , determine the multiplication result as the second classification probability, if the second classification probability is greater than the preset threshold, then use the spoken event type corresponding to the text term to perform spoken processing on the text term, determine the processing result as the updated term of the corresponding text term, form a spoken text based on all updated terms, input the spoken text into the trained speech synthesis model to obtain the speech synthesis result, and correct the classification probability of the text term according to the similarity between the local semantic features and the global semantic features of the text term to obtain a spoken event type that is more in line with language habits, process the text term according to the spoken event type to obtain a spoken text, and then synthesize a more realistic spoken voice, thereby improving the simulation degree of speech synthesis, and thus improving the user's inquiry experience in the financial service platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1This is a schematic diagram of an application environment of an artificial intelligence-based speech synthesis method provided in Example 1 of the present invention;

[0024] Figure 2 This is a flow chart of a speech synthesis method based on artificial intelligence provided in the first embodiment of the present invention;

[0025] Figure 3 This is a structural diagram of a speech synthesis device based on artificial intelligence provided in the second embodiment of the present invention;

[0026] Figure 4 This is a structural diagram of a computer device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0027] In the following description, specific details such as particular system structures and techniques are provided for purposes of illustration, not limitation, to facilitate a thorough understanding of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the present invention may be practiced in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

[0028] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0029] It will also be understood that the term "and / or" used in the present description and appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0030] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0031] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" in the present specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present invention. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] Embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0034] Fundamental AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biological retrieval, speech processing, natural language processing, and machine learning / deep learning.

[0035] It should be understood that the order of execution of the steps in the following embodiments does not necessarily mean the order in which they are executed. The order in which each process is executed should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0036] In order to illustrate the technical solution of the present invention, specific embodiments are provided below.

[0037] The first embodiment of the present invention provides a method for speech synthesis based on artificial intelligence, which can be applied in the following situations: Figure 1The application environment in which the client communicates with the server. The client includes but is not limited to computer devices such as PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0038] The client and server can be deployed in a financial service platform, which can be an insurance system, banking system, transaction system, order system, etc. The financial service platform can support functions such as shopping, social networking, interactive games, resource transfer, etc. It can also have functions such as applying for loans, credit cards, or purchasing insurance and financial products. The client provides a speech synthesis service so that a highly simulated machine customer service voice can be synthesized in the customer service service of the financial service platform to improve the user's inquiry experience in the financial service platform.

[0039] See also Figure 2 , is a flow chart of a speech synthesis method based on artificial intelligence provided by the first embodiment of the present invention, the above-mentioned speech synthesis method can be applied to Figure 1 The client in the client, the computer device corresponding to the client connects to the server to obtain the text to be synthesized from the server. The text to be synthesized can refer to the text to be converted into synthesized speech. The computer device corresponding to the client can be deployed with a trained semantic extraction model, a trained event classification model and a trained speech synthesis model. The trained semantic extraction model can be used to extract the semantic features of the input text, the trained event classification model can be used to classify the spoken event type of each word in the input text, and the trained speech synthesis model can be used to convert the input text into a synthesized speech signal. Figure 2 As shown, the speech synthesis method may include the following steps:

[0040] Step S201, obtain a text to be synthesized formed by N text terms, for any text term, determine the adjacent terms of the text term from the text to be synthesized based on the distance between the text terms, and form a subtext of the corresponding text term from all adjacent terms and the text terms.

[0041] Wherein, N is an integer greater than zero, the text to be synthesized may refer to the original text that needs to be speech synthesized. In this embodiment, the text to be synthesized may refer to the customer service script text in the financial service platform. The text to be synthesized may include N text terms, and one text term corresponds to one term position in the text to be synthesized. The distance between two text terms can be determined based on the distance between the corresponding term positions. Adjacent terms may refer to terms near the text terms, and subtext may refer to local text information near the text terms.

[0042] Specifically, corresponding term positions can be assigned to text terms in order from left to right. For example, the term position corresponding to the leftmost text term is 1. Similarly, the term position corresponding to the nth text term from the left is n. The absolute value of the difference between the term positions is used as the distance between the term positions to determine the distance between the text terms.

[0043] In this embodiment, when determining adjacent terms, a term quantity threshold may be set for screening, that is, several other text terms closest to the text term are selected as adjacent terms.

[0044] In one embodiment, when determining adjacent terms, a distance threshold may be set for screening, that is, a number of other text terms whose distance to the text term is less than the distance threshold are selected as adjacent terms.

[0045] It should be noted that when all adjacent terms and text terms form a sub-text of the corresponding text term, a splicing method can be adopted to determine the splicing order according to the term positions corresponding to all adjacent terms and text terms respectively. For example, if the term position of the adjacent term is 1 and the term position of the text term is 2, then the adjacent term in the sub-text is on the left and the text term is on the right.

[0046] Optionally, determining adjacent terms of a text term from the text to be synthesized based on the distance between the text terms includes:

[0047] According to the distance between text terms, the first K reference terms with the closest distance to the text terms are determined from all text terms contained in the text to be synthesized;

[0048] Each reference term is combined with a text term to form a term pair, the frequency of occurrence of the term pair in a preset corpus is obtained, and the reference term in the term pair whose frequency of occurrence is greater than a preset frequency threshold is determined as an adjacent term of the text term.

[0049] Wherein, K is an integer greater than zero and less than N, the reference term may refer to other text terms adjacent to the text term, a single term pair may include a reference term and a text term, the preset corpus may contain several stored texts, the frequency of occurrence may ensure the frequency of occurrence of the term pair in the preset corpus, and the preset frequency threshold may be used to measure whether the term pair is a commonly used term combination.

[0050] Specifically, for any term pair, the number of occurrences of the term pair in all stored texts in the preset corpus is obtained, and the number of occurrences is compared with the number of term combinations in all stored texts to obtain the frequency of occurrence of the corresponding term pair. In this embodiment, the preset frequency threshold is set to 1 / 10000. The implementer can adjust the preset frequency threshold according to the richness of the text stored in the preset corpus. For example, the more stored texts and the longer the text length, the smaller the preset frequency threshold.

[0051] In this embodiment, the reference terms in the term pair whose occurrence frequency is greater than the preset frequency threshold are determined as adjacent terms of the text term. At this time, the sub-text formed by the adjacent terms and the text terms is more in line with common language habits, which facilitates the subsequent determination of the importance of the text terms and avoids the interference of irrelevant terms, thereby improving the simulation degree of subsequent speech synthesis.

[0052] The above-mentioned step of obtaining the text to be synthesized formed by N text terms, for any text term, determining the adjacent terms of the text term from the text to be synthesized based on the distance between the text terms, and forming a sub-text of the corresponding text term from all adjacent terms and text terms, extracting the sub-text containing the text term, facilitates the subsequent analysis of the semantic representation ability of the text term on the text to be synthesized, and further provides guidance information for the spoken processing of the text term, thereby improving the simulation degree of subsequent speech synthesis.

[0053] Step S202: input the text to be synthesized into a trained semantic extraction model to obtain global semantic features, and input all sub-texts into the trained semantic extraction model respectively to obtain local semantic features of the corresponding sub-texts.

[0054] Among them, the trained semantic extraction model can be used to extract the semantic features of the input text, the global semantic features can represent the semantic information of the entire text to be synthesized, and the local semantic features can represent the semantic information of the sub-text.

[0055] Specifically, the trained semantic extraction model can adopt the encoder part of a trained text reconstruction model, a trained text classification model, and other models. The encoder part is mainly composed of convolutional layers and pooling layers, which can be used to extract high-level features of the input text.

[0056] The above steps of inputting the text to be synthesized into the trained semantic extraction model to obtain global semantic features, and inputting all sub-texts into the trained semantic extraction model respectively to obtain local semantic features of the corresponding sub-texts, represent the text semantic information in the form of feature vectors, facilitate the subsequent calculation of semantic information similarity, improve the convenience of calculation, and thus improve the efficiency of speech synthesis.

[0057] Step S203: input the text to be synthesized into the trained event classification model, and output the colloquial event type corresponding to each text term and its first classification probability.

[0058] Among them, the trained event classification model can be used to classify and predict the colloquial event type of each text item in the synthesized text. The colloquial event type can refer to the colloquial processing method corresponding to the text item. For example, the colloquial event type can include repetition type, pause type, extension type and non-processing type, etc. The first classification probability can refer to the probability that the text item is processed colloquially using the colloquial event type.

[0059] Specifically, in this embodiment, the trained event classification model can be a multi-classification model. The training data set of the multi-classification model is composed of several historical spoken texts, and the spoken event types corresponding to the spoken terms in the spoken texts are labeled. The cross-entropy loss function is used for training, that is, the historical spoken texts are input into the event classification model to obtain the predicted spoken event type of each historical text term in the historical spoken text, forming a predicted spoken event type sequence, and the classification loss is calculated according to the predicted spoken event type sequence, the labeled sequence and the cross-entropy loss function. According to the classification loss, the parameters of the event classification model are adjusted using the gradient descent method until the classification loss converges to the minimum, thereby obtaining a trained event classification model.

[0060] In one embodiment, the trained event classification model can also be multiple binary classification models, each binary classification model corresponds to a spoken event type, and the binary classification can include spoken event types and unprocessed types. Since spoken terms are usually less than text terms and belong to sparse labels after annotation, weighted binary cross entropy can be used for training, thereby reducing the impact of the difference between the number of spoken terms and the number of text terms in the training data set on the training process.

[0061] Optionally, the text to be synthesized is input into a trained event classification model, and the colloquial event type corresponding to each text term and its first classification probability are output, including:

[0062] For any text term, the text to be synthesized is input into the trained event classification model, and the output text term corresponds to the classification prediction value of M preset event types;

[0063] Normalize all classification prediction values ​​to obtain the classification probability corresponding to each classification prediction value, determine that the maximum classification probability is the first classification probability corresponding to the text term, and the preset event type corresponding to the maximum classification probability is the spoken event type corresponding to the text term;

[0064] Traverse all text terms and obtain the colloquial event type and its first classification probability corresponding to each text term.

[0065] Wherein, M is an integer greater than zero, and the preset event type may refer to a pre-set colloquial event type. In this case, the trained event classification model can only classify text terms into M preset event types. The classification prediction value may represent the prediction value of the text term being classified into the corresponding preset event type, and the classification probability may represent the probability of the text term being classified into the corresponding preset event type.

[0066] Specifically, any text term corresponds to M classification prediction values. At this time, the exponential normalization function can be used for normalization to obtain M normalized classification probabilities. The preset event type corresponding to the maximum value of the M classification probabilities is the preset event type to which the text term is most likely to be classified.

[0067] The above-mentioned step of inputting the text to be synthesized into the trained event classification model and outputting the colloquial event type corresponding to each text term and its first classification probability can provide a colloquial processing method for the text term, which is convenient for subsequent judgment of whether to perform colloquial processing on the text term, that is, to perform colloquial processing on text terms that meet language habits, thereby improving the simulation degree of speech synthesis.

[0068] In step S204, for the local semantic features corresponding to any text term, the similarity between the local semantic features and the global semantic features is calculated, the similarity is multiplied by the first classification probability corresponding to the text term, and the multiplication result is determined to be the second classification probability. If the second classification probability is greater than a preset threshold, the text term is colloquially processed using the colloquial event type corresponding to the text term, and the processing result is determined to be the updated term of the corresponding text term.

[0069] Among them, similarity can be used to characterize the degree of similarity between local semantic features and global semantic features, the second classification probability can refer to the modified first classification probability, the preset threshold can be used to determine whether the second classification probability is sufficient to perform colloquial processing, and the updated term can refer to the text term that has been colloquially processed.

[0070] Specifically, in this embodiment, the preset threshold can be set to 0.5, and the implementer can adjust the preset threshold according to actual conditions. When the second classification probability is greater than the preset threshold, it means that the probability that the text term should adopt the corresponding spoken event type is large enough, so the text term can be processed in a spoken language.

[0071] Optionally, multiplying the similarity and the first classification probability corresponding to the text term to determine the multiplication result as the second classification probability includes:

[0072] A preset mapping table is used to map similarity into weight coefficients;

[0073] The weight coefficient is multiplied by the first classification probability, and the multiplication result is determined to be the second classification probability.

[0074] Among them, the mapping table contains the mapping relationship between similarity and weight coefficient. The value range of the weight coefficient is [0, 1]. The similarity can be represented by the Euclidean distance. The larger the Euclidean distance, the smaller the similarity, and the smaller the weight coefficient obtained by mapping. Correspondingly, the smaller the Euclidean distance, the greater the similarity, and the larger the weight coefficient obtained by mapping.

[0075] Specifically, the greater the similarity, the greater the semantic contribution of the text term to the text to be synthesized, that is, the text term can be regarded as important information at this time. In general language habits, the degree of colloquialism of a term is related to its importance. For example, for important terms, extension, repetition, pauses, etc. are often used to achieve colloquialism in order to emphasize the importance of the term.

[0076] In this embodiment, the classification probability is corrected by similarity so that the corrected second classification probability can include information on semantic importance, thereby improving the authenticity of the spoken language processing, being more in line with language habits, and improving the simulation degree of speech synthesis.

[0077] Optionally, after determining that the multiplication result is the second classification probability, the method further includes:

[0078] Comparing the second classification probability with a preset threshold to obtain a comparison result;

[0079] If the comparison result is that the second classification probability is less than or equal to the preset threshold, the text term itself is used as the updated term.

[0080] The comparison result may include that the second classification probability is less than or equal to a preset threshold, and that the second classification probability is greater than a preset threshold.

[0081] Specifically, when the second classification probability is less than or equal to a preset threshold, it indicates that the semantic importance of the text term may be relatively low at this time, or the probabilities of classifying it into multiple preset event types may be relatively close, and it is impossible to determine the colloquial event type that should be adopted. For example, the classification probability of the text term for the non-processing type is 0.32, the classification probability for the repetition type is 0.36, and the classification probability for the extension type is 0.32. At this time, although the repetition type is used as the colloquial event type, the probability that it belongs to other types is also relatively large, which may lead to incorrect processing. Therefore, the text term itself is used as the updated term, that is, no colloquial processing is performed. <s

[0082] Optionally, the colloquial event types include a pause event type, a repetition event type, and an extension event type;

[0083] Performing colloquial processing on the text term using the corresponding colloquial event type of the text term, and determining that the processing result is an updated term, includes:

[0084] If the corresponding colloquial event type of the text term is a pause event type, a filler term is added after the text term, and the updated term is formed by the text term and the filler term;

[0085] If the corresponding colloquial event type of the text term is a repetition event type, the text term is copied to obtain a copied term, and the updated term is formed by the text term and the copied term;

[0086] If the corresponding colloquial event type of the text term is an extension event type, the character at the end of the text term is copied to obtain at least one extended character, and the updated term is formed by the text term and all the extended characters. [[ID=!7]]

[0087] Among them, the pause event type may refer to a pause occurring near the text term, the repetition event type may refer to the text term being continuously repeated multiple times, the extension event type may refer to the sound at the end of the text term being elongated, the filler term can be used to represent a pause operation, the copied term is the same as the text term, and the extended character can be the same as the character at the end of the text term.

[0088] Specifically, in this embodiment, a filler term can be added after the text term, and the filler term can include "um", "ah", "uh", "mhm", etc., and the updated term is formed by the text term and the filler term.

[0089] [[ID=!4]]In one implementation manner, a filler term can also be added before the text term, and the updated term is formed by the text term and the filler term.

[0090] When copying the character at the end of the text term, it can be copied multiple times to obtain several extended characters, and then the updated term is formed by the text term and all the extended characters.

[0091] In this embodiment, a variety of colloquial event types are used to perform colloquial processing on text items, so that the colloquial processing results are more realistic and have a higher degree of simulation.

[0092] The above-mentioned steps are as follows: for any local semantic feature corresponding to a text term, the similarity between the local semantic feature and the global semantic feature is calculated, the similarity is multiplied by the first classification probability corresponding to the text term, and the multiplication result is determined to be the second classification probability. If the second classification probability is greater than a preset threshold, the text term is colloquially processed using the colloquial event type corresponding to the text term, and the processing result is determined to be the updated term of the corresponding text term. The first classification probability is corrected by the similarity so that the corrected second classification probability can cover information on semantic importance, thereby improving the authenticity of the text after colloquial processing and being more in line with colloquial language habits, thereby improving the simulation degree of subsequent speech synthesis.

[0093] Step S205 , forming a spoken text based on all updated terms, and inputting the spoken text into a trained speech synthesis model to obtain a speech synthesis result.

[0094] Among them, the spoken text may refer to the text to be converted into synthesized speech with spoken features, the speech synthesis result may refer to the synthesized speech with spoken features, and the trained speech synthesis model may adopt a trained FastSpeech model, a trained Tactron model, etc.

[0095] Optionally, a spoken text is formed from all updated terms, including:

[0096] Set the number of updated terms to C, and randomly select C updated terms from all updated terms that have undergone colloquial processing;

[0097] The corresponding text terms in the text to be synthesized are replaced with C update terms to obtain the spoken text.

[0098] Wherein, C is an integer greater than zero, and the number of updated terms may refer to the number of text terms that need to be processed colloquially.

[0099] Specifically, in this embodiment, C can be set to 3, and the implementer can adjust the number of term updates according to actual conditions. Since the updated terms include text terms that have been colloquially processed and text terms that have not been processed, only the text terms that have been colloquially processed are randomly selected at this time. During the random selection, all updated terms that have been colloquially processed can be assigned the same sampling probability and sampled without replacement to obtain C sampled terms.

[0100] It should be noted that since the spoken language processing requires the addition of words or characters, after replacing the corresponding text words in the synthesized text with C updated words, the replacement results need to be normalized in size, that is, a maximum size is set and the size of the replacement results is expanded to the maximum size. The expansion method can be zero-padding.

[0101] In this embodiment, by setting the number of updated terms, the number of text terms that are colloquially processed is controlled, thereby avoiding excessive colloquial terms, being more in line with language habits, and improving the simulation degree of colloquial text.

[0102] The above steps of forming a spoken text based on all updated terms, inputting the spoken text into a trained speech synthesis model, and obtaining a speech synthesis result can synthesize a more realistic spoken speech based on the spoken text, thereby improving the simulation level of the speech synthesis.

[0103] In this embodiment, the classification probability of the text term is corrected based on the similarity between the local semantic features and the global semantic features of the text term to obtain a colloquial event type that is more in line with language habits. The text term is processed according to the colloquial event type to obtain a colloquial text, and then a more realistic colloquial speech is synthesized, thereby improving the simulation degree of speech synthesis and improving the user's inquiry experience in the financial service platform.

[0104] Corresponding to the artificial intelligence-based speech synthesis method of the above embodiment, Figure 3 The following is a block diagram of the structure of an artificial intelligence-based speech synthesis device provided in the second embodiment of the present invention. The speech synthesis device is applied to a client. The computer device corresponding to the client is connected to the server to obtain the text to be synthesized from the server. The text to be synthesized can refer to the text to be converted into synthesized speech. The computer device corresponding to the client can be deployed with a trained semantic extraction model, a trained event classification model, and a trained speech synthesis model. The trained semantic extraction model can be used to extract semantic features of the input text, the trained event classification model can be used to classify the colloquial event type of each word in the input text, and the trained speech synthesis model can be used to convert the input text into a synthesized speech signal. For ease of explanation, only the parts related to the embodiment of the present invention are shown.

[0105] See also Figure 3 , the speech synthesis device comprises:

[0106] The subtext construction module 31 is configured to obtain a text to be synthesized consisting of N text terms, and for each text term, determine the adjacent terms of the text term from the text to be synthesized based on the distance between the text terms, and form a subtext of the corresponding text term from all the adjacent terms and the text term, where N is an integer greater than zero.

[0107] The feature extraction module 32 is used to input the text to be synthesized into the trained semantic extraction model to obtain global semantic features, and input all sub-texts into the trained semantic extraction model to obtain local semantic features of the corresponding sub-texts;

[0108] An event classification module 33 is used to input the text to be synthesized into a trained event classification model and output the colloquial event type corresponding to each text term and its first classification probability;

[0109] The term update module 34 is configured to calculate, for each text term, a similarity between the local semantic feature and the global semantic feature, multiply the similarity by the first classification probability corresponding to the text term, and determine the multiplication result as a second classification probability. If the second classification probability is greater than a preset threshold, the text term is colloquially processed using the colloquial event type corresponding to the text term, and the processing result is determined as an updated term for the corresponding text term.

[0110] The speech synthesis module 35 is used to form a spoken text based on all updated terms, and input the spoken text into a trained speech synthesis model to obtain a speech synthesis result.

[0111] Optionally, the subtext construction module 31 includes:

[0112] a term screening unit for determining, based on the distances between the text terms, the first K reference terms that are closest to the text terms from all the text terms contained in the text to be synthesized, where K is an integer greater than zero and less than N;

[0113] The term pair construction unit is used to combine each reference term with a text term to form a term pair, obtain the frequency of occurrence of the term pair in a preset corpus, and determine the reference term in the term pair whose frequency of occurrence is greater than a preset frequency threshold as an adjacent term of the text term.

[0114] Optionally, the event classification module 33 includes:

[0115] A type prediction unit is used to input the text to be synthesized into a trained event classification model for any text term, and output classification prediction values ​​corresponding to M preset event types of the text term, where M is an integer greater than zero;

[0116] a normalization processing unit, configured to perform normalization processing on all classification prediction values ​​to obtain a classification probability corresponding to each classification prediction value, determine that the maximum classification probability is the first classification probability corresponding to the text term, and determine that the preset event type corresponding to the maximum classification probability is the spoken event type corresponding to the text term;

[0117] The term traversal unit is used to traverse all text terms to obtain the colloquial event type corresponding to each text term and its first classification probability.

[0118] Optionally, the term updating module 34 includes:

[0119] A weight mapping unit, configured to map the similarity into a weight coefficient using a preset mapping table, wherein the mapping table includes a mapping relationship between the similarity and the weight coefficient;

[0120] The weighted calculation unit is used to multiply the weight coefficient and the first classification probability to determine the multiplication result as the second classification probability.

[0121] Optionally, the term updating module 34 further includes:

[0122] a threshold comparison unit, configured to compare the second classification probability with a preset threshold to obtain a comparison result;

[0123] The term updating unit is configured to use the text term itself as an updated term if the comparison result shows that the second classification probability is less than or equal to a preset threshold.

[0124] Optionally, the spoken event type includes a pause event type, a repetition event type, and an extension event type;

[0125] The term update module 34 includes:

[0126] a first updating unit configured to add a filler term after the text term if the colloquial event type corresponding to the text term is a pause event type, so as to form an updated term from the text term and the filler term;

[0127] a second updating unit configured to copy the text term to obtain a copied term if the colloquial event type corresponding to the text term is a repeated event type, and to form an updated term from the text term and the copied term;

[0128] The third updating unit is used to copy the characters at the end of the text term if the colloquial event type corresponding to the text term is an extended event type, to obtain at least one extended character, and to form an updated term by the text term and all the extended characters.

[0129] Optionally, the speech synthesis module 35 includes:

[0130] a term selection unit, configured to set the number of updated terms to C, and randomly select C updated terms from all updated terms that have undergone colloquial processing, where C is an integer greater than zero;

[0131] The term replacement unit is used to replace the corresponding text terms in the to-be-synthesized text with C updated terms to obtain a spoken text.

[0132] It should be noted that the information interaction, execution process, etc. between the above-mentioned modules and units are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0133] Figure 4 This is a schematic diagram of the structure of a computer device provided in Example 3 of the present invention. Figure 4 As shown, the computer device of this embodiment includes: at least one processor ( Figure 4 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned embodiments of the speech synthesis method are implemented.

[0134] The computer device may include, but is not limited to, a processor and a memory. It will be understood by those skilled in the art that Figure 4 The above is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include a network interface, a display screen, and an input device.

[0135] The processor may be a CPU, or other general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. A general-purpose processor may be a microprocessor, or any conventional processor.

[0136] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory can be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium can be the hard disk of the computer device, and in other embodiments, it can also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Furthermore, the memory can also include both the internal storage unit of the computer device and the external storage device. The memory is used to store the operating system, application programs, boot loaders (BootLoader), data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or is about to be output.

[0137] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned method embodiment. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include at least: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0138] The present invention may implement all or part of the processes in the above-mentioned method embodiments, and may also be completed through a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0139] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0140] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0141] In the embodiments provided by the present invention, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely illustrative. For example, the division of modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0142] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0143] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A speech synthesis method based on artificial intelligence, characterized in that: The speech synthesis method comprises: Obtain a text to be synthesized consisting of N text terms, and for any text term, determine adjacent terms of the text term from the text to be synthesized based on the distance between the text terms, and form a subtext corresponding to the text term from all adjacent terms and the text term, where N is an integer greater than zero; Input the text to be synthesized into a trained semantic extraction model to obtain global semantic features, and input all sub-texts into the trained semantic extraction model to obtain local semantic features of the corresponding sub-texts; Input the text to be synthesized into a trained event classification model, and output the colloquial event type corresponding to each text term and its first classification probability; For any local semantic feature corresponding to a text term, calculating the similarity between the local semantic feature and the global semantic feature, multiplying the similarity by the first classification probability corresponding to the text term, and determining the multiplication result as a second classification probability; if the second classification probability is greater than a preset threshold, performing colloquial processing on the text term using the colloquial event type corresponding to the text term, and determining the processing result as an updated term corresponding to the text term; A spoken text is formed based on all updated terms, and the spoken text is input into a trained speech synthesis model to obtain a speech synthesis result.

2. The speech synthesis method according to claim 1, wherein: The step of determining adjacent terms of the text terms from the text to be synthesized based on the distance between the text terms includes: According to the distances between the text terms, determining the first K reference terms that are closest to the text terms from all the text terms contained in the text to be synthesized, where K is an integer greater than zero and less than N; Each reference term is combined with a text term to form a term pair, the occurrence frequency of the term pair in a preset corpus is obtained, and the reference term in the term pair whose occurrence frequency is greater than a preset frequency threshold is determined as an adjacent term of the text term.

3. The speech synthesis method according to claim 1, wherein: The step of inputting the text to be synthesized into a trained event classification model and outputting the colloquial event type corresponding to each text term and its first classification probability includes: For any text term, input the text to be synthesized into a trained event classification model, and output classification prediction values ​​corresponding to M preset event types of the text term, where M is an integer greater than zero; Normalizing all classification prediction values ​​to obtain a classification probability corresponding to each classification prediction value, determining that the maximum classification probability is the first classification probability corresponding to the text term, and the preset event type corresponding to the maximum classification probability is the spoken event type corresponding to the text term; Traverse all text terms and obtain the colloquial event type and its first classification probability corresponding to each text term.

4. The speech synthesis method according to claim 1, wherein: The multiplying the similarity and the first classification probability corresponding to the text term to determine the multiplication result as the second classification probability includes: Mapping the similarity into a weight coefficient using a preset mapping table, wherein the mapping table includes a mapping relationship between the similarity and the weight coefficient; The weight coefficient and the first classification probability are multiplied together to determine a multiplication result as the second classification probability.

5. The speech synthesis method according to claim 1, wherein: After determining that the multiplication result is the second classification probability, the method further includes: Comparing the second classification probability with the preset threshold to obtain a comparison result; If the comparison result is that the second classification probability is less than or equal to the preset threshold, the text term itself is used as the updated term.

6. The speech synthesis method according to claim 1, wherein: The spoken event types include pause event types, repetition event types and extension event types; The performing colloquial processing on the text term using the colloquial event type corresponding to the text term and determining that the processing result is an updated term includes: If the spoken event type corresponding to the text term is the pause event type, then adding a filler term after the text term to form the updated term from the text term and the filler term; If the spoken event type corresponding to the text term is the repeated event type, the text term is copied to obtain a copied term, and the updated term is formed by the text term and the copied term; If the spoken event type corresponding to the text term is the extension event type, the characters at the end of the text term are copied to obtain at least one extended character, and the updated term is formed by the text term and all the extended characters.

7. The speech synthesis method according to any one of claims 1 to 6, characterized in that: The forming of spoken text based on all updated terms includes: Set the number of updated terms to C, and randomly select C updated terms from all updated terms that have been colloquially processed, where C is an integer greater than zero; The corresponding text terms in the text to be synthesized are replaced with the C updated terms respectively to obtain the spoken text.

8. A speech synthesis device based on artificial intelligence, characterized in that: The speech synthesis device comprises: a subtext construction module for obtaining a text to be synthesized consisting of N text terms, and for any text term, determining adjacent terms of the text term from the text to be synthesized based on the distance between the text terms, and forming a subtext corresponding to the text term from all adjacent terms and the text term, where N is an integer greater than zero; A feature extraction module is used to input the text to be synthesized into a trained semantic extraction model to obtain global semantic features, and input all sub-texts into the trained semantic extraction model to obtain local semantic features of the corresponding sub-texts; An event classification module, configured to input the text to be synthesized into a trained event classification model and output the colloquial event type corresponding to each text term and its first classification probability; a term update module configured to calculate, for each text term, a local semantic feature corresponding to the local semantic feature and the global semantic feature, multiply the similarity by a first classification probability corresponding to the text term, determine the multiplication result as a second classification probability, and, if the second classification probability is greater than a preset threshold, perform colloquial processing on the text term using the colloquial event type corresponding to the text term, and determine the processing result as an updated term corresponding to the text term; The speech synthesis module is used to form a spoken text based on all updated terms, and input the spoken text into a trained speech synthesis model to obtain a speech synthesis result.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.

Images