A speech synthesis method based on syntax and prosody and related equipment
Through the syntax-pronunciation method, the recursive sound system structure is constructed and the trained model is called, which solves the problem of poor pronunciation quality in traditional Chinese pronunciation synthesis methods, and achieves a more natural and high-quality pronunciation synthesis effect.
Patent Information
- Application Number
- CN202510032215.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-08
- Publication Date
- 2025-04-08
AI Technical Summary
Traditional Chinese pronunciation synthesis methods have problems such as poor pronunciation quality, not natural enough, and not outstanding enough.
A syntax-pronunciation method is used to construct a recursive sound system structure, including phoneme sequences, accent marking data, pronunciation phrase marking data and pronunciation clause marking data, and a trained recursive sound system model is called for pronunciation synthesis.
A more natural and higher quality speech synthesis is achieved, accurately capturing the initial down-tonation in the speech, and successfully reproducing syntactic differences, building a hierarchy of speech.
Smart Images
Figure CN119785769A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech synthesis technology, and is applicable to the financial or medical fields, and in particular to a syntax-prosody-based speech synthesis method and related equipment. Background Art
[0002] End-to-end text-to-speech (TTS) systems have significantly improved the quality of speech synthesis for alphabet-based languages such as English. Unlike English, which only contains 26 different alphabets, Chinese presents unique challenges due to its large character set and inconsistency in pronunciation.
[0003] To cope with this complexity, Chinese end-to-end speech synthesis methods rely on using phoneme sequences to achieve the purpose of synthesizing speech.
[0004] However, the applicant has found that the traditional speech synthesis method still has the problem that the synthesized speech quality is poor, unnatural, and the effect is not prominent enough. Summary of the invention
[0005] The purpose of the embodiments of the present application is to propose a syntax-prosody based speech synthesis method and related equipment to solve the problems that the traditional speech synthesis method still has poor quality, unnaturalness and insufficient effect of synthesized speech.
[0006] In order to solve the above technical problems, the present application provides a syntax-prosody-based speech synthesis method, which adopts the following technical solution:
[0007] Receiving a speech synthesis request carrying speech synthesis text sent by a user terminal;
[0008] A recursive phonological structure is constructed according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data, and phonological clause marking data, and the recursive phonological structure PPhrase n It is expressed as:
[0009] PPhrase n =PPhrase n-1 +PClause n
[0010] Among them, PPhrase n Indicates the nth phonological phrase annotation data, PClause n Indicates the nth phonological clause annotation data;
[0011] The trained recursive phonological model is called, and the recursive phonological structure is input into the trained recursive phonological model to perform a speech synthesis operation to obtain target audio data.
[0012] Furthermore, the step of constructing a recursive phonological structure according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data, and phonological clause marking data, specifically includes the following steps:
[0013] Performing a syntactic parsing operation on the speech synthesis text according to a syntactic parser to obtain syntactic structure data;
[0014] Converting the syntactic structure data into the phoneme sequence and the syntactic structure data carrying the accent marking data;
[0015] The syntactic structure data carrying the stress marking data is converted into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure.
[0016] Furthermore, the step of converting the syntactic structure data carrying the stress marking data into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure specifically includes the following steps:
[0017] The particle phrases controlling the noun phrases and the verb phrases in the syntactic structure data are converted into the phonological phrase annotation data, wherein the phonological phrase annotation data PPhrase is represented as:
[0018] PPhrase=PP+NP+VP
[0019] Wherein, PP represents the particle phrase, NP represents the noun phrase, and VP represents the verb phrase;
[0020] The inflectional phrases or compound structures in the syntactic structure data are marked as the phonological clause annotation data, wherein the clause annotation data PClause is represented as:
[0021] PClause=IP or PClause=CP
[0022] Wherein, IP represents the inflectional phrase, and CP represents the compound structure.
[0023] Furthermore, after the step of converting the syntactic structure data carrying the stress marking data into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure, the following step is also included:
[0024] The recursive phonological structure is subjected to a phonological boundary insertion operation according to the stress apex principle and the anti-spacing constraint principle.
[0025] Furthermore, after the step of inserting the phonological boundary into the recursive phonological structure according to the stress apex principle and the anti-spacing constraint principle, the following step is also included:
[0026] The redundant structure elimination operation is performed on the recursive phonological structure after the phonological boundary insertion operation.
[0027] Furthermore, before the step of calling the trained recursive phonological model and inputting the recursive phonological structure into the trained recursive phonological model to perform speech synthesis operation to obtain target audio data, the following steps are also included:
[0028] Reading a system database, and acquiring model training data in the system database, wherein the model training data includes training text data and training voice data corresponding to the training text data;
[0029] converting the training text data into a training recursive phonological structure;
[0030] The original text-to-speech model is called, and the training recursive phonological structure is used as input data and the training speech data is used as output data to perform a model training operation on the original text-to-speech model to obtain the trained recursive phonological model.
[0031] In order to solve the above technical problems, the embodiment of the present application further provides a speech synthesis device based on syntax-prosody, which adopts the following technical solution:
[0032] A request receiving module, used for receiving a speech synthesis request carrying speech synthesis text sent by a user terminal;
[0033] A recursive phonological structure construction module is used to construct a recursive phonological structure according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data and phonological clause marking data, and the recursive phonological structure PPhrase n It is expressed as:
[0034] PPhrase n =PPhrase n-1 +PClause n
[0035] Among them, PPhrase n Indicates the nth phonological phrase annotation data, PClause n Indicates the nth phonological clause annotation data;
[0036] The speech synthesis module is used to call the trained recursive phonology model and input the recursive phonology structure into the trained recursive phonology model to perform speech synthesis operation to obtain target audio data.
[0037] Furthermore, the recursive phonological structure building module includes:
[0038] A syntactic parsing submodule, used for performing a syntactic parsing operation on the speech synthesis text according to a syntactic parser to obtain syntactic structure data;
[0039] A first syntactic structure conversion submodule, configured to convert the syntactic structure data into the phoneme sequence and syntactic structure data carrying stress marking data;
[0040] The second syntactic structure conversion submodule is used to convert the syntactic structure data carrying the stress marking data into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure.
[0041] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0042] The invention comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned syntax-prosody-based speech synthesis method when executing the computer-readable instructions.
[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0044] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the syntax-prosody based speech synthesis method as described above are implemented.
[0045] The present application provides a syntax-prosody-based speech synthesis method, comprising: receiving a speech synthesis request carrying a speech synthesis text sent by a user terminal; constructing a recursive phonological structure according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data, and phonological clause marking data, and the recursive phonological structure PPhrase n Expressed as: PPhrase n =PPhrase n-1 +PClause n , among which, PPhrase n Indicates the nth phonological phrase annotation data, PClause nIndicates the nth phonological clause annotation data; calls the trained recursive phonological model, and inputs the recursive phonological structure into the trained recursive phonological model for speech synthesis operation to obtain target audio data. Compared with the prior art, the present application introduces a recursive phonological model, which can accurately capture the initial falling tone in speech and successfully reproduce syntactic differences, effectively construct the hierarchical structure of speech, and generate more natural speech. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0047] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0048] Figure 2 is a flow chart of an implementation of a syntax-prosody-based speech synthesis method provided in an embodiment of the present application;
[0049] Figure 3 is a structural schematic diagram of a syntax-prosody-based speech synthesis device provided in an embodiment of the present application;
[0050] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0052] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0053] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0054] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0055] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0056] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.
[0057] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .
[0058] It should be noted that the syntax-prosody-based speech synthesis method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the syntax-prosody-based speech synthesis device is generally arranged in the server / terminal device.
[0059] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0060] Continue to refer Figure 2, shows a flow chart of an embodiment of a syntax-prosody-based speech synthesis method according to the present application. The syntax-prosody-based speech synthesis method comprises: step S201, step S202 and step S203.
[0061] In step S201, a speech synthesis request carrying speech synthesis text sent by a user terminal is received.
[0062] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for the convenience of understanding and are not used to limit the present application.
[0063] In the embodiment of the present application, the user inputs a speech synthesis request through their terminal device (such as a mobile phone, computer, etc.), and the input is received by the system. The speech synthesis text is the specific content that the user wants the system to synthesize. Specifically, the speech synthesis text can be "transaction data or payment data or business data or purchase data" related to a financial institution (such as a bank, etc.). The speech synthesis text can also be medical data related to medical scenarios, such as personal health records, prescriptions, examination reports and other data. It should be understood that the examples of speech synthesis text here are only for the convenience of understanding and are not used to limit the present application.
[0064] In step S202, a recursive phonological structure is constructed according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data, and phonological clause marking data. The recursive phonological structure PPhrase n It is expressed as:
[0065] PPhrase n =PPhrase n-1 +PClause n
[0066] Among them, PPhrase n Indicates the nth phonological phrase annotation data, PClause n Represents the nth phonological clause annotation data.
[0067] In an embodiment of the present application, a method for constructing a recursive phonological structure based on a speech synthesis text may be to perform a syntactic parsing operation on the speech synthesis text according to a syntactic parser to obtain syntactic structure data; convert the syntactic structure data into a phoneme sequence and syntactic structure data carrying stress marking data; and convert the syntactic structure data carrying stress marking data into phonological phrase marking data and phonological clause marking data according to a syntax-prosody mapping relationship to obtain a recursive phonological structure.
[0068] In step S203, the trained recursive phonological model is called, and the recursive phonological structure is input into the trained recursive phonological model for speech synthesis operation to obtain target audio data.
[0069] In the embodiment of the present application, the sentence end marker "." represents a declarative sentence, "?" represents an interrogative sentence, and "," represents a pause in the middle of a sentence.
[0070] In an embodiment of the present application, a new speech synthesis method called a recursive phonological model is provided. The model integrates the insights of various phonological theories, is mainly based on the phonological structure proposed by the (syntax-prosodic mapping hypothesis), and combines the boundary-driven theory and the prosodic legitimacy constraint. The uniqueness of the model is that, in addition to using phoneme symbols, specific symbols are introduced to represent stress, phonological phrases and phonological clauses, which are represented by "\", "[]" and "{}" respectively. Since the present application emphasizes the recursiveness of PPhrases in this method, the present application is called a recursive phonological model.
[0071] In the embodiments of the present application, through the above method, the model of the present application can effectively capture the complex prosodic features in the speech and generate more natural speech.
[0072] In an embodiment of the present application, a syntax-prosody-based speech synthesis method is provided, comprising: receiving a speech synthesis request carrying a speech synthesis text sent by a user terminal; constructing a recursive phonological structure according to the speech synthesis text, wherein the recursive phonological structure comprises a phoneme sequence, stress marking data, phonological phrase marking data, and phonological clause marking data, and the recursive phonological structure PPhrase n Expressed as: PPhrase n =PPhrase n-1 +PClause n , among which, PPhrase n Indicates the nth phonological phrase annotation data, PClause nIndicates the nth phonological clause annotation data; calls the trained recursive phonological model, and inputs the recursive phonological structure into the trained recursive phonological model for speech synthesis operation to obtain target audio data. Compared with the prior art, the present application introduces a recursive phonological model, which can accurately capture the initial falling tone in speech and successfully reproduce syntactic differences, effectively construct the hierarchical structure of speech, and generate more natural speech.
[0073] In some optional implementations of the embodiments of the present application, the above-mentioned steps of constructing a recursive phonological structure according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data, and phonological clause marking data, specifically include the following steps:
[0074] Performing syntactic parsing operation on the speech synthesis text according to the syntactic parser to obtain syntactic structure data;
[0075] Converting the syntactic structure data into a phoneme sequence and syntactic structure data carrying stress marking data;
[0076] According to the syntax-prosody mapping relationship, the syntactic structure data carrying the stress marking data is converted into phonological phrase marking data and phonological clause marking data to obtain a recursive phonological structure.
[0077] In the embodiment of the present application, the present application first uses a syntactic parser to automatically extract the syntactic structure, and then generates a phoneme sequence and stress by converting the text, wherein the output of the syntactic parser is converted into a representation of phonemes and stresses.
[0078] In some optional implementations of the embodiments of the present application, the step of converting the syntactic structure data carrying the stress marking data into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure specifically includes the following steps:
[0079] The particle phrases that dominate the noun phrases and verb phrases in the syntactic structure data are converted into phonological phrase annotation data, wherein the phonological phrase annotation data PPhrase is represented as:
[0080] PPhrase=PP+NP+VP
[0081] Among them, PP represents a particle phrase, NP represents a noun phrase, and VP represents a verb phrase;
[0082] The inflectional phrases or compound structures in the syntactic structure data are marked as phonological clause annotation data, wherein the clause annotation data PClause is represented as:
[0083] PClause=IP or PClause=CP
[0084] Among them, IP represents an inflectional phrase and CP represents a compound structure.
[0085] In the embodiment of the present application, according to the syntax-prosodic mapping hypothesis, in the syntactic structure, the particle phrase (PP) that dominates the noun phrase (NP) and the verb phrase (VP) is converted into a phonological phrase (PPhrases), represented by "[]". In order to realize the phonological clause (PClauses), the present application marks the IP output by the syntax parser as PClause, represented by "{}". In addition, if the CP dominates the IP, only the CP is marked as PClause, and the IP is replaced by "{}", specifically:
[0086] Phonological phrases (PPhrases) construction:
[0087] PPhrase=PP+NP+VP
[0088] Among them, PP represents a particle phrase, NP represents a noun phrase, and VP represents a verb phrase;
[0089] Construction of PClauses:
[0090] PClause=IP or PClause=CP
[0091] Among them, IP represents an inflectional phrase and CP represents a compound structure.
[0092] In some optional implementations of the embodiments of the present application, after the step of converting the syntactic structure data carrying the stress marking data into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure, the following steps are also included:
[0093] The phonological boundary insertion operation is performed on the recursive phonological structure according to the principle of stress apex and the principle of anti-interval constraint.
[0094] In the embodiment of the present application, according to the principles of stress apex and anti-spacing constraints, a phonological boundary is required after each stress. This means that the AA (stress + stress) sequence is separated due to stress apex, while the AU (stress + unstressed) sequence is separated due to anti-spacing constraints. In contrast, UA and UU sequences can form a PPhrase, such as [UA] and [UU]. However, AA and AU sequences must be divided into [A][A] and [A][U], respectively. For example, in the first process, the AU sequence is represented as "[(\)()]", which is converted to "[[\][]]" by inserting a PPhrase boundary. On the other hand, the UA sequence is represented as "[()(\)]", which does not violate stress apex or anti-spacing constraints, so there is no need to insert a PPhrase boundary and is converted to a single PPhrase "[\]", specifically:
[0095] AA sequence:
[0096] AA→[A][A]
[0097] AU sequence:
[0098] AU→[A][U]
[0099] UA and UU sequences:
[0100] UA→[UA], UU→[UU]
[0101] In some optional implementations of the embodiments of the present application, after the step of inserting a phonological boundary into the recursive phonological structure according to the stress apex principle and the anti-spacing constraint principle, the following steps are also included:
[0102] The redundant structure elimination operation is performed on the recursive phonological structure after the phonological boundary insertion operation.
[0103] In the embodiment of the present application, the redundant structure elimination operation means that according to the constraint, the internal PPhrase is acceptable only when there are multiple PPhrases in the external PPhrase. If there is only one PPhrase in the external PPhrase, the external PPhrase is removed.
[0104] In some optional implementations of the embodiments of the present application, before the step of calling the trained recursive phonological model and inputting the recursive phonological structure into the trained recursive phonological model for speech synthesis operation to obtain the target audio data, the following steps are also included:
[0105] Reading a system database, and obtaining model training data from the system database, wherein the model training data includes training text data and training speech data corresponding to the training text data;
[0106] Converting training text data into training recursive phonological structures;
[0107] The original text-to-speech model is called, and the training recursive phonological structure is used as input data and the training speech data is used as output data to perform a model training operation on the original text-to-speech model to obtain a trained recursive phonological model.
[0108] In the embodiment of the present application, the training process of the model of the present application can be summarized into the following steps:
[0109] 1. Input text preprocessing: Convert the input text into a phoneme sequence and mark the stress, phonological phrases and phonological clauses.
[0110] 2. Recursive phonological structure construction: Recursively construct phonological phrases to form a complete phonological structure.
[0111] 3. Model training: Use the recursive phonological structure and the corresponding speech data to train the TTS model.
[0112] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0113] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0115] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0116] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device based on syntax-prosody, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0117] like Figure 3 As shown, the syntax-prosody based speech synthesis device 200 of the embodiment of the present application includes:
[0118] The request receiving module 210 is used to receive a speech synthesis request carrying speech synthesis text sent by a user terminal;
[0119] The recursive phonological structure construction module 220 is used to construct a recursive phonological structure according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data and phonological clause marking data, and the recursive phonological structure PPhrase n It is expressed as:
[0120] PPhrase n =PPhrase n-1 +PClause n
[0121] Among them, PPhrase n Indicates the nth phonological phrase annotation data, PClause n Indicates the nth phonological clause annotation data;
[0122] The speech synthesis module 230 is used to call the trained recursive phonology model and input the recursive phonology structure into the trained recursive phonology model to perform speech synthesis operation to obtain target audio data.
[0123] In an embodiment of the present application, a syntax-prosody-based speech synthesis device 200 is provided, comprising: a request receiving module 210, for receiving a speech synthesis request carrying a speech synthesis text sent by a user terminal; a recursive phonological structure construction module 220, for constructing a recursive phonological structure according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data, and phonological clause marking data, and the recursive phonological structure PPhrase n Expressed as: PPhrase n =PPhrase n-1 +PClause n , among which, PPhrase n Indicates the nth phonological phrase annotation data, PClause n The nth phonological clause annotation data is represented; the speech synthesis module 230 is used to call the trained recursive phonological model, and input the recursive phonological structure into the trained recursive phonological model for speech synthesis operation to obtain the target audio data. Compared with the prior art, the present application introduces a recursive phonological model, which can accurately capture the initial falling tone in the speech, successfully reproduce the syntactic differences, effectively construct the hierarchical structure of the speech, and generate more natural speech.
[0124] In some optional implementations of the embodiments of the present application, the recursive phonological structure building module includes:
[0125] The syntactic parsing submodule is used to perform syntactic parsing operations on the speech synthesis text according to the syntactic parser to obtain syntactic structure data;
[0126] A first syntactic structure conversion submodule, used for converting syntactic structure data into a phoneme sequence and syntactic structure data carrying stress marking data;
[0127] The second syntactic structure conversion submodule is used to convert the syntactic structure data carrying the stress marking data into phonological phrase marking data and phonological clause marking data according to the syntax-prosody mapping relationship to obtain a recursive phonological structure.
[0128] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.
[0129] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0130] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0131] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the syntax-prosody based speech synthesis method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.
[0132] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as computer-readable instructions for running the syntax-prosody-based speech synthesis method.
[0133] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.
[0134] The computer device provided by the present application introduces a recursive phonological model, which can accurately capture the initial falling tone in speech and successfully reproduce syntactic differences, and can effectively construct the hierarchical structure of speech and generate more natural speech.
[0135] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the syntax-prosody-based speech synthesis method as described above.
[0136] The computer-readable storage medium provided by the present application introduces a recursive phonological model, which can accurately capture the initial falling tone in speech and successfully reproduce syntactic differences, and can effectively construct the hierarchical structure of speech and generate more natural speech.
[0137] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0138] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A method for speech synthesis based on syntax and prosody, characterized in that: The steps include: Receiving a speech synthesis request carrying speech synthesis text sent by a user terminal; A recursive phonological structure is constructed according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data, and phonological clause marking data, and the recursive phonological structure PPhrase n It is expressed as: PPhrase n =PPhrase n-1 +PClause n Among them, PPhrase n Indicates the nth phonological phrase annotation data, PClause n Indicates the nth phonological clause annotation data; The trained recursive phonological model is called, and the recursive phonological structure is input into the trained recursive phonological model to perform a speech synthesis operation to obtain target audio data.
2. The method for speech synthesis based on syntax and prosody according to claim 1, characterized in that: The step of constructing a recursive phonological structure according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data, and phonological clause marking data, specifically includes the following steps: Performing a syntactic parsing operation on the speech synthesis text according to a syntactic parser to obtain syntactic structure data; Converting the syntactic structure data into the phoneme sequence and the syntactic structure data carrying the accent marking data; The syntactic structure data carrying the stress marking data is converted into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure.
3. The method for speech synthesis based on syntax and prosody according to claim 2, characterized in that: The step of converting the syntactic structure data carrying the stress marking data into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure specifically includes the following steps: The particle phrases controlling the noun phrases and the verb phrases in the syntactic structure data are converted into the phonological phrase annotation data, wherein the phonological phrase annotation data PPhrase is represented as: PPhrase=PP+NP+VP Wherein, PP represents the particle phrase, NP represents the noun phrase, and VP represents the verb phrase; The inflectional phrases or compound structures in the syntactic structure data are marked as the phonological clause annotation data, wherein the clause annotation data PClause is represented as: PClause=IP or PClause=CP Wherein, IP represents the inflectional phrase, and CP represents the compound structure.
4. The method for speech synthesis based on syntax and prosody according to claim 2, characterized in that: After the step of converting the syntactic structure data carrying the stress marking data into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure, the following step is also included: The recursive phonological structure is subjected to a phonological boundary insertion operation according to the stress apex principle and the anti-spacing constraint principle.
5. The method for speech synthesis based on syntax and prosody according to claim 5, characterized in that: After the step of inserting the phonological boundary into the recursive phonological structure according to the stress apex principle and the anti-spacing constraint principle, the following step is also included: The redundant structure elimination operation is performed on the recursive phonological structure after the phonological boundary insertion operation.
6. The method for speech synthesis based on syntax and prosody according to claim 1, characterized in that: Before the step of calling the trained recursive phonological model and inputting the recursive phonological structure into the trained recursive phonological model to perform speech synthesis operation to obtain target audio data, the method further includes the following steps: Reading a system database, and acquiring model training data in the system database, wherein the model training data includes training text data and training voice data corresponding to the training text data; converting the training text data into a training recursive phonological structure; The original text-to-speech model is called, and the training recursive phonological structure is used as input data and the training speech data is used as output data to perform a model training operation on the original text-to-speech model to obtain the trained recursive phonological model.
7. A speech synthesis device based on syntax and prosody, characterized in that: include: A request receiving module, used for receiving a speech synthesis request carrying speech synthesis text sent by a user terminal; A recursive phonological structure construction module is used to construct a recursive phonological structure according to the speech synthesis text, wherein the recursive phonological structure includes a phoneme sequence, stress marking data, phonological phrase marking data and phonological clause marking data, and the recursive phonological structure PPhrase n It is expressed as: PPhrase n =PPhrase n-1 +PClause n Among them, PPhrase n Indicates the nth phonological phrase annotation data, PClause n Indicates the nth phonological clause annotation data; The speech synthesis module is used to call the trained recursive phonology model and input the recursive phonology structure into the trained recursive phonology model to perform speech synthesis operation to obtain target audio data.
8. The syntax-prosody based speech synthesis device according to claim 7, characterized in that: The recursive phonological structure building module includes: A syntactic parsing submodule, used for performing a syntactic parsing operation on the speech synthesis text according to a syntactic parser to obtain syntactic structure data; A first syntactic structure conversion submodule, configured to convert the syntactic structure data into the phoneme sequence and syntactic structure data carrying stress marking data; The second syntactic structure conversion submodule is used to convert the syntactic structure data carrying the stress marking data into the phonological phrase marking data and the phonological clause marking data according to the syntax-prosody mapping relationship to obtain the recursive phonological structure.
9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the syntax-prosody based speech synthesis method according to any one of claims 1 to 6 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the syntax-prosody-based speech synthesis method according to any one of claims 1 to 6.