Speech synthesis method and device based on accent, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice synthesis method and device based on accent, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining target text information and target rereading information corresponding to the target text information; performing rereading syllable feature extraction processing on the target rereading information to obtain rereading feature data; performing text character feature extraction processing on the target text information to obtain text feature data; performing feature fusion based on the text feature data and the reread feature data to obtain reread text data; performing acoustic feature analysis on the reread text data through a pre-trained speech feature conversion model to obtain an acoustic feature sequence; and performing voice generation based on the acoustic feature sequence and the reread feature data to obtain a target synthetic voice. According to the embodiment of the invention, accurate accent can be generated during speech synthesis.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for speech synthesis based on accent. Background Art

[0002] Speech synthesis technology refers to the technology of synthesizing text content into corresponding speech content, but there are still many different ways to generate speech. In the financial field, speech synthesis technology is widely used, especially in customer service, automated report generation, etc., which need to accurately and efficiently convey key information. The role of stress in speech expression is very important. It not only affects the comprehensibility of speech, but also conveys additional emotional and intentional information. The same text synthesis under different emphasis on words will express different speech semantics. Stress can change the meaning of the sentence and emphasize specific words or phrases, thereby affecting the listener's understanding and feelings.

[0003] The speech synthesis technology in the related art mainly focuses on the intelligibility, clarity and fluency of the synthesized speech, and pays less attention to the related contents of speech expressiveness such as stress. Therefore, how to generate accurate stress in speech synthesis has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a method, device, electronic device and storage medium for speech synthesis based on accent, aiming to generate accurate accent during speech synthesis.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a method for speech synthesis based on accent, the method comprising:

[0006] Acquire target text information and target rereading information corresponding to the target text information;

[0007] Performing stressed syllable feature extraction processing on the target stressed syllable information to obtain stressed syllable feature data;

[0008] Performing text feature extraction processing on the target text information to obtain text feature data;

[0009] Perform feature fusion based on the text feature data and the reread feature data to obtain reread text data;

[0010] Performing acoustic feature analysis on the reread text data through a pre-trained speech feature conversion model to obtain an acoustic feature sequence;

[0011] Speech generation is performed based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech.

[0012] In some embodiments, before performing acoustic feature analysis on the reread text data using a pre-trained speech feature conversion model to obtain an acoustic feature sequence, the speech feature conversion model is also pre-trained, specifically including:

[0013] Acquire a training data set and an original speech feature conversion model, wherein the training data set includes text sample data and acoustic sample data corresponding to the text sample data, and the text sample data is configured with corresponding stress annotation information;

[0014] Performing acoustic feature analysis on the text sample data and the stress marking information corresponding to the text sample data through the original speech feature conversion model to obtain acoustic training features;

[0015] Performing feature comparison based on the acoustic training feature and the acoustic sample data to obtain comparison deviation data;

[0016] The original speech feature conversion model is updated based on the comparison deviation data, and the acoustic feature analysis of the text sample data and the stress annotation information corresponding to the text sample data is performed back through the original speech feature conversion model until the original speech feature conversion model meets a preset training termination condition, thereby obtaining the pre-trained speech feature conversion model.

[0017] In some embodiments, obtaining a training data set includes:

[0018] Obtain multiple audio training samples;

[0019] Extracting spectrum features for each of the audio training samples to obtain the acoustic sample data;

[0020] Performing text semantic feature extraction processing on each of the audio training samples to obtain the text sample data;

[0021] Performing stress feature labeling processing on each of the audio training samples to obtain the stress labeling information;

[0022] The acoustic sample data, the rereading annotation information and the text sample data corresponding to the same audio training sample are integrated to obtain the training data set.

[0023] In some embodiments, the performing acoustic feature analysis on the reread text data using a pre-trained speech feature conversion model to obtain an acoustic feature sequence includes:

[0024] Performing phoneme duration analysis based on the stressed text data to obtain phoneme duration data;

[0025] The acoustic feature analysis is performed on the stressed text data and the phoneme duration data using a pre-trained speech feature conversion model to obtain an acoustic feature sequence.

[0026] In some embodiments, before performing acoustic feature analysis on the text sample data and the stress marking information corresponding to the text sample data through the original speech feature conversion model to obtain acoustic training features, the method further includes:

[0027] Based on the acoustic sample data, determining phoneme mapping information corresponding to the text sample data;

[0028] Based on the phoneme mapping information and the acoustic sample data, generating phoneme duration data corresponding to the text sample data;

[0029] The method of performing acoustic feature analysis on the text sample data and the stress mark information corresponding to the text sample data through the original speech feature conversion model to obtain acoustic training features includes:

[0030] The original speech feature conversion model is used to perform acoustic feature analysis on the text sample data, the stress marking information corresponding to the text sample data, and the phoneme duration data to obtain acoustic training features.

[0031] In some embodiments, after performing speech generation based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech, the method further includes:

[0032] Extract stress features based on the target synthesized speech to obtain speech stress information;

[0033] Performing a stress feature comparison between the speech stress information and the stress feature data to obtain stress accuracy;

[0034] The speech feature conversion model is adjusted based on the stress accuracy to generate updated synthesized speech adapted to the stress feature data.

[0035] In some embodiments, the rereading feature data includes rereading position information and rereading degree information, and the step of performing feature fusion based on the text feature data and the rereading feature data to obtain the rereading text data includes:

[0036] Based on the rereading position information, performing rereading position marking processing on the text feature data to obtain first text data;

[0037] Based on the stress degree information, performing stress character encoding processing on the first text data, the stress text data;

[0038] The generating speech based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech comprises:

[0039] Based on the stress degree information, performing stress speech coding processing on the acoustic feature sequence to obtain a stress speech feature sequence;

[0040] The stressed speech feature sequence is used for speech generation to obtain the target synthesized speech.

[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech synthesis device based on stress, the device comprising:

[0042] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the stress-based speech synthesis method described in the first aspect when executing the computer program.

[0043] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the stress-based speech synthesis method described in the first aspect.

[0044] The stress-based speech synthesis method, device, electronic device and storage medium proposed in the present application obtain target text information and target stress information corresponding to the target text information, perform stress syllable feature extraction processing based on the target stress information to obtain stress feature data, perform text feature extraction processing based on the target text information to obtain text feature data, then perform feature fusion based on the text feature data and the stress feature data to obtain stress text data, and input the stress text data into a pre-trained speech feature conversion model for acoustic feature analysis to obtain an acoustic feature sequence, and then generate speech based on the acoustic feature sequence and the stress feature data to obtain the target synthesized speech. It can be seen from this that the present application obtains a target synthesized speech with high accuracy by fusing the stress feature data with the text feature data and inputting them into the speech feature conversion model to obtain an acoustic feature sequence, and then using the stress feature data and the acoustic feature sequence for speech generation again. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a method for speech synthesis based on accent provided in an embodiment of the present application;

[0046] Figure 2 yes Figure 1 The flowchart before step S105 in FIG.

[0047] Figure 3yes Figure 2 Flow chart of step S201 in FIG.

[0048] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0049] Figure 5 yes Figure 2 The flowchart before step S202 and step S202 in FIG.

[0050] Figure 6 yes Figure 1 Flowchart of steps S104 to S106 in FIG.

[0051] Figure 7 yes Figure 1 The flowchart after step S106 in FIG.

[0052] Figure 8 is a structural schematic diagram of a speech synthesis device based on accent provided in an embodiment of the present application;

[0053] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0057] First, some nouns involved in this application are analyzed:

[0058] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0059] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0060] Text-to-Speech (TTS) is a technology that can convert text into speech. In the financial field, speech synthesis technology is widely used, especially in customer service, automated report generation, etc., which need to accurately and efficiently convey key information.

[0061] The embodiments of the present application provide a method, device, electronic device and storage medium for speech synthesis based on accent, aiming to generate accurate accent during speech synthesis.

[0062] The stress-based speech synthesis method, device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the stress-based speech synthesis method in the embodiments of the present application is described.

[0063] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0064] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0065] The stress-based speech synthesis method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The stress-based speech synthesis method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a stress-based speech synthesis method, etc., but is not limited to the above forms.

[0066] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0067] Figure 1is an optional flowchart of the stress-based speech synthesis method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0068] Step S101, obtaining target text information and corresponding target rereading information;

[0069] Step S102, extracting stressed syllable features from target stressed information to obtain stressed feature data;

[0070] Step S103, performing text feature extraction processing on the target text information to obtain text feature data;

[0071] Step S104, performing feature fusion based on the text feature data and the reread feature data to obtain reread text data;

[0072] Step S105, performing acoustic feature analysis on the reread text data using a pre-trained speech feature conversion model to obtain an acoustic feature sequence;

[0073] Step S106, generating speech based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech.

[0074] Steps S101 to S106 shown in the embodiment of the present application are to obtain target text information and target stress information corresponding to the target text information, perform stress syllable feature extraction based on the target stress information to obtain stress feature data, perform text feature extraction based on the target text information to obtain text feature data, then perform feature fusion based on the text feature data and the stress feature data to obtain stress text data, and input the stress text data into a pre-trained speech feature conversion model for acoustic feature analysis to obtain an acoustic feature sequence, and then perform speech generation based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech. It can be seen from this that the present application obtains a target synthesized speech with high accuracy by fusing the stress feature data with the text feature data and inputting them into the speech feature conversion model to obtain an acoustic feature sequence, and then using the stress feature data and the acoustic feature sequence for speech generation again.

[0075] In step S101 of some embodiments, target text information that needs to be converted into speech is obtained, and target stress information corresponding to the target text information is obtained, wherein the target stress information is used to describe the content that needs to be stressed in the target text information, and the target stress information can be pre-defined or obtained by performing phonetic rules, lexical analysis, etc. on the target text information through a neural network model.

[0076] In step S102 of some embodiments, an accent encoder is used to extract stressed syllable features from target accent information to obtain accent feature data converted into feature vectors. The accent feature data includes feature information such as pitch, length, volume, etc., and also indicates the positions in the corresponding target text information where accents need to be added.

[0077] In step S103 of some embodiments, a text encoder is used to perform text feature extraction processing on the target text information to obtain text feature data converted into feature vectors.

[0078] In step S104 of some embodiments, the text feature data and the stress feature data are subjected to feature fusion, which refers to combining the extracted text feature data and the stress feature data to obtain the stress text data. The text feature data contains the features of the stress feature data, that is, the position of the stress and the feature information of the stress in the text feature data. This process will adjust the pronunciation, rhythm and intonation of the text to ensure that the speech that meets the requirements is obtained in the subsequent speech synthesis process.

[0079] In step S105 of some embodiments, the pre-trained speech feature conversion model is used to perform acoustic feature analysis on the feature-fused reread text data to obtain an acoustic feature sequence. The acoustic feature sequence includes specific parameters for synthesizing natural sounds obtained by converting the reread text data, such as fundamental frequency, formant frequency, amplitude, etc., including position information and feature information of stress.

[0080] In step S106 of some embodiments, a final audio file is generated according to the acoustic feature sequence and the rereading feature data to obtain the target synthesized speech. This step can be implemented by a neural network model such as a speech synthesis model. This involves waveform generation technology, such as waveform splicing, parameter synthesis or end-to-end neural network methods, which can produce high-quality synthesized speech.

[0081] It should be noted that, due to the feature fusion and acoustic feature conversion of the stress feature data, it is possible that the feature information at some stress levels is lost or weakened in these processes. Therefore, in order to accurately generate the target synthesized speech that meets the target stress information, the stress feature data is used again as one of the input information during speech generation.

[0082] See also Figure 2 In some embodiments, before step S105, the speech feature conversion model is pre-trained, which may include but is not limited to steps S201 to S204:

[0083] Step S201, obtaining a training data set and an original speech feature conversion model, wherein the training data set includes text sample data and acoustic sample data corresponding to the text sample data, and the text sample data is configured with corresponding stress annotation information;

[0084] Step S202, performing acoustic feature analysis on the text sample data and the rereading annotation information corresponding to the text sample data through the original speech feature conversion model to obtain acoustic training features;

[0085] Step S203, performing feature comparison based on the acoustic training features and the acoustic sample data to obtain comparison deviation data;

[0086] Step S204, updates the original speech feature conversion model based on the comparison deviation data, and returns to execute acoustic feature analysis of the text sample data and the rereading annotation information corresponding to the text sample data through the original speech feature conversion model until the original speech feature conversion model meets the preset training termination condition, thereby obtaining a pre-trained speech feature conversion model.

[0087] In step S201 of some embodiments, a training data set is obtained, and an untrained original speech feature conversion model is obtained, the training data set includes two parts, one part is the text content of the synthesized speech, that is, the text sample data, and is configured with corresponding stress annotation information, the stress annotation information marks the position of the stress and the syllable characteristics in the text sample data. The other part is the acoustic sample data corresponding to the text sample data, which can be obtained by performing acoustic feature conversion on the speech data corresponding to the text sample data, or can be preset.

[0088] In step S202 of some embodiments, the text sample data and its stress annotation information are input into an original speech feature conversion model that has not yet been trained. The task of the speech feature conversion model is to predict a set of acoustic training features based on the input text sample data and the corresponding stress annotation information. The acoustic training features describe the important information that is ultimately required to synthesize the speech.

[0089] In some embodiments, the speech feature conversion model can be a flow-based generative model (Flow-basedGenerative Model l), also known as a flow model, which allows continuous variables to be modeled directly, which makes the flow model very suitable for generating high-quality speech waveforms. Since the speech signal is a continuous time series, the flow model can capture this characteristic well. And the flow model can directly calculate the probability density function of the data, which is very important for evaluating the quality of the generated sample. This means that we can quantify the authenticity of the generated speech and can use likelihood as an optimization target. The flow model can be easily extended to a conditional generation model, which means that they can generate specific speech based on input conditions, such as text or speech features.

[0090] In step S203 of some embodiments, the acoustic training features predicted by the original speech feature conversion model are compared with the real acoustic sample data in the training data set. By calculating the difference between the two, such as the mean square error or other loss function, the comparison deviation data is obtained, which is an indicator to measure the accuracy of the model prediction.

[0091] In step S204 of some embodiments, we use a back propagation algorithm to update the parameters of the original speech feature conversion model based on the comparison deviation data, so that the original speech feature conversion model can more accurately predict the acoustic training features. This process is repeated, and each iteration uses new model parameters to re-predict the acoustic training features and recalculate the comparison deviation data. This continues until the speech feature conversion model reaches a predetermined training termination condition, such as when the performance of the model is no longer significantly improved or a predetermined number of iterations is reached, and a pre-trained speech feature conversion model is obtained.

[0092] Through steps S201 to S204, a speech feature conversion model capable of generating a sound feature sequence based on text sample data and stress marking data can be trained, so that the speech feature conversion model can accurately generate an acoustic feature sequence based on stressed text data when applied.

[0093] See also Figure 3 In some embodiments, step S201 may include but is not limited to steps S301 to S305:

[0094] Step S301, obtaining multiple audio training samples;

[0095] Step S302, extracting spectrum features for each audio training sample to obtain acoustic sample data;

[0096] Step S303, performing text semantic feature extraction processing on each audio training sample to obtain text sample data;

[0097] Step S304, performing stress feature labeling processing on each audio training sample to obtain stress labeling information;

[0098] Step S305 , integrating the acoustic sample data, rereading annotation information and text sample data corresponding to the same audio training sample to obtain a training data set.

[0099] In step S301 of some embodiments, a plurality of audio training samples required for training are obtained. These audio training samples may come from different people and contain different contents and emotional expressions, and contain stress features.

[0100] In step S302 of some embodiments, each audio training sample is converted into an audio sample linear spectrum, which is a frequency domain representation that shows the energy distribution of the audio signal at different frequencies. Afterwards, feature extraction is performed on the audio sample linear spectrum, such as extracting Mel-Frequency Cepstrum Coefficients (MFCCs), thereby capturing acoustic sample data of the audio signal.

[0101] In step S303 of some embodiments, the corresponding speech-converted text is generated for each audio training sample, which can be realized by a speech recognition model, or the corresponding text content can be configured when the audio training sample is obtained. Then, the speech-converted text is subjected to text semantic feature extraction processing, feature information in the speech-converted text is extracted, and key information in the speech-converted text, such as context information, grammatical structure, etc., is extracted to obtain text sample data.

[0102] In step S304 of some embodiments, stress features are extracted for each audio training sample to determine stress marking information corresponding to the text sample data.

[0103] In step S305 of some embodiments, the acoustic sample data, text sample data and stress marking information corresponding to the same audio training sample are integrated to form a complete training sample for training the speech feature conversion model.

[0104] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S402:

[0105] Step S401, performing phoneme duration analysis based on the reread text data to obtain phoneme duration data;

[0106] Step S402: Perform acoustic feature analysis on the stressed text data and the phoneme duration data using a pre-trained speech feature conversion model to obtain an acoustic feature sequence.

[0107] In step S401 of some embodiments, the phoneme duration analysis is performed on the stressed text data by using the trained duration predictor to obtain the duration of each phoneme, i.e., phoneme duration data, wherein the phoneme is the basic unit of speech. In some embodiments, before step S401, the duration predictor is also pre-trained.

[0108] In step S402 of some embodiments, the phoneme duration data and the stressed text data are input into a pre-trained speech feature conversion model for acoustic feature analysis, so that the speech feature conversion model generates an acoustic feature sequence that is more in line with user needs and has a higher accuracy based on the phoneme duration data and the stressed text data.

[0109] Through steps S401 to S402, phoneme duration analysis is performed based on the stressed text data to obtain phoneme duration data. The specific duration of the phoneme corresponding to each character in the stressed text data can be obtained and input into the speech feature conversion model for acoustic feature analysis, so that when generating speech, a more accurate target synthesized speech can be obtained at the stress duration level.

[0110] See also Figure 5 In some embodiments, the steps before step 202 may include but are not limited to steps S501 to S502:

[0111] Step S501, determining phoneme mapping information corresponding to text sample data based on acoustic sample data;

[0112] Step S502, generating phoneme duration data corresponding to the text sample data based on the phoneme mapping information and the acoustic sample data;

[0113] Step S202 may include but is not limited to step S503:

[0114] Step S503: Perform acoustic feature analysis on the text sample data, the stress marking information corresponding to the text sample data, and the phoneme duration data through the original speech feature conversion model to obtain acoustic training features.

[0115] In step S501 of some embodiments, based on the acoustic sample data, the phoneme corresponding to each character in the text sample data is determined to obtain phoneme mapping information, so that the acoustic representation of each phoneme in the text can be accurately identified.

[0116] In step S502 of some embodiments, based on the phoneme mapping information and the acoustic sample data, the phoneme duration data corresponding to the text sample data is determined, and the duration of each phoneme is determined, so that the duration of each phoneme in actual pronunciation can be quantified.

[0117] In step S503 of some embodiments, when training the speech feature conversion model, the text sample data, the stress annotation information corresponding to the text sample data, and the phoneme duration data are input into the original speech feature conversion model for acoustic feature analysis, so that the trained speech feature conversion model can obtain an acoustic feature sequence that meets user requirements in terms of the duration and acoustic performance of each phoneme when the phoneme duration data is input.

[0118] Through steps S501 to S503, when training the speech feature conversion model, the phoneme duration data corresponding to the text sample data is also input into the speech feature conversion model, so that the speech feature conversion model can generate a target synthesized speech that is more accurate in the stress duration level and meets user needs based on the phoneme duration data.

[0119] See also Figure 6 In some embodiments, the rereading characteristic data includes rereading position information and rereading degree information. Step S104 may include but is not limited to steps S601 to S602:

[0120] Step S601, based on the rereading position information, performing rereading position marking processing on the text feature data to obtain first text data;

[0121] Step S602, based on the stress degree information, performing stress character encoding processing on the first text data to stress the text data;

[0122] Step S106 includes but is not limited to steps S603 to S604:

[0123] Step S603, based on the stress degree information, performing stress speech coding processing on the acoustic feature sequence to obtain a stress speech feature sequence;

[0124] Step S604, performing speech generation on the reread speech feature sequence to obtain a target synthesized speech.

[0125] In step S601 of some embodiments, based on the stress position information, stress position marking processing is performed on the text feature data to mark the position of the stress, thereby obtaining first text data.

[0126] In step S602 of some embodiments, based on the stress degree information, encoding is performed on the first text data according to different stress degrees, thereby obtaining stressed text data carrying the stress degree information.

[0127] In steps S603 to S604 of some embodiments, during speech generation, the acoustic feature sequence is adjusted based on the stress degree information to reflect the effects of the stress degree, such as pitch, length, volume, etc., to simulate the stress phenomenon that meets the user's needs and obtain a stress speech feature sequence. Then, speech generation is performed based on the stress speech feature sequence to obtain the target synthesized speech.

[0128] It should be noted that, due to the feature fusion and acoustic feature conversion of the stress feature data, it is possible that the feature information at some stress levels is lost or weakened in these processes. Therefore, in order to obtain the target synthesized speech that meets the requirements and conforms to the stress information, the stress feature data is used again as one of the input information during speech generation.

[0129] Through steps S601 to S604, the stress position information and stress degree information included in the stress feature data are used to mark the stress position and encode the stress degree on the text feature data, so that the stressed text data can more accurately include stress information. When generating speech, the acoustic feature sequence is adjusted based on the stress degree information to prevent the loss of stress information, thereby generating a target synthesized speech that more accurately meets the user's needs.

[0130] See also Figure 7 In some embodiments, step S106 may include but is not limited to steps S701 to S703:

[0131] Step S701, extracting stress features based on the target synthesized speech to obtain speech stress information;

[0132] Step S702, comparing the speech re-reading information with the re-reading feature data to obtain the re-reading accuracy;

[0133] Step S703, adjusting the speech feature conversion model based on the stress accuracy to generate updated synthesized speech adapted to the stress feature data;

[0134] In step S701 of some embodiments, stress feature extraction is performed on the generated target synthesized speech to extract actual stress information and obtain speech stress information, which may include stress position information and stress degree information.

[0135] In step S702 of some embodiments, the extracted speech stress information is compared with preset stress feature data to evaluate the stress accuracy of the target synthesized speech to determine whether it accurately reflects the expected requirements.

[0136] In step S703 of some embodiments, the speech feature conversion model may be adjusted accordingly based on the evaluation result of the rereading accuracy, and the speech generation part may also be adjusted to generate adapted rereading feature data, that is, updated synthesized speech that meets the user's expected needs. This process may involve fine-tuning model parameters until the updated synthesized speech meets the expected requirements.

[0137] Through steps S701 to S703, stress features are extracted through the target synthesized speech to obtain speech stress information, which is then compared with the stress feature data to evaluate stress accuracy. Then, the speech feature conversion model is adjusted based on the stress accuracy to generate updated synthesized speech that is adapted to the stress feature data, thereby ensuring that accurate stress is generated during speech synthesis.

[0138] The embodiment of the present application obtains target text information and target stress information corresponding to the target text information, performs stress syllable feature extraction processing based on the target stress information to obtain stress feature data, performs text character feature extraction processing based on the target text information to obtain text feature data, then performs feature fusion based on the text feature data and the stress feature data to obtain stress text data, and inputs the stress text data into a pre-trained speech feature conversion model for acoustic feature analysis to obtain an acoustic feature sequence, and then generates speech based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech. It can be seen from this that the present application obtains a target synthesized speech with high accuracy by fusing the stress feature data with the text feature data and inputting them into the speech feature conversion model to obtain an acoustic feature sequence, and then using the stress feature data and the acoustic feature sequence for speech generation again.

[0139] See also Figure 8 The embodiment of the present application further provides a speech synthesis device based on stress, which can implement the above-mentioned speech synthesis method based on stress, and the device includes:

[0140] A data acquisition module, used for acquiring target text information and target rereading information corresponding to the target text information;

[0141] A stress feature extraction module is used to extract stressed syllable features from target stress information to obtain stress feature data;

[0142] The text feature extraction module is used to extract text features from the target text information to obtain text feature data;

[0143] A feature fusion module, used for performing feature fusion based on text feature data and reread feature data to obtain reread text data;

[0144] An acoustic analysis module, used to perform acoustic feature analysis on the reread text data through a pre-trained speech feature conversion model to obtain an acoustic feature sequence;

[0145] The speech synthesis module is used to generate speech based on the acoustic feature sequence and the stress feature data to obtain the target synthesized speech.

[0146] Among them, the rereading feature extraction module can be a rereading encoder, the text feature extraction module can be a text encoder, and the speech synthesis module can be a speech synthesis model.

[0147] The specific implementation of the stress-based speech synthesis device is substantially the same as the specific implementation of the stress-based speech synthesis method described above, and will not be described in detail herein.

[0148] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned accent-based speech synthesis method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0149] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0150] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0151] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the accent-based speech synthesis method of the embodiment of the present application;

[0152] Input / output interface 903, used to implement information input and output;

[0153] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0154] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0155] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0156] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned stress-based speech synthesis method is implemented.

[0157] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The stress-based speech synthesis method, device, electronic device and storage medium provided in the embodiment of the present application obtain target text information and target stress information corresponding to the target text information, perform stress syllable feature extraction processing based on the target stress information to obtain stress feature data, perform text feature extraction processing based on the target text information to obtain text feature data, then perform feature fusion based on the text feature data and the stress feature data to obtain stress text data, and input the stress text data into a pre-trained speech feature conversion model for acoustic feature analysis to obtain an acoustic feature sequence, and then generate speech based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech. It can be seen from this that the present application obtains a target synthesized speech with high accuracy by fusing the stress feature data with the text feature data and inputting them into the speech feature conversion model to obtain an acoustic feature sequence, and then using the stress feature data and the acoustic feature sequence for speech generation again.

[0159] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0160] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0161] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0163] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0164] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0165] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0166] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0167] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0168] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0169] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speech synthesis method based on accent, It is characterized in that The method comprises: Acquire target text information and target rereading information corresponding to the target text information; Performing stressed syllable feature extraction processing on the target stressed syllable information to obtain stressed syllable feature data; Performing text feature extraction processing on the target text information to obtain text feature data; Perform feature fusion based on the text feature data and the reread feature data to obtain reread text data; Performing acoustic feature analysis on the reread text data through a pre-trained speech feature conversion model to obtain an acoustic feature sequence; Speech generation is performed based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech.

2. The method according to claim 1, It is characterized in that Before performing acoustic feature analysis on the reread text data by using the pre-trained speech feature conversion model to obtain an acoustic feature sequence, the method further includes pre-training the speech feature conversion model, specifically including: Acquire a training data set and an original speech feature conversion model, wherein the training data set includes text sample data and acoustic sample data corresponding to the text sample data, and the text sample data is configured with corresponding stress annotation information; Performing acoustic feature analysis on the text sample data and the stress marking information corresponding to the text sample data through the original speech feature conversion model to obtain acoustic training features; Performing feature comparison based on the acoustic training feature and the acoustic sample data to obtain comparison deviation data; The original speech feature conversion model is updated based on the comparison deviation data, and the acoustic feature analysis of the text sample data and the stress annotation information corresponding to the text sample data is performed back through the original speech feature conversion model until the original speech feature conversion model meets a preset training termination condition, thereby obtaining the pre-trained speech feature conversion model.

3. The method according to claim 2, It is characterized in that The step of obtaining a training data set includes: Obtain multiple audio training samples; Extracting spectrum features for each of the audio training samples to obtain the acoustic sample data; Performing text semantic feature extraction processing on each of the audio training samples to obtain the text sample data; Performing stress feature labeling processing on each of the audio training samples to obtain the stress labeling information; The acoustic sample data, the rereading annotation information and the text sample data corresponding to the same audio training sample are integrated to obtain the training data set.

4. The method according to claim 2, It is characterized in that The acoustic feature analysis of the reread text data is performed by using a pre-trained speech feature conversion model to obtain an acoustic feature sequence, including: Performing phoneme duration analysis based on the stressed text data to obtain phoneme duration data; The acoustic feature analysis is performed on the stressed text data and the phoneme duration data using a pre-trained speech feature conversion model to obtain an acoustic feature sequence.

5. The method according to claim 4, It is characterized in that Before performing acoustic feature analysis on the text sample data and the stress marking information corresponding to the text sample data through the original speech feature conversion model to obtain acoustic training features, the method further includes: Based on the acoustic sample data, determining phoneme mapping information corresponding to the text sample data; Based on the phoneme mapping information and the acoustic sample data, generating phoneme duration data corresponding to the text sample data; The method of performing acoustic feature analysis on the text sample data and the stress mark information corresponding to the text sample data through the original speech feature conversion model to obtain acoustic training features includes: The original speech feature conversion model is used to perform acoustic feature analysis on the text sample data, the stress marking information corresponding to the text sample data, and the phoneme duration data to obtain acoustic training features.

6. The method according to claim 1, It is characterized in that After the speech generation is performed based on the acoustic feature sequence and the stress feature data to obtain the target synthesized speech, the method further includes: Extract stress features based on the target synthesized speech to obtain speech stress information; Performing a stress feature comparison between the speech stress feature information and the stress feature data to obtain stress accuracy; The speech feature conversion model is adjusted based on the stress accuracy to generate updated synthesized speech adapted to the stress feature data.

7. The method according to claim 1, It is characterized in that The rereading feature data includes rereading position information and rereading degree information, and the feature fusion based on the text feature data and the rereading feature data to obtain the rereading text data includes: Based on the rereading position information, performing rereading position marking processing on the text feature data to obtain first text data; Based on the stress degree information, performing stress character encoding processing on the first text data, the stress text data; The generating speech based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech comprises: Based on the stress degree information, performing stress speech coding processing on the acoustic feature sequence to obtain a stress speech feature sequence; The stressed speech feature sequence is used for speech generation to obtain the target synthesized speech.

8. A speech synthesis device based on stress, It is characterized in that The device comprises: A data acquisition module, used for acquiring target text information and target rereading information corresponding to the target text information; A stress feature extraction module, used for performing stress syllable feature extraction processing on the target stress information to obtain stress feature data; A text feature extraction module is used to extract text features from the target text information to obtain text feature data; A feature fusion module, used for performing feature fusion based on the text feature data and the reread feature data to obtain reread text data; An acoustic analysis module, used for performing acoustic feature analysis on the reread text data through a pre-trained speech feature conversion model to obtain an acoustic feature sequence; The speech synthesis module is used to generate speech based on the acoustic feature sequence and the stress feature data to obtain a target synthesized speech.

9. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the stress-based speech synthesis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the accent-based speech synthesis method according to any one of claims 1 to 7 is implemented.

Images