Speech synthesis method and device based on conditional matching flow, equipment and medium

What is AI technical title?
AI technical title is built by Patsnap AI team. It summarizes the technical point description of the patent document.
By introducing a conditional matching flow module into the speech synthesis model, the feature distribution adjustment of speech sample data is solved, and the problem of low efficiency in generating speech data in the existing speech synthesis model is solved, and more efficient speech data generation is achieved.

CN119920236APending Publication Date: 2025-05-02PING AN TECH (SHENZHEN) CO LTD

View PDF 0 Cites 0 Cited by

Patent Information

Application Number: CN202510112861.7
Authority / Receiving Office: CN · China
Patent Type: Applications(China)
Current Assignee / Owner
Filing Date: 2025-01-23
Publication Date: 2025-05-02

AI Technical Summary

Technical Problem

The existing speech synthesis model is less efficient in generating speech data, resulting in high inference costs and slow time, reducing the possibility of model online deployment.

Method used

A speech synthesis method based on conditional matching stream is adopted, and a speech synthesis model is constructed by obtaining speech sample data for text annotation and preprocessing, and a speech synthesis model is constructed, and the audio feature vector is adjusted using the conditional matching stream module. Finally, the adjusted feature vector is input to the decoder to generate speech data.

Benefits of technology

It significantly improves the efficiency of the speech synthesis model to generate speech data, reduces errors caused by input mismatch, improves the working efficiency of the decoder, and shortens the generation time of speech data.

✦ Generated by Eureka AI based on patent content.

Smart Images

Figure CN119920236A_ABST

Patent Text Reader

Abstract

The invention belongs to the technical field of speech processing, is suitable for the medical field and the financial field, and discloses a speech synthesis method, device and equipment based on a conditional matching stream, and a medium, and the method comprises the steps: inputting second text data into an encoder of a speech synthesis model, and carrying out the audio feature extraction, performing mask processing on the extracted audio feature vector; obtaining third text data to be synthesized, inputting the first audio feature vector and the third text data to be synthesized into a conditional matching stream module of a speech synthesis model for feature distribution adjustment, and obtaining a second audio feature vector based on a vector field of the conditional matching stream module and the second audio feature vector, converting the first audio feature vector and third text data to be synthesized into a third audio feature vector; and inputting the third audio feature vector into a decoder of a speech synthesis model for decoding to obtain speech data generated by third text data to be synthesized. According to the invention, the speech data generation efficiency of the speech synthesis model is effectively improved.

Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of speech processing technology, is applicable to the medical field and the financial field, and particularly relates to a speech synthesis method, device, equipment and medium based on conditional matching stream. Background Art

[0002] The naturalness displayed by the speech synthesis model can be said to have reached the human level, and it has reached an extremely leading level in terms of speaker similarity and comprehensibility. When the speech synthesis model was originally designed, its purpose was to make fuller use of a larger corpus and sound library, so as to get rid of the embarrassing situation that the traditional speech synthesis model needs to spend a long time on corpus. In medical scenarios, the application of speech synthesis technology is particularly extensive and critical. For example, in the hospital's intelligent guidance system, patients can enter text information such as symptoms or department requirements, and the speech synthesis model will quickly convert the corresponding guidance content into clear and easy-to-understand voice, providing patients with accurate guidance, including department location, treatment process and precautions. Especially for elderly patients or patients with poor eyesight, they can successfully complete the preparation before the consultation by listening to the voice prompts, which greatly improves the medical experience. In financial scenarios, speech synthesis technology also plays an important role. For example, the intelligent customer service system can quickly respond to customer inquiries, whether it is financial product information or business handling procedures. After the customer enters the text, the speech synthesis system will reply with a professional and friendly voice, providing a convenient service experience. For example, when customers ask about detailed information such as the interest rate and repayment method of a loan product, intelligent customer service can answer them quickly and accurately, enhancing customer trust and satisfaction.

[0003] The speech synthesis models proposed in the past often use large language models to match the discrete symbols of text and speech, and generate discrete symbols in an autoregressive format. The advantage of this method is that it can build the association between text and semantic features in a discretized way, and the generated speech rhythm and diversity are richer. However, since the autoregressive model requires the generation of the previous symbol to drive the generation of the next symbol, the inference cost of generating speech using the autoregressive model is high, and the inference time is slow, which greatly reduces the possibility of online deployment of the model.

[0004] In view of this, how to improve the efficiency of speech synthesis models in generating speech data is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a speech synthesis method, device, equipment and medium based on conditional matching flow, so as to solve the technical problem that the existing speech synthesis model has low efficiency in generating speech data.

[0006] In a first aspect, the present invention provides a speech synthesis method based on conditional matching stream, comprising:

[0007] Acquire first voice sample data, and perform text annotation on the acquired first voice sample data to obtain first text data corresponding to the first voice sample data;

[0008] Performing data preprocessing on first text data corresponding to the first voice sample data to obtain second text data corresponding to the first voice sample data;

[0009] Constructing a speech synthesis model, and inputting the second text data into an encoder of the speech synthesis model to extract audio features, so as to obtain an audio feature vector of the first speech sample data;

[0010] Performing mask processing on the audio feature vector of the first speech sample data, and dividing the masked audio feature vector of the first speech sample data into a first audio feature vector and a second audio feature vector;

[0011] Acquire third text data to be synthesized, input the first audio feature vector and the third text data to be synthesized into a conditional matching stream module of a speech synthesis model to adjust feature distribution, and convert the first audio feature vector and the third text data to be synthesized into a third audio feature vector based on the vector field of the conditional matching stream module and the second audio feature vector;

[0012] The third audio feature vector is input into a decoder of a speech synthesis model for decoding to obtain speech data generated by the third text data to be synthesized.

[0013] In a second aspect, the present invention provides a speech synthesis device based on a conditional matching stream, the device being used to implement the speech synthesis method based on a conditional matching stream as described in the first aspect above, comprising:

[0014] An acquisition module, used to acquire first voice sample data, and perform text annotation on the acquired first voice sample data to obtain first text data corresponding to the first voice sample data;

[0015] A first processing module, configured to perform data preprocessing on first text data corresponding to the first voice sample data to obtain second text data corresponding to the first voice sample data;

[0016] A feature extraction module, used for constructing a speech synthesis model, and inputting the second text data into an encoder of the speech synthesis model to extract audio features, so as to obtain an audio feature vector of the first speech sample data;

[0017] A second processing module, configured to perform mask processing on the audio feature vector of the first speech sample data, and divide the audio feature vector of the first speech sample data after the mask processing into a first audio feature vector and a second audio feature vector;

[0018] a conversion module, used for acquiring third text data to be synthesized, inputting the first audio feature vector and the third text data to be synthesized into a conditional matching stream module of a speech synthesis model for adjusting feature distribution, and converting the first audio feature vector and the third text data to be synthesized into a third audio feature vector based on the vector field of the conditional matching stream module and the second audio feature vector;

[0019] A decoding module is used to input the third audio feature vector into a decoder of a speech synthesis model for decoding to obtain speech data generated by the third text data to be synthesized.

[0020] In a third aspect, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned speech synthesis method based on conditional matching stream when executing the computer program.

[0021] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which implements the steps of the above-mentioned speech synthesis method based on conditional matching stream when executed by a processor.

[0022] In the scheme implemented by the above-mentioned speech synthesis method, device, equipment and medium based on conditional matching stream, first speech sample data can be obtained through the client, and text annotation is performed on the obtained first speech sample data to obtain first text data corresponding to the first speech sample data; data preprocessing is performed on the first text data corresponding to the first speech sample data to obtain second text data corresponding to the first speech sample data; a speech synthesis model is constructed, and the second text data is input into the encoder of the speech synthesis model for audio feature extraction to obtain an audio feature vector of the first speech sample data; the audio feature vector of the first speech sample data is masked, and the audio feature vector of the masked first speech sample data is divided into a first audio feature vector and a second audio feature vector; the third text data to be synthesized is obtained, and the first audio feature vector and the third text data to be synthesized are input into the conditional matching stream module of the speech synthesis model for feature distribution adjustment, and based on The first audio feature vector and the third text data to be synthesized are converted into a third audio feature vector through the vector field and the second audio feature vector of the conditional matching flow module; the third audio feature vector is input into the decoder of the speech synthesis model for decoding to obtain the speech data generated by the third text data to be synthesized. In the present invention, the first audio feature vector and the third text data to be synthesized are converted into a third audio feature vector through the vector field and the second audio feature vector of the conditional matching flow module, the feature distribution of the input data is optimized by using the conditional matching flow module to make it more in line with the requirements of subsequent speech synthesis, and then the third audio feature vector is input into the decoder of the speech synthesis model for decoding to obtain the speech data generated by the third text data to be synthesized. The adjusted third audio feature vector reduces errors caused by input mismatch, can significantly improve the working efficiency of the decoder, greatly shortens the generation time of speech data, and effectively improves the efficiency of the speech synthesis model in generating speech data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0024] Figure 1 It is a schematic diagram of an application environment of a speech synthesis method based on conditional matching stream in one embodiment of the present invention;

[0025] Figure 2 It is a flow chart of a speech synthesis method based on conditional matching stream in one embodiment of the present invention;

[0026] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S10;

[0027] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S40;

[0028] Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S50;

[0029] Figure 6 yes Figure 2 A schematic flow chart of a specific implementation of step S60;

[0030] Figure 7 is a structural schematic diagram of a speech synthesis device based on conditional matching stream in one embodiment of the present invention;

[0031] Figure 8 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0032] Fig. 9 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] The speech synthesis method based on conditional matching stream provided by the embodiment of the present invention can be applied in Figure 1 In the application environment, Figure 1This is a schematic diagram of an application environment of a speech synthesis method based on conditional matching flow in one embodiment of the present invention; wherein, the client communicates with the server through a network. The server can obtain first speech sample data through the client, and perform text annotation on the obtained first speech sample data to obtain first text data corresponding to the first speech sample data; perform data preprocessing on the first text data corresponding to the first speech sample data to obtain second text data corresponding to the first speech sample data; construct a speech synthesis model, and input the second text data into the encoder of the speech synthesis model for audio feature extraction to obtain an audio feature vector of the first speech sample data; perform masking on the audio feature vector of the first speech sample data, and divide the audio feature vector of the masked first speech sample data into a first audio feature vector and a second audio feature vector; obtain the third text data to be synthesized, and input the first audio feature vector and the third text data to be synthesized into the conditional matching flow module of the speech synthesis model for feature extraction. The first audio feature vector and the third text data to be synthesized are converted into a third audio feature vector based on the vector field and the second audio feature vector of the conditional matching stream module; the third audio feature vector is input into the decoder of the speech synthesis model for decoding to obtain the speech data generated by the third text data to be synthesized. In the present invention, the first audio feature vector in the speech data sample and the third text data to be synthesized are first input into the conditional matching stream module of the speech synthesis model, and based on the vector field and the second audio feature vector of the conditional matching stream module, the first audio feature vector and the third text data to be synthesized are converted into the third audio feature vector, and finally the third audio feature vector is input into the decoder of the speech synthesis model for decoding to obtain the speech data generated by the third text data to be synthesized, which effectively improves the efficiency of the speech synthesis model in generating speech data. The present invention is described in detail below through specific embodiments.

[0035] See also Figure 2 As shown, Figure 2 A flow chart of a speech synthesis method based on conditional matching stream provided in an embodiment of the present invention is provided. The speech synthesis method based on conditional matching stream specifically comprises the following steps:

[0036] S10: Acquire first voice sample data, and perform text annotation on the acquired first voice sample data to obtain first text data corresponding to the first voice sample data.

[0037] When performing speech synthesis, the present invention first collects first speech sample data. The first speech sample data can be obtained through a variety of channels. For example, in the medical field, speech samples related to symptom descriptions, diagnostic terms, etc. can be selected from public medical speech data sets, or they can be obtained by self-recording, such as using professional recording equipment to allow doctors and patients to describe and record their condition, treatment plan, etc. according to specific requirements in different departments of the hospital. In financial scenarios, for speech synthesis such as financial product introductions and customer consultation responses, the first speech sample data also needs to be obtained. Relevant samples can be selected from public financial speech data sets, or they can be recorded by themselves. For example, professional recording equipment can be used in different financial environments such as banks and stock exchanges to allow financial practitioners and customers to explain and record financial services, investment consultations, etc. according to specific requirements. Next, the acquired first speech sample data needs to be text-annotated, and each segment of speech sample data is annotated with the corresponding text content, thereby establishing an association between speech data and text data. Specifically, if Figure 3 As stated, Figure 3 yes Figure 2 A specific implementation flow diagram of step S10 in the embodiment of the present invention specifically includes the following steps S11-S12:

[0038] S11: Perform text conversion processing on the acquired first voice sample data through the automatic speech recognition model to obtain Chinese text data corresponding to the first voice sample data. Specifically, after acquiring the first voice sample data, the present invention can input these first voice sample data into the automatic speech recognition model. The automatic speech recognition model will first perform feature extraction on the voice sample, extracting feature information such as audio, spectrum, and sound intensity of the first voice sample data. These extracted features can reflect the characteristics of the first voice sample data in terms of pitch, timbre, and sound length. Then, the feature information of the extracted first voice data sample is analyzed, and the first voice data is converted into Chinese text data.

[0039] S12: Performing pinyin conversion processing on the Chinese text data corresponding to the first voice sample data through the phoneme conversion model to obtain the first text data corresponding to the first voice sample data after the pinyin conversion. Specifically, after the Chinese text data corresponding to the first voice sample data obtained by step S11 of the present invention can be input into the phoneme conversion model, the phoneme conversion model analyzes the input Chinese text data, and determines the pinyin corresponding to each Chinese text data according to a pre-set Chinese-pinyin mapping rule, so as to obtain the first text data corresponding to the first voice sample data after the pinyin conversion.

[0040] S20: Perform data preprocessing on the first text data corresponding to the first voice sample data to obtain second text data corresponding to the first voice sample data. After the present invention performs text annotation on the acquired first voice sample data in step S10, there will be a large difference in length between the first text data and the first voice sample data, which will interfere with the subsequent understanding of the semantics of the text data. Therefore, it is necessary to perform data preprocessing on the first text data corresponding to the first voice sample data. The specific steps are shown in step S21:

[0041] S21: Perform padding and completion processing on the first text data corresponding to the first voice sample data to obtain the second text data corresponding to the first voice sample data. Generally, in the actual correspondence between voice data and text data, the acoustic feature frame number T of the voice is often greater than the text length t. In the actual expression process of voice, factors such as the duration of pronunciation and rhythm changes will make the corresponding acoustic feature frame number longer, while the text data is relatively concise and compact, and the text content can be expressed with fewer characters or words. Therefore, the method of padding and completion can be adopted to align the lengths of the first voice sample data and the second text data.

[0042] In one embodiment of the present invention, performing padding processing on the first text data corresponding to the first voice sample data to obtain the second text data corresponding to the first voice sample data specifically includes the following steps:

[0043] S211: Obtain the text length of the second text data corresponding to the first voice sample data, and extract the frame number of the acoustic feature of the first voice sample. Specifically, in the present invention, when processing the first voice sample data, the entire first voice sample data can be divided into multiple frames according to a specific time interval, and the total number of divided frames is the frame number of the acoustic feature. For example, a segment of speech is divided into 50 time frames after processing, then the frame number of the acoustic feature here is 50. In addition, the text length of the second text data can be calculated according to the number of characters.

[0044] S212: Calculate the length difference between the number of frames of the acoustic feature of the first voice sample and the text length of the second text data corresponding to the first voice sample data. After obtaining the number of frames of the acoustic feature of the first voice sample and the text length of the second text data corresponding to the first voice sample data, it is necessary to calculate the difference between the two. The length difference between the two can be obtained by subtracting the text length of the second text data corresponding to the first voice sample data from the number of frames of the acoustic feature of the first voice sample. For example, if the number of frames of the acoustic feature is 50 and the text length is 5 according to the number of characters, then the length difference between the two is 45.

[0045] S213: adding blank characters to the first text data according to the calculated length difference to obtain second text data corresponding to the first voice sample data. Specifically, in the present invention, the length of the second text data can be increased by adding blank characters, so that the length of the second text data can eventually be equal to the number of frames of the acoustic feature of the first voice sample, so as to achieve the effect of filling the length.

[0046] S30: Construct a speech synthesis model, and input the second text data into the encoder of the speech synthesis model for audio feature extraction to obtain the audio feature vector of the first speech sample data. Specifically, the present invention can first construct a speech synthesis model, and input the second text data into the encoder of the speech synthesis model for audio feature extraction, and the acoustic feature information of the first speech sample data can be extracted by the encoder. For example, in a medical scenario, the medical-related second text data is input into the encoder of the speech synthesis model for audio feature extraction. In this process, the encoder can extract the acoustic feature information of the previous doctor's diagnosis speech samples of the same disease, and then further combine, abstract and transform the extracted acoustic feature information through its internal multi-layer network structure, and gradually extract more medical semantics and high-level audio feature vectors to obtain the audio feature vector of the first speech sample data, thereby providing a basis for the subsequent synthesis of accurate and easy-to-understand medical speech. In a financial scenario, the present invention can first construct a speech synthesis model that fits the financial business scenario, and input the financial-related second text data into the encoder of the speech synthesis model for audio feature extraction, providing a basis for synthesizing professional and clear financial speech.

[0047] S40: Mask the audio feature vector of the first speech sample data, and divide the masked audio feature vector of the first speech sample data into a first audio feature vector and a second audio feature vector. Figure 4 As stated, Figure 4 yes Figure 2 A specific implementation flow diagram of step S40 in the embodiment of the present invention specifically includes the following steps S41-S42:

[0048] S41: Preset the mask ratio of the audio feature vector of the first voice sample data, and select the mask position of the audio feature vector of the first voice sample data through a random number generator. Specifically, in the present invention, the mask ratio of the audio feature vector of the first voice sample data can be preset first to determine how many elements in the audio feature vector of the first voice sample data need to be masked. For example, the mask ratio of the audio feature vector of the first voice sample data is preset to 30%, indicating that 30% of the elements in the audio feature vector of the first voice sample data need to be masked; then use the random number generator to randomly determine which positions in the audio feature vector need to be masked according to the preset mask ratio, and mark the specific positions of the subsequent elements to be masked.

[0049] S42: Hide the corresponding elements in the mask position of the audio feature vector of the selected first speech sample data to obtain the audio feature vector of the first speech sample data after mask processing. In the present invention, based on the mask position selected in the above step S41, the elements in the audio feature vector corresponding to these mask positions can be hidden so that the hidden elements are not directly seen or used in subsequent processing to obtain the audio feature vector of the first speech sample data after mask processing.

[0050] S50: Obtain the third text data to be synthesized, input the first audio feature vector and the third text data to be synthesized into the conditional matching stream module of the speech synthesis model to adjust the feature distribution, and convert the first audio feature vector and the third text data to be synthesized into a third audio feature vector based on the vector field of the conditional matching stream module and the second audio feature vector. Specifically, Figure 5 As stated, Figure 5 yes Figure 2 A specific implementation flow diagram of step S50 in the embodiment of the present invention specifically includes the following steps S51-S53:

[0051] S51: Analyze the feature distribution of the first audio feature vector and the third text data to be synthesized, and the feature distribution of the second audio feature vector. Specifically, the present invention needs to first analyze the initial feature distribution P0 of the first audio feature vector and the third text data to be synthesized, and the target feature distribution P1 of the second audio feature vector.

[0052] S52: The feature distribution of the first audio feature vector and the third text data to be synthesized is adjusted to the feature distribution of the second audio feature vector through the vector field of the conditional matching flow module, so as to obtain the second audio feature vector after adjusting the feature distribution and the third text data to be synthesized. Specifically, the present invention performs corresponding transformation operations on the first audio feature vector and the third text data according to the pattern presented by the target feature distribution P1 of the second audio feature vector and the conversion rules in the vector field. For example, the values of the pitch and rhythm-related dimensions in the first audio feature vector can be adjusted to make it closer to the feature expression of the second audio feature vector; for the third text data, the presentation form of its semantic structure features can be adjusted to make its final feature distribution match the feature distribution of the second audio feature vector, so as to obtain the second audio feature vector after adjusting the feature distribution and the third text data to be synthesized.

[0053] S53: Performing audio conversion processing on the second audio feature vector and the third text data to be synthesized to obtain a third audio feature vector. Specifically, the present invention performs audio conversion processing on the second audio feature vector and the third text data to be synthesized to obtain a third audio feature vector, which combines the second audio feature vector after adjusting the feature distribution and the third text data to be synthesized, and a decoding operation can be further performed based on the third audio feature vector.

[0054] S60: Input the third audio feature vector into the decoder of the speech synthesis model for decoding, and obtain speech data generated by the third text data to be synthesized. In the medical scenario, the third audio feature vector contains information such as the pronunciation characteristics of medical terms and the intonation characteristics of different disease descriptions. Through the work of the decoder, speech data generated by the third text data related to medical knowledge is finally obtained, such as detailed diagnosis results for a certain disease, follow-up treatment recommendations and other speech content, to facilitate patients to understand medical information. In the financial scenario, the third audio feature vector contains key information such as the pronunciation of financial professional vocabulary and interest rates. After being decoded by the decoder, speech data generated by the third text data in the financial field is obtained, such as the income description of financial products, loan default risk warnings and other voices, to help customers clearly understand financial business. Specifically, such as Figure 6 As stated, Figure 6 yes Figure 2 A specific implementation flow diagram of step S60 in the embodiment of the present invention specifically includes the following steps S61-S62:

[0055] S61: truncate the second audio feature vector in the third audio feature vector to obtain the audio feature vector of the third text data to be synthesized. Specifically, in the present invention, in the above step S50, the first audio feature vector and the third text data to be synthesized are converted into a third audio feature vector, and the third audio feature vector contains the second audio feature vector in the first speech sample data. Therefore, it is necessary to truncate the second audio feature vector in the third audio feature vector to obtain the audio feature vector of the third text data to be synthesized.

[0056] S62: Decode the audio feature vector of the third text data to be synthesized to obtain the voice data generated by the third text data to be synthesized. Specifically, after obtaining the audio feature vector of the third text data to be synthesized in step S61, the present invention uses a decoder to restore and transform the information contained in the audio feature vector of the third text data to be synthesized, and finally generates the voice data corresponding to the third text data to be synthesized.

[0057] It can be seen that in the above scheme, when performing speech synthesis, the first audio feature vector in the speech data sample and the third text data to be synthesized are first input into the conditional matching flow module of the speech synthesis model, and based on the vector field and the second audio feature vector of the conditional matching flow module, the first audio feature vector and the third text data to be synthesized are converted into a third audio feature vector, and finally the third audio feature vector is input into the decoder of the speech synthesis model for decoding, so as to obtain the speech data generated by the third text data to be synthesized, thereby effectively improving the efficiency of the speech synthesis model in generating speech data.

[0058] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0059] In one embodiment, a speech synthesis device based on conditional matching stream is provided, and the speech synthesis device based on conditional matching stream corresponds to the speech synthesis method based on conditional matching stream in the above embodiment. Figure 7 As shown, Figure 7 7 is a structural diagram of a speech synthesis device based on conditional matching stream in one embodiment of the present invention, the speech synthesis device based on conditional matching stream includes an acquisition module 71, a first processing module 72, a feature extraction module 73, a second processing module 74, a conversion module 75, and a decoding module 76. The functional modules are described in detail as follows:

[0060] An acquisition module 71 is used to acquire first voice sample data, and perform text annotation on the acquired first voice sample data to obtain first text data corresponding to the first voice sample data;

[0061] A first processing module 72, configured to perform data preprocessing on the first text data corresponding to the first voice sample data to obtain second text data corresponding to the first voice sample data;

[0062] A feature extraction module 73 is used to construct a speech synthesis model, and input the second text data into an encoder of the speech synthesis model to extract audio features, so as to obtain an audio feature vector of the first speech sample data;

[0063] A second processing module 74 is used to perform mask processing on the audio feature vector of the first speech sample data, and divide the audio feature vector of the first speech sample data after the mask processing into a first audio feature vector and a second audio feature vector;

[0064] a conversion module 75, configured to obtain third text data to be synthesized, input the first audio feature vector and the third text data to be synthesized into a conditional matching stream module of a speech synthesis model to adjust feature distribution, and convert the first audio feature vector and the third text data to be synthesized into a third audio feature vector based on the vector field of the conditional matching stream module and the second audio feature vector;

[0065] The decoding module 76 is used to input the third audio feature vector into the decoder of the speech synthesis model for decoding to obtain speech data generated by the third text data to be synthesized.

[0066] In one embodiment, the acquisition module 71 is specifically configured to:

[0067] Performing text conversion processing on the acquired first voice sample data through an automatic speech recognition model to obtain Chinese text data corresponding to the first voice sample data;

[0068] The Chinese text data corresponding to the first voice sample data is processed by pinyin conversion through the phoneme conversion model to obtain first text data corresponding to the first voice sample data after pinyin conversion.

[0069] In one embodiment, the first processing module 72 is specifically configured to:

[0070] The first text data corresponding to the first voice sample data is filled and completed to obtain the second text data corresponding to the first voice sample data.

[0071] In one embodiment, the first processing module 72 is specifically configured to:

[0072] Obtaining the text length of the second text data corresponding to the first voice sample data, and extracting the frame number of the acoustic feature of the first voice sample;

[0073] Calculating a length difference between the number of frames of the acoustic feature of the first voice sample and the text length of the second text data corresponding to the first voice sample data;

[0074] Adding blank characters to the first text data according to the calculated length difference to obtain second text data corresponding to the first voice sample data.

[0075] In one embodiment, the second processing module 74 is specifically configured to:

[0076] Presetting a mask ratio of the audio feature vector of the first speech sample data, and selecting a mask position of the audio feature vector of the first speech sample data through a random number generator;

[0077] The corresponding elements in the mask position of the audio feature vector of the selected first speech sample data are hidden to obtain the audio feature vector of the first speech sample data after mask processing.

[0078] In one embodiment, the conversion module 75 is specifically configured to:

[0079] Analyzing the feature distribution of the first audio feature vector and the third text data to be synthesized, and the feature distribution of the second audio feature vector;

[0080] Adjusting the feature distribution of the first audio feature vector and the third text data to be synthesized to the feature distribution of the second audio feature vector through the vector field of the conditional matching stream module, so as to obtain the second audio feature vector after the feature distribution is adjusted and the third text data to be synthesized;

[0081] The second audio feature vector and the third text data to be synthesized are subjected to audio conversion processing to obtain a third audio feature vector.

[0082] In one embodiment, the decoding module 76 is specifically configured to:

[0083] Truncating the second audio feature vector in the third audio feature vector to obtain the audio feature vector of the third text data to be synthesized;

[0084] The audio feature vector of the third text data to be synthesized is decoded to obtain speech data generated by the third text data to be synthesized.

[0085] The present invention provides a speech synthesis device based on conditional matching stream. When performing speech synthesis, the first audio feature vector in the speech data sample and the third text data to be synthesized are first input into the conditional matching stream module of the speech synthesis model to adjust the feature distribution, and based on the vector field and the second audio feature vector of the conditional matching stream module, the first audio feature vector and the third text data to be synthesized are converted into a third audio feature vector, and finally the third audio feature vector is input into the decoder of the speech synthesis model for decoding, so as to obtain the speech data generated by the third text data to be synthesized, thereby effectively improving the efficiency of the speech synthesis model in generating speech data.

[0086] For the specific definition of the speech synthesis device based on the conditional matching stream, please refer to the definition of the speech synthesis method based on the conditional matching stream above, which will not be repeated here. The various modules in the above-mentioned speech synthesis device based on the conditional matching stream can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0087] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, Figure 8 1 is a schematic diagram of the structure of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a speech synthesis method based on conditional matching streams are implemented.

[0088] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig. 9 As shown, Fig. 91 is another structural diagram of a computer device in one embodiment of the present invention. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, the functions or steps on the client side of a speech synthesis method based on conditional matching streams are implemented.

[0089] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0090] Acquire first voice sample data, and perform text annotation on the acquired first voice sample data to obtain first text data corresponding to the first voice sample data;

[0091] Performing data preprocessing on first text data corresponding to the first voice sample data to obtain second text data corresponding to the first voice sample data;

[0092] Constructing a speech synthesis model, and inputting the second text data into an encoder of the speech synthesis model to extract audio features, so as to obtain an audio feature vector of the first speech sample data;

[0093] Performing mask processing on the audio feature vector of the first speech sample data, and dividing the masked audio feature vector of the first speech sample data into a first audio feature vector and a second audio feature vector;

[0094] Acquire third text data to be synthesized, input the first audio feature vector and the third text data to be synthesized into a conditional matching stream module of a speech synthesis model to adjust feature distribution, and convert the first audio feature vector and the third text data to be synthesized into a third audio feature vector based on the vector field of the conditional matching stream module and the second audio feature vector;

[0095] The third audio feature vector is input into a decoder of a speech synthesis model for decoding to obtain speech data generated by the third text data to be synthesized.

[0096] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0097] Acquire first voice sample data, and perform text annotation on the acquired first voice sample data to obtain first text data corresponding to the first voice sample data;

[0098] Performing data preprocessing on first text data corresponding to the first voice sample data to obtain second text data corresponding to the first voice sample data;

[0099] Constructing a speech synthesis model, and inputting the second text data into an encoder of the speech synthesis model to extract audio features, so as to obtain an audio feature vector of the first speech sample data;

[0100] Performing mask processing on the audio feature vector of the first speech sample data, and dividing the masked audio feature vector of the first speech sample data into a first audio feature vector and a second audio feature vector;

[0101] Acquire third text data to be synthesized, input the first audio feature vector and the third text data to be synthesized into a conditional matching stream module of a speech synthesis model to adjust feature distribution, and convert the first audio feature vector and the third text data to be synthesized into a third audio feature vector based on the vector field of the conditional matching stream module and the second audio feature vector;

[0102] The third audio feature vector is input into a decoder of a speech synthesis model for decoding to obtain speech data generated by the third text data to be synthesized.

[0103] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0104] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0105] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0106] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech synthesis method based on conditional matching stream, characterized in that: include: Acquire first voice sample data, and perform text annotation on the acquired first voice sample data to obtain first text data corresponding to the first voice sample data; Performing data preprocessing on first text data corresponding to the first voice sample data to obtain second text data corresponding to the first voice sample data; Constructing a speech synthesis model, and inputting the second text data into an encoder of the speech synthesis model to extract audio features, so as to obtain an audio feature vector of the first speech sample data; Performing mask processing on the audio feature vector of the first speech sample data, and dividing the masked audio feature vector of the first speech sample data into a first audio feature vector and a second audio feature vector; Acquire third text data to be synthesized, input the first audio feature vector and the third text data to be synthesized into a conditional matching stream module of a speech synthesis model to adjust feature distribution, and convert the first audio feature vector and the third text data to be synthesized into a third audio feature vector based on the vector field of the conditional matching stream module and the second audio feature vector; The third audio feature vector is input into a decoder of a speech synthesis model for decoding to obtain speech data generated by the third text data to be synthesized.

2. The speech synthesis method based on conditional matching stream according to claim 1, characterized in that: The step of performing text annotation on the acquired first voice sample data to obtain first text data corresponding to the first voice sample data comprises: Performing text conversion processing on the acquired first voice sample data through an automatic speech recognition model to obtain Chinese text data corresponding to the first voice sample data; The Chinese text data corresponding to the first voice sample data is processed by pinyin conversion through the phoneme conversion model to obtain first text data corresponding to the first voice sample data after pinyin conversion.

3. The speech synthesis method based on conditional matching stream according to claim 1, characterized in that: The performing data preprocessing on the first text data corresponding to the first voice sample data to obtain the second text data corresponding to the first voice sample data includes: The first text data corresponding to the first voice sample data is filled and completed to obtain the second text data corresponding to the first voice sample data.

4. The speech synthesis method based on conditional matching stream according to claim 3, characterized in that: The step of performing padding processing on the first text data corresponding to the first voice sample data to obtain the second text data corresponding to the first voice sample data includes: Obtaining the text length of the second text data corresponding to the first voice sample data, and extracting the frame number of the acoustic feature of the first voice sample; Calculating a length difference between the number of frames of the acoustic feature of the first voice sample and the text length of the second text data corresponding to the first voice sample data; Adding blank characters to the first text data according to the calculated length difference to obtain second text data corresponding to the first voice sample data.

5. The speech synthesis method based on conditional matching stream according to claim 1, characterized in that: Performing mask processing on the audio feature vector of the first speech sample data, and dividing the masked audio feature vector of the first speech sample data into a first audio feature vector and a second audio feature vector, including: Presetting a mask ratio of the audio feature vector of the first speech sample data, and selecting a mask position of the audio feature vector of the first speech sample data through a random number generator; The corresponding elements in the mask position of the audio feature vector of the selected first speech sample data are hidden to obtain the audio feature vector of the first speech sample data after mask processing.

6. The speech synthesis method based on conditional matching stream according to claim 1, characterized in that: The step of obtaining the third text data to be synthesized, inputting the first audio feature vector and the third text data to be synthesized into a conditional matching stream module of a speech synthesis model to adjust feature distribution, and converting the first audio feature vector and the third text data to be synthesized into a third audio feature vector based on the vector field of the conditional matching stream module and the second audio feature vector, includes: Analyzing the feature distribution of the first audio feature vector and the third text data to be synthesized, and the feature distribution of the second audio feature vector; Adjusting the feature distribution of the first audio feature vector and the third text data to be synthesized to the feature distribution of the second audio feature vector through the vector field of the conditional matching stream module, so as to obtain the second audio feature vector after the feature distribution is adjusted and the third text data to be synthesized; The second audio feature vector after adjusting the feature distribution and the third text data to be synthesized are subjected to audio conversion processing to obtain a third audio feature vector.

7. The speech synthesis method based on conditional matching stream according to claim 1, characterized in that: The step of inputting the third audio feature vector into a decoder of a speech synthesis model for decoding to obtain speech data generated by the third text data to be synthesized includes: Truncating the second audio feature vector in the third audio feature vector to obtain the audio feature vector of the third text data to be synthesized; The audio feature vector of the third text data to be synthesized is decoded to obtain speech data generated by the third text data to be synthesized.

8. A speech synthesis device based on conditional matching stream, characterized in that: include: An acquisition module, used to acquire first voice sample data, and perform text annotation on the acquired first voice sample data to obtain first text data corresponding to the first voice sample data; A first processing module, configured to perform data preprocessing on first text data corresponding to the first voice sample data to obtain second text data corresponding to the first voice sample data; A feature extraction module, used for constructing a speech synthesis model, and inputting the second text data into an encoder of the speech synthesis model to extract audio features, so as to obtain an audio feature vector of the first speech sample data; A first processing module, configured to perform mask processing on the audio feature vector of the first speech sample data, and divide the audio feature vector of the first speech sample data after the mask processing into a first audio feature vector and a second audio feature vector; a conversion module, used for acquiring third text data to be synthesized, inputting the first audio feature vector and the third text data to be synthesized into a conditional matching stream module of a speech synthesis model for adjusting feature distribution, and converting the first audio feature vector and the third text data to be synthesized into a third audio feature vector based on the vector field of the conditional matching stream module and the second audio feature vector; A decoding module is used to input the third audio feature vector into a decoder of a speech synthesis model for decoding to obtain speech data generated by the third text data to be synthesized.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the speech synthesis method based on conditional matching stream as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the speech synthesis method based on conditional matching stream as claimed in any one of claims 1 to 7 are implemented.