Voice generation method and device, electronic equipment and readable storage medium
By judging and inserting phoneme interval features and language habit features in the speech generation model, combined with deep learning model training, the problem that the existing technology cannot express special language habits is solved, and more anthropomorphic speech generation is achieved, improving user experience.
Patent Information
- Application Number
- CN202411133619.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-26
AI Technical Summary
The existing speech generation model cannot effectively display output results with distinctive language habits, resulting in the user being unable to obtain the desired voice effect during use.
By converting text into phoneme files and determining whether phoneme interval features and language habit features need to be inserted before and after each phoneme, these features are trained in combination with deep learning models to generate more anthropomorphic voice files.
It effectively improves the anthropomorphic level of voice generation, improves user experience, and ensures the synthesis quality of voice files.
Smart Images

Figure CN119028316A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence technology, and in particular relates to a speech generation method, device, electronic device and readable storage medium. Background Art
[0002] The speech generation model is generated by training the neural network corresponding to the speech synthesis model. Due to individual or regional differences, the existing speech generation model cannot well express the output results with characteristic language habits, so that users cannot get the desired speech effect during use. Therefore, a new technology is needed to solve the above problems. Summary of the invention
[0003] The object of the present invention is to provide a speech generation method, device, electronic device and readable storage medium to solve the problems raised in the above background technology.
[0004] To achieve the above object, a technical solution adopted by the present invention is: a speech generation method, comprising the following steps:
[0005] S1: Input the text file into the speech generation model;
[0006] S2: Convert text into phoneme files;
[0007] S3: extracting the time length of each phoneme, and arranging the phonemes in the order of the text content of the text file;
[0008] S4: Determine whether a phoneme interval feature needs to be inserted before and after each phoneme, wherein the phoneme interval feature includes at least one of a phoneme extension sound or a blank sound before and after each phoneme;
[0009] S5: Determine whether a language habit feature needs to be inserted before and after each phoneme, wherein the language habit feature is the oral habit before and after each phoneme in the entire text file language environment;
[0010] S6: synthesize a voice file according to the generated results of S3, S4 and S5.
[0011] Preferably, the speech generation model includes training for phoneme interval features and training for language habit features, and the training method is as follows:
[0012] S01: Input the training voice file;
[0013] S02: using a natural language processing system capable of processing a standard phoneme set to obtain the time length of each phoneme in the voice file, and at the same time, under the condition that there is a gap time length before and after each phoneme, obtaining a gap time length one before and after each phoneme, wherein the gap time length one is at least one of a phoneme extension sound, a blank sound, or a mouth habit, and the mouth habit is a phoneme-level feature of the non-standard phoneme set;
[0014] S03: Obtain a feature set of oral habits in all training voice files, use a statistical method to obtain a set of oral habits with a certain confidence level, wherein the confidence level is at least 80%, and then obtain a phoneme set of oral habits;
[0015] S04: Add the phoneme set of the spoken habit to the standard phoneme set to form a phoneme collection;
[0016] S05: inputting the training voice file into a natural language processing system capable of processing a phoneme collection, obtaining the time length of each phoneme in the voice file, and obtaining the second gap duration before and after each phoneme, wherein the second gap duration is at least one of a phoneme extension sound or a blank sound;
[0017] S06: Obtain a time length collection of each phoneme and a gap duration collection of all training speech files, and use a statistical method to obtain a gap duration collection with a certain confidence level, wherein the confidence level is at least 80%, and then obtain a phoneme collection of gap duration collection;
[0018] S07: Input the data obtained in S06 into the deep learning model to learn the phonemes before and after each oral habit feature, the time length and frequency of each phoneme in any adjacent phonemes, the phoneme extension sound or blank sound before and after each phoneme, and the duration of the phoneme extension sound or blank sound.
[0019] Preferably, in step S03, after obtaining a set of oral habits that meet the confidence level, it is necessary to determine whether the oral habits are real oral habits through manual verification.
[0020] Preferably, in steps S03 and S05, the phonemes and the phoneme-extended sounds in the gap duration of each habit feature need to be unified, and the steps of unifying the features are as follows:
[0021] SA1: Convert the phonemes of each habit feature and the phoneme extension sounds in the gap duration 2 into spectrograms respectively;
[0022] SA2: Arrange and group all the extended phonemes or phoneme durations in the second phoneme of each habit feature according to their similarity;
[0023] SA3: Determine the condition of feature uniformity, and determine the spectrum of oral habits or the spectrum of extended sounds of two phonemes in gap duration based on the condition.
[0024] Preferably, in step SA2, it is necessary to balance the materials in the group, and in step S07, it is necessary to balance the time length and frequency of each phoneme, the phoneme extension sound and the blank sound before and after the phoneme.
[0025] Preferably, before step S01, segmentation conditions need to be set, and each training voice file needs to be segmented when the segmentation conditions exist. The segmentation condition is the set duration of blank sounds. When a voice file has a blank sound that is not less than the set duration, the voice file is segmented at the blank sound.
[0026] The present invention also provides a speech generating device, comprising:
[0027] Input module, used to input text files;
[0028] A processing module is used to convert the text file into a phoneme level file, and at the same time determine whether to insert a phoneme extension sound, a blank sound or a mouth habit before and after each phoneme, and perform processing;
[0029] Generate module, used to merge and output processed text files.
[0030] The present invention also provides a computer-readable storage medium on which a program is stored. When the program is run, the computer executes the steps in the speech generation method.
[0031] The present invention also provides an electronic device, comprising:
[0032] a memory configured to store a computer program;
[0033] The processor is configured to make the computer execute the steps in the speech generation method when the program is executed.
[0034] Beneficial effects of the present invention: In the scheme of the present invention, before synthesizing speech with phoneme-level features, it is determined whether a phoneme extension sound or a blank sound or a mouth habit needs to be inserted before and after each phoneme, which can effectively ensure the degree of anthropomorphism of the generated speech and improve the user experience;
[0035] Among them, by unifying the features of the phonemes of each mouth-habit feature and the phoneme extension sound in the gap duration, when inserting the phoneme extension sound or the mouth-habit feature, the graph with unified features can be directly inserted;
[0036] Among them, by balancing the various materials in the group in step SA2, it can be effectively ensured that the characteristics of the idiom or the extended sound can be trained and learned. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 It is a flow chart of the speech generation method in the present invention. DETAILED DESCRIPTION
[0038] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.
[0039] Example:
[0040] A speech generation method, a device, an electronic device and a readable storage medium are provided. A speech generation method, a speech generation device, a computer-readable storage medium and an electronic device are provided.
[0041] For details, see Figure 1 , a speech generation method, comprising the following steps:
[0042] S1: Input the text file into the speech generation model;
[0043] S2: Convert text into phoneme files;
[0044] S3: extracting the time length of each phoneme, and arranging the phonemes in the order of the text content of the text file;
[0045] S4: Determine whether a phoneme interval feature needs to be inserted before and after each phoneme, wherein the phoneme interval feature includes at least one of a phoneme extension sound or a blank sound before and after each phoneme;
[0046] S5: Determine whether a language habit feature needs to be inserted before and after each phoneme, wherein the language habit feature is the oral habit before and after each phoneme in the entire text file language environment;
[0047] S6: synthesize a voice file according to the generated results of S3, S4 and S5.
[0048] The speech generation model includes training of phoneme interval features and training of language habit features, and the training method is as follows:
[0049] S01: Input a training voice file; wherein, there are several training voice files, assuming 1000. Before this step, set segmentation conditions. Each training voice file needs to be segmented when the segmentation conditions exist. The segmentation condition is the set duration of blank sound. When a voice file has a blank sound not less than the set duration, the voice file is segmented at the blank sound. For example, segmentation is performed at the place where the blank sound of adjacent phonemes is greater than 1 second, and two voice files are obtained that do not include the blank sound. The preferred segmentation effect is that the blank sound can be cut off, which can reduce the total duration of training materials and improve training efficiency.
[0050] S02: using a natural language processing system capable of processing a standard phoneme set to obtain the time length of each phoneme in the voice file, and at the same time, under the condition that there is a gap time length before and after each phoneme, obtaining a gap time length one before and after each phoneme, wherein the gap time length one is at least one of a phoneme extension sound, a blank sound, or a mouth habit, and the mouth habit is a phoneme-level feature of a non-standard phoneme set. In this embodiment, a whisper framework can be used, and the gap time length one is a segment that cannot be recognized by the whisper framework, which is also the main training material for extracting the characteristic language habits;
[0051] Taking the hypothetical 3-second language segment "we say" as an example, after outputting it through the whisper framework, we can get the following results in Table 1 (the data in the table are all hypothetical):
[0052] Table 1
[0053]
[0054] Since oral habits are phoneme-level features that can be represented by phonemes and do not exist in the standard phoneme set, the oral habits in gap duration 1 and the time length of the phonemes that constitute the oral habits can be output through the whisper framework, and the phoneme set of the oral habits can be obtained by setting the phonemes by oneself.
[0055] S03: Obtain a feature set of oral habits in all training voice files, and use statistical methods to obtain a set of oral habits with a certain confidence level, wherein the confidence level is at least 80%. At the same time, it is necessary to determine whether the oral habits are real oral habits through manual verification, and obtain a phoneme set of oral habits after converting the real oral habits into phonemes. Manual verification can play a role in compensating for the content learned by machine learning and avoid errors.
[0056] In this embodiment, since the oral habit feature is a feature with a time period located before and after the phoneme, the feature collection of oral habit is actually a collection of time segments. That is to say, according to the various oral habit features included in the collection, it can be calculated by statistical methods (each oral habit feature can be regarded as a probability form that conforms to the 0-1 distribution), so as to obtain a set of oral habits that meet the set confidence level.
[0057] S04: Add the phoneme set of the spoken habit to the standard phoneme set to form a phoneme collection;
[0058] S05: inputting the training voice file into a natural language processing system capable of processing a phoneme collection, obtaining the time length of each phoneme in the voice file, and obtaining the second gap duration before and after each phoneme, wherein the second gap duration is at least one of a phoneme extension sound or a blank sound;
[0059] S06: Obtain a time length collection of each phoneme and a gap duration collection of all training speech files, and use a statistical method to obtain a gap duration collection with a certain confidence level, wherein the confidence level is at least 80%, and then obtain a phoneme collection of gap duration collection;
[0060] S07: Input the data obtained in S06 into, for example, a Transformer neural network model for learning to obtain the phonemes before and after each oral habit feature, the time length and frequency of each phoneme in any adjacent phonemes, the extended phoneme sound or blank sound before and after each phoneme, and the duration of the extended phoneme sound or blank sound; wherein, in this step, the time length and frequency of each phoneme, the extended phoneme sound and blank sound before and after the phoneme need to be balanced, for example, a diffusion model may be used;
[0061] In steps S03 and S05, the phonemes and the extended sounds of the phonemes in the gap duration of each habit feature need to be unified. The steps of unifying the features are as follows:
[0062] SA1: Convert the phonemes of each habit feature and the phoneme extension sounds in the gap duration 2 into spectrograms respectively;
[0063] SA2: Arrange and group all the extended phonemes or phoneme sounds in the second gap duration of each habit feature according to their similarity. After grouping, balance the materials in each group by using methods such as diffusion model. The grouping condition can be to group the spectra with a certain value of similarity into one group, for example, the spectra with an overall similarity of more than 95% into one group, 85%-95% into one group, etc.; or to group a fixed number of spectra according to the similarity of the spectra, such as 10 in one group; or to group the spectra with a certain value of similarity determined by a certain segment of each spectra, for example, to select the spectra within 40% of the central axis of the spectra for determination, or to select the spectra with the same time axis on both sides of the central axis of the spectra for similarity determination, etc.;
[0064] SA3: Determine the condition for feature unification, and determine the spectrum of oral habits or the spectrum of phoneme extension sounds in gap duration 2 determined under the condition according to the condition for feature unification. Specifically, you can choose to select consistent segments within a fixed time axis within each group of selected spectra as the condition for feature unification, or select 95% of each spectrum in each group for balancing as the condition for feature unification. In actual use, when it is necessary to insert the spectrum of oral habits or the spectrum of phoneme extension sounds in gap duration 2, just insert the above-mentioned spectrum with unified features.
[0065] In addition, in this scheme, when training the phoneme interval features and the language habit features, it is necessary to retain some training voice files as verification sets, and adjust the hyperparameters based on the verification results of the verification sets to achieve optimal adjustment of the training results. This step is a routine operation of deep learning and will not be elaborated on here.
[0066] In addition, the speech generating device provided by the present invention comprises:
[0067] Input module, used to input text files;
[0068] A processing module is used to convert the text file into a phoneme level file, and at the same time determine whether to insert a phoneme extension sound, a blank sound or a mouth habit before and after each phoneme, and perform processing;
[0069] Generate module, used to merge and output processed text files.
[0070] The computer-readable storage medium provided by the present invention stores a program thereon, and when the program is executed, the computer executes the above-mentioned speech generation method.
[0071] The electronic device provided by the present invention comprises:
[0072] a memory configured to store a computer program;
[0073] The processor is configured to enable the computer to execute the above-mentioned speech generation method when the program is running.
[0074] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A speech generation method, characterized in that: The following steps are involved: S1: Input the text file into the speech generation model; S2: Convert text into phoneme files; S3: extracting the time length of each phoneme, and arranging the phonemes in the order of the text content of the text file; S4: Determine whether a phoneme interval feature needs to be inserted before and after each phoneme, wherein the phoneme interval feature includes at least one of a phoneme extension sound or a blank sound before and after each phoneme; S5: Determine whether a language habit feature needs to be inserted before and after each phoneme, wherein the language habit feature is the oral habit before and after each phoneme in the entire text file language environment, and the oral habit is a phoneme-level feature; S6: synthesize a voice file according to the generated results of S3, S4 and S5.
2. A speech generation method according to claim 1, characterized in that: The speech generation model includes training of phoneme interval features and training of language habit features, and the training method is as follows: S01: Input the training voice file; S02: using a natural language processing system capable of processing a standard phoneme set to obtain the time length of each phoneme in the voice file, and at the same time, under the condition that there is a gap time length before and after each phoneme, obtaining a gap time length one before and after each phoneme, wherein the gap time length one is at least one of a phoneme extension sound, a blank sound, or a mouth habit, and the mouth habit is a phoneme-level feature of the non-standard phoneme set; S03: Obtain a feature set of oral habits in all training voice files, use a statistical method to obtain a set of oral habits with a certain confidence level, wherein the confidence level is at least 80%, and then obtain a phoneme set of oral habits; S04: Add the phoneme set of the spoken habit to the standard phoneme set to form a phoneme collection; S05: inputting the training voice file into a natural language processing system capable of processing a phoneme collection, obtaining the time length of each phoneme in the voice file, and obtaining the second gap duration before and after each phoneme, wherein the second gap duration is at least one of a phoneme extension sound or a blank sound; S06: Obtain a set of time lengths of each phoneme and a set of gap durations 2 of all training speech files, and use a statistical method to obtain a set of gap durations 2 with a certain confidence level, wherein the confidence level is at least 80%; S07: Input the data obtained in S06 into the deep learning model to learn the phonemes before and after each oral habit feature, the time length and frequency of each phoneme in any adjacent phonemes, the phoneme extension sound or blank sound before and after each phoneme, and the duration of the phoneme extension sound or blank sound.
3. A speech generation method according to claim 2, characterized in that: In step S03, after obtaining a set of oral habits that meet the confidence level, it is necessary to determine whether the oral habits are real oral habits through manual verification.
4. A speech generation method according to claim 2, characterized in that: In steps S03 and S05, the features of the phonemes and the extended sounds of the phonemes in the gap duration of each habit feature need to be unified. The steps of the feature unification are as follows: SA1: Convert the phonemes of each habit feature and the phoneme extension sounds in the gap duration 2 into spectrograms respectively; SA2: Arrange and group all the extended phonemes or phoneme durations in the second phoneme of each habit feature according to their similarity; SA3: Determine the condition of feature uniformity, and determine the spectrum of oral habits or the spectrum of extended sounds of two phonemes in gap duration based on the condition.
5. A speech generation method according to claim 4, characterized in that: In step SA2, it is necessary to balance each material in the group, and in step S07, it is necessary to balance the time length and frequency of each phoneme, and the phoneme extension sound and blank sound before and after the phoneme.
6. A speech generation method according to claim 2, characterized in that: Before step S01, segmentation conditions need to be set. Each training voice file needs to be segmented when the segmentation conditions exist. The segmentation condition is the set duration of blank sounds. When a voice file has a blank sound that is not less than the set duration, the voice file is segmented at the blank sound.
7. A speech generating device, which generates speech according to the method described in claims 1-6, characterized in that: include: Input module, used to input text files; A processing module is used to convert the text file into a phoneme level file, and at the same time determine whether to insert a phoneme extension sound, a blank sound or a mouth habit before and after each phoneme, and perform processing; Generate module, used to merge and output processed text files.
8. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed, the computer is enabled to execute the method according to any one of claims 1 to 6.
9. An electronic device, characterized in that: include: a memory configured to store a computer program; The processor is configured to make the computer execute the method according to any one of claims 1 to 6 when the program is executed.
