Voice text training data set generation method and device, storage medium and product
By splitting Chinese text based on construction grammar rules and extracting acoustic feature encoding of speech data, pause and stress information are identified to form a speech-text training dataset. This solves the problem of information loss when converting speech to text and achieves more natural and emotional speech synthesis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE M2M
- Filing Date
- 2025-12-17
- Publication Date
- 2026-05-01
AI Technical Summary
Existing speech-to-text training datasets cannot effectively recognize pitch changes and pauses in text content, resulting in information loss or distortion and a lack of emotional expression when speech is converted into text.
Based on construction grammar rules, Chinese text is split into multiple construction structure grouped texts, acoustic feature encoding of speech data is obtained, pause information and stress information are identified, and these information are labeled in the text to form a speech-text training dataset.
The generated speech-text training dataset can better preserve the text structure and incorporate speech emotional information, thereby improving the naturalness of speech synthesis and the accuracy of emotional expression.
Smart Images
Figure CN121963699A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of IT application technology, and in particular to a method, device, storage medium, and product for generating speech and text training datasets. Background Technology
[0002] Current systems based on large-scale TTS (Text-To-Speech) models have achieved remarkable results. In addition to the continuous optimization of the models, the scale of the training datasets has also been continuously expanded. However, the ability to use the training dataset to train TTS models, to train TTS models to recognize text content, and to convert the corresponding text content into speech according to its true meaning, with pitch changes and pauses, is a capability that current speech and text training datasets do not possess. Summary of the Invention
[0003] To address the aforementioned issues, this disclosure provides a method, device, storage medium, and product for generating speech-text training datasets.
[0004] According to a first aspect of the present disclosure, a method for generating a speech-text training dataset is provided, the method comprising: Based on construction grammar rules, the target Chinese text is split into multiple construction structure grouped texts; For the target configuration group text among the multiple configuration group texts, obtain the speech data corresponding to the target configuration group text, and extract the acoustic features of the speech data for encoding to obtain the acoustic feature encoding of the target configuration group text; The emotional information in the acoustic feature encoding is obtained, and the emotional information includes pause information and accent information; By annotating the corresponding positions of the pause information and the stress information in the target configuration structure grouped text, text-speech pair samples with speech structure tags are obtained to form a speech-text training dataset.
[0005] In one implementation, the step of splitting the Chinese text into multiple constructional grouped texts based on construction grammar rules includes: A construction grammar rule base is constructed based on part-of-speech tagging rules, and the construction grammar rule base includes multiple construction grammar rules; The Chinese text is processed using word segmentation and part-of-speech tagging tools to obtain the part-of-speech connection sequence of the Chinese text; The part-of-speech linking sequence is matched with the construction grammar rules in the construction grammar rule base to output multiple construction structure grouped texts.
[0006] In one embodiment, the step of acquiring speech data corresponding to a target configuration group text among the plurality of configuration group texts, and extracting acoustic features from the speech data for encoding to obtain the acoustic feature encoding of the target configuration group text, includes: Acquire the speech data corresponding to the grouped text of the target configuration structure; A pyramid structure encoder composed of convolutional layers and a self-attention network is used to perform temporal downsampling and acoustic feature extraction on the speech data to obtain the acoustic feature encoding of the target configuration structure grouped text.
[0007] In one implementation, obtaining the emotional information from the acoustic feature encoding includes: Using a continuous integral gating mechanism, the acoustic feature encoding is time-aligned with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text; Based on the duration array, pause information and stress information in the speech are identified.
[0008] In one implementation, the step of using a continuous integral gating mechanism to time-align the acoustic feature encoding with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text includes: Calculate the weight corresponding to each time step in the acoustic feature encoding, whereby the weight represents the amount of speech information contained in the corresponding time step; Perform continuous gated integration on the weights corresponding to each time step; If the first cumulative weight value obtained by continuous gating integration reaches a set threshold, then the first target acoustic feature encoding component corresponding to the first cumulative weight value is determined to be a sound segment containing valid characters.
[0009] In one implementation, the step of using a continuous integral gating mechanism to time-align the acoustic feature encoding with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text further includes: Calculate the weight corresponding to each time step in the acoustic feature encoding, whereby the weight represents the amount of speech information contained in the corresponding time step; Perform continuous gated integration on the weights corresponding to each time step; If the second cumulative weight value obtained by continuous gating integration does not reach the set threshold, it is determined that the second target acoustic feature coding component corresponding to the second cumulative weight value corresponds to a sound segment containing invalid characters and not containing valid characters. Reduce the gating duration of the continuous gating integral, divide the text corresponding to the acoustic feature encoding of the target configuration structure grouped text into valid and invalid characters, and obtain an array containing the duration of valid characters and the duration of invalid characters based on the number of tokens of the target configuration structure grouped text.
[0010] In one implementation, identifying pause and stress information in speech based on the duration array includes: Obtain the mode and variance of the duration of all invalid characters in the duration array; Invalid characters whose duration is greater than or equal to the mode and whose error exceeds the variance are identified as pause information; For invalid characters whose duration is less than the mode and whose error exceeds the variance, words before and after them are combined and read together. Abnormal regions of energy or pitch in the duration of valid characters in the duration array are identified as accent information.
[0011] In one implementation, the step of annotating the corresponding positions of the pause information and the stress information in the target configuration structure grouped text to obtain text-speech pair samples with speech structure tags, thereby forming a speech-text training dataset, includes: The pause information and the accent information are inserted symbolically into the corresponding positions in the target configuration structure grouped text to form enhanced text with speech structure markers; The enhanced text is paired with the corresponding speech segments with pauses and stresses to obtain text-speech pair samples with speech structure tags, thus forming a speech-text training dataset.
[0012] According to a second aspect of the present disclosure, a speech-text training dataset generation apparatus is provided, the apparatus comprising: The splitting module is used to split target Chinese text into multiple structured grouped texts based on construction grammar rules; The encoding module is used to obtain speech data corresponding to the target configuration group text in the plurality of configuration group texts, and extract the acoustic features of the speech data for encoding to obtain the acoustic feature encoding of the target configuration group text. The acquisition module is used to acquire the emotional information in the acoustic feature encoding, the emotional information including pause information and accent information; The generation module is used to annotate the corresponding positions of the pause information and the stress information in the target configuration structure grouped text to obtain text-speech pair samples with speech structure tags, so as to form a speech-text training dataset.
[0013] According to a third aspect of the present disclosure, an electronic device is provided, comprising: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method of any one of the first aspects.
[0014] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method described in any of the first aspects.
[0015] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in any one of the first aspects.
[0016] The embodiments of this disclosure achieve the following beneficial effects: This disclosure provides a method for generating a speech-text training dataset, comprising: splitting a target Chinese text into multiple construction structure grouped texts based on construction grammar rules; for a target construction structure grouped text among the multiple construction structure grouped texts, acquiring speech data corresponding to the target construction structure grouped text, and extracting acoustic features from the speech data for encoding to obtain acoustic feature encoding of the target construction structure grouped text; then acquiring emotional information from the acoustic feature encoding, including pause information and stress information; and then marking the corresponding positions of pause information and stress information in the target construction structure grouped text to obtain text-speech pair samples with speech structure tags, thereby forming a speech-text training dataset. This method can solve the problem of speech information loss or distortion (lack of emotional color) when converting speech to text. The TTS speech model generated by training on the speech-text training dataset provided in this disclosure can make the synthesized speech more natural and emotional.
[0017] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description
[0018] The accompanying drawings, which are included to provide a further understanding of this disclosure and form part of this disclosure, illustrate exemplary embodiments of the present disclosure and are used to explain the disclosure, but do not constitute an undue limitation of the disclosure. In the drawings: Figure 1 This is a flowchart of a method for generating a speech-text training dataset according to an embodiment of this disclosure.
[0019] Figure 2 This is a flowchart of a method for generating a speech-text training dataset according to an embodiment of this disclosure.
[0020] Figure 3This is a flowchart of a method for generating a speech-text training dataset according to an embodiment of this disclosure.
[0021] Figure 4 This is a flowchart of a method for generating a speech-text training dataset according to an embodiment of this disclosure.
[0022] Figure 5 This is a flowchart of a method for generating a speech-text training dataset according to an embodiment of this disclosure.
[0023] Figure 6 This is a flowchart of a method for generating a speech-text training dataset according to an embodiment of this disclosure.
[0024] Figure 7 This is a flowchart of a method for generating a speech-text training dataset according to an embodiment of this disclosure.
[0025] Figure 8 This is a flowchart of a method for generating a speech-text training dataset according to an embodiment of this disclosure.
[0026] Figure 9 This is a block diagram of a speech-text training dataset generation device provided in one embodiment of the present disclosure.
[0027] Figure 10 This is a block diagram of an electronic device provided in one embodiment of the present disclosure. Detailed Implementation
[0028] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.
[0029] It should be understood that the term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description.
[0030] It should be noted that the concepts of "first," "second," etc., mentioned in this disclosure are used only to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. The modifiers "a" and "a plurality of" mentioned in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated in the context, they should be understood as "one or more." In the description of this disclosure, unless otherwise stated, "a plurality of" means two or more, and other quantifiers are similar; "at least one," "one or more," or similar expressions refer to any combination of these items, including any combination of single or multiple items.
[0031] Although operations or steps are described in a specific order in the accompanying drawings in the embodiments of this disclosure, it should not be construed as requiring these operations or steps to be performed in the specific order or serial order shown, or requiring all of the shown operations or steps to be performed to obtain the desired result. In the embodiments of this disclosure, these operations or steps may be performed serially; they may be performed in parallel; or a portion of these operations or steps may be performed.
[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of these messages or information. It is understood that before using the technical solutions disclosed in the embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.
[0033] Figure 1 This is a flowchart illustrating a method for generating a speech-text training dataset according to an embodiment of this disclosure. Figure 1 As shown in the figure, this disclosure provides a method for generating a speech-text training dataset, which may include the following steps: In step S10, the target Chinese text is split into multiple construction structure grouped texts based on construction grammar rules.
[0034] In this step, the target Chinese text is split into multiple construction structure groups based on construction grammar rules. For example, a construction grammar rule library can be built first based on part-of-speech tagging rules, which includes multiple construction grammar rules. Then, word segmentation and part-of-speech tagging tools are used to process the Chinese text to obtain the part-of-speech connection sequences of the Chinese text. Finally, the part-of-speech connection sequences are matched with the construction grammar rules in the construction grammar rule library to output multiple construction structure groups of text.
[0035] In step S20, for the target configuration group text among the multiple configuration group texts, the speech data corresponding to the target configuration group text is obtained, and the acoustic features of the speech data are extracted and encoded to obtain the acoustic feature encoding of the target configuration group text.
[0036] In this step, for a target configuration structure group text among multiple configuration structure group texts, the speech data corresponding to the target configuration structure group text is obtained, and the acoustic features of the speech data are extracted and encoded to obtain the acoustic feature encoding of the target configuration structure group text. For example, the speech data corresponding to the target configuration structure group text can be obtained first, and then a pyramid structure encoder composed of convolutional layers and a self-attention network can be used to perform temporal downsampling and acoustic feature extraction on the speech data to obtain the acoustic feature encoding of the target configuration structure group text.
[0037] In step S30, the emotional information in the acoustic feature encoding is obtained, and the emotional information includes pause information and accent information.
[0038] In this step, emotional information from the acoustic feature encoding is obtained, including pause and stress information. For example, a continuous integral gating mechanism can be used to time-align the acoustic feature encoding with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text. Then, based on the duration array, pause and stress information in the speech is identified.
[0039] In step S40, the corresponding positions of the pause information and the stress information are marked in the target configuration structure grouped text to obtain text-speech pair samples with speech structure tags, so as to form a speech-text training dataset.
[0040] In this step, pause and stress information are labeled at corresponding positions in the target configuration structure grouped text to obtain text-speech pair samples with speech structure tags, thus forming a speech-text training dataset. For example, pause and stress information can first be inserted symbolically into the corresponding positions in the target configuration structure grouped text to form enhanced text with speech structure tags. Then, the enhanced text is paired with the corresponding speech segments with pauses and stresses to obtain text-speech pair samples with speech structure tags, thus forming a speech-text training dataset.
[0041] This process allows the generated dataset to retain both the textual structure and the emotional information of the speech, which helps subsequent models to more accurately convert speech to text, avoid information loss, and improve the naturalness of speech synthesis and the accuracy of emotional expression.
[0042] This disclosure provides a method for generating a speech-text training dataset, comprising: splitting a target Chinese text into multiple construction structure group texts based on construction grammar rules; for a target construction structure group text among the multiple construction structure group texts, acquiring speech data corresponding to the target construction structure group text, and extracting acoustic features from the speech data for encoding to obtain acoustic feature encoding of the target construction structure group text; then acquiring emotional information from the acoustic feature encoding, including pause information and stress information; and then marking the corresponding positions of pause information and stress information in the target construction structure group text to obtain text-speech pair samples with speech structure tags, thereby forming a speech-text training dataset. This method can solve the problem of speech information loss or distortion (lack of emotional color) during speech-to-text conversion. The TTS speech model generated by training on the speech-text training dataset provided in this disclosure can make the synthesized speech more natural and emotional.
[0043] Figure 2 This is a flowchart illustrating a method for generating a speech-text training dataset according to an embodiment of this disclosure. Figure 2 As shown, splitting the Chinese text into multiple construction structure groups based on construction grammar rules may include the following steps: In step S101, a construction grammar rule base is constructed based on part-of-speech tagging rules, and the construction grammar rule base includes multiple construction grammar rules.
[0044] In step S102, the Chinese text is processed using word segmentation and part-of-speech tagging tools to obtain the part-of-speech connection sequence of the Chinese text.
[0045] In step S103, the part-of-speech linking sequence is matched with the construction grammar rules in the construction grammar rule base to output multiple construction structure grouped texts.
[0046] For example, we can first assemble corresponding construction grammar rules according to the nature of Chinese characters (pronouns, verbs, nouns, etc.) and their construction structures. Then, we can input the corresponding construction grammar rules into a construction grammar rule table based on the part-of-speech construction structure information of v+n (verb + noun), thus constructing a Chinese construction grammar rule library. Next, we can split the Chinese text into multiple text fragments according to character symbols. Then, we can compare the split text fragments with the stored construction structures. If the structure already exists, we can directly return its corresponding and a list of splittable construction structure phrases. Otherwise, we can use the LTP tool from Harbin Institute of Technology to split the Chinese text information, outputting the word segmentation information and part-of-speech tagging information. Then, we can read the part-of-speech tagging list, connect the part-of-speech tagging information with plus signs, and match the complete connection information with the construction grammar rule table to output multiple grouped texts of construction structures.
[0047] For example, for the Chinese text: "He asked Tom to get his coat," the following are multiple possible grouped text structures that can be output: ['He told Tom to get his coat,' 'He told Tom,' 'get his coat,' 'go get his coat,' 'Tom went to get his coat,' 'told Tom to get his coat,' 'told Tom to get his coat,' 'told Tom.']
[0048] By using a clear rule base and professional tools, we ensure the standardization and accuracy of text segmentation, laying the foundation for accurate correspondence between speech and text in the future, and making the segmented text more consistent with the inherent logic of speech expression.
[0049] Figure 3 This is a flowchart illustrating a method for generating a speech-text training dataset according to an embodiment of this disclosure. Figure 3 As shown, the step of obtaining speech data corresponding to the target configuration structure group text in the plurality of configuration structure group texts, and extracting the acoustic features of the speech data for encoding to obtain the acoustic feature encoding of the target configuration structure group text, may include the following steps: In step S201, the speech data corresponding to the target configuration structure grouped text is obtained.
[0050] For example, real human speech data corresponding to the target configuration structure grouped text can be collected, or speech data corresponding to the target configuration structure grouped text can be output by the TTS model.
[0051] In step S202, a pyramid structure encoder composed of convolutional layers and a self-attention network is used to perform temporal downsampling and acoustic feature extraction on the speech data to obtain the acoustic feature encoding of the target configuration structure grouped text.
[0052] In this step, a pyramid-structured encoder composed of convolutional layers and a self-attention network is used to perform temporal downsampling and acoustic feature extraction on the speech data to obtain the acoustic feature encoding of the target configuration structure grouped text. For example, the encoder can employ a pyramid structure composed of convolutions and self-attention (SANs). Convolutions are used for temporal downsampling, multiplication units (MUs) are used to further capture acoustic details, and the two-dimensional output is projected as input to the pyramid structure composed of SANs. Two pooling layers with a width of 2 are evenly inserted between the stacked SANs to encourage effective encoding at each temporal resolution, further reducing the original temporal sampling rate to 1 / 8. Finally, the acoustic feature encoding hk of the target configuration structure grouped text is obtained.
[0053] In this embodiment, the encoder structure described above efficiently captures key acoustic features of speech. Temporal downsampling improves processing efficiency, and the precisely extracted acoustic features provide high-quality data support for subsequent emotion extraction and text-to-speech correspondence.
[0054] Figure 4 This is a flowchart illustrating a method for generating a speech-text training dataset according to an embodiment of this disclosure. Figure 4 As shown, obtaining the emotional information from the acoustic feature encoding may include the following steps: In step S301, the acoustic feature encoding is time-aligned with the characters of the target configuration structure grouped text using a continuous integral gating mechanism to obtain a duration array based on the number of tokens in the target configuration structure grouped text.
[0055] For example, the weights corresponding to each time step in the acoustic feature encoding can be calculated first, with each weight representing the amount of speech information contained in the corresponding time step. Then, continuous gating integration is performed on the weights corresponding to each time step. If the first cumulative weight value obtained by continuous gating integration reaches a set threshold, the first target acoustic feature encoding component corresponding to the first cumulative weight value is determined to be a sound segment containing valid characters. If the second cumulative weight value obtained by continuous gating integration does not reach the set threshold, the second target acoustic feature encoding component corresponding to the second cumulative weight value is determined to be a sound segment containing invalid characters and not containing valid characters. Finally, the gating duration of continuous gating integration is reduced, and the text corresponding to the acoustic feature encoding of the target configuration structure grouped text is divided into valid characters and invalid characters. An array containing the duration of valid characters and the duration of invalid characters based on the number of tokens in the target configuration structure grouped text is obtained.
[0056] In step S302, pause information and stress information in the speech are identified based on the duration array.
[0057] For example, the mode and variance of the duration of all invalid characters in the duration array can be obtained first. Then, invalid characters whose duration is greater than or equal to the mode and whose error exceeds the variance can be identified as pause information. Then, words before and after invalid characters whose duration is less than the mode and whose error exceeds the variance can be linked together for continuous reading. Finally, abnormal areas of energy or pitch of the duration of valid characters in the duration array can be identified as accent information.
[0058] In the embodiments of this disclosure, time alignment ensures an accurate correspondence between emotional information and text characters, and the duration array provides a reliable basis for emotional information recognition, making the extracted pauses and stress information more consistent with the actual speech expression.
[0059] Figure 5This is a flowchart illustrating a method for generating a speech-text training dataset according to an embodiment of this disclosure. Figure 5 As shown, the step of using a continuous integral gating mechanism to time-align the acoustic feature encoding with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text may include the following steps: In step S3011, the weight corresponding to each time step in the acoustic feature encoding is calculated, and the weight represents the amount of speech information contained in the corresponding time step.
[0060] In this step, the weight αk corresponding to each time step in the acoustic feature encoding is calculated. The weight αk represents the amount of speech information contained in the corresponding time step. For example, the time step length can be 0.1 seconds, but it can also be other lengths, such as 0.05 seconds or 0.5 seconds. This disclosure does not impose any restrictions on this.
[0061] In step S3012, continuous gated integration is performed on the weights corresponding to each time step.
[0062] In this step, continuous gated integration is performed on the weight αk corresponding to each time step. For example, continuous gated integration involves accumulating the weight αk corresponding to multiple time steps over time until the accumulated weight value reaches a set threshold, at which point integration stops, and the integration result is output, which includes the accumulated integration time and the accumulated integration weight.
[0063] In step S3013, if the first cumulative weight value obtained by continuous gating integration reaches a set threshold... If the first cumulative weight value corresponds to the first target acoustic feature coding component, it is determined that the first target acoustic feature coding component is a sound segment containing valid characters, and the coding length of the first target acoustic feature coding component is the length of the valid character pronunciation.
[0064] For example, setting a threshold It can be obtained from the following formula: Formula 1 in, The full weights corresponding to the acoustic feature encoding hk are given. The number of valid characters in the target configuration structure grouped text, where m is not zero.
[0065] By calculating weights and performing integral judgments, the encoded components corresponding to valid characters are accurately selected, providing core data for the construction of the duration array and ensuring that the duration array can accurately reflect the time distribution of valid characters.
[0066] Figure 6 This is a flowchart illustrating a method for generating a speech-text training dataset according to an embodiment of this disclosure. Figure 6As shown, the step of using a continuous integral gating mechanism to time-align the acoustic feature encoding with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text may further include the following steps: In step S3014, the weight corresponding to each time step in the acoustic feature encoding is calculated, and the weight represents the amount of speech information contained in the corresponding time step.
[0067] In step S3015, continuous gated integration is performed on the weights corresponding to each time step.
[0068] The implementation methods for steps S3014 to S3015 can be referred to steps S3011 to S3012, and will not be repeated here.
[0069] In step S3016, if the second cumulative weight value obtained by continuous gating integration does not reach the set threshold, it is determined that the second target acoustic feature coding component corresponding to the second cumulative weight value corresponds to a sound segment containing invalid characters and not containing valid characters.
[0070] In this step, if the second cumulative weight value obtained by continuous gating integration using the cumulative integration duration determined in step S3012 as the gating duration does not reach the set threshold... If the second cumulative weight value corresponds to the second target acoustic feature coding component, it is determined that the second target acoustic feature coding component corresponds to a sound segment containing invalid characters and not containing valid characters. The coding length of the second target acoustic feature coding component is the time length of the invalid characters (pause).
[0071] In step S3017, the gate duration of the continuous gate integral is reduced, the text corresponding to the acoustic feature encoding of the target configuration structure grouped text is divided into valid characters and invalid characters, and an array containing the duration of valid characters and the duration of invalid characters based on the number of tokens of the target configuration structure grouped text is obtained.
[0072] In this step, if the gate duration of the continuous gate integral is too long, the truncated acoustic feature encoding components may correspond to more than one valid character or one invalid character. Therefore, the gate duration of the continuous gate integral can be reduced to accurately segment the text corresponding to the acoustic feature encoding of the target configuration structure grouped text into valid and invalid characters, and to obtain an array containing the durations of valid and invalid characters based on the number of tokens in the target configuration structure grouped text. For example, the array containing the durations of valid and invalid characters based on the number of tokens in the target configuration structure grouped text can be as follows: [' ', ' ',' ', ' '...' ', ' '] in, The duration of the valid characters. Indicates the duration of invalid characters.
[0073] In this embodiment of the disclosure, through detailed character segmentation and duration statistics, the duration array can more comprehensively present the temporal characteristics of the speech, providing detailed data support for subsequent pause and stress information recognition.
[0074] Figure 7 This is a flowchart illustrating a method for generating a speech-text training dataset according to an embodiment of this disclosure. Figure 7 As shown, the step of identifying pause and stress information in speech based on the duration array may include the following steps: In step S3021, the mode and variance of the duration of all invalid characters in the duration array are obtained.
[0075] For example, if the duration of all invalid characters in the duration array is (0.05, 0.08, 0.07, 0.07, 0.12), then its mode is 0.07 and its variance is 0.0005.
[0076] In step S3022, invalid characters whose duration is greater than or equal to the mode and whose error exceeds the variance are identified as pause information.
[0077] In this step, invalid characters whose duration is greater than or equal to the mode of the invalid character's duration (e.g., 0.07), and whose error between duration and mode (duration - mode) exceeds the variance, are identified as pause information. For example, in the sequence (0.05, 0.08, 0.07, 0.07, 0.12), the durations corresponding to 0.07, 0.07, 0.08, and 0.12 are pauses.
[0078] In step S3023, words formed by combining invalid characters whose duration is less than the mode and whose error exceeds the variance are processed for continuous reading.
[0079] In this step, words formed before and after the invalid character whose duration is less than the mode of the invalid character's duration (e.g., 0.07), and whose error between duration and mode (mode - duration) exceeds the variance, are processed for continuous reading. For example, taking (0.05, 0.08, 0.07, 0.07, 0.12) as an example, the duration corresponding to 0.05 cannot be determined as a pause, and the words formed before and after this duration should be processed for continuous reading.
[0080] In step S3024, abnormal regions of energy or pitch in the duration of valid characters in the duration array are identified as accent information.
[0081] In this step, the energy or pitch anomaly regions corresponding to the acoustic feature encoding of the effective character duration in the duration array are identified as accent information.
[0082] Through data statistics and feature analysis, we have achieved accurate recognition of pauses and stress information, making the recognition results more consistent with the actual rules of speech expression and providing an accurate basis for text annotation.
[0083] Figure 8 This is a flowchart illustrating a method for generating a speech-text training dataset according to an embodiment of this disclosure. Figure 8 As shown, the step of annotating the corresponding positions of the pause information and the stress information in the target configuration structure grouped text to obtain text-speech pair samples with speech structure tags to form a speech-text training dataset may include the following steps: In step S401, the pause information and the accent information are inserted in symbolic form into the corresponding positions in the target configuration structure grouped text to form enhanced text with speech structure markers.
[0084] An example of enhanced text with speech structure tags can be shown below: He asked Tom to get his coat. In this context, \ indicates a break, \\ indicates a long pause, and . indicates an accent.
[0085] In step S402, the enhanced text is paired with the corresponding speech segments with pauses and stresses to obtain text-speech pair samples with speech structure tags, so as to form a speech-text training dataset.
[0086] The enhanced text enriches the speech-related emotional information of the text, and the accurate pairing of text and speech segments ensures the high quality of the dataset, providing data that fits the actual application scenario for subsequent TTS model training. This enables the TTS speech model generated by training on the speech-text training dataset provided in this embodiment to produce more natural and emotional synthesized speech.
[0087] Figure 9 This is a block diagram of a speech-text training dataset generation apparatus provided in one embodiment of this disclosure. Figure 1 As shown, this disclosure provides a speech-text training dataset generation device 900, which may include the following modules: The splitting module 910 is used to split target Chinese text into multiple structured grouped texts based on construction grammar rules.
[0088] The encoding module 920 is used to acquire speech data corresponding to the target configuration group text in the plurality of configuration group texts, and extract the acoustic features of the speech data for encoding to obtain the acoustic feature encoding of the target configuration group text.
[0089] The acquisition module 930 is used to acquire the emotional information in the acoustic feature encoding, the emotional information including pause information and accent information.
[0090] The generation module 940 is used to mark the corresponding positions of the pause information and the stress information in the target configuration structure grouped text to obtain text-speech pair samples with speech structure tags, so as to form a speech-text training dataset.
[0091] In one possible implementation, the splitting module 910 is further configured to: A construction grammar rule base is constructed based on part-of-speech tagging rules, and the construction grammar rule base includes multiple construction grammar rules; The Chinese text is processed using word segmentation and part-of-speech tagging tools to obtain the part-of-speech connection sequence of the Chinese text; The part-of-speech linking sequence is matched with the construction grammar rules in the construction grammar rule base to output multiple construction structure grouped texts.
[0092] In one possible implementation, the encoding module 920 is further configured to: Acquire the speech data corresponding to the grouped text of the target configuration structure; A pyramid structure encoder composed of convolutional layers and a self-attention network is used to perform temporal downsampling and acoustic feature extraction on the speech data to obtain the acoustic feature encoding of the target configuration structure grouped text.
[0093] In one possible implementation, the acquisition module 930 is further configured to: Using a continuous integral gating mechanism, the acoustic feature encoding is time-aligned with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text; Based on the duration array, pause information and stress information in the speech are identified.
[0094] In one possible implementation, the acquisition module 930 is further configured to: Calculate the weight corresponding to each time step in the acoustic feature encoding, whereby the weight represents the amount of speech information contained in the corresponding time step; Perform continuous gated integration on the weights corresponding to each time step; If the first cumulative weight value obtained by continuous gating integration reaches a set threshold, then the first target acoustic feature encoding component corresponding to the first cumulative weight value is determined to be a sound segment containing valid characters.
[0095] In one possible implementation, the acquisition module 930 is further configured to: Calculate the weight corresponding to each time step in the acoustic feature encoding, whereby the weight represents the amount of speech information contained in the corresponding time step; Perform continuous gated integration on the weights corresponding to each time step; If the second cumulative weight value obtained by continuous gating integration does not reach the set threshold, it is determined that the second target acoustic feature coding component corresponding to the second cumulative weight value corresponds to a sound segment containing invalid characters and not containing valid characters. Reduce the gating duration of the continuous gating integral, divide the text corresponding to the acoustic feature encoding of the target configuration structure grouped text into valid and invalid characters, and obtain an array containing the duration of valid characters and the duration of invalid characters based on the number of tokens of the target configuration structure grouped text.
[0096] In one possible implementation, the acquisition module 930 is further configured to: Obtain the mode and variance of the duration of all invalid characters in the duration array; Invalid characters whose duration is greater than or equal to the mode and whose error exceeds the variance are identified as pause information; For invalid characters whose duration is less than the mode and whose error exceeds the variance, words before and after them are combined and read together. Abnormal regions of energy or pitch in the duration of valid characters in the duration array are identified as accent information.
[0097] In one possible implementation, the generation module 940 is further configured to: The pause information and the accent information are inserted symbolically into the corresponding positions in the target configuration structure grouped text to form enhanced text with speech structure markers; The enhanced text is paired with the corresponding speech segments with pauses and stresses to obtain text-speech pair samples with speech structure tags, thus forming a speech-text training dataset.
[0098] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.
[0099] The splitting module splits the target Chinese text according to construction grammar rules, outputting multiple construction structure grouped texts. The encoding module acquires the corresponding speech data for each construction structure grouped text, extracts acoustic features, and encodes them to obtain acoustic feature codes. The acquisition module extracts emotional information related to pauses and stresses from the acoustic feature codes. The generation module annotates this information at corresponding positions in the target text, forming text-speech pair samples, ultimately constructing a speech-text training dataset.
[0100] The clear division of labor and collaborative work among the modules ensure the efficiency and accuracy of dataset generation, enabling the generated dataset to better meet the training needs of subsequent speech processing models. This allows the TTS speech model trained using the speech-text training dataset provided in this embodiment to produce more natural and emotional synthesized speech.
[0101] Figure 10 This is a block diagram of an electronic device provided in one embodiment of this disclosure. For example... Figure 10 As shown, the electronic device 1000 may include: a processor 1001 and a memory 1002. The electronic device 1000 may also include one or more of a multimedia component 1003, an input / output (I / O) interface 1004, and a communication component 1005.
[0102] The processor 1001 controls the overall operation of the electronic device 1000 to complete all or part of the steps in the above-described method for generating the voice-text training dataset. The memory 1002 stores various types of data to support the operation of the electronic device 1000. This data may include, for example, instructions for any application or method operating on the electronic device 1000, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 1003 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 1002 or transmitted via communication component 1005. The audio component also includes at least one speaker for outputting audio signals. I / O interface 1004 provides an interface between processor 1001 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 1005 is used for wired or wireless communication between the electronic device 1000 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 1005 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.
[0103] In an exemplary embodiment, the electronic device 1000 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described method for generating speech-text training datasets.
[0104] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described speech-text training dataset generation method. For example, the computer-readable storage medium may be the memory 1002 including the program instructions, which may be executed by the processor 1001 of the electronic device 1000 to complete the above-described speech-text training dataset generation method.
[0105] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described speech-text training dataset generation method when executed by the programmable device.
[0106] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.
[0107] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.
[0108] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.
Claims
1. A method for generating a speech-text training dataset, characterized in that, The method includes: Based on construction grammar rules, the target Chinese text is split into multiple construction structure grouped texts; For the target configuration group text among the multiple configuration group texts, obtain the speech data corresponding to the target configuration group text, and extract the acoustic features of the speech data for encoding to obtain the acoustic feature encoding of the target configuration group text; The emotional information in the acoustic feature encoding is obtained, and the emotional information includes pause information and accent information; By annotating the corresponding positions of the pause information and the stress information in the target configuration structure grouped text, text-speech pair samples with speech structure tags are obtained to form a speech-text training dataset.
2. The method according to claim 1, characterized in that, The method based on construction grammar rules splits the Chinese text into multiple construction structure groups, including: A construction grammar rule base is constructed based on part-of-speech tagging rules, and the construction grammar rule base includes multiple construction grammar rules; The Chinese text is processed using word segmentation and part-of-speech tagging tools to obtain the part-of-speech connection sequence of the Chinese text; The part-of-speech linking sequence is matched with the construction grammar rules in the construction grammar rule base to output multiple construction structure grouped texts.
3. The method according to claim 1, characterized in that, The step of acquiring speech data corresponding to the target configuration group text in the plurality of configuration group texts, and extracting the acoustic features of the speech data for encoding to obtain the acoustic feature encoding of the target configuration group text, includes: Acquire the speech data corresponding to the grouped text of the target configuration structure; A pyramid structure encoder composed of convolutional layers and a self-attention network is used to perform temporal downsampling and acoustic feature extraction on the speech data to obtain the acoustic feature encoding of the target configuration structure grouped text.
4. The method according to claim 1, characterized in that, The step of obtaining the emotional information from the acoustic feature encoding includes: Using a continuous integral gating mechanism, the acoustic feature encoding is time-aligned with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text; Based on the duration array, pause information and stress information in the speech are identified.
5. The method according to claim 4, characterized in that, The step of using a continuous integral gating mechanism to time-align the acoustic feature encoding with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text includes: Calculate the weight corresponding to each time step in the acoustic feature encoding, whereby the weight represents the amount of speech information contained in the corresponding time step; Perform continuous gated integration on the weights corresponding to each time step; If the first cumulative weight value obtained by continuous gating integration reaches a set threshold, then the first target acoustic feature encoding component corresponding to the first cumulative weight value is determined to be a sound segment containing valid characters.
6. The method according to claim 5, characterized in that, The method of using a continuous integral gating mechanism to time-align the acoustic feature encoding with the characters of the target configuration structure grouped text to obtain a duration array based on the number of tokens in the target configuration structure grouped text further includes: Calculate the weight corresponding to each time step in the acoustic feature encoding, whereby the weight represents the amount of speech information contained in the corresponding time step; Perform continuous gated integration on the weights corresponding to each time step; If the second cumulative weight value obtained by continuous gating integration does not reach the set threshold, it is determined that the second target acoustic feature coding component corresponding to the second cumulative weight value corresponds to a sound segment containing invalid characters and not containing valid characters. Reduce the gating duration of the continuous gating integral, divide the text corresponding to the acoustic feature encoding of the target configuration structure grouped text into valid and invalid characters, and obtain an array containing the duration of valid characters and the duration of invalid characters based on the number of tokens of the target configuration structure grouped text.
7. The method according to claim 4, characterized in that, The step of identifying pause and stress information in speech based on the duration array includes: Obtain the mode and variance of the duration of all invalid characters in the duration array; Invalid characters whose duration is greater than or equal to the mode and whose error exceeds the variance are identified as pause information; For invalid characters whose duration is less than the mode and whose error exceeds the variance, words before and after them are combined and read together. Abnormal regions of energy or pitch in the duration of valid characters in the duration array are identified as accent information.
8. The method according to claim 1, characterized in that, The step of annotating the corresponding positions of the pause information and the stress information in the target configuration structure grouped text to obtain text-speech pair samples with speech structure tags, thereby forming a speech-text training dataset, including: The pause information and the accent information are inserted symbolically into the corresponding positions in the target configuration structure grouped text to form enhanced text with speech structure markers; The enhanced text is paired with the corresponding speech segments with pauses and stresses to obtain text-speech pair samples with speech structure tags, thus forming a speech-text training dataset.
9. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-8.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-8.
11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.